Skip to content

Added UI Automation Crawler to Onboarding Agent #2294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robgruen merged 29 commits into from
May 6, 2026
Merged

Added UI Automation Crawler to Onboarding Agent #2294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
75087f8
Onboarding agent: grammarGen and scaffolder updates
robgruen May 4, 2026
da98aa4
UI Automation helper and TS client (slice 1)
robgruen May 4, 2026
5e0406d
Slice 2: full verb set, find, events.idle, selector fallbacks
robgruen May 4, 2026
b2e96b8
Slice 3a: snapshot capture/restore mechanism
robgruen May 4, 2026
6e063f1
Slice 4: dynamic-controls calibration + rules-aware fingerprint
robgruen May 4, 2026
2660de8
Slice 5: record mode (UIA event subscription + JSONL output)
robgruen May 4, 2026
75a98fe
Slice 6a: autonomous explore loop mechanism (stub oracle)
robgruen May 4, 2026
f13d307
Slice 6b: LLM oracle (typechat) plugged into the explore loop
robgruen May 4, 2026
5f6a1b0
Slice 7: synthesis pipeline (graph → discoveredActions.json)
robgruen May 4, 2026
d2b841e
End-to-end Clock proof: crawl → synthesize → replay → real alarm created
robgruen May 4, 2026
b7dfb6c
Synthesis quality lift: GPT-5 + tighter prompts + validation pass + m…
robgruen May 4, 2026
08a534d
Vision-driven reconnaissance: per-tab + iterative
robgruen May 4, 2026
156610e
README: status section + reconnaissance docs + remaining work
robgruen May 4, 2026
4a920e8
Scaffolder for runtime UI-Automation agents
robgruen May 5, 2026
78188d8
TypeAgent integration: windowsClock agent driven via NL through dispa…
robgruen May 5, 2026
ffbb746
Multi-tab focused crawls: 14 actions across all 5 Clock tabs
robgruen May 5, 2026
93ed038
README: capture known issues from the multi-tab crawl + integration
robgruen May 5, 2026
59cf152
marked UI crawling as experimental
robgruen May 5, 2026
c5f9f27
added choice for building uiautomation handler directly from shell
robgruen May 5, 2026
c71432a
desktop agent will now autobuild if use consents
robgruen May 5, 2026
342a306
updated clock handler to handle neutral state
robgruen May 5, 2026
405bef2
lint
robgruen May 5, 2026
fe1b471
updated status messaging during build action
robgruen May 5, 2026
750df17
Merge origin/main: resolve conflicts keeping both windowsClock and vi…
Copilot May 5, 2026
e1297f6
Fix repo policy check failures: lowercase package name, add metadata,…
Copilot May 5, 2026
baefc8a
fix: align windowsclock-agent lockfile specifier casing
Copilot May 5, 2026
039c7ce
test(shell): align completion toggle tests with @ command dropdown be…
Copilot May 5, 2026
33a6e98
test(shell): update completion toggle spec for command dropdown behavior
Copilot May 6, 2026
0b5856b
fix(shell): remove incorrect autocomplete-container check from wrong …
Copilot May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions dotnet/uiAutomationHelper/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
bin/
obj/
*.user
*.suo
.vs/
27 changes: 27 additions & 0 deletions dotnet/uiAutomationHelper/UiAutomationHelper.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0-windows</TargetFramework>
<OutputType>Exe</OutputType>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>latest</LangVersion>
<RootNamespace>UiAutomationHelper</RootNamespace>
<AssemblyName>UiAutomationHelper</AssemblyName>
<Version>0.1.0</Version>
<OutputPath>bin\$(Configuration)</OutputPath>
<AppendTargetFrameworkToOutputPath>false</AppendTargetFrameworkToOutputPath>
<NoWarn>$(NoWarn);SYSLIB1054;CA1508;CA1859</NoWarn>
</PropertyGroup>
<ItemGroup>
<Compile Remove="test/**" />
<None Remove="test/**" />
<EmbeddedResource Remove="test/**" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="FlaUI.Core" Version="4.0.0" />
<PackageReference Include="FlaUI.UIA3" Version="4.0.0" />
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="UiAutomationHelper.Tests" />
</ItemGroup>
</Project>
56 changes: 56 additions & 0 deletions dotnet/uiAutomationHelper/UiAutomationHelper.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 17
VisualStudioVersion = 17.8.34408.163
MinimumVisualStudioVersion = 10.0.40219.1
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "UiAutomationHelper", "UiAutomationHelper.csproj", "{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "test", "test", "{0C88DD14-F956-CE84-757C-A364CCF449FC}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "UiAutomationHelper.Tests", "test\UiAutomationHelper.Tests.csproj", "{04DFE25D-177A-4CD2-99A8-39B68BB1674E}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Debug|x64 = Debug|x64
Debug|x86 = Debug|x86
Release|Any CPU = Release|Any CPU
Release|x64 = Release|x64
Release|x86 = Release|x86
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Debug|Any CPU.Build.0 = Debug|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Debug|x64.ActiveCfg = Debug|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Debug|x64.Build.0 = Debug|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Debug|x86.ActiveCfg = Debug|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Debug|x86.Build.0 = Debug|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Release|Any CPU.ActiveCfg = Release|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Release|Any CPU.Build.0 = Release|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Release|x64.ActiveCfg = Release|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Release|x64.Build.0 = Release|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Release|x86.ActiveCfg = Release|Any CPU
{A1B2C3D4-E5F6-7890-ABCD-1234567890AB}.Release|x86.Build.0 = Release|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Debug|Any CPU.Build.0 = Debug|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Debug|x64.ActiveCfg = Debug|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Debug|x64.Build.0 = Debug|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Debug|x86.ActiveCfg = Debug|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Debug|x86.Build.0 = Debug|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Release|Any CPU.ActiveCfg = Release|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Release|Any CPU.Build.0 = Release|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Release|x64.ActiveCfg = Release|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Release|x64.Build.0 = Release|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Release|x86.ActiveCfg = Release|Any CPU
{04DFE25D-177A-4CD2-99A8-39B68BB1674E}.Release|x86.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{04DFE25D-177A-4CD2-99A8-39B68BB1674E} = {0C88DD14-F956-CE84-757C-A364CCF449FC}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {B1C2D3E4-F5A6-7890-BCDE-2345678901BC}
EndGlobalSection
EndGlobal
Loading
Loading