The automation problem.
A computer-use loop becomes dangerous when the model’s suggestion and the program’s authority are treated as the same thing. A prompt can describe a goal; it should not silently grant filesystem, shell or desktop access.
The project needed to expose only capabilities the current runtime can support, enforce an operator-selected permission tier and require observation after every action.
The boundaries the runtime enforces.
The agent is designed around explicit limits:
- The default observe tier exposes runtime checks and read-only file access inside approved paths.
- Desktop, browser, file and system tools appear only when both the runtime and permission tier allow them.
- There is no general-purpose shell. The bounded program launcher invokes one allowlisted executable directly and rejects pipelines, substitutions and command chaining.
- A task cannot report completion until the orchestration loop has a verified post-action observation.
Separate reasoning from authority.
The model should decide which declared action it wants to request. It should not decide whether that action is permitted, how long it may run or how much output enters the next prompt.
I put those decisions in a policy-gated tool registry and bounded orchestration layer. Unknown tools fail, retries and task duration are limited, and audit events pass through redaction before reaching disk.
An observe–decide–gate–verify loop.
Perception captures a screenshot or accessibility tree. Gemini returns one declared function call. The policy layer checks runtime support, tier and allowlists before a tool runs. The resulting observation becomes the next turn’s evidence.
Decisions that narrow the attack surface.
Capability is treated as configuration and code, not prompt etiquette.
Build the registry at runtime
Unsupported tools are omitted instead of being advertised and allowed to fail after the model chooses them.
The model sees a smaller toolset, which is preferable to pretending every environment has the same powers.
Use independent gates
System tools require the system tier and a separate dangerous-action confirmation flag, with path, application and executable allowlists.
Configuration takes longer, but one broad switch cannot expose every high-impact action.
Authenticate the local console
The web control plane exchanges an operator token for a short-lived HttpOnly session and applies origin, rate, upload and concurrency limits.
It remains a single-process local control plane, not a public multi-tenant service.
Runtime and release verification.
Portable tests run on Linux while Windows CI exercises desktop-aware coverage. Static analysis, dependency audit, site validation and Docker smoke tests cover the surrounding delivery paths.
Container releases are promoted by digest only after vulnerability, SBOM and provenance checks. Native desktop control remains intentionally outside Docker; the public Pages site is a presentation, not an operator console.
What the alpha proves.
The agent can operate declared desktop and browser tools through a loop whose authority is visible and bounded outside the model response.
It does not claim general autonomy. UI quality, window focus, third-party latency and canvas-heavy interfaces still limit reliability, so the project belongs in disposable or carefully bounded environments.
Evidence ledger.
The alpha’s claims are tied to checked limits rather than autonomous-task anecdotes:
- Verification
- 155 unit-test declarations and a 70% repository coverage gate.
- Task bounds
- 50 turns, 900 seconds per task, 120 seconds per model request and 45 seconds per action by default.
- Control plane
- A minimum 24-character operator token, at most eight WebSockets and two native streams by default.
- Boundary
- Gemini is a cloud dependency; Docker cannot control the host desktop, and native reliability depends on UI accessibility and focus.