FerrisGrid
Agent docsIf you are an agent, follow this link for raw documentation.Open raw docs

FERRISGRIDVisual control for local agents

Turn screens into coordinates, and coordinates into action.

Get Started
Commands
Docker Workspace
Architecture
GitHub
TypeScript npm mirror

Single-step execution

Every invocation observes once or executes one validated action, then exits. The agent owns the reasoning loop.

Coordinate-first screenshots

Screenshots include deterministic metadata so image-space choices map back to native screen coordinates.

Eyes plus a map

FerrisGrid turns desktop pixels into structured observations an LLM can reason over without hiding the underlying screenshot.

Local traces

Screenshots, metadata, action requests, parsed actions, results, and recap artifacts stay under .ferrisgrid by default.

Docker workspace

Run a Linux desktop in the background, watch it through noVNC, and keep agent input away from your main desktop.

Compact Markdown protocol

FerrisGrid prints agent-readable Markdown for observations, actions, errors, and doctor checks.

Cross-platform shape

The same observe/act interface is designed to span macOS, Linux, and Windows as platform backends mature.

View raw Markdown
$ ferrisgrid observe
## FerrisGrid Observation
- session: .ferrisgrid/session-...
- step: 1
- coordinate_mode: normalized-1000
- screen_id: screen-1
- screenshot: .ferrisgrid/.../screen-1.png

$ ferrisgrid act --file .ferrisgrid/action.md
status: executed

The idea ​

Computers already show the state an agent needs: pixels, windows, buttons, text fields, menus. FerrisGrid makes that state actionable by pairing screenshots with deterministic coordinates, constrained actions, and local traces.

FerrisGrid gives the machine eyes, gives the model a map, and lets Rust move fast.

Install ​

bash
cargo install ferrisgrid-cli
ferrisgrid doctor

An equivalent TypeScript npm package is available from BrunoV21/FerrisGrid-CLI-ts:

bash
npm install -g ferrisgrid-cli
ferrisgrid doctor

Feature requests and protocol changes belong in BrunoV21/FerrisGrid-CLI.

First observe/action loop ​

bash
ferrisgrid observe

The agent reads the returned screenshot path and coordinate metadata, decides one action, then calls:

bash
ferrisgrid act --file .ferrisgrid/action.md

Development from source ​

bash
git clone https://github.com/BrunoV21/FerrisGrid-CLI.git
cd FerrisGrid-CLI
cargo build
cargo test --workspace
cargo run -q -p ferrisgrid-cli -- doctor

Next steps ​

  1. Read the getting started guide.
  2. Learn the command surface.
  3. Run FerrisGrid inside a Docker Linux workspace.
  4. Open issues and feature requests on GitHub.

FerrisGrid - terminal-first visual control for local agents