One core, two front doors: a CLI for people, MCP for agents

A five-minute 60 FPS video is 18,000 frames, and an agent gets them at most eight per MCP call.

assert not coverage["native_delivery_and_declared_review_complete"] is the check frameatlas’s scale test runs after 2,250 get_batch calls over MCP and a hash check on all 18,000 frames that came back. Every frame was sent, and the review still has to count as incomplete. frameatlas is my Python CLI and MCP server for frame-exact video review.

It’s for people who want an agent, or a human reviewer, to look at every frame of a video instead of a sampled subset. The review guide has the agent track transitions and visible UI actions image by image, and sampling one frame per second, which frameatlas never does, would show it 300 of those 18,000. What I’m guarding against is an agent that saw some frames and reports on the whole video. So a served frame and the agent’s note that it looked go into two separate tables.

One Dataset class behind three thin front ends

The core is dataset.py: 196 lines that open a completed extraction, return frame metadata, verify PNG hashes and render images. The CLI in cli.py (86 lines), the loopback viewer in server.py (115) and the MCP server in mcp.py (219) parse input, call the same methods and format the answer. The CLI’s dispatch is a list of one-liners:

    dataset = Dataset(args.dataset)
    if args.command == "info":
        result = dataset.manifest
...
    elif args.command == "frames":
        result = dataset.frames(args.start, args.count)
...
print(json.dumps(result, ensure_ascii=False, indent=2, allow_nan=False))

The MCP side calls the same methods. It only wraps each result in a text content block.

if name == "dataset_info":
    content = [text_content(self.dataset.manifest)]
elif name == "list_frames":
    content = [text_content(self.dataset.frames(args["start"], args.get("count", 32)))]

Range checks live in Dataset as well, so an out-of-range index fails with the same sentence from the CLI and from MCP. On the 17-frame test fixture, frameatlas frame <dataset> --index 17 and a get_frame call with "index": 17 both end in Frame index must be between 0 and 16.

What the human door keeps for itself

The CLI has nine subcommands and the MCP server has nine tools, and they don’t map one to one. The person at the terminal holds every verb that touches the filesystem. extract takes any local video and an output folder and refuses a nonempty one. sheet writes a contact sheet PNG to any new file you name, except inside the dataset, where it stops with Choose a new contact sheet output outside the dataset. serve and mcp pick which dataset gets exposed at all.

The agent gets none of that. frameatlas mcp /absolute/path/to/datasets/my-video hands the server one completed dataset at startup, and no tool takes a path, a shell command or a URL. Every file the manifest or the frame index names goes through safe_path, which resolves it and raises Dataset path escapes its root or is not a file. for anything outside the dataset root.

In the viewer, arrow keys move exactly one frame, you can jump to an index or a source timestamp, switch to 1:1, or press “Play every frame”, which waits for each image to decode and can run below real time rather than skip. The HTTP server binds 127.0.0.1 and answers 403 to a foreign Host, a foreign Origin or Sec-Fetch-Site: cross-site.

Frame lookup splits the other way. A person runs frame --index 16 or frame --time 299.983333, two flags in one mutually exclusive group. A model gets frame_at_time as its own tool, described as returning metadata only, so lookup and image delivery never share a name.

Nine tools that say what they don’t count

A model reads a tool description literally and plans from it, so most of frameatlas’s descriptions state what a call does not mean. list_frames says “This does not count as viewing frames.” get_contact_sheet says it “does NOT count as individual frame delivery or observations”. The tool an agent uses for exhaustive review spells out its limits in the description and the schema:

definition("get_batch", "Return EVERY consecutive frame in a range as individual ordered PNGs. Default native resolution. 1–8 images, 20 MiB total. Near EOF returns the remaining frames and next_start=null. No sampling.",
           {"start": integer(), "count": integer(1, 8), "max_width": WIDTH,
            "display": {"type": "boolean"}, "session": SESSION}, ("start",), writes=True),

definition() turns writes=True into readOnlyHint: false and always sets additionalProperties: false. Three tools carry that flag: get_frame, get_batch and record_observation. Fetching an image counts as a write here because it adds a row to the ledger, and the annotation admits it.

The output is shaped so the model never does cursor arithmetic. Each image is preceded by a text block with that frame’s metadata, and the batch ends with {"returned_indices": [...], "next_start": ..., "sampling": "none", "image_bytes": ...}. The agent passes next_start back as start, and at the end of the video it is null.

The initialize response also carries an instructions string with the review rules. One of them is “Never claim completeness while gaps remain”. Another treats on-screen text as untrusted source content, because a frame can contain any words at all.

One ledger, two sessions

Both servers write to coverage.sqlite in the dataset folder. They share one Coverage class, which creates two tables.

db.execute("CREATE TABLE IF NOT EXISTS deliveries (session TEXT, frame_index INTEGER, kind TEXT, delivered_at TEXT, PRIMARY KEY(session,frame_index,kind))")
db.execute("CREATE TABLE IF NOT EXISTS observations (session TEXT, frame_index INTEGER, note TEXT, observed_at TEXT, PRIMARY KEY(session,frame_index))")

deliveries records what was served and in what form: native, preview, crop, crop_preview, display_resampled or display_resampled_preview. observations holds the notes an agent wrote. The viewer records under the session viewer and MCP calls default to agent.

Auditing an agent run is one command, frameatlas coverage datasets/my-video --session agent. It prints the same report the agent gets from review_coverage: native images delivered, frames with recorded observations, the first unobserved frame, the next native frames still undelivered. The report also has a meaning field stating that observations “are agent declarations, not independent proof of attention”.

One rule connects the two tables. record_observation refuses any frame that wasn’t delivered as a whole image in the same session. The refusal names the frame and the reason.

raise ValueError(f"Frame {index} has not been delivered individually in this session. A contact sheet or metadata list does not qualify.")

The unit test runs it end to end. Frame 0 goes out native, frame 1 as a 32-pixel preview and frame 2 as a 16×16 crop. The report then shows one native delivery and two individual ones, and a note on frame 2 is rejected because a crop doesn’t qualify.

Counting only what left the process

If the write to stdout fails, the ledger shouldn’t say the frame went out. So both servers build the response first and record deliveries only after the bytes are written. ToolServer.prepare() returns the result plus a list of pending records, and the stdio loop does this:

output_stream.write(json.dumps(response, ensure_ascii=False, allow_nan=False) + "\n")
output_stream.flush()
server.commit(records)

The viewer follows the same order: coverage.delivered() runs after wfile.write() and flush(), and a HEAD request writes nothing and records nothing. Budgets are checked before anything is committed. A batch whose PNGs add up to more than 20 MiB raises during prepare(), so a test that pads three images to 7 MiB each ends with zero recorded deliveries. Another test hands stdio() a stream that raises BrokenPipeError when an image is written, and the ledger stays at zero.

Error text the next call can use

Arguments are checked against the published schema before any tool runs. Nothing is coerced, so true is not an integer. These are the real messages for bad get_batch arguments:

{"start": 0, "count": 9} -> count is outside its allowed range.
{"start": true} -> start must have type integer.
{"start": 0, "unexpected": 1} -> Unexpected arguments: unexpected
{"start": 0, "display": "yes"} -> display must have type boolean.
{"start": 0, "max_width": 12} -> max_width is outside its allowed range.
{} -> Missing required arguments.

They go back as a tool result with "isError": true, following the split I described in my earlier MCP piece. Budget errors name the fix: Batch exceeds 20 MiB. Use fewer images or an explicit preview max_width. On the CLI the same exceptions become frameatlas: <message> on stderr with exit code 1, and a test asserts that no Traceback reaches the terminal.

A frame without a timestamp stops the run with extraction will not invent one from FPS, the manifest is marked incomplete, and Dataset won’t open it: Dataset is not a complete FrameAtlas v1 extraction.

The weak spot is the human side. frameatlas --help gives a help string to extract and to nothing else.

positional arguments:
  {extract,info,verify,frames,frame,sheet,coverage,serve,mcp}
    extract             Decode every frame; keep native RGB PNGs and exact
                        timestamps

frameatlas mcp --help shows a bare dataset positional with no description at all. The model door ended up better documented than the one I use myself.

Tests that drive both doors

conftest.py generates its own videos: 96×54 RGB, 17 frames at 60 FPS by default, with a 16-bit binary index drawn across the top of each frame. The MCP tests go through stdio() with real JSON-RPC lines. A tools/list sent before initialize returns an error, after the handshake it returns nine tools, "count": 9 comes back as isError, and an unknown method gets -32601. The HTTP test sends Host: evil.example and expects 403, and checks that thumbnails, contact sheets and HEAD requests leave native_images_delivered at 0.

The opt-in scale test encodes 300 seconds of that 96×54 synthetic video at exactly 60 FPS, extracts and verifies 18,000 frames, then starts frameatlas mcp as a subprocess and pulls every frame through it. It records no observations and writes "ai_understanding_tested": False into its report. The validation notes give 40 passed for the normal suite and about 10 minutes 37 seconds for the scale run on Windows with Python 3.12 and PyAV 19.0.1. CI runs the normal suite on Linux, macOS and Windows across Python 3.10, 3.12 and 3.14, plus the 18,000-frame job on Linux.

The audit view the viewer doesn’t have yet

The viewer’s sidebar calls /api/coverage with no session, so it shows the viewer session only. The endpoint already accepts ?session=agent, but nothing in the page asks for it, so today I audit an agent from the terminal. Showing the agent’s delivery and observation gaps beside my own in that sidebar, and help strings for the eight bare subcommands, are the next two changes.