Developer API

Every action the model can return uses an action_type from the table below, paired with a params object. Your executor switches on the type and applies the parameters. The terminal types — done and fail — set the response status and signal you to stop looping.

Server-enforced action policy

Prompt rules steer the model; action_policy enforces normalized structured output after the model responds and before an action is returned or dispatched. Enforcement is atomic: if one action in a proposed batch violates policy, none of that batch is admitted. Omit the field for the unrestricted compatibility default.

{
  "action_policy": {
    "blocked_keys": [
      "escape"
    ],
    "block_window_close": true,
    "max_actions": 5
  }
}

Supply the policy on POST /v1/predict, session creation, /v1/parse, direct machine action/batch requests, Task Run creation, saved or ad-hoc Workflow Run creation, and Schedule creation. These surfaces preserve their create-time policy across turns, task steps, retries, recovery, and nested delegation. Constraints that require structured inspection reject raw/uninspectable code rather than guessing whether it is safe.

Action names are surface-specific. Inference and Parse policies use prediction action names such as type_text; direct machine action policies use machine command names such as type. Copy names from the action catalog for the endpoint you are calling.

Completion behavior

The [done] label is an optional prompt convention for making one completion condition easy to spot, for example [done] when: the saved note is visible. It is not a wire-level command, parser token, SSE marker, or guarantee. The API does not complete a step merely because that text appears in an instruction.

{
  "status": "done",
  "actions": [
    {
      "action_type": "done",
      "params": {},
      "description": "",
      "raw_code": ""
    }
  ],
  "reasoning": "The saved note is visible in the current screenshot."
}

response.status is the authoritative loop-control field. Stop the ordinary action loop when it is done or fail. A done action, when present, is a terminal signal with no keyboard or mouse effect; a valid done response is not required to contain that action. The model emits a completion claim when it believes either the observed screenshot already satisfies the condition or the proposed action batch will produce the requested result. That claim does not close a session and is not proof that your business operation persisted.

Keyboard and clipboard semantics

All three operations are focus-sensitive. They go to the application and control that own OS focus when your executor runs them. A screenshot can show likely focus, but the prediction API cannot inspect the clipboard or prove focus. For POST /v1/machines/{id}/actions, map prediction type_text to machine command type;key_press and key_combo keep their names. Prediction done and fail are client terminal signals, not machine commands.

Use lowercase canonical key names on the wire: ctrl, alt, shift, cmd, escape, and enter. The server policy normalizes esc, return, control/ctl, command/super/win/windows, and option. meta is not a current server alias: emit cmd and normalize meta to cmd in your client-side guard for executor portability. Also normalize side-specific executor names such as ctrl_l/ctrl_r, command_l/command_r, and option_l/option_r before applying a client policy. Send canonical wire names whenever you create an API action.

Troubleshooting an Escape loop

Prompt prohibitions are advisory, not enforcement. A model or recovery strategy can still propose Escape to dismiss a suspected modal or clear focus. Set session create-time action_policy with blocked_keys: ["escape"] and block_window_close: true for fail-closed server enforcement. Also inspect every structured action in your executor, reject prohibited keys case-insensitively, and never evaluate raw_code; this client check protects older deployments and actions from another source. The tools allowlist can disable the wholekey_press class, but it cannot ban only Escape while retaining Tab and Enter.

// 1. Create the session with action_policy.blocked_keys=["escape"] and
//    block_window_close=true. Still validate every returned action locally.
// 2. If key_press/key_combo contains "escape" or "esc", DO NOT EXECUTE it.
// 3. Capture a fresh screenshot, then send this to the SAME session:
{
  "screenshot": freshScreenshotBase64,
  "instruction": "CONTROL FEEDBACK: the previous key_press([escape]) was rejected and NOT executed. The current screenshot is authoritative. Continue without Escape. Use one key_combo([ctrl,v]) only after focus is visibly in the note body."
}
// 4. The body changed, so use a NEW Idempotency-Key. Reuse the old key only
//    for an identical transport retry of the original request.

If the same policy violation is proposed three times, stop the paid loop and surface the current screenshot for human review or reset the session. Repeatedly sending an unchanged screen without explicit rejection feedback can reinforce the same recovery plan.