Eval Tool Python Backend

The Python execution stack in packages/coding-agent: tool behavior, runner lifecycle, environment handling, execution semantics, output rendering, supported magics, and operational failure modes.

Scope and Key Files

• Tool surface: src/tools/shell/eval.ts
• Session/per-call kernel orchestration: src/eval/py/executor.ts
• Subprocess kernel client: src/eval/py/kernel.ts
• Python wrapper / NDJSON server: src/eval/py/runner.py
• Prelude helpers loaded into every kernel: src/eval/py/prelude.py
• Host-side agent helper bridge: src/eval/agent-bridge.ts
• MIME bundle renderer (text + structured outputs): src/eval/py/display.ts
• Interactive-mode renderer for user-triggered Python runs: src/modes/terminal/components/transcript/eval-execution.ts
• Runtime/env filtering and Python resolution: src/eval/py/runtime.ts

What eval’s Python backend is

The eval tool executes one Python cell per call inside a retained python subprocess that speaks NDJSON over stdin/stdout. No Jupyter gateway and no extra pip dependencies are required, a vanilla Python 3.8+ interpreter is enough. Rich display() output (PIL, pandas, plotly, matplotlib figures) keeps working because the wrapper implements MIME-bundle dispatch. State persists across calls in the retained kernel, so define helpers and datasets in one call and reuse them in the next.

Tool params (one cell per call):

{
  language: "py" | "js" | "rb" | "jl";  // enum narrowed per session to the enabled backends
  code: string;
  title?: string;
  timeout?: number; // seconds, clamped to 1..3600, default 30. Inactivity budget, see "Cell timeout".
  reset?: boolean; // reset this language's kernel before execution
}

The tool is concurrency = "exclusive" for a session, so calls do not overlap.

Kernel lifecycle

Each Python kernel is a single subprocess: <resolved-python> -u <runner.py>. The runner is bundled with the host binary (Bun text import) and cached as runner-<hash>.py in cache/python-runner under the veyyon config root, or under $XDG_CACHE_HOME/veyyon when that is set. Every spawn checks the file is present and writes it again when it was removed. The script is written to a sibling file and renamed into place, so a second process never runs a partly written runner.

Importing the SDK or session class does not load evaluator implementations. The first $ Python command loads its executor through the shell domain manifest. Constructing eval and reading its metadata do not load evaluators; the first enabled request loads only its selected language. Each loaded evaluator registers process-exit cleanup beside its resource pool. Session disposal releases that session’s resources without loading unused evaluators.

Kernel startup sequence:

1. Availability check (checkPythonKernelAvailability): verifies that a Python interpreter resolves and runs.
2. Spawn python -u runner.py with filtered env and cwd.
3. Send an init request that runs os.chdir(cwd), injects env entries, and adds cwd to sys.path.
4. Execute PYTHON_PRELUDE (idempotent: only initializes once per process).

Kernel shutdown:

• Send {"type": "exit"} over stdin.
• Wait for process exit with SHUTDOWN_GRACE_MS budget.
• Escalate to SIGTERM and finally SIGKILL if the process does not exit in time.

Between cells: on Linux with glibc, the runner calls malloc_trim(0) after each cell and before the cell’s done frame. glibc keeps a freed block under its mmap threshold inside its heap and returns only the top of the heap to the system; the trim returns the free pages inside the heap as well. A kernel that has reported a cell done holds the memory its namespace references and the interpreter’s own. The walk takes under 1 ms on a heap with nothing to return and about 25 ms to return 1.4 GB. Other platforms and C libraries skip the trim.

Wire protocol (NDJSON, host ↔ runner)

One JSON object per line, UTF-8, \n terminated.

Host → runner:

{"id": "<reqId>", "code": "<source>", "silent": false, "storeHistory": true, "cwd": "<optional>", "env": {"KEY": "VAL"}}
{"type": "exit"}

Runner → host:

{"type": "started",  "id": "<reqId>"}
{"type": "stdout",   "id": "<reqId>", "data": "..."}
{"type": "stderr",   "id": "<reqId>", "data": "..."}
{"type": "display",  "id": "<reqId>", "bundle": {<mime>: <value>}}
{"type": "result",   "id": "<reqId>", "bundle": {<mime>: <value>}}
{"type": "error",    "id": "<reqId>", "ename": "...", "evalue": "...", "traceback": ["..."]}
{"type": "done",     "id": "<reqId>", "status": "ok"|"error", "executionCount": N, "cancelled": false}

Status events the prelude emits (e.g. _emit_status("find", count=…)) ship inside display bundles under application/x-veyyon-status so the existing TUI status renderer keeps working.

Magics

The runner’s source transformer rewrites IPython-style magics to plain Python calls before parsing. Supported set:

Unknown magic names raise NameError: UsageError: ... inside the cell.

Session persistence semantics

python.kernelMode controls retained kernel reuse:

• session (default)
  • Reuses kernel sessions keyed by namespaced eval session id and interpreter. The eval session id of a saved session is its session file, so a directory change keeps the kernel and its names; each cell runs in the session’s current cwd, which is also the first sys.path entry. A directory with its own .venv or venv gets its own kernel, because that venv selects the interpreter.
  • Multiple owners can share the same retained kernel for that key.
  • /new, /resume of another session and a handoff shut down the kernels the session’s cells started; the next cell starts a new kernel. A kernel a spawned agent still shares stays running for that agent. A reload of the same session keeps its kernel.
  • Calls through the tool are exclusive, so tool invocations do not overlap.
  • A dead retained subprocess is replaced before execution.
  • If the subprocess dies during execution, it is replaced and the cell is retried once.
• per-call
  • Spawns a fresh subprocess for each request.
  • Shuts the subprocess down after the request.
  • No cross-call state persistence.

Multi-cell behavior in a single tool call

A call contains one cell. To build up state, make successive calls: the retained kernel keeps every defined name between them. If a cell fails, earlier state remains in memory and the tool returns a targeted error.

reset=true resets that language’s kernel before the cell executes.

Environment filtering and runtime resolution

Environment is filtered before launching the runner:

• Allowlist includes core vars like PATH, HOME, locale vars, VIRTUAL_ENV, PYTHONPATH, etc.
• Allow-prefixes: LC_, XDG_, VEYYON_
• Denylist strips common API keys (OpenAI/Anthropic/Gemini/etc.)

Runtime selection order (skipped entirely when the python.interpreter setting specifies an explicit executable):

1. Active/located venv (VIRTUAL_ENV, then CONDA_PREFIX, then <cwd>/.venv, <cwd>/venv)
2. Managed venv at ~/.veyyon/python-env
3. python or python3 on PATH

When a venv is selected, its bin/Scripts path is prepended to PATH.

The runner additionally receives PYTHONUNBUFFERED=1 and PYTHONIOENCODING=utf-8 so streamed output reaches the host promptly.

Tool availability and mode selection

eval.py / eval.js (both default true) plus optional boolean env flags VEYYON_PY / VEYYON_JS control eval backend exposure. Ruby and Julia backends (language: "rb" / "jl") also exist behind eval.rb / eval.jl (both default false) and the VEYYON_RB / VEYYON_JL flags; the language enum is narrowed per session to the enabled backends:

• Python backend only (eval.py=true, eval.js=false, or VEYYON_PY=1 VEYYON_JS=0)
• JavaScript backend only (eval.py=false, eval.js=true, or VEYYON_PY=0 VEYYON_JS=1)
• both backends (eval.py=true, eval.js=true, or VEYYON_PY=1 VEYYON_JS=1)

VEYYON_PY and VEYYON_JS use normal boolean flag parsing. Each flag, when set, overrides only its own setting; an unset flag falls back to its setting (eval.py / eval.js, both default true).

eval.pyWorkspace defaults to false. When enabled, the model-facing eval description instructs the agent to retain large tool.* results in Python variables, transform them in the kernel, reuse helper functions, and display only compact conclusions. It adds guidance only; it does not add Python APIs or change cell execution.

If Python preflight fails and eval.js is enabled, eval remains available for js cells; py cells fail with a Python-backend availability error.

Python prelude helpers include agent(prompt, *, agent="deep", model=None, label=None, schema=None, handle=False, isolated=None, apply=None, merge=None). It synchronously calls the host bridge, runs one agent through the task executor, and returns the final text. When schema is supplied, the helper parses the agent’s JSON output and returns the object. When handle=True, it instead returns a DAG node dict ({"text", "output", "handle", "id", "agent"}) whose handle is the spawned agent’s recoverable agent://<id> URI (the parsed object lands under "data" when schema is also set), so a downstream pipeline/parallel stage can reference the transcript by handle instead of re-inlining it.

Persisted helper state

JavaScript and Python cells expose kv for JSON values that must survive a kernel reset or session continuation. The store is scoped to the session under its artifacts directory and is shared across both runtimes:

kv.set("cursor_handle", {"id": "callback-17"})
saved = kv.get("cursor_handle")
keys = kv.list()
kv.delete("cursor_handle")

kv.get(name, default=None) accepts a Python default value. JavaScript returns undefined for a missing key. Keys contain 1–256 characters without /, \, or NUL. One encoded value is limited to 256 KiB and the store is limited to 4 MiB. Concurrent JavaScript and Python writes preserve updates to different keys.

defs() returns up to 200 sorted names defined by user cells with a short value shape: function <name> for a callable, the type and length for a list, tuple, dict or set (list[1000000]), and the type and at most 60 characters of the repr for any other value. Prelude and runtime names are omitted. Use it to check retained kernel state before redefining a helper.

Execution flow and cancellation/timeout

Cell timeout

Each eval cell timeout is in seconds, defaults to 30, and is clamped to 1..3600. It is a wall-clock budget on the cell’s own work that the watchdog (IdleTimeout, src/eval/idle-timeout.ts) enforces, but it is suspended while a host-side agent()/parallel()/completion() bridge call is in flight: those calls emit synthetic pause/resume timeout-control status events (withBridgeTimeoutPause, src/eval/bridge-timeout.ts) that pause the watchdog entirely and start a fresh timeout window when control returns to the runtime, so a long fanout or a slow completion runs to completion instead of being killed mid-stream. Pause is reference-counted because parallel() can have multiple bridge calls in flight at once.

The pause/resume events are the sole mechanism that suspends the budget. Everything else the cell does, compute, stdout/stderr, log()/phase(), and ordinary (non-agent) tool calls, counts against timeout, so a cell that is not delegating to an agent/completion is bounded by a plain wall-clock timeout. The tool combines the caller abort signal, the session abort signal, and the watchdog’s signal with AbortSignal.any(...); no wall-clock deadline is passed to the backend, so neither runtime arms a competing fixed timer.

Kernel execution cancellation

On abort/timeout:

• The host sends kill("SIGINT") to the runner subprocess.
• The runner’s exec-time signal handler raises KeyboardInterrupt inside the running cell.
• Result includes cancelled=true; a kernel timeout is annotated as eval cell timed out after <n>s; kernel interrupted but remains running. Reset the kernel via { reset: true } if state appears corrupted.
• Between requests the runner installs SIG_IGN for SIGINT so a stray cancel does not tear down the kernel.

If the runner does not emit done within 5s of the interrupt (INTERRUPT_ESCALATION_MS, e.g. stuck in C code holding the GIL), the host shuts the subprocess down (escalating exit → SIGTERM → SIGKILL), the cell is annotated as kernel-killed, and the kernel is recreated on the next call.

stdin behavior

Interactive stdin is not supported. The runner does not forward input() prompts; user code that calls input() blocks until cancellation.

Output capture and rendering

Captured output classes

From runner frames:

• stdout / stderr → plain text chunks
• display / result → rich display handling (MIME bundle)
• error → traceback text, from the first frame of cell code inward; the runner frames that dispatched the cell are omitted, and a cell that does not compile shows its SyntaxError at its line in <cell> with no frames
• application/x-veyyon-status MIME inside display → structured status events

Display MIME precedence:

1. text/markdown
2. text/plain
3. text/html (converted to basic markdown)

Additionally captured as structured outputs:

• application/json → JSON tree data
• image/png / image/jpeg → image payloads
• application/x-veyyon-status → status events

Matplotlib

The runner sets MPLBACKEND=Agg as an environ default so figures render off-screen. After every cell, pyplot.get_fignums() is iterated; each figure is saved to PNG, emitted as an image/png display, and closed.

Storage and truncation

Output is streamed through OutputSink and may be persisted to artifact storage. Tool results can include truncation metadata and artifact://<id> for full output recovery.

Renderer behavior

• Tool renderer (eval-render.ts, re-exported from eval.ts):
  • shows code-cell blocks with per-cell status
  • collapsed preview defaults to 10 lines
  • supports expanded mode for all output retained in the tool result
• Interactive renderer (eval-execution.ts):
  • used for user-triggered Python execution in TUI
  • collapsed preview defaults to 20 lines (EXECUTION_PREVIEW_LINES)
  • clamps a very long individual line to 4000 terminal columns, not characters (EXECUTION_MAX_DISPLAY_COLUMNS). Columns are what the terminal has to fit, so ANSI colour codes are not counted and a wide character counts as two. The clamped line ends with … [N visible columns omitted].
  • keeps at most 100 output lines while a cell is streaming (EXECUTION_STREAMING_LINE_CAP, five screenfuls). When more arrive the oldest are dropped and the footer states it: … N earlier lines dropped while streaming. That note is separate from the … N more lines (ctrl+o to expand) hint, because expanding reveals hidden lines and cannot bring back dropped ones. Once the cell finishes, the full output replaces what streaming kept, so the note disappears.
  • shows cancellation/error/truncation notices

The bash execution block (bash-execution.ts) shares all three of those limits, along with the clamp itself, through modes/terminal/components/transcript/execution-shared.ts.

Operational troubleshooting

• Python backend not available: Check eval.py, VEYYON_PY, and that python/python3 is on PATH. If preflight fails and eval.js is enabled, use a js cell.
• No Python on PATH: Install a system Python 3.8+ or place a venv at ~/.veyyon/python-env. veyyon setup python --check reports the resolved interpreter.
• Execution hangs then times out: Increase tool timeout (max 3600s) if workload is legitimate. For stuck native code, cancellation triggers SIGINT first then escalates; the session restarts on the next request.
• Python kernel init failed: Python kernel exited with code N: The interpreter exited before it answered the init request. The interpreter’s stderr is in the log as Python runner stderr.
• stdin/input prompts in Python code: input() is not supported; pass data programmatically.
• Working directory errors: Tool validates cwd exists and is a directory before execution.

Relevant environment variables

Each variable is read directly from the process environment under its VEYYON_ name; there is no legacy alias resolution.

• VEYYON_PY / VEYYON_JS: eval backend exposure overrides
• VEYYON_PYTHON_SKIP_CHECK=1: bypass Python preflight/warm checks
• VEYYON_PYTHON_INTEGRATION=1: enable the opt-in integration tests that spawn a real Python
• VEYYON_PYTHON_IPC_TRACE=1: log NDJSON frames exchanged with the runner subprocess