Custom LLM Harness

Make an agent run observable

The harness connects model serving to an interactive agent environment and a repeatable evaluation path. It manages endpoint warm-up and shutdown, reports billing information with its reporting delay, preserves run metadata and traces, and distinguishes infrastructure failures from scored task attempts.

The hosted gateway checks access credentials and separates its management credentials from the agent process. A dedicated operating-system user and service restrictions limit the hosted agent's local access. These are scoped deployment controls; they are not a claim of a general-purpose secure sandbox.

A negative result from an agent comparison

One study compared a baseline configuration with an optional delegation-oriented workflow on a frozen 30-task Terminal-Bench subset, using three attempts per task and arm. Task tests supplied the success signal; the analysis retained infrastructure-invalid attempts separately.

The 180 scored trials gave average success rates of 55.6% and 57.8%. The paired difference was +2.2 percentage points, with a 95% interval from −7.8 to +12.2: the experiment did not establish an improvement. Trace analysis also found no invocation of the secondary worker CLIs in either arm. Making delegation available had not demonstrated delegation in use.

What is original here

My contribution is the harness and the experimental accounting around it: keeping interactive workloads separate from benchmark endpoints, bounding runs, recording failures, and checking what the agent actually did. The model weights, inference engine, base agent, benchmark environments, and task checkers are credited upstream components.