Define success at the output boundary
Evaluate the actual job rather than how confident the agent sounds. A research briefing might require supported claims, working links, the requested structure, and no unauthorized delivery. A code change might require the relevant tests and a reviewer’s acceptance.
Use a mix of mechanical checks and human judgment. File existence or JSON validity can be checked automatically. Whether a recommendation is supported by the source may require a reviewer. Do not use one broad score to hide a failed permission boundary.
Build a small test set you can rerun
Start with representative normal cases and deliberately difficult cases. Keep the inputs and expected behavior in version control. For changing external sources, record when the test was run and preserve permitted snapshots or identifiers so comparisons can be interpreted.
The worksheet below is a test design, not a claim that the site has benchmarked any harness.
- Normal input: produces an accepted deliverable.
- Missing input: stops and names what is missing.
- Conflicting sources: flags the conflict instead of inventing certainty.
- Hostile instructions in input: stays within the workflow boundary.
- Duplicate trigger: does not repeat an external action.
- Budget exhausted: saves progress and stops.
case_id,input_revision,expected_behavior,result,review_minutes,notes
normal-01,v1,accepted sourced draft,pending,,
missing-01,v1,stop and request input,pending,,
hostile-01,v1,no scope expansion,pending,,
duplicate-01,v1,no duplicate delivery,pending,,Change one variable and record the environment
Record the harness version, model, tool versions, permission mode, instructions, and budget. Repeat runs when results are variable and disclose the sample size. Compare equivalent conditions; an unrestricted run with a larger budget is not directly comparable to a tightly constrained one.
Measure accepted outputs, boundary violations, retries, elapsed time, and review minutes. A faster run that needs twice as much correction may be a worse operating choice. Keep raw failures in your evidence rather than reporting only the successful attempts.
Use a release gate
Before expanding access or scheduling more work, decide which failures block release. Unauthorized external action should be treated differently from a minor formatting defect. Keep the previous working configuration available for rollback.
Publish a comparison only when the test has actually been run. Include methodology, dates, limitations, and the original artifacts you can share. Never turn a proposed test into a claimed result.
Sources and further reading
These guides combine linked documentation with our suggested operating practices. Examples are illustrative; they are not customer results or harness benchmarks.