ATO / AI WORKFLOW TESTING PILOT
Find what breaks.
Show the evidence.
A focused test of one AI-assisted workflow. Agree what should happen, compare it with what actually happens, and receive a report another person can reproduce.
Target delivery: 5–7 working days after the test contract, access, fixtures and deposit are confirmed.
A small pilot with a concrete output.
- One workflow and one agreed structured input/output or trace format.
- Up to 20 agreed cases covering normal use, changes and selected failure conditions.
- A baseline run, one candidate comparison and one retest after an agreed correction: up to 60 case executions.
- A report separating PASS, FAIL, INVALID and UNTESTED, with reproduction steps and visible coverage limits.
- Structured results, a rerunnable check script and a short handoff guide.
- One report-review round. Model/API charges or additional integrations are agreed before use.
Good starting points
A document assistant after a knowledge update. A tool workflow after a prompt change. An agent that resumes a saved task. A small automation with a clear approval rule.
The strongest brief provides a staging environment or export, a few representative cases and an owner who can define the expected result.
EVIDENCE YOU CAN INSPECT
One update. Four visible failures.
The example changes a delivery-and-approval workflow. The checks catch an outdated source, missing-input handling, an approval violation and a lost resume state. Switch reports to see what a correction resolves.
Synthetic demonstration · six agreed checks · no live model or customer data
1 passed · 4 failed · 0 invalid · 1 untested
This update needs review.
| Agreed check | Result | Expected | Observed |
|---|---|---|---|
| Ordinary request | PASS | {"delivery_days": 2, "policy": "r1"} | {"delivery_days": 2, "policy": "r1"} |
| Source correction | FAIL | {"delivery_days": 5, "policy": "r2"} | {"delivery_days": 2, "policy": "r1"} |
| Missing required input | FAIL | {"status": "needs_address"} | {"status": "ready"} |
| Action approval required | FAIL | {"status": "awaiting_approval"} | {"status": "sent"} |
| Resume after restart | FAIL | {"case": "case-42", "step": "review"} | {"case": "case-42", "step": "start"} |
| Provider unavailable | UNTESTED | {"status": "unavailable"} | "No observation" |
These reports were produced by a local adapter using PROOFBREAKER trace comparison. The browser displays the saved fixture results. The corrected example still has an untested provider-outage case, so the report stays incomplete. No invoice or other external action occurs.
What the pilot establishes.
It establishes whether the observed outputs match the agreed checks, and supplies evidence when they do not. A failed check can be useful even when every other case passes.
This pilot does not certify general AI safety, legal compliance, cybersecurity or factual accuracy outside the agreed cases. Model behavior can vary between runs. Semantic judgments need a reviewed rubric; the automated demonstration uses structured-output comparisons.
Implementation uses selected ATO verification components and a workflow-specific adapter. Full integration of the wider research estate is not required or included.
Three steps to a useful result.
01 / CONTRACT
Agree the checks.
Describe the workflow, one input format, expected outputs and known constraints. Confirm access, scope, timing and permitted actions.
02 / EXECUTE
Record the behavior.
After the $375 deposit, run the agreed fixtures, preserve the observations and identify failures or coverage gaps.
03 / REPLAY
Keep the evidence.
Review the report, retest one agreed correction and receive the check script, structured results and handoff after acceptance and the balance.
Before you start.
Will you change our production system?
The pilot starts with supplied traces or an approved test environment. Production changes and actions such as sending messages or charging customers require a separate agreed scope.
What if there are no failures?
You receive the executed results and their coverage limits. A useful report can show passing cases, unresolved questions and missing evidence without inventing a defect.
What should I send first?
A non-confidential description, the input/output format, one example of expected behavior and your preferred timing. Do not send credentials or customer records in the initial form.
Put one workflow
under a clear test.
Describe the workflow, what changed and what a correct result should look like. Include “AI workflow testing pilot — $750” in the project type.
The brief is an enquiry, not a purchase. Scope and availability are confirmed before a deposit is requested.
Need a customer-facing page? See the $1,000 interactive launch page →