Evaluate an AI Workflow Before Production: Golden Sets, Schemas, and a Kill Switch
A demo that looks clever is not a production job. Build a golden set, pin the output to a JSON Schema, decide who may approve a write, and refuse to ship if you cannot fail closed.
A founder watches the model fill a spreadsheet once and calls it automated. Then a missing stock keeping unit (SKU), a joke in the notes, or a second language produces a row that looks finished and is wrong. Production is not “it worked on Tuesday.” Production is “we know what it does on the worst Tuesday.”
Security comes first: the small-business threat model. After this page, you log and replay with observability. The approval pattern itself is human-in-the-loop (HITL) in n8n.
Pass/fail before you connect the write
flowchart TD
A["Golden set"] --> B["Run workflow"]
B --> C{"Schema valid?"}
C -->|No| D["Fail closed / log"]
C -->|Yes| E{"HITL required?"}
E -->|Yes| F["Human approve"]
E -->|No| G["Write"]
F -->|Reject| D
F -->|Approve| G
G --> H["Kill switch still off?"]Build the golden set from real mess
- Export twenty real inputs. Strip secrets. Keep the typos.
- For each row, write the accepted output by hand, or mark REJECT with a reason.
- Store them next to the workflow, versioned. A Google Sheet is enough. A git JSON file is better.
- Run the workflow against the set after every prompt, model, or tool change. Count schema failures and human-would-reject cases separately.
Do not generate the golden set with the same model you are testing. That is grading your own homework. A person who does the job today should mark the answers.
Pin the shape with JSON Schema
JSON Schema Draft 2020-12 is the current dialect on json-schema.org. Use it as a contract: required fields, types, enums, max lengths. Validate in the workflow before any HTTP write. Sourced: JSON Schema 2020-12 (checked 16 September 2026).
JSON Schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"additionalProperties": false,
"required": ["company", "intent", "next_step"],
"properties": {
"company": { "type": "string", "minLength": 1, "maxLength": 120 },
"intent": {
"type": "string",
"enum": ["quote", "support", "other"]
},
"next_step": { "type": "string", "maxLength": 280 }
}
}Node
import Ajv from 'ajv';
const ajv = new Ajv({ allErrors: true });
const validate = ajv.compile(schema);
if (!validate(payload)) {
throw new Error('schema_fail');
}Hypothetical intake object. additionalProperties false so a chatty model cannot smuggle extra keys into your CRM.
If your stack cannot run Ajv, a Code node that checks required keys and enum membership is still a schema. Do not skip the check because “the prompt says return JSON.” Prompts are not parsers.
The four questions that kill a demo
| Question | Fail closed if |
|---|---|
| Who may approve a write? | The answer is “the workflow” or “whoever is in Slack.” |
| What happens on schema fail? | It retries into production, or it emails the customer anyway. |
| Where is the pause? | You need four logins to stop runs. |
| What is the blast radius of one bad row? | It can update every customer, send to a list, or charge a card. |
If any row fails, stay on the HITL pattern. Do not “just this once” connect the write because a demo is booked.
Filled hypothetical: invoice line items
Hypothetical. A studio asks a model to turn a voice note into line items. The demo is perfect. The golden set includes a note that mentions “the usual retainer” with no number. Schema requires amount as a number. The row fails closed. That is a pass for the evaluation, even though it is a fail for the model. Shipping without that row would have posted a zero or a hallucinated fee.
What this does not prove
A green golden set is not a guarantee on tomorrow’s input. It is a regression net. Observability is how you catch the twenty-first case. Accuracy percentages from a vendor demo are not your golden set. Run yours.
If you want a second pair of eyes on the contract before the write goes live, send the schema and three ugly rows. That is enough to say ship, gate, or stop.
Frequently asked questions
What is a golden set?
A small, versioned collection of real inputs with the output a trusted person already accepted (or rejected). You run the workflow against it before each prompt or model change. If you cannot name twenty cases, you are not evaluating. You are demoing.
Why JSON Schema instead of “the model usually returns JSON”?
Because “usually” is how you post a half-empty invoice. JSON Schema Draft 2020-12 is the current IETF specification for describing the shape of JSON. Validate before any write. If the payload fails the schema, do not retry into the live system. Log it and stop.
When is human-in-the-loop required?
Whenever the action can spend money, change a customer record, send a message that looks like it came from a person, or delete anything. The human-in-the-loop (HITL) spoke covers n8n wait/approve patterns. This page is the gate: no golden set, no schema, no production.
What is a kill switch?
A single flag or paused workflow that stops new runs without hunting through five tools. If you cannot pause it in one minute, it is not a switch. It is a treasure hunt.
Can I skip evals because the task is “just drafting”?
If a human always pastes the draft, evals can be lighter: a weekly sample of ten outputs. If the draft can reach a customer without that paste, it is production. Treat it as such.
How large does the golden set need to be?
Large enough to include the ugly cases: empty fields, mixed languages, attachments, sarcasm, and the one customer who writes novels in the notes. Twenty honest rows beat two hundred synthetic ones you never read.