Tool — Model Evaluation
Noema Eval Harness
Runs Noema model eval suites and gates promotion decisions. A model version must pass its eval gate before it is promoted in ArcRouter. Ratified in NOEMA.0 as D-N03. Source: brain/noema/eval_harness.py.
What it does
The eval harness runs a set of task-class evaluations against a Noema model version, compares scores to the base model and a prompt-only baseline, and produces a pass/fail verdict. A fail blocks promotion in ArcRouter — the model stays in shadow mode until it passes.
- Loads eval suite YAML from
brain/noema/evals/ - Runs each eval against the target model via ArcRouter (shadow alias)
- Compares scores to base model and prompt-only baseline
- Checks for general regression on non-specialty tasks
- Outputs a JSON report and a pass/fail verdict
- On pass: emits a promotion event to ArcRouter; on fail: writes failure details to eval log
Eval gates (NOEMA.0 D-N03)
G2 — Evals and data
Eval set per seed specialty; classified corpus
Each specialty must have an eval set before its adapter enters training. The harness validates that eval sets exist and are properly formatted.
G3 — Department adapters
Beats base + prompt on specialty eval; no general regression
Each department LoRA must score better than the base model + prompt engineering on its specialty eval, AND must not regress on the general task eval suite.
G4 — Specialty adapters
Same as G3 per specialty
Applies the same criteria to each specialty adapter added in NOEMA.4.
Usage
# Run eval suite for a model version
python3 brain/noema/eval_harness.py \
--model noema-dept-operations-v0.1 \
--suite department-operations \
--base AR-D1 \
--output evals/results/operations-v0.1.json
# Check promotion status
python3 brain/noema/eval_harness.py \
--report evals/results/operations-v0.1.json \
--gate G3Eval suite format
# brain/noema/evals/department-operations.yaml
name: department-operations
version: "1"
cases:
- id: ops-001
prompt: "Draft a project status update for a client..."
judge: llm # uses ArcRouter to score 1-5
min_score: 3.5
- id: ops-002
prompt: "What are the next steps for this project?"
judge: llm
min_score: 3.5
regression_suite: general-v1 # must not regress on this