Tool — Model Evaluation

Noema Eval Harness

Runs Noema model eval suites and gates promotion decisions. A model version must pass its eval gate before it is promoted in ArcRouter. Ratified in NOEMA.0 as D-N03. Source: brain/noema/eval_harness.py.

PythonCLINOEMA.0 ratifiedbrain/noema/eval_harness.py

The eval harness runs a set of task-class evaluations against a Noema model version, compares scores to the base model and a prompt-only baseline, and produces a pass/fail verdict. A fail blocks promotion in ArcRouter — the model stays in shadow mode until it passes.

  • Loads eval suite YAML from brain/noema/evals/
  • Runs each eval against the target model via ArcRouter (shadow alias)
  • Compares scores to base model and prompt-only baseline
  • Checks for general regression on non-specialty tasks
  • Outputs a JSON report and a pass/fail verdict
  • On pass: emits a promotion event to ArcRouter; on fail: writes failure details to eval log
G2 — Evals and data
Eval set per seed specialty; classified corpus
Each specialty must have an eval set before its adapter enters training. The harness validates that eval sets exist and are properly formatted.
G3 — Department adapters
Beats base + prompt on specialty eval; no general regression
Each department LoRA must score better than the base model + prompt engineering on its specialty eval, AND must not regress on the general task eval suite.
G4 — Specialty adapters
Same as G3 per specialty
Applies the same criteria to each specialty adapter added in NOEMA.4.
# Run eval suite for a model version python3 brain/noema/eval_harness.py \ --model noema-dept-operations-v0.1 \ --suite department-operations \ --base AR-D1 \ --output evals/results/operations-v0.1.json # Check promotion status python3 brain/noema/eval_harness.py \ --report evals/results/operations-v0.1.json \ --gate G3
# brain/noema/evals/department-operations.yaml name: department-operations version: "1" cases: - id: ops-001 prompt: "Draft a project status update for a client..." judge: llm # uses ArcRouter to score 1-5 min_score: 3.5 - id: ops-002 prompt: "What are the next steps for this project?" judge: llm min_score: 3.5 regression_suite: general-v1 # must not regress on this