Neura Market
Back to .md Directory
EVALS.md

Training Readiness Checklist

Lists five gates to verify before launching a training or tuning run, with checkboxes for each item.

m-cahill
May 2, 2026
0 downloads
1 views
ai rag eval workflow guardrails
View source

What this file does

Lists five gates to verify before launching a training or tuning run, with checkboxes for each item.

When to use it

  • Starting a large-scale training or tuning run
  • Auditing evaluation pipeline completeness
  • Onboarding new team members to release process

Assumes this stack

Pythonbackend.tools.seed_golden_dataset

Training Readiness Checklist

Before launching a large-scale training or tuning run, verify the following gates are closed.

1. Evaluation Semantics Locked

  • Primary Metric defined: answer_correctness (Binary 0/1).
  • Evaluator implemented: AnswerCorrectnessJudge in services/judges.py.
  • Versioning: Evaluator output includes judge_version.

2. Dataset Canonicalization

  • Golden Set: golden-v1 exists and is importable.
  • Validation: Dataset builder prevents empty datasets.
  • Ground Truth: golden-v1 manifest includes verified ground truth answers.

3. Metric Persistence

  • Runs: Metrics stored in TunixRunEvaluation.
  • Registry: Metrics promoted to ModelVersion.
  • UI: Metrics visible in Run History and Model Registry.

4. Guardrails

  • Tuning: Block tuning if metric is undefined or judge is missing.
  • Registry: Prevent promoting failed runs.

5. CI/CD

  • Tests: Backend coverage restored.
  • Frontend: "Promote Best" workflow tested.

Usage

To start a training run:

  1. Ensure golden-v1 (or your target dataset) is seeded: python -m backend.tools.seed_golden_dataset
  2. Select answer_correctness as the metric.
  3. Verify results in Leaderboard.

What's inside

5 sections with 12 checklist items, plus a usage section with 3 steps.

Change this for your project

  • Replace golden-v1 with your dataset name
  • Replace AnswerCorrectnessJudge with your judge class
  • Replace TunixRunEvaluation with your metric storage model
  • Replace m-cahill/tunix-rt with your repository

Where it goes

Keep alongside your test suite. Used to define and score model evaluations.

Related Documents

EVALS.md

IATA BCBP Standard Compliance

Documents which IATA BCBP fields and barcode formats a Swift library implements, including Version 8 gender code support.

ai
1
64
anomaddev
EVALS.md

Evaluating AI Agent Systems: Metrics, Benchmarks, and Quality Assurance (2024-2026)

Surveys 2024-2026 metrics, benchmarks, and monitoring tools for evaluating AI agent systems, with recommendations for a self-improving coding agent.

aiagentllm
0
40
oddurs
EVALS.md

Voice AI Leaderboards, Benchmarks, and Evaluation Gaps (Jan 2025 -- Feb 2026)

Surveys 20+ voice AI benchmarks from Jan 2025, Feb 2026, identifies evaluation gaps, and provides leaderboard data for STT, TTS, and end-to-end voice agents.

aiagentllm
0
36
petteriTeikari
EVALS.md

LLM Evaluation — Deep Dive

Covers why LLM evaluation is hard, benchmark taxonomy, LLM-as-judge methodology, contamination, robustness, and designing an eval suite for a product.

aiagentllm
0
28
ffaisal93