Neura Market
Back to .md Directory
EVALS.md

Training Readiness Checklist

Lists five gates to verify before launching a training or tuning run, with checkboxes for each item.

m-cahill
May 2, 2026
0 downloads
0 views
ai rag eval workflow guardrails
View source

What this file does

Lists five gates to verify before launching a training or tuning run, with checkboxes for each item.

When to use it

  • Starting a large-scale training or tuning run
  • Auditing evaluation pipeline completeness
  • Onboarding new team members to release process

Assumes this stack

Pythonbackend.tools.seed_golden_dataset

Training Readiness Checklist

Before launching a large-scale training or tuning run, verify the following gates are closed.

1. Evaluation Semantics Locked

  • Primary Metric defined: answer_correctness (Binary 0/1).
  • Evaluator implemented: AnswerCorrectnessJudge in services/judges.py.
  • Versioning: Evaluator output includes judge_version.

2. Dataset Canonicalization

  • Golden Set: golden-v1 exists and is importable.
  • Validation: Dataset builder prevents empty datasets.
  • Ground Truth: golden-v1 manifest includes verified ground truth answers.

3. Metric Persistence

  • Runs: Metrics stored in TunixRunEvaluation.
  • Registry: Metrics promoted to ModelVersion.
  • UI: Metrics visible in Run History and Model Registry.

4. Guardrails

  • Tuning: Block tuning if metric is undefined or judge is missing.
  • Registry: Prevent promoting failed runs.

5. CI/CD

  • Tests: Backend coverage restored.
  • Frontend: "Promote Best" workflow tested.

Usage

To start a training run:

  1. Ensure golden-v1 (or your target dataset) is seeded: python -m backend.tools.seed_golden_dataset
  2. Select answer_correctness as the metric.
  3. Verify results in Leaderboard.

What's inside

5 sections with 12 checklist items, plus a usage section with 3 steps.

Change this for your project

  • Replace golden-v1 with your dataset name
  • Replace AnswerCorrectnessJudge with your judge class
  • Replace TunixRunEvaluation with your metric storage model
  • Replace m-cahill/tunix-rt with your repository

Where it goes

Keep alongside your test suite. Used to define and score model evaluations.

Related Documents

EVALS.md

AI Tools for Developers

Curates a personal reference of AI coding tools, models, and setup instructions for VS Code, Xcode, and Cursor.

aillmeval
0
27
MichaelLedger
EVALS.md

Voice AI Leaderboards, Benchmarks, and Evaluation Gaps (Jan 2025 -- Feb 2026)

Surveys 20+ voice AI benchmarks from Jan 2025, Feb 2026, identifies evaluation gaps, and provides leaderboard data for STT, TTS, and end-to-end voice agents.

aiagentllm
0
18
petteriTeikari
EVALS.md

Evaluating AI Agent Systems: Metrics, Benchmarks, and Quality Assurance (2024-2026)

Surveys 2024-2026 metrics, benchmarks, and monitoring tools for evaluating AI agent systems, with recommendations for a self-improving coding agent.

aiagentllm
0
18
oddurs
EVALS.md

IATA BCBP Standard Compliance

Documents which IATA BCBP fields and barcode formats a Swift library implements, including Version 8 gender code support.

ai
1
17
anomaddev