Research & Science
EVALS.md · 22 documents
Intelligent Research Assistant - Technical Documentation
Documents the architecture, components, API endpoints, and security features of a multi-agent research assistant platform.
Research Report: Using LLMs as Oracle for Entity Matching Ground Truth
Summarises research on using LLMs (DeepSeek, GPT-4, Claude) for entity matching ground truth, covering accuracy, prompt engineering, ensembles, cost analysis, validation, and regression test conversion.
RAG System Testing Methodologies: A Comprehensive Guide
Surveys RAG evaluation frameworks, metrics, and testing methodologies with implementation guidance and tool comparisons.
GenAI Benchmarks & Evaluation — Product-Based Companies
Understanding how to **benchmark, evaluate, and compare LLMs** is essential for roles at Google, OpenAI, Anthropic, Cohere, and AI research teams. This file covers the most important benchmarks, evaluation methodologies, and how to build custom evaluation harnesses.
Arxiv Papers in cs.CV on 2023-04-20
Lists 12 computer vision papers from arXiv on a single date with metadata and summaries.
Arxiv Papers in cs.CV on 2024-07-15
Lists 12 computer vision papers from arXiv on a single day with metadata and summaries.
Analysis of Paper 50
Structured analysis of a research paper's use of BigCloneBench, answering 12 specific questions about dataset usage and validity.
Voice AI Leaderboards, Benchmarks, and Evaluation Gaps (Jan 2025 -- Feb 2026)
Surveys 20+ voice AI benchmarks from Jan 2025, Feb 2026, identifies evaluation gaps, and provides leaderboard data for STT, TTS, and end-to-end voice agents.
LLM Evaluation — Deep Dive
Covers why LLM evaluation is hard, benchmark taxonomy, LLM-as-judge methodology, contamination, robustness, and designing an eval suite for a product.
RAG Evaluation Frameworks: Comprehensive Analysis for VERA
Surveys RAG evaluation frameworks and metrics, focusing on grounding verification for a verification engine called VERA.
Discussion Report: Flat Band Evaluation Methodology & Strategic Direction
Documents a team's decision to replace manual weight tuning with inverse estimation from expert-labeled flat band pairs.
Evaluating generative systems
Explains reference-based and reference-free evaluation workflows for generative LLM systems, including metric types and dataset-level vs row-level scoring.
ProtoExtract — Evaluation Approach Using OmniDocBench Methodology
Defines a multi-level evaluation framework for clinical protocol table extraction, adapting OmniDocBench metrics and adding domain-specific accuracy measures.
American Mass-Market Brand Psychology
Defines a mass-market brand identity for a word game called Hintly, modeled on McDonald's psychology and American consumer triggers.
UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation
Introduces a self-evaluation framework for unified multimodal models that uses the model's own understanding to assess its generation, eliminating extra models and labeled images.
🔬 Open Deep Research
Provides a configurable, open-source deep research agent that runs across multiple model providers, search tools, and MCP servers.
VaultGres Project Context
Documents the architecture, build commands, configuration, and development conventions for a Rust-based PostgreSQL-compatible RDBMS called VaultGres.
State-of-the-Art in PII/PHI Detection for Handwritten Medical Documents: A Comprehensive Review
Reviews datasets, models, and metrics for detecting PII/PHI in handwritten medical documents, identifying a critical research gap.
Universal and Transferable Adversarial Attacks on Aligned Language Models
Summarizes a paper on generating adversarial suffixes that cause aligned language models to produce harmful content, with high transferability to commercial models.
Enhancing Budget-Aware Gating for Retrieval Augmented Generation (RAG)
Describes a thesis that designs and validates a Budget-Aware Uncertainty Gate (BAUG) to reduce token consumption and latency in multi-round RAG systems while maintaining answer quality.
New Papers
Collects and categorises 100+ papers on tool use, reasoning, and RL for LLMs, with personal notes and links.
Codex [OpenAI] [2021.07] [Close]
Summarises the Codex paper: a GPT model fine-tuned on GitHub code for Python generation, evaluated on the new HumanEval benchmark.