Overview

# Overview

Compare Models is a daily-updated leaderboard system that aggregates LLM performance data from the Chatbot Arena with pricing information from multiple sources. The application automatically scrapes model rankings from OpenLM.ai's Chatbot Arena+ leaderboard, fetches pricing data from OpenRouter and LiteLLM, and produces a unified CSV dataset. It includes a web interface for searching models, calculating costs, and comparing quality-to-price ratios. The system runs automated daily updates via GitHub Actions to maintain current data.

# Recent Changes (December 2025)

- **Vercel-Inspired Redesign**: Complete visual overhaul with Vercel-style dark theme
  - Deep black backgrounds (#000, #0a0a0a) with pure white text
  - Blue/purple accent gradient for primary actions
  - CSS custom properties for consistent theming
  - Ghost buttons with subtle hover states
  - Minimal borders and cleaner visual hierarchy
- **Token Converter**: New utility card for converting between pages, words, and tokens
  - Bidirectional conversion: edit pages or words, see estimated tokens
  - Formula: 1 token ≈ 0.75 words ≈ 4 characters, 1 page ≈ 500 words
  - Shows character count alongside token estimate
- **Horizontal Scenarios**: Quick Scenarios now display in a clean horizontal scrollable row
- **Enhanced Table**: Improved typography with tabular numbers and cleaner row styling

# Previous Changes (November 2025)

- **Select All Button**: New button to quickly select or deselect all visible models in the table. Button toggles between "Select All" and "Deselect All" states.
- **Top 30 Button**: One-click button to select the top 30 models in the current table view for quick comparisons.
- **Chart Visualization**: New "Generate Chart" button creates an interactive scatter plot showing Arena Score vs Average API Cost for selected models. Each model appears as a uniquely colored dot with its name displayed directly next to it. Chart displays in a modal with tooltips showing model details.
- **Compact UI Design**: Reduced padding and font sizes across all cards and sections for more efficient space usage
- **Quick Scenarios Modal**: Clicking scenario buttons now opens a modal dialog asking users to choose between:
  - **Read**: Calculates input cost only (processing existing text)
  - **Write**: Calculates output cost only (generating new text)
  - **Summarize**: Calculates input cost + 30% output cost (reading and summarizing)
- **Currency Formatting**: Large numbers display with space separators (e.g., $30 000 000.00) for better readability
- **Hide N/A Models**: When costs are calculated, models without pricing are automatically hidden. Toggle "Show models without pricing" checkbox to view them
- **Fixed Scenario Values**: Corrected token amounts for NYT Archive (25B), U.S. Law (24B), Google Code (24B), and World Law (300B) scenarios
- **Referrer Tracking**: "Try in Blend" URLs include `ref=pricecomparisonlive` parameter for traffic attribution
- **Sortable Columns**: Click column headers to sort by Score, Max In/Out tokens, In/Out costs, and Total Cost
- **Persistent Selection**: Selected models remain visible even when filtered out by search
- **Cost Display**: Shows "N/A" instead of $0.00 for models without pricing data
- **Reduced Decimal Places**: Total costs display with 2 decimal places instead of 4

# User Preferences

Preferred communication style: Simple, everyday language.

# System Architecture

## Data Processing Pipeline
The core architecture centers around a Python-based ETL pipeline (`update_leaderboard.py`) that:
- **Data Extraction**: Scrapes HTML from OpenLM.ai using BeautifulSoup to parse the sortable leaderboard table
- **Data Enrichment**: Fetches pricing data from OpenRouter and LiteLLM APIs, then normalizes model names for matching
- **Data Output**: Generates both CSV files for programmatic use and markdown for human-readable previews

## Automation Strategy
Uses GitHub Actions for daily automated updates with a push-based deployment model:
- **Schedule**: Daily cron job triggers the update script
- **Persistence**: Commits generated CSV directly to the repository for easy consumption
- **Distribution**: Leverages GitHub's CDN for fast, reliable data access

## Frontend Architecture
Static HTML/CSS/JavaScript web interface that:
- **Data Source**: Fetches CSV directly from GitHub's raw content URL
- **Client-Side Processing**: All filtering, searching, and cost calculations happen in browser JavaScript
- **User Experience**: Provides search, filtering, model comparison, and cost calculation tools

## File Structure Design
- **Single Responsibility**: Each file has a clear, focused purpose
- **Minimal Dependencies**: Uses only essential Python packages (pandas, requests, beautifulsoup4, tabulate)
- **Output Separation**: Distinguishes between machine-readable (CSV) and human-readable (markdown) formats

# External Dependencies

## Data Sources
- **OpenLM.ai Chatbot Arena+**: Primary source for model rankings and benchmark scores via HTML scraping
- **OpenRouter API**: Model pricing data for cost calculations
- **LiteLLM**: Additional pricing and provider information

## Infrastructure Services
- **GitHub Actions**: Automated daily updates and CI/CD pipeline
- **GitHub Pages**: Static site hosting for the web interface
- **GitHub Raw Content**: CDN for serving CSV data to the web interface

## Python Libraries
- **pandas**: Data manipulation and CSV generation
- **requests**: HTTP client for API calls and web scraping
- **beautifulsoup4**: HTML parsing for leaderboard extraction
- **tabulate**: Formatted table output for markdown generation
- **gradio_client**: Additional model information retrieval

## Web Technologies
- **Vanilla JavaScript**: Client-side data processing and UI interactions
- **Chart.js**: Interactive data visualization library for scatter plot charts
- **CSS3**: Modern styling with gradients and responsive design
- **HTML5**: Static web interface structure