NYCGO Project - Claude Code Guide

This workspace contains two related repositories for managing NYC Governance Organizations data.

Multi-repo setup: If you're working with both nyc-governance-organizations and nycgo-admin-ui, create a parent folder containing both repos and copy this CLAUDE.md there. This gives Claude Code context across both repositories.

mkdir nycgo && cd nycgo
git clone <nyc-governance-organizations-url>
git clone <nycgo-admin-ui-url>
cp nyc-governance-organizations/CLAUDE.md .

Sprint Planning

Sprint documentation lives in nyc-governance-organizations/docs/sprints/:

Completed (in completed/ subfolder):

• SPRINT_1.md - Admin UI Infrastructure
• SPRINT_2.md - Pipeline Testing & Validation
• SPRINT_3.md - v1.1.2 Release Fix & Production Pipeline Test
• SPRINT_4.md - Edit Submission Rate Limiting
• SPRINT_4.5.md - Release Notes & Smart Versioning
• SPRINT_5.md - Data Quality & Standardization
• SPRINT_6.md - Directory Logic Transparency & QA (Dec 2024, released v1.6.0)

Active/Planned (in root folder):

• SPRINT_7.md - Cleanup & Technical Debt (schema docs, externalize configs, etc.)
• PHASE_II.md - Phase II Data Release (46-field schema, 9 new orgs - deferred indefinitely)

---

Repository Structure

/nycgo/
├── nyc-governance-organizations/  # Data pipeline & golden dataset
└── nycgo-admin-ui/                # Web-based admin interface

Both are symlinks to the actual repos.

---

Repository Overview

nyc-governance-organizations (Pipeline Repo)

Purpose: Data pipeline for processing, validating, and publishing the NYC Governance Organizations dataset.

Key Directories:

• data/input/ - Source QA edit files
• data/published/latest/ - Single source of truth for golden datasets
• data/audit/runs/ - Run artifacts with inputs/outputs/review
• data/changelog.csv - Master append-only changelog
• src/nycgo_pipeline/ - Python pipeline package
• scripts/pipeline/ - CLI entrypoints (run_pipeline.py, publish_run.py)
• .github/workflows/publish-release.yml - Automated release workflow

Current Versions:

• v1.6.0 - Latest published (434 records, 38 fields, snake_case column names)
• v1.2.0 - Development (443 records, 46 fields, Phase II schema - deferred)

Pipeline Commands:

# Run pipeline
make run-pipeline GOLDEN=... QA=... DESCRIPTOR=...

# Publish a run
make publish-run RUN_ID=... VERSION=...

# Run tests
make test

nycgo-admin-ui (Admin UI Repo)

Purpose: Web-based interface for submitting individual organization edits.

Key Files:

• js/config.js - Configuration (data sources, editable fields, GitHub settings)
• js/app.js - Main application logic
• js/data.js - Data loading and parsing
• pending-edits/ - Edits waiting to be processed
• processed-edits/ - Archive of processed edits
• scheduled-edits/ - Future-dated edits
• .github/workflows/process-edit.yml - Workflow to process edits

Current Configuration (config.js):

• Data source: GitHub raw file from data/published/latest/NYCGO_golden_dataset_latest.csv
• Batch review interface: review-edits.html for reviewing CSV uploads before commit
• Workflow mode: Check repo variable WORKFLOW_MODE (test or production)

⚠️ IMPORTANT: Cache-Busting for JS Changes

When modifying any JavaScript files in nycgo-admin-ui/js/, you MUST update the cache-busting version in BOTH HTML files to ensure users get the new code:

1. Edit index.html - update all ?v=YYYYMMDD params to today's date
2. Edit review-edits.html - update all ?v=YYYYMMDD params to today's date

Example: Change js/app.js?v=20260101 to js/app.js?v=20260108

Without this step, users may continue loading old cached JavaScript.

---

Data Flow

User submits edit in Admin UI
        ↓
CSV file committed to nycgo-admin-ui/pending-edits/
        ↓
process-edit.yml workflow runs
        ↓
Pipeline processes edit using nyc-governance-organizations
        ↓
Output committed to main branch
        ↓
publish-release.yml creates GitHub release (if test/prod mode)
        ↓
Artifacts available in data/published/latest/

---

Edit Format

Edits use a CSV format with these columns:

record_id,record_name,field_name,action,justification,evidence_url

Actions:

• direct_set - Set field to specific value
• append_to_list - Append to semicolon-separated list
• remove_from_list - Remove from list
• generate_recordid - Auto-generate new RecordID

---

Workflow Modes

The process-edit.yml workflow has two modes (controlled by a single repo variable):

Mode	Variable	Target Branch	Release
Test	WORKFLOW_MODE=test (default)	main	Draft
Production	WORKFLOW_MODE=production	main	Real

---

Sprint 6 Completed (Dec 2024)

Sprint 6 focused on Directory Logic Transparency, field standardization, and QA:

1. Directory rules module: src/nycgo_pipeline/directory_rules.py - single source of truth
2. Regression tests: 70 test cases covering all organization types and edge cases
3. snake_case standardization: All column names converted from PascalCase to snake_case
4. Column ordering: Golden dataset columns aligned with published export order
5. Edit UI enhancement: Shows directory eligibility status with reasoning
6. Admin UI fixes: BOM handling in CSV parser, snake_case column support
7. Workflow fix: Automatic release run detection uses name-based sorting (not mtime)
8. QA fixes: URL corrections, release attribution, UI polish

Released: v1.6.0 with snake_case columns

Sprint 1 Completed (Dec 2024)

1. Admin UI data source: Now uses NYCGO_golden_dataset_latest.csv from data/published/latest/
2. Batch edit review: New review-edits.html interface for reviewing CSV uploads before commit
3. Single source of truth: data/published/latest/ is now the only golden dataset location
4. data/working/ removed: Eliminated to prevent divergence; backup at backup/pre-working-removal-*

---

Common Tasks

Check current data source

cat nycgo-admin-ui/js/config.js | grep github:

Find latest golden dataset

ls -la nyc-governance-organizations/data/published/latest/NYCGO_golden_dataset_*.csv

View pending edits

ls -la nycgo-admin-ui/pending-edits/

Test the admin UI locally

cd nycgo-admin-ui && python3 -m http.server 8000

---

Safety Protocols for Schema/Breaking Changes

When making changes to the golden dataset schema, field names, or other breaking changes:

1. Commit Before Breaking Changes

cd nyc-governance-organizations
git add -A && git commit -m "chore: Complete work before [change description]"

2. Create Baseline Tag

git tag v1.X.Y-pre-[change-name] -m "Stable baseline before [change description]"

This enables easy rollback if something goes wrong.

3. Make Changes

Update all affected files:

• data/published/latest/NYCGO_golden_dataset_latest.csv - Data file
• schemas/nycgo_golden_dataset.tableschema.json - Schema definition
• scripts/process/export_dataset.py - Export logic
• src/nycgo_pipeline/*.py - Pipeline code
• tests/ - Test fixtures and assertions

4. Run Tests

make test

Verify no new failures (pre-existing failures are documented in SPRINT_7.md).

5. Pipeline Smoke Test

echo "record_id,record_name,field_name,action,justification,evidence_url" > /tmp/empty_qa.csv
make run-pipeline GOLDEN=data/published/latest/NYCGO_golden_dataset_latest.csv \
                  QA=/tmp/empty_qa.csv \
                  DESCRIPTOR="post-change-test"

Verify: Golden=38 fields, Published=17 fields (or expected counts).

6. Clean Up Test Artifacts

rm -rf data/audit/runs/*post-change-test*

7. Commit Changes

Include the baseline tag reference in the commit message for traceability.

---

Schema Reference

Current (v1.8.x): 38 fields, snake_case column names, record_id format NYC_GOID_XXXXXX Phase II: 46 fields (deferred indefinitely)

See nyc-governance-organizations/docs/SCHEMA.md for complete field documentation.

Key fields:

• record_id, name, name_alphabetized
• operational_status, organization_type, url
• in_org_chart, listed_in_nyc_gov_agency_directory
• principal_officer_full_name, principal_officer_first_name, principal_officer_last_name, principal_officer_title

---

Release Assets & Artifacts

GitHub Release Assets (attached to each release):

• NYCGO_golden_dataset_v{X.Y.Z}.csv - Versioned golden dataset
• NYCGO_golden_dataset_latest.csv - Same file, stable name
• NYCGovernanceOrganizations_v{X.Y.Z}.csv - Public export (directory-eligible only)
• NYCGovernanceOrganizations_latest.csv - Same file, stable name

Pipeline Run Artifacts (data/audit/runs/<run_id>/):

<run_id>/
├── inputs/           # Copies of input files used
├── outputs/          # Generated files
│   ├── golden_pre-release.csv
│   ├── public_export.csv
│   └── run_summary.json
└── review/           # Diff and changelog files

Data Dictionary: docs/NYC_Agencies_and_Governance_Organizations_Data_Dictionary.xlsx

Primary Key: record_id (format: NYC_GOID_XXXXXX, unique per record)

---

Related Documentation

• nyc-governance-organizations/README.md - Main pipeline docs
• nyc-governance-organizations/docs/ - Detailed documentation
• docs/PHASE_I_SCHEMA.md - Field definitions and types
• nycgo-admin-ui/README.md - Admin UI docs