AGENTS.md

This file provides guidance to AI coding assistants when working with dictionary data in this directory.

Overview

The Source/Data/ directory contains all dictionary data files and the build system for generating McBopomofo's language model data. These files define the mapping between Bopomofo phonetic input and Traditional Chinese characters/phrases, along with frequency data and special features like macros and symbols.

For comprehensive dictionary development workflows, see Wiki: 詞庫開發說明. This document provides technical reference for AI assistants.

GitHub Copilot users: See .github/instructions/Data.instructions.md for path-specific Copilot instructions for this directory.

General guidelines (emoji, date/time format, Conventional Commits, etc.) are defined in the root AGENTS.md and .github/copilot-instructions.md.

Key Principles:

• In most cases, you'll be adding new characters or phrases rather than deleting existing ones

File Descriptions

Source Data Files (Input)

File	Purpose	Format
BPMFBase.txt	Single character Bopomofo mappings	character bopomofo pinyin tone tag
BPMFMappings.txt	Multi-character phrases (2-6 chars)	phrase bpmf1 bpmf2 ... (space-separated)
BPMFPunctuations.txt	Punctuation marks mapping	Similar to BPMFBase
phrase.occ	Phrase frequency/occurrence data	phrase frequency (tab-separated)
heterophony1.list	Primary heterophony readings	character bopomofo
heterophony2.list	Secondary heterophony readings	character bopomofo
heterophony3.list	Tertiary heterophony readings	character bopomofo
exclusion.txt	Phrase frequency exclusions	phrase context_to_exclude (tab-separated)
Symbols.txt	Special symbols (era names, etc.)	symbol bopomofo score
Macros.txt	Text macros (date/time)	MACRO@NAME bopomofo score
associated-punctuation.txt	Punctuation for phrase associations	Special format for derive script

Generated Output Files

File	Generated By	Purpose
data.txt	make all	Main language model data for McBopomofo
data-plain-bpmf.txt	make all	Data for traditional Bopomofo IME mode
associated-phrases-v2.txt	make all	Associated phrase suggestions
PhraseFreq.txt	make all	Compiled frequency data

Important: Generated output files are built locally and NOT committed to git (listed in .gitignore).

Common Commands

# Essential Make Targets
make all           # Generate all output files
make sort          # Sort all data files with correct C locale
make check         # Validate data integrity
make tidy          # Clean up formatting issues
make clean         # Clean generated files

# Recommended workflow: format, sort, validate, and build
make tidy sort check all

Building via Xcode

xcodebuild -project ../McBopomofo.xcodeproj -target Data -configuration Debug build
# Or select "Data" scheme in Xcode and build (⌘+B)

This runs make all as part of the Xcode build process. For dictionary development, using make directly is recommended.

Critical: C Locale Sorting

All data files MUST be sorted with C locale for binary search compatibility:

# Primary data files
LC_ALL=C sort -o BPMFMappings.txt BPMFMappings.txt
LC_ALL=C sort -o phrase.occ phrase.occ

# Heterophony lists
env LANG=C sort -k1 heterophony1.list | uniq > tmp && mv tmp heterophony1.list
env LANG=C sort -k1 heterophony2.list | uniq > tmp && mv tmp heterophony2.list
env LANG=C sort -k1 heterophony3.list | uniq > tmp && mv tmp heterophony3.list

Workflow: Adding/Modifying Phrases

Adding a New Multi-Character Phrase

1. Add to BPMFMappings.txt:

   新詞彙 ㄒㄧㄣ ㄘˊ ㄏㄨㄟˋ
   

   Format: phrase, then Bopomofo for each character (space-separated)

2. Add to phrase.occ with frequency:

   新詞彙	100
   

   • Frequency must be a positive integer (0 is acceptable, negative values are NOT)
   • Use tab separator between phrase and frequency
   • Higher frequency = more common phrase

3. Sort both files:

   make sort
   # Or manually:
   LC_ALL=C sort -o BPMFMappings.txt BPMFMappings.txt
   LC_ALL=C sort -o phrase.occ phrase.occ
   

4. Validate and build:

   make check    # Validate integrity
   make all      # Generate output files
   

Adding a Single Character

Single characters go into BPMFBase.txt:

字 ㄗˋ zi4 -4 big5

Format: character bopomofo pinyin tone tag

Handling Heterophony Characters (破音字)

Do NOT suggest any changes to heterophony1.list, heterophony2.list or heterophony3.list. If a PR contains any changes to those files, highlight them and ask human reviewers to pay attention to them.

For Mandarin heteronyms, an entry in heterophony1.list sets the primary reading and causes other readings of the same character to be demoted. This can be problematic for high-frequency characters whose other readings are also common. To compensate, heterophony2.list and heterophony3.list "promote" those readings back, but with a frequency discount to ensure they don't outrank the primary one.

This is a complex language model tuning mechanism, so agents must NOT suggest changes on their own.

Adding Emojis and Symbols

Emojis/symbols are allowed but MUST NOT be the default candidate. Add to Symbols.txt with negative score (e.g., -8).

File Format Specifications

File	Format	Rules	Example
BPMFMappings.txt	phrase bpmf1 bpmf2 ...	Space-separated; character count = Bopomofo count; C locale sorted	小麥注音 ㄒㄧㄠˇ ㄇㄞˋ ㄓㄨˋ ㄧㄣ
phrase.occ	phrase<TAB>frequency	Tab-separated; frequency ≥ 0 (no negatives); C locale sorted	小麥<TAB>120
heterophony.list*	character bopomofo	One reading per line; sorted by character	中 ㄓㄨㄥ
exclusion.txt	phrase<TAB>context	Tab-separated; excludes phrase when in context	一下<TAB>國一下
Symbols.txt	symbol bopomofo score	Space-separated; negative score for low priority	平成 ㄆㄧㄥˊ-ㄔㄥˊ -8
Macros.txt	MACRO@NAME bopomofo score	Space-separated; runtime expansion	MACRO@DATE_TODAY ㄐㄧㄣ-ㄊㄧㄢ -8

Critical Format Rules:

• BPMFMappings.txt and phrase.occ use different separators (space vs tab)
• Character count in phrase must equal number of Bopomofo readings
• Frequencies must be non-negative integers (0 acceptable, negatives NOT allowed)
• All files must be C locale sorted for binary search compatibility

Python Package Structure

The curation/ package contains library modules organized into submodules:

Submodule	Purpose
curation.builders	Data building and processing tools (frequency_builder, phrase_deriver)
curation.compilers	Data compilation tools (main_compiler, plain_bpmf_compiler)
curation.validators	Validation and analysis tools (score_validator)
curation.utils	General utilities (text_filter)

Scripts with side effects are located in scripts/:

Script	Purpose
count_occurrences.py	Counts phrase occurrences in text corpus
analyze_data.py	Analyzes dictionary data and generates reports
map_bpmf.py	Helper for automatic Bopomofo mapping

Python Development Guidelines

CRITICAL RULES for AI coding assistants:

1. Module Organization Rules

Library modules (in curation/ package) MUST NOT have side effects at module level:

• PROHIBITED: Opening files, reading/writing data, printing output at module level
• PROHIBITED: Executing code immediately when module is imported
• REQUIRED: All initialization must be in functions
• REQUIRED: Module must be importable without executing code

Example of BAD module (violates rules):

# BAD: Has side effects at import time
import configparser
config = configparser.ConfigParser()
config.read('config.ini')  # WRONG: Reads file at import!
corpus = open('corpus.txt').read()  # WRONG: Opens file at import!

Example of GOOD module:

# GOOD: No side effects, importable as library
import configparser

def load_config(config_path='config.ini'):
    """Load configuration from file."""
    config = configparser.ConfigParser()
    config.read(config_path)
    return config

def main():
    """Main entry point for CLI usage."""
    config = load_config()
    # ... rest of logic

if __name__ == '__main__':
    main()

2. Import Guidelines

• REQUIRED: All imports at top of file
• NEVER use inline imports (unless explicitly necessary for specific technical reasons)
• Use relative imports within package (e.g., from .compiler_utils import HEADER)
• Use absolute imports for external packages (e.g., import argparse)

Example of BAD imports:

# BAD: Inline import
def process_data():
    import pandas as pd  # WRONG: NEVER do this!
    return pd.DataFrame(data)

Example of GOOD imports:

# GOOD: All imports at top
import argparse
import sys
from typing import List, Dict
from .compiler_utils import HEADER

def process_data(input_data):
    result = []
    for item in input_data:
        result.append(item.upper())
    return result

3. Side Effect Management

Scripts with side effects belong in scripts/ directory, NOT in curation/ package.

Scripts that do any of the following must be in scripts/:

• Read configuration files at module level
• Open and process data files at module level
• Print output or generate reports at module level
• Execute analysis immediately when imported

When to use scripts/ vs curation/:

Location	Purpose	Characteristics
scripts/	Pure CLI tools	Has side effects; not importable as library; immediate execution
curation/	Library modules	No side effects; importable; reusable functions

4. Script vs Library Separation

Library modules in curation/ should:

• Provide reusable functions and classes
• Have a main() function for CLI usage
• Be declared in pyproject.toml [project.scripts]
• Follow pattern: mcbpmf-tool-name = "curation.module:main"

Scripts in scripts/ should:

• Be standalone executables
• Have all logic in if __name__ == '__main__': block
• Be called directly: python3 scripts/script_name.py
• NOT be imported by other modules

5. PEP-8 Naming Conventions

• Module names: lowercase_with_underscores.py
• Function names: lowercase_with_underscores()
• Class names: CapitalizedWords
• Constants: UPPERCASE_WITH_UNDERSCORES

Examples:

• GOOD: frequency_builder.py, main_compiler.py, text_filter.py
• BAD: buildFreq.py, nonCJK_filter.py, cook-plain-bpmf.py

6. Package Installation

Install as editable package for development:

pip install -e .              # Install package
pip install -e ".[dev]"       # Install with dev dependencies
pip install -e ".[notebook]"  # Install with notebook dependencies

After installation, use console scripts:

mcbpmf-build-freq              # Instead of: python3 -m curation.builders.frequency_builder
mcbpmf-compile                 # Instead of: python3 -m curation.compilers.main_compiler
mcbpmf-validate-scores         # Instead of: python3 -m curation.validators.score_validator

Project Path Configuration

All scripts and modules use centralized path constants from the curation package:

from curation import PROJECT_ROOT, CONFIG_FILE

# PROJECT_ROOT = Source/Data/ directory (where pyproject.toml lives)
# CONFIG_FILE = Source/Data/textpool.rc

# Example usage in scripts
config = configparser.ConfigParser()
config.read(CONFIG_FILE)
corpus_path = Path(config.get('data', 'corpus_path')).expanduser()

Do NOT compute paths relatively in individual scripts:

• BAD: Path(__file__).parent.parent
• BAD: os.path.abspath(sys.argv[0]).split('/')
• GOOD: from curation import PROJECT_ROOT

This ensures:

• Single source of truth for project structure
• Easy refactoring if directory structure changes
• Consistent behavior across all tools

Historical Context: Tool Evolution (2012-2025)

Migration from bin/ to curation/ (October 2024)

Prior to October 2024, all Python tools were located in the bin/ directory (now renamed to bin_legacy/). This directory accumulated tools over 13+ years with contributions from multiple developers.

Tool Creation Timeline

• 2012-08-06: cook.py created by Mengjuei Hsieh, replacing Ruby implementation
• 2012-09-16: buildFreq.py created, replacing bash version
• 2013-01-02: self-score-test.py added for quality validation
• 2013-01-21: C version moved to C_Version/ subdirectory ("phasing out")
• 2024-03-15: derive_associated_phrases.py added by Lukhnos Liu (v2 system)
• 2024-08-25: audit_encoding.swift added by zonble
• 2025-03-08: cook.py modernized with Black formatting and argparse

Why Migration Was Needed

The bin/ structure had accumulated issues:

1. Flat organization (~30 files, no logical grouping)
2. Mixed concerns (library code, CLI scripts, config files, legacy tools)
3. Inconsistent naming conventions
4. Some modules had side effects at import time
5. Not installable as proper Python package
6. Each script calculated paths differently

What Was Migrated vs Preserved

Migrated to curation/ package (actively used):

• All compilation and build tools
• Frequency calculation
• Data validation
• Text processing utilities

Moved to scripts/ (CLI-only, with side effects):

• Corpus occurrence counting
• Data analysis reports
• BPMF mapping helpers

Preserved in bin_legacy/ (historical reference):

• audit_encoding.swift - Still usable standalone tool (2024)
• C_Version/ - Fast C implementation, phased out in 2013
• Sample_Prep/ - Corpus preparation methodology
• disabled/ - Legacy Perl/Ruby/Bash implementations

Path Configuration Changed

• Before: Each script calculated paths relatively
• After: Import from curation package: from curation import PROJECT_ROOT, CONFIG_FILE

Using Legacy Tools

The audit_encoding.swift tool is still functional:

cd bin_legacy
swift audit_encoding.swift  # Validates BPMFBase.txt encoding categories

C version (for performance comparison):

cd bin_legacy/C_Version
export TEXTPOOL=/path/to/corpus
./count.bash 測試詞彙

For complete migration history and tool details, see bin_legacy/DEPRECATED.md.

Key Contributors

• Mengjuei Hsieh: Original Python implementation (2012-2013)
• Lukhnos Liu: Modernization and associated phrases v2 (2024-2025)
• zonble: Encoding audit tool (2024)

Data Generation Pipeline

The build process flows: phrase.occ + exclusion.txt → buildFreq.py → PhraseFreq.txt → cook.py (+ other inputs) → data.txt → derive_associated_phrases.py → associated-phrases-v2.txt.

For detailed pipeline diagram, frequency calculation algorithms, and heterophony processing logic, see algorithm.md section "字典資料的生成與使用".

Editorial Guidelines

For editorial policies, see Wiki: 詞庫開發說明.

Phrase Quality Control

Check phrase rarity: site:.tw "phrase" (under 1,000 results → likely safe to remove)

Data Integrity Checks

# Character count matches Bopomofo count (empty output = good)
awk 'length($1)/3!=NF-1' BPMFMappings.txt

# Phrase consistency check
diff -u <(awk '{print $1}' BPMFMappings.txt|sort -u) \
        <(awk 'length($1)>3{print $1}' phrase.occ|sort -u)

Testing Your Changes

make tidy sort    # Format and sort
make check        # Validate integrity
make all          # Build output files
make _install     # Install to ~/Library/Input Methods/McBopomofo.app/
pkill -HUP McBopomofo  # Restart

Common Issues and Solutions

Issue	Solution
Sort order is wrong	Always use C locale: LC_ALL=C sort -o file file
Phrase in BPMFMappings.txt not showing	Verify phrase exists in phrase.occ with non-zero frequency
make check fails (character count)	Ensure character count = Bopomofo reading count
Heterophony shows wrong default	Place common reading in heterophony1.list, less common in heterophony2.list
Emoji appears as first candidate	Add to Symbols.txt with negative score (e.g., -8)

Reference Files for Context

• Root AGENTS.md: Overall project architecture and build system
• algorithm.md: Detailed algorithm explanation (Chinese)
• Wiki: 程式架構: Program architecture
• Wiki: 詞庫開發說明: Dictionary development guide
• Wiki: 使用手冊: User manual