Book PDF to Structured JSON

Produce a traceable electronic edition whose hierarchy comes from the printed table of contents and whose remote copy can be proven identical to the reviewed local JSON.

Read the right reference

• Read references/pipeline.md before planning or executing a book conversion.
• Use scripts/compare_tree_export.py after a full remote export. It compares titles, bodies, levels, order, and parent relationships and emits reproducible SHA-256 fingerprints.

Core rules

1. Treat the rendered printed TOC as the structural authority. Do not infer the final tree only from OCR body headings.
2. Preserve source evidence: page boundaries, source-page mapping, heading-match score, and the original extracted text.
3. Separate detection from correction. OCR models and rare-character audits propose candidates; only reviewed, context-safe rules may change text.
4. Make the JSON tree the authoritative deliverable. Generate TXT, TSV, chapter files, and upload payloads from that same tree.
5. Fail closed on uncertain headings, missing parents, duplicate keys, cycles, unexpected node counts, empty leaf content, or unexplained text loss.
6. Never mutate a remote environment until the user has authorized upload and the exact environment, equipment/library ID, and book set are known.
7. Export a full remote backup before replacement. Pass credentials through environment variables or standard input; never place tokens in commands, reports, or logs.
8. After upload, export full remote bodies and compare normalized content and structure. Counts alone are not acceptance evidence.

Access and authorization

• Read only the source books and extraction evidence the user placed in scope.
• Write only to the approved local output and audit locations.
• Use network access only when the user explicitly requests upload or remote read-back verification.
• Do not create background jobs, persistence, credential stores, or privilege changes.

Execution outline

1. Inventory sources and classify each as text PDF, scanned PDF, DOCX, or TXT.
2. Extract page-aware UTF-8 text; use OCR only where the text layer is absent or demonstrably worse.
3. Render and transcribe the printed TOC, including hierarchy, printed page number, and expected item count.
4. Locate every TOC heading near its expected source page; record score and offsets, and stop for ambiguous matches.
5. Slice content between adjacent located headings and clean only layout artifacts and reviewed OCR errors.
6. Build and validate the canonical tree, then generate all derivative artifacts.
7. Run static, structural, semantic-candidate, and cross-artifact audits. Review candidate corrections and rerun validation.
8. If upload is requested, back up the target, import only the authorized books, read back the full dataset, and run the deterministic comparator.
9. Report the exact local version directory, book/node totals, audit results, backup path, remote target, and per-book fingerprints.

Completion standard

Do not call the work complete unless:

• every printed TOC entry maps to exactly one node in the intended order;
• every parent exists and the hierarchy is acyclic and level-consistent;
• expected content-bearing nodes are nonempty;
• no unexplained replacement characters, mojibake, private-use glyphs, page headers, or page markers remain;
• running headers are removed even when the PDF text layer merges them into an adjacent body line, and footnotes do not interrupt or reorder a sentence continued on the next page;
• the final included node ends at a reviewed terminal boundary rather than blindly consuming the PDF tail;
• JSON parses and all derivative files are regenerated from it;
• the remote full export matches the reviewed local tree when an upload occurred.