TODOs (in priority order)
Tracks 13 pending and 13 completed development tasks for a Python toolkit, prioritised and with deferred items noted.
What this file does
Tracks 13 pending and 13 completed development tasks for a Python toolkit, prioritised and with deferred items noted.
When to use it
- Planning your own project backlog in priority order
- Reviewing what features are still missing before a release
- Deciding which tasks to assign to contributors
TODOs (in priority order)
- Use custom
clickargument types to support theconfiglibfor functions. - Design a module for benchmark building pipelines
- Use reflection for automated logger configuration for the log module
- Implement
log.watchdog - Figure out how to strictly separate
load_configandload_envforconfigurableclasses - Simplify the usage pattern of
load_configandload_env - Add error messages if using a configuration setting before it is loaded
- Develop a standard operating procedure (sop) for redacting the identity information for anonymous artifact submission in paper reviewing
- Refine the ergonomics of
prompt - Add format specifiers to
prompt - Add a
py-tree-sitterwrapper module - Move in the LLM pricing tracker
- Develop a sop to integrate Python code with native language code (such as Rust and Zig) to accommodate memory/time critical scenarios
-
We should implementDeferred: The current status of Zig toolchains is immatureIntervalTreein thealgmodule in Zig, as the Python implementation consumes too much memory -
subshellsometimes misses outputs, especially whenPHDKIT_SIMPLE_SUBSHELLis on -
Consider how to useconfiglibfor the context manager protocol - Explore how to handle optional settings in
configlib - Add
postloadfunctionalities forconfiglib - Add docs for default setting values in
configlib - Implement the
subshellfunctionality of therichwrapper module - Make
infix_fntype-safe - Audit this whole "vibe-coding" interval tree implementation for correctness and performance
- Consider how to handle configurations of base classes in
configlib -
Implement theAlready supported bydyntqdmfunctionality of therichwrapper modulerich - Test whether
configlibhandles arrays in TOML correctly - Add a flag to
subshellto switch it to a simple interface when interacting with coding agents
What's inside
26 checklist items: 13 open, 12 completed, 1 deferred, each with a short description.
Change this for your project
- Replace
phdkitreferences with your own project name - Replace
cychen2021/phdkitrepository URL with your own - Replace
PHDKIT_SIMPLE_SUBSHELLenv var name with yours
Where it goes
Keep alongside your test suite. Used to define and score model evaluations.
Related Documents
IATA BCBP Standard Compliance
Documents which IATA BCBP fields and barcode formats a Swift library implements, including Version 8 gender code support.
Evaluating AI Agent Systems: Metrics, Benchmarks, and Quality Assurance (2024-2026)
Surveys 2024-2026 metrics, benchmarks, and monitoring tools for evaluating AI agent systems, with recommendations for a self-improving coding agent.
Voice AI Leaderboards, Benchmarks, and Evaluation Gaps (Jan 2025 -- Feb 2026)
Surveys 20+ voice AI benchmarks from Jan 2025, Feb 2026, identifies evaluation gaps, and provides leaderboard data for STT, TTS, and end-to-end voice agents.
LLM Evaluation — Deep Dive
Covers why LLM evaluation is hard, benchmark taxonomy, LLM-as-judge methodology, contamination, robustness, and designing an eval suite for a product.