Phase 1: Detailed Candidate Identification Plan
Provides step-by-step queries and methods to discover candidate open-source projects across 8 categories for analysis.
What this file does
Provides step-by-step queries and methods to discover candidate open-source projects across 8 categories for analysis.
When to use it
- Starting a systematic project selection process for code review or benchmarking
- Needing concrete search queries and API calls for GitHub and package managers
- Planning a two-week automated then manual candidate discovery pipeline
- Requiring category-specific evaluation criteria for diverse project types
Assumes this stack
Phase 1: Detailed Candidate Identification Plan
Parent: Project Design Related: Project Selection Methodology - High-level selection approach
Overview
This document provides specific implementation details for identifying candidate projects across all 8 categories using multiple discovery methods. This is the detailed execution plan that implements the high-level methodology described in README.md.
Relationship to README.md:
- README.md: High-level strategy, criteria, and timeline
- CANDIDATE_IDENTIFICATION_PLAN.md: Detailed implementation steps and specific queries
- AUTOMATION_OPPORTUNITIES.md: Which tasks can be automated vs. require human judgment
Method 1: Knowledge-Based Selection
Training Data Mining
Implementation:
- Query knowledge base for "best practices" projects in each category
- Search for projects mentioned in architectural discussions
- Identify projects from "awesome lists" and curated collections
- Reference projects from educational materials and tutorials
Specific Queries to Execute:
- "What are the most well-regarded Chrome extensions for developers?"
- "Which Rust libraries are considered best-in-class for performance?"
- "What are the most popular MCP server implementations?"
- "Which full-stack frameworks are known for their architecture?"
- "What are the most respected data science libraries?"
- "Which CLI tools are considered gold standards?"
- "What mobile app frameworks are most recommended?"
- "Which documentation platforms are most praised?"
Expected Output: 5-10 candidates per category from knowledge base
Community Recognition
Implementation:
- Search for projects mentioned in "best practices" discussions
- Identify projects from Stack Overflow "most loved" technologies
- Reference projects from developer surveys and reports
- Find projects mentioned in conference talks and presentations
Specific Sources:
- Stack Overflow Developer Survey results
- State of JS, State of CSS, State of Rust surveys
- GitHub's "Trending" and "Most Starred" lists
- Conference talk abstracts and slides
- Blog posts about "best practices" and "architecture"
Expected Output: 3-5 additional candidates per category
Method 2: Search-Based Discovery
GitHub Trending Analysis
Implementation:
- Use GitHub API to query trending repositories by language/topic
- Filter by category-specific topics and tags
- Analyze trending over different time periods (weekly, monthly, yearly)
- Cross-reference with star growth rates
Specific GitHub Searches:
# Chrome Extensions
topic:chrome-extension
topic:browser-extension
language:JavaScript topic:extension
# MCP Servers
topic:mcp-server
topic:model-context-protocol
language:Python topic:mcp
# Rust Libraries
language:Rust
topic:rust-library
topic:crate
# Full-Stack Systems
topic:fullstack
topic:web-application
topic:api
# Data Science & ML
topic:machine-learning
topic:data-science
topic:ml
language:Python topic:data
# CLI Tools
topic:cli
topic:command-line
topic:terminal
# Mobile Applications
topic:mobile
topic:react-native
topic:flutter
topic:mobile-app
# Documentation Sites
topic:documentation
topic:docs
topic:docusaurus
topic:gitbook
Expected Output: 10-15 candidates per category
Star Count Analysis
Implementation:
- Query GitHub API for repositories sorted by star count
- Filter by category-specific criteria
- Analyze star-to-fork ratios for community engagement
- Cross-reference with recent activity
API Queries:
# Example GitHub API calls
curl "https://api.github.com/search/repositories?q=language:rust+stars:>1000&sort=stars&order=desc"
curl "https://api.github.com/search/repositories?q=topic:chrome-extension+stars:>500&sort=stars&order=desc"
Expected Output: 5-8 candidates per category
Fork Analysis
Implementation:
- Analyze fork-to-star ratios to identify community adoption
- Look for projects with high fork counts relative to stars
- Identify projects with active fork communities
- Cross-reference with recent fork activity
Metrics to Calculate:
- Fork-to-star ratio
- Recent fork activity (last 6 months)
- Fork engagement (issues, PRs from forks)
- Fork maintenance (active forks vs. stale forks)
Expected Output: 3-5 candidates per category
Method 3: Community-Based Research
Developer Survey Analysis
Implementation:
- Analyze Stack Overflow Developer Survey results
- Reference State of JS, State of CSS, State of Rust surveys
- Cross-reference with GitHub's "Most Loved" and "Most Wanted" lists
- Identify projects from "Tools & Technologies" sections
Specific Survey Sources:
- Stack Overflow Developer Survey 2023/2024
- State of JS 2023
- State of CSS 2023
- State of Rust 2023
- GitHub's "Most Loved" repositories
- Developer satisfaction surveys
Expected Output: 2-3 candidates per category
Industry Report Analysis
Implementation:
- Search for "best practices" and "architecture" blog posts
- Identify projects from conference presentations
- Reference projects from technical books and courses
- Analyze projects from "awesome lists" and curated collections
Specific Sources:
- Conference talk abstracts (JSConf, RustConf, PyCon, etc.)
- Technical blog posts about architecture
- "Awesome" lists on GitHub
- Technical book references
- Online course materials
Expected Output: 3-5 candidates per category
Method 4: Category-Specific Discovery
Chrome Extensions
Specific Searches:
- Chrome Web Store "Featured" and "Popular" sections
- GitHub topics:
chrome-extension,browser-extension - NPM packages with
chrome-extensionkeyword - Projects mentioned in Chrome extension development guides
Evaluation Criteria:
- Chrome Web Store ratings and reviews
- User count and active installations
- Security audit results
- Chrome API compatibility
MCP Servers
Specific Searches:
- GitHub topics:
mcp-server,model-context-protocol - Projects from MCP documentation and examples
- AI/ML community discussions about MCP
- Projects referenced in MCP tutorials
Evaluation Criteria:
- MCP protocol compliance
- AI model integration quality
- Documentation completeness
- Community adoption
Rust Libraries
Specific Searches:
- crates.io by download count and rating
- GitHub topics:
rust-library,crate - Projects from Rust "awesome" lists
- Projects mentioned in Rust community discussions
Evaluation Criteria:
- crates.io download statistics
- Reverse dependency count
- Performance benchmarks
- Memory safety practices
Full-Stack Systems
Specific Searches:
- GitHub topics:
fullstack,web-application,api - Projects from full-stack development guides
- Projects mentioned in architecture discussions
- Projects from deployment and scaling guides
Evaluation Criteria:
- Architecture complexity and quality
- Scalability practices
- Deployment strategies
- Monitoring and observability
Data Science & ML Projects
Specific Searches:
- PyPI packages by download count
- GitHub topics:
machine-learning,data-science,ml - Projects from ML community discussions
- Projects mentioned in data science tutorials
Evaluation Criteria:
- PyPI download statistics
- Academic citations
- Reproducibility practices
- Documentation quality
CLI Tools & Applications
Specific Searches:
- GitHub topics:
cli,command-line,terminal - Projects from CLI development guides
- Projects mentioned in developer productivity discussions
- Projects from terminal and shell guides
Evaluation Criteria:
- User experience and interface design
- Performance and resource usage
- Cross-platform compatibility
- Package manager adoption
Mobile Applications
Specific Searches:
- GitHub topics:
mobile,react-native,flutter,mobile-app - Projects from mobile development guides
- Projects mentioned in mobile architecture discussions
- Projects from app store listings
Evaluation Criteria:
- App store ratings and reviews
- User experience quality
- Performance optimization
- Platform compatibility
Documentation Sites
Specific Searches:
- GitHub topics:
documentation,docs,docusaurus,gitbook - Projects from documentation platform guides
- Projects mentioned in technical writing discussions
- Projects from content management guides
Evaluation Criteria:
- Content quality and accuracy
- User experience and navigation
- Accessibility compliance
- Community contribution
Implementation Timeline
Week 1: Automated Discovery (Fully Automated)
Automation Tools:
- GitHub API Collector - Execute all GitHub API searches
- Trending Analyzer - Analyze trending repositories
- Metrics Calculator - Calculate quantitative metrics
- Package Manager Collector - Collect package ecosystem data
- Pattern Matcher - Detect documentation and security files
Automated Tasks:
- Execute GitHub API searches for all categories
- Analyze trending repositories across time periods
- Calculate star growth rates, activity scores, community health metrics
- Collect NPM, PyPI, crates.io download statistics
- Detect presence of README, CONTRIBUTING, LICENSE files
- Detect security files (SECURITY.md, dependabot configs)
- Generate quantitative scoring reports
- Compile initial candidate lists with metrics
Week 2: Human Judgment & Selection
Human Tasks:
- Review automated results and metrics
- Apply qualitative assessment criteria
- Analyze developer surveys and community recognition
- Review conference presentations and industry reports
- Cross-reference with community discussions
- Apply diversity requirements (scale, approach, community)
- Make final selection decisions
Week 3: Final Selection & Documentation
Human Tasks:
- Final project selection and ranking
- Category representation assessment
- Selection rationale documentation
- Quality assurance review
- Handoff preparation for Phase 2
Expected Output
Target Numbers:
- 20-30 candidates per category (160-240 total)
- 5-10 from knowledge base
- 10-15 from GitHub trending
- 5-8 from star count analysis
- 3-5 from fork analysis
- 2-3 from developer surveys
- 3-5 from industry reports
Automation Benefits:
- 70% Token Reduction: From ~50,000 to ~15,000 tokens
- Faster Processing: Automated data collection vs. manual queries
- Consistent Metrics: Standardized quantitative analysis
- Human Focus: Judgment tasks only, not mechanistic data collection
Quality Assurance:
- All candidates meet minimum activity thresholds (automated)
- All candidates have comprehensive documentation (automated detection)
- All candidates demonstrate active community engagement (automated metrics)
- All candidates represent different scales and approaches (human judgment)
This detailed plan ensures systematic discovery of high-quality candidates across all categories while maintaining diversity in scale, approach, and community dynamics.
What's inside
4 discovery methods, 8 category-specific sections, 12 code/query examples, and a 3-week timeline
Change this for your project
- Replace
jmalicki/repo-intelwith your own repository path - Replace
../../projects/01-project-selection/with your own project directory structure - Replace category-specific topics like
topic:chrome-extensionwith your own search terms
Where it goes
Keep it in your repository where the agent or team that needs it will read it.
Worth borrowing
- Separating automated metric collection from human qualitative judgment across two weeks
- Using fork-to-star ratio and recent fork activity as community health signals
- Defining category-specific evaluation criteria alongside generic discovery methods
Related Documents
YouTube Video Script: Train OccWorld on Tokyo 3D City Data with Lambda Cloud
Provides a complete script and visual plan for a YouTube tutorial on training the OccWorld 3D world model on Tokyo city data using Lambda Cloud GPUs.
Agents and LLMs
Serves as a personal reference notebook covering LLM families, fine-tuning, RAG, agents, evaluation, and applied AI topics.
Model Documentation
Documents the transfer learning strategy, architecture, training procedure, and explainability method for a CodeBERT-based human-vs-LLM code classifier.
Transformer Model Technical Research Analysis
Explains hyperparameters and architectural choices of a small Transformer decoder, including benefits and trade-offs of increasing each.