Instructions to the Evaluator (delete this section)
Provides a peer-evaluation rubric for student teams to assess and improve each other's open-source project materials.
What this file does
Provides a peer-evaluation rubric for student teams to assess and improve each other's open-source project materials.
When to use it
- Running a course where teams evaluate each other's projects
- Creating a structured peer-review process for open-source assignments
- Setting deadlines and deliverables for iterative project feedback
- Guiding evaluators to test prototypes and give actionable suggestions
Instructions to the Evaluator (delete this section)
This rubric is meant to be "filled out" by the evaluator. Your goal is to assist the team being evaluated with a first draft of materials that might be used for proposals (e.g., to Big Ideas, grants, etc.), recruitment/on-boarding, and generally having a good public face on their online project description. This will lead to information added to their class project page and GitHub repositories by the end of class.
Timeline
- Nov 12: Initial check-in between teams. See the
progress.htmlpage for assigned teams. - Nov 19: Evaluator delivers rubric to the team they are helping. By default, use the GitHub issue tracker for the team's project. If feedback is NOT in the issue tracker by Nov 19, please contact @davclark and explain the situation.
- Dec 3: Final presentations, with a brief tour of the project. Evaluation should be incorporated into class website and repo as appropriate.
- Dec 10: Alternate date for presentations.
This file is meant to ultimately land (with appropriate edits) into the project description page (in this projects directory).
Please adopt a friendly but critical perspective. You are trying to help the other team, and no one is going to "lose points" for any shortcomings. The goal is to get everyone as far towards their goals as possible!
Essential to a good "open" project is discoverability by a friendly netizen. Ideally, someone would be able to get up to speed on a project without bothering you until they have something useful or constructive to share! Thus, please provide URLs below (where appropriate) where someone could find this information on the internet.
Real-world challenge
What new understanding will be obtained? What public good would be served? What business opportunities are available? Who would benefit, who are consumers/users of your data or technology?
Why are you motivated to work on this project?
Data / Materials
What data sources or data collection methods have you used? Are they automatic/numerically collected, or manually collected (as with a survey)? Where / how is data stored? What will you do / have you done to get data? Is there important metadata (e.g., time of collection, source, etc.)? Why is the data you have or expect to have sufficient or insufficient to meet your challenge?
Approach
What are the risks? What technical approaches have been explored, and which remain to be explored? Where does the prototype "run?" What methods (e.g., statistics, signal processing, transformation) and tooling (e.g., python libraries, hardware platforms) are being used/evaluated/considered?
When does analysis come in.
What skills are needed, and what skills remain to be learned by your team? How would someone learn these skills?
Project Management
What is your first milestone? What is your current milestone? What is your "final" milestone for completing this project (even if that milestone is currently unreasonable ;).
How do you evaluate progress and work remaining? How are next tasks determined and divided between team members?
Who did what / what role did people play, and how could you find out (e.g., from the issue tracker, meeting notes, git commits, etc.)?
Other
Please provide any feedback you find useful! In particular, please help your partner team to identify potential problems that might cause their project to "fail." Also recommend any resources that you think may be useful.
Ideally, also "execute" or "test" the current prototype. This could range from assembling the bill of materials (i.e., shopping cart) to build the hardware to actually running some code on your own computer. If this is not possible, please describe what would be necessary to do so, and why you are unable to.
What's inside
5 sections (Timeline, Real-world challenge, Data/Materials, Approach, Project Management, Other) plus a header note
Change this for your project
- Replace
Nov 12,Nov 19,Dec 3,Dec 10with your own course dates - Replace
@davclarkwith the instructor's GitHub handle - Replace
progress.htmlwith the actual URL or page name for team assignments
Where it goes
Use as the system prompt for your model call, or as the base instructions in your agent framework.
Worth borrowing
- Asking evaluators to test the prototype and report what prevented them from doing so
- Requiring evaluators to provide URLs for each section to enforce discoverability
Related Documents
Character Persona
Defines a Fallout 2 roleplay persona for an LLM agent: charming rogue with social-first combat and moral code.
How to implement tool use
Explains how to define, configure, and control tool use with the Anthropic API, including model selection, tool definitions, output control, and parallel calls.
Gemini by Example
Collects 20+ Gemini API code examples covering text, images, audio, video, PDFs, agents, and token counting.
Claude Tool Use (Function Calling) Documentation
Teaches how to define and use Claude tool use (function calling) with Python, including schema rules, execution patterns, and best practices.