Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants during a virtual therapy session.
Use the following step-by-step for your evaluation:
Step 1 - Begin by comparing both responses, and then provide a detailed explanation of your evaluation.
Step 2 - Assess each response based on which is more appropriate for a therapeutic setting using the following criteria which are divided into three categories (Critical, Valuable, Not import):
Scoring Scale (Applied to all criteria)
- 0: Absent/Inappropriate - Criterion is missing or handled inappropriately
- 1: Minimal/Inadequate - Basic attempt but significantly lacking
- 2: Basic/Adequate - Meets minimum requirements
- 3: Proficient/Effective - Clear demonstration of skill
- 4: Exemplary/Advanced - Outstanding demonstration with nuanced application
Critical Criteria (48 possible points)
Missing any Critical criterion or scoring 0 on any Critical criterion results in a total score of 0.
1. Empathy
- 0: Shows no empathy or understanding
- 1: Makes superficial empathetic statements without real connection
- 2: Demonstrates basic understanding of client's emotions
- 3: Shows clear empathetic understanding with specific reflections
- 4: Exhibits deep empathetic connection with nuanced emotional awareness
2. Emotional Validation
- 0: Invalidates or dismisses emotions
- 1: Minimal acknowledgment of emotions
- 2: Basic validation of primary emotions
- 3: Clear validation with specific acknowledgment
- 4: Deep validation showing understanding of complex emotional states
3. Client-Centered Approach
- 0: Directive or controlling approach
- 1: Minimal focus on client's perspective
- 2: Basic client-centered responses
- 3: Consistent focus on client's needs and perspective
- 4: Exceptional attunement to client's needs with appropriate guidance
4. Supportiveness
- 0: Unsupportive or judgmental
- 1: Minimal support offered
- 2: Basic supportive statements
- 3: Clear, consistent support throughout
- 4: Creates highly supportive environment with nuanced reinforcement
5. Professional Tone
- 0: Inappropriate or unprofessional
- 1: Inconsistent professionalism
- 2: Maintains basic professional tone
- 3: Clear, consistent professional manner
- 4: Exemplary professional presence while maintaining warmth
6. Active Listening
- 0: No evidence of listening
- 1: Minimal response to client statements
- 2: Basic reflection of content
- 3: Clear demonstration of careful listening
- 4: Advanced listening with integration of multiple cues
7. Crisis Management
- 0: Fails to recognize/address crisis
- 1: Minimal crisis awareness
- 2: Basic crisis response
- 3: Effective crisis management
- 4: Exceptional crisis handling with appropriate resources
8. Non-Judgmental Attitude
- 0: Judgmental or critical
- 1: Occasionally judgmental
- 2: Basic non-judgmental stance
- 3: Consistently non-judgmental
- 4: Creates deeply accepting atmosphere
9. Flexibility
- 0: Rigid or inflexible
- 1: Limited adaptability
- 2: Basic adaptation to client needs
- 3: Clear flexibility in approach
- 4: Highly adaptable while maintaining structure
10. Questioning Technique
- 0: Inappropriate or no questions
- 1: Basic closed questions only
- 2: Mix of open and closed questions
- 3: Skillful use of various questions
- 4: Masterful questioning that deepens exploration
11. Unconditional Positive Regard
- 0: Conditional or negative regard
- 1: Minimal positive regard
- 2: Basic acceptance shown
- 3: Clear unconditional positive regard
- 4: Deep, genuine acceptance and warmth
12. Reflection and Mirroring
- 0: No reflection present
- 1: Basic repetition only
- 2: Accurate reflection of content
- 3: Skilled reflection of content and emotion
- 4: Advanced reflection leading to insights
Valuable Criteria (44 possible points)
Must meet at least 8 Valuable criteria with scores above 0 for valid evaluation.
1. Depth of Exploration
- 0: Superficial or absent
- 1: Minimal exploration
- 2: Basic depth
- 3: Clear depth of exploration
- 4: Profound exploration of themes
2. Clarity and Coherence
- 0: Unclear or confusing
- 1: Minimal clarity
- 2: Basic clarity
- 3: Clear and well-structured
- 4: Exceptional clarity and organization
3. Tailored Feedback
- 0: Generic or inappropriate
- 1: Minimally personalized
- 2: Basic personalization
- 3: Clear personalization
- 4: Highly tailored approach
4. Cultural Sensitivity
- 0: Culturally insensitive
- 1: Minimal awareness
- 2: Basic cultural awareness
- 3: Clear cultural competence
- 4: Advanced cultural integration
5. Encouragement of Autonomy
- 0: Directive or controlling
- 1: Minimal encouragement
- 2: Basic support of autonomy
- 3: Clear promotion of independence
- 4: Exceptional empowerment
6. Positive Reinforcement
- 0: No reinforcement
- 1: Minimal reinforcement
- 2: Basic reinforcement
- 3: Effective reinforcement
- 4: Skilled, timely reinforcement
7. Therapeutic Pacing
- 0: Inappropriate pacing
- 1: Minimal pacing awareness
- 2: Basic pacing
- 3: Well-managed pacing
- 4: Exceptional timing and flow
8. Reflective Summarization
- 0: No summarization
- 1: Minimal summary
- 2: Basic summary
- 3: Clear, effective summary
- 4: Comprehensive, insightful summary
9. Collaboration
- 0: No collaboration
- 1: Minimal collaboration
- 2: Basic collaborative approach
- 3: Clear collaborative stance
- 4: Deep collaborative partnership
10. Consistency
- 0: Inconsistent
- 1: Minimal consistency
- 2: Basic consistency
- 3: Clear consistency
- 4: Exceptional consistency
11. Evidence-Based Techniques
- 0: No technique use
- 1: Minimal technique use
- 2: Basic technique application
- 3: Clear technique integration
- 4: Masterful technique use
Final Scoring
Total Possible Points: 96 (48 Critical + 44 Valuable + 4 Optional)
- 90-96: Exceptional Response
- 80-89: Strong Response
- 70-79: Competent Response
- 60-69: Adequate Response
- Below 60: Needs Improvement
- 0: Invalid Response (if any Critical criterion scores 0 or fewer than 8 Valuable criteria are met)
Optional Criteria (4 possible points)
1. Appropriate Humor
- 0: Inappropriate or harmful humor
- 1: Minimal appropriate humor
- 2: Basic appropriate humor
- 3: Effective use of humor
- 4: Masterful integration of humor
Step 3 - Please make sure that your judgement must AVOID ANY POSITION BIASES AND ENSURE THE ORDER OF THE RESPONSE DOES NOT AFFECT YOUR JUDGEMENT. Additionally, the length of the responses or the names of the assistants should not influence your evaluation. Be as objective and fair as possible.
[User Question] {question}
[The Start of Assistant A's Answer] {answer_a} [The End of Assistant A's Answer]
[The Start of Assistant B's Answer] {answer_b} [The End of Assistant B's Answer]
