QualRubric

## PhD Qualifying Exams

The goal of a Qualifying Exam ("qual") is for a student to *effectively demonstrate that they have the knowledge and skills that will be needed to conduct meaningful research in their chosen subfield*. There are a number of key phrases in this sentence:

* **effectively demonstrate** : the student should give a presentation of a research topic in a way that is organized, clear, correct, and convincing, as judged by the examiners.

* **knowledge and skills** : the student should demonstrate during the exam that they understand fundamental topics in ECE that are directly needed for and/or related to their research.

* **meaningful research** : the student should be able to contextualize the topic that they present to describe why the problem/question is important and why the proposed solution or analysis is appropriate/effective.

* **chosen subfield** : the student may give a talk related to their own research but the exam is about evaluating their command of the key ideas and tools in a subfield/sub area of ECE.

The goal of the committee/examiners is to:

1. Determine if the student has effectively demonstrated the skills.
2. Provide feedback to the student about areas in which they need to improve.
3. Make a recommendation to the Graduate Director as to whether the student has passed, conditionally passed subject to some requirements, or should retake the exam.

The goal of a rubric is to make it clearer to the examinees and examiners what issues must be addressed by the presentation and on what basis the recommendation will be made.

### Questions to answer

To make questions easier to state, let's assume the student is presenting their own research, perhaps based on a recently submitted or accepted conference paper. It's important to note that "the presentation" is both the prepared materials (slides) and the questions and answers given by the student. 

It's tempting to make a numerical rubric (e.g. 1-5) for each of the questions below. This would invite a score-based approach with a cutoff for passing etc. This is not aligned with the intent of the exam, which is to evaluate whether the student is prepared to do doctoral research. Taking in the feedback/answers to the questions can give a more constructive approach to a conditional pass or a failure so that the student can fix the things that were lacking.

#### Context and importance

There are at least four ways in which the presentation should provide context for the project being presented. 

1. Did the presentation justify the need for the research in terms of a compelling application?
2. Did the presentation explain desired outcomes and design constraints?
3. Did the presentation describe (with citation) what prior work has been done and how the project differs from/is similar to the prior art?
4. Did the presentation make clear how the tools and ideas used in the project are related to fundamentals of the subfield/concentration area?
5. Did the presentation "close the loop" by explaining the degree to which the project addressed the problem in the identified application and what further work would be needed?

**Example:** in a project about designing a new sensor, the presentation should explain where such a sensor would be used, why it is needed, what kinds of sensors are available now, what limitations are there on the design, and how the solution relates to topics learned in introductory graduate-level classes.

#### Organization, accessibility, and credit

The presentation should be organized so that the examiners can follow what the student is describing. In particular, the presentation should not be a conference talk for specialists because the examiners will not in general be specialists in the particular project being presented.

1. Did the presentation have a clear structure and organization?
2. Was the presentation understandable by researchers in the general field of study?
3. Did the presentation make effective use of visual/graphical elements?
4. Were sources of figures or other material acknowledged?

**Example:** in a project about designing error correcting codes for molecular communication systems, the presentation should be understandable to an examiner who knows about signal processing in wireless communication systems. The presentation should describe the problem of cells communicating to each other using chemicals and "quorum sensing" in enough mathematical detail for someone who is familiar with the mathematical tools but not the chemistry would be able to follow.

#### Knowledge and skills

The presentation should convince the committee that they have a solid foundation on which to further extend their knowledge and skills to address the future research questions 

1. Did the presentation explain the tools and techniques used in sufficient detail?
2. Did the presentation explain the central insight that enabled the project?
3. Did the presentation connect the project to foundational works and concepts?
4. Did the presentation explain contribution/novelty/innovation of the project in a sufficiently technical way?

**Example:** in a project about optimizing computations involving heterogeneous computing platforms, a presentation should explain the key "trick" that allows for a more efficient solution. The differences between the setup of the application and the "textbook" version of parallel and distributed computing should be clearly stated.

### The exam process

It is tempting to think that a "perfect" presentation could address all of these questions without the need for further questioning by the examiners. However, that is not a realistic objective and in fact is *not a desirable outcome*. A better goal is for everyone to learn something fromt the process. The examiners can learn more about a research topic in which they are not an expert. The student can get valuable feedback from experienced researchers with similar domain expertise. This feedback could be:

* *Pointers to research related to future work.* The goal for the student should be understand how their work/presentation connects to adjacent areas but sometimes there are research areas or techniques which are more tangentially connected but might still be useful. For example, there may be more advanced ideas from another field (mathematics, physics, etc.) that could be useful for future work.
* *Advice on how to make the "story" of the research more accessible.* Part of learning to be a researcher is learning how to communicate technical ideas to a variety of audiences. Even if a student can make their ideas clear to some professors in related areas, there might be some ideas on how to communicate to other audiences.
* *Recommendations for other application domains for the research.* A student might be have been motivated by one application setting but perhaps the ideas could be applicable to other applications that they have not considered.

#### Preparing the slides and materials

There are some bad strategies one could use in preparing for the qualifying exam:

* Making a 100 slide deck with backup slides on every antipated question.
* Taking a 20 minute conference talk and basically giving that with no major changes.
* Trying to create the perfect presentation and focusing on addressing each of the questions listed above one by one.

The first is bad because it's a huge time sink. The second is bad because it assumes the examiners are already in the narrow subfield of the topic. The third is bad because it priotizes passing the exam instead of telling the "story" of the research