CS 2112 Peer Evaluation

After all deliverables have been submitted to CMSX for group projects, we will ask you to fill out an online form to officially evaluate your teammates. The purpose of this peer evaluation is to evaluate team citizenship, not technical capability.

You will rate each team member, including yourself, on the same scale. These ratings should reflect each individual’s level of participation and effort and their sense of team responsibility. The scale is as follows:

Excellent: Consistently carried more than their fair share of the workload
Very good: Consistently did what they were supposed to do, very well prepared and cooperative
Satisfactory: Usually did what they were supposed to do, acceptably prepared and cooperative
Ordinary: Often did what they were supposed to do, minimally prepared and cooperative
Marginal: Sometimes failed to show up or complete assignments, rarely prepared
Deficient: Often failed to show up or complete assignments, rarely prepared
Unsatisfactory: Consistently failed to show up or complete assignments, rarely prepared
Superficial: Practically no participation
No show: No participation at all

You will also write a brief comment to justify your ratings. Your comments will not be revealed to your teammates. Only the professor and perhaps head TAs will see your comments.

The ratings you give your peers and yourself will be transformed into a numeric score for each team member. That score will be used to scale the scores group members receive on the project. So, the ratings you submit will not be directly revealed to your team members, but some function of them will be.

Just do your best to honestly assess yourself and your teammates. The rating system we are using is robust and works well in practice. We have used it for a few years in this course with success. It has also been studied academically; see the citations at the end of the document for details.

The rest of this document describes how scores will be calculated. The details are here as reassurance, rather than because you need to know them. It’s fine to stop reading here.

Calculation of Scores

Each qualitative rating will be transformed into a quantitative rating as follows:

Excellent: 100
Very Good: 90
Satisfactory: 80
Ordinary: 70
Marginal: 60
Deficient: 50
Unsatisfactory: 40
Superficial: 25
No Show: 0

The individual rating for a team member is their average quantitative rating, including their own self-rating.

The team rating is the average of all the quantitative ratings for all team members.

The individual adjustment factor (henceforth, factor) is an individual’s rating divided by the team rating, times 100. The factor is capped at 105.

The individual adjustment factor score will be used to scale scores on the project assignments.

If the course staff decides that peer evaluation is penalizing students scores too harshly, we may choose to apply the adjustment to less than 100% of the assignment score. Also, we will not apply it to any part of the assignment that group members do separately, such as individually completed written problems.

Examples

Next, we discuss some situations that might arise and how this scoring system handles them.

Everyone gives the same rating to everyone. Then everyone gets an adjustment factor of 1. For example:

Name	Vote 1	Vote 2	Vote 3	Individual Rating	Adjustment Factor
David	70	70	70	70	100
Anne	70	70	70	70	100
Michael	70	70	70	70	100
			Team:	70

Note that it doesn’t matter whether everyone used 70 or 100 or 25 for their votes. As long as everyone agrees, everyone gets the adjustment factor of 100.

One person dislikes the rest of the team. Then the other team members’ scores go down, but not by much.

Name	Vote 1	Vote 2	Vote 3	Individual Rating	Adustment Factor
David	100	100	0	67	86
Anne	100	100	0	67	86
Michael	100	100	100	100	105
			Team:	78

Whoever submitted Vote 3 (probably Michael) has caused David and Anne’s scores to go down. Out of their final grade in the entire course, this makes little difference. We rarely see such wide disparities in ratings, and in such cases the course staff will examine the justifications submitted to keep things honest.

The rest of the team dislikes one person. That person’s score goes down by about half.

Name	Vote 1	Vote 2	Vote 3	Individual Rating	Adjustment Factor
David	100	100	100	100	105
Anne	100	100	100	100	105
Michael	0	0	100	33	43
			Team:	78

It looks like David and Anne don’t like Michael's contributions. This could have a significant effect on his score, so again, the course staff will examine the written comments of all group members and make appropriate adjustments.

The dislike is mutual. Then the outcome doesn’t change by much.

Name	Vote 1	Vote 2	Vote 3	Individual Rating	Adjustment Factor
David	100	100	0	67	105
Anne	100	100	0	67	105
Michael	0	0	100	33	60
			Team:	56

This time Michael dislikes David and Anne, too. Their scores remain unchanged; his goes up by a little.

Team members fail to provide ratings. If a team member fails to vote, that person’s column will be filled automatically. A 0 will imputed to any team member who didn’t vote (including themself), and a 25 to those who did. For example, suppose that Michael failed to vote. Then his vote (#3 below) would be filled in with a 25 for Anne and David and a 0 for himself:

Name	Vote 1	Vote 2	Vote 3	Individual Rating	Adjustment Factor
David	100	100	25	75	104
Anne	100	100	25	75	104
Michael	100	100	0	67	92
			Team:	72

About a 10% deduction for Michael results: it is important to fill out peer evaluations.

Acknowledgment

The core of this rating scheme has been examined and found to be highly useful and infrequently problematic in three academic publications:

R.W. Brown. Autorating: Getting Individual Marks from Team Marks and Enhancing Teamwork. IEEE Frontiers in Education Conference, 1995.
D.B. Kaufman, R.M. Felder, H.Fuller. Accounting for Individual Effort in Cooperative Learning Teams. J. Engr. Education 89(2): 133-140, 2000.
B. Oakley, R.M. Felder, R. Brent, I. Elhajj. Turning Student Groups into Effective Teams. J. Student Centered Learning 2(1): 9-34, 2004.