Orthogonal Rubrics
March 9th, 2010As I was putting together a grading rubric for a recent student assignment, I had no trouble turning the stated criteria into grading criteria. There are 15 points possible for the assignment, so I broke down the assignment’s expectations into 15 pieces, each worth one point. This has made grading papers a breeze. Does the student provide a good definition of the concept? 1 point for yes, 0 for no, and .5 for a valid effort. But as I graded more papers using the rubric, I started to notice some issues with my criteria. I hadn’t considered whether or not all concepts were truly orthogonal.
First, some explanation of the concept of orthogonality. It’s a statistical concept that asks whether each factor being measured (or each test performed) is providing solely unique information. For example, cutting up a pizza is an orthogonal task. There’s no way that part of one slice can also be a part of another slice. No slices can overlap. This is important in statistics as it allows you to make stronger conclusions about the effect of one factor and because it allows you to keep your chance of error to a minimum.
So how does a statistical concept relate to building a rubric? For each of the standards that I put on my grading rubric, I expected to be able to evaluate them independently. Having an interesting introduction has nothing to do with providing a proper definition; neither of those factors relate to whether the student used proper grammar. Thus each of those factors is orthogonal (or independent). But what about these two items: appropriate choice of concept and successful application of concept. These concepts are not always orthogonal. For example, if the student chooses a concept that is not valid because it isn’t part of the course, then that student can’t apply the concept to explain her group’s interaction. In that case, the student loses two points for one mistake.
What’s the big deal?, you might ask. The rubric is perfectly clear on how many points are awarded for these two factors. That’s perfectly true. Except that if the concepts are not orthogonal, then I can’t tell what I’m really measuring. Does the student understand what an appropriate concept is? Maybe, but not based on her paper. Does the student understand how to apply a concept? Maybe, but I can only tell if the student chooses an appropriate concept in the first place. So the student’s answer can’t actually tell me what the student understands. To award points when I don’t have the required information to make the decision is not ethical.
Before considering what might be done to solve this problem, let me recognize that one of the issues is my specific point-by-point system, which allows these concepts to be considered in an orthogonal sense in the first place. A more subjective grading system might not have this problem. But because I believe that these types of rubrics are most transparent and helpful, I think it is useful to consider whether or not your concepts are orthogonal.
What should be done instead? After assembling your point rubric, take a glance over it. Are there areas where you expect that some students will fall into a pit of lost points? Are there items that you think are tied a bit too closely together? If there are, you need to revise the rubric and adjust the points awarded. After taking a look, you may still need to adjust after grading a few papers. I didn’t recognize this problem until I found a couple of students who had picked incorrect concepts. Thankfully I recognized the issue right away and made adjustments in my grading (though not to the rubric itself).
On a broader scale, these kinds of adjustments may be required for a wide variety of ways that we treat students. For example, let’s say a student asks for an extension to a paper one hour before a paper is due and another student asks one hour after. Our inclination might be to give an extension to the student who asked prior to the due date and not to the student who asked after. But what other factors may be at work here? Does the timing of the request also correlate with the quality of student? If it does, then we may be granting extra time to good students and not to bad students.
Yet, how much does the extra time influence good students versus bad students? I’m guessing that it’s not equal. For a good student, extra time may be put to good use. For a bad student, extra time may not make a difference at all. So to grant extra time based on factors that ALSO correlate with good students versus bad students may be to grade the bad student even more harshly and the good student even more leniently. This may further the gap between good students and bad students.
I’m not suggesting, then, that grading policies be made hard and fast or that all students be treated to exacting standards. These too tend to favor good students over bad. Rather, I’m suggesting that, within reason, instructors stay flexible. Be flexible in grading and be mindful of ways that your grading rubric may systematically favor some students over others, above and beyond their assignment quality. Be flexible in allowing for extra time, knowing that the only component of assignments that should be graded is the quality of the work. Of course, this doesn’t mean that all students get As or that students can turn in a paper whenever they want. After all, it’s a school, not a zoo. But to not be aware of systematic, nonorthogonal, discriminatory factors that disadvantage some students and not others - that’s just plain ignorance.