Application of rubrics to Goal Aware Activities
Ever since we conceptualized and built the Sakai Goal Management tool we have heard a need for a rubric built into the tool. However, it has not been very clear how we should manage and apply rubrics. Today, in the OSP 2.5 planning meeting, I heard of a possible way we could design the tools that made some sense to me. However, I think that the implication that I drew from the conversation makes we wonder if the idea is implementable.
Our current design for “Goal Aware” tools allows a user to create (and centrally manage) a tag (a Goal, in our parlance) for each learning outcome for a class. Another type of tag that schools might find a lot of value in would be a managed “Stage” type tag. As it was explained to me, for program assessment, there needs to be a managed system of “Outcomes”, “Stages” and rubrics for each combination of the two.
While we allow multiple “Goals” to be attached to any one activity (an assignment for example), we wouldn’t expect activities to apply to more than one “Stage”. As it was stated in this meeting, there would be the expectation that at any one stage, only one rubric should be used when evaluating student performance (ratings, in our case) for an outcome of the activity. This would help support the investigation into inter-rater reliability when more than one person is rating students.
It looks like this:
| "Stage" type tags | |||||||
|---|---|---|---|---|---|---|---|
| Stage 1 | Stage 2 | Stage 3 | |||||
|
"Goal" |
Outcome 1 | Stage 1 Rubric for Outcome 1 | Stage 2 Rubric for Outcome 1 | Stage 3 Rubric for Outcome 1 | |||
| Outcome 2 | Stage 1 Rubric for Outcome 2 | Stage 2 Rubric for Outcome 2 | Stage 3 Rubric for Outcome 2 | ||||
| Outcome 3 | Stage 1 Rubric for Outcome 3 | Stage 2 Rubric for Outcome 3 | Stage 3 Rubric for Outcome 3 | ||||
| Outcome 4 | Stage 1 Rubric for Outcome 4 | Stage 2 Rubric for Outcome 4 | Stage 3 Rubric for Outcome 4 | ||||
| Outcome 5 | Stage 1 Rubric for Outcome 5 | Stage 2 Rubric for Outcome 5 | Stage 3 Rubric for Outcome 5 | ||||
So far, so good. That makes sense to me, if the rubrics are simple rating scales that inform the "rating" process (ie: result in a single number), but my understanding is that they are not. The usual definition of a rubric is yet another matrix of "performance indicators" and a "rating scale" for each of those indicators. Typically the results of each of the ratings figure into the overall "grade" of an assignment, but in this case only result into a single rating one just one goal that an activity was supposed to address.
Since I am not a teacher, I can't say for sure, but this seems like a lot of work to do to assess a single student. My feeling is that teachers/faculty will largely oppose that level of granularity when doing assessment.