The Living Schoolbook

Disclaimer: For those of you who do not know me, I am not a teacher. I provide technical support for grant funded educational projects in the School of Education at Syracuse University. Take this perspective with that grain of salt.

Over the past three years of working with the Open Source Portfolio and Sakai communities, I have heard lots of people talk about how they plan to implement the portfolio tools. Some of the uses highlight the ability of the tools to create a standard presentation (like a resume or curriculum-vitae) by using a template. In a later entry, I’ll be walking you through the steps I would follow in order to create a set of forms, wizards and a template that will allow users to pop out a presentation like that. This entry isn’t about those types of uses. I am sure that everyone understands the purpose of a portfolio like that. I want to take a moment to talk about the types of portfolios that can be useful in education settings. This entry doesn't really talk about how to imlpement an accreditation portfolio. I want to set the stage first.

The outcomes based accreditation or program review portfolio

Some schools need a means to review student learning/progress/achievement of ALL of the students in their programs in a very comprehensive way. The prerequisite to a system like this is that the faculty has decided upon an assessment framework that will be applied to all of their students and all of their classes. A “portfolio” effort like this has several components.

Establishment of a framework for evaluation

In order to set this up, a set of outcomes must be agreed to by the faculty. This is a process that many schools apparently do not engage in very often. I haven’t heard of a faculty that spontaneously decides to engage in this process. They usually have more interesting things to do than debate the purpose of the programs in which they teach. Usually it is a mandate from an accrediting body that initiates the process. It was for our faculty.

Identification of key program assessments

A plan has to be designed for measuring the performance/achievement of the students in the program (and in so doing, the effectiveness of the program). A natural tool to use for this purpose is the assessments already in place in the program’s courses. Chances are, a subset or cross section of the work the students already do in their classes would provide the information needed to see how the students and the program are doing. By highlighting the assessments already in place in the program’s curriculum and matching them to the outcomes that they address we can aggregate student work aggregated into FACULTY DESIGNED portfolios that serve the purpose of the program.

Assessment of student work in light of the program outcomes

Once the a faculty member has identified an assessment (an assignment usually) in their course that is intended to assess (at least in part) one or more of the program outcomes, the student’s work will need to be assessed against each of those outcomes. Whether the instructor of the course performs that assessment while grading papers or a "program review panel" performs the assessment later on, the process requires standardized rating scales and rubrics to ensure that all of the faculty are on the same page when assessing the student work. Establishing standard rubrics that ensure inter-rater reliability is another huge task that takes place outside of the “system”. The axiom “garbage in – garbage out” applies here.

Analysis of the Data

It may surprise you to see the multiplicative effects that adding new assessments, outcomes and students to this structure has on the amount of data that there is to look at and analyze. A cohort of 100 students going through a program with 30 highlighted assessments each identified as addressing on average of 2 outcome each provides 6000 pieces of data for analysis. If assessment is continuously performed by instructors (rather than by a “panel” at the end of the semester) it might be helpful if both students and the faculty could have real time access to this assessment portfolio data to help identify problems and trends in student performance and hopefully correct the problem with relevant formative feedback, counseling, etc. If the data analysis is done at the end of the semester, the possibility of formative feedback may be forfeit and the focus on the data analysis would then be to get an overview of the performance of various aspects of the program in order to improve it.

Implementing Change

Once the data analysis is completed, some issues may be identified. The faculty may decide to change its rubrics to tune the assessment process, rephrase the program outcomes, restructure the program curriculum and/or pick different assessments for review. Accreditation bodies are particularly interested in this process for overall review and data driven decision-making.

Without going into details about the implimentation of a solution in Sakai and OSP, it should be clear enough that a mandate to engage in this level of program review requires changes in the organization of a school. In our school, a new full time position was created to manage the assessment process. The assessment coordinator is charged with setting up the assessment framework, collecting the data and running reports.

Goal Management

When the LSB started using OSP to address our program review portfolio needs, we quickly identified a gap between the tool set's capabilities and design philosophy. Most everything in OSP is geared towards making the process initiated by students. In the OSP matrix tool, the STUDENT gets to pick the items that meet the criteria of the matrix. While we were interested in any additional material that students felt was relevent to present as part of their own evidence of mastery of the program outcomes, we knew that asking them to resubmit all of the work that the faculty had identified was a process that was unreliable and unnecessary.

The Goal Management tool released as a “contrib” tool for the 2.4 release was an effort to provide a means for a user to articulate multiple sets of outcomes (goals) and to share those goals with a number of class worksites in Sakai. Instructors who create assignments in their classes can identify the outcomes that pertain to each assignment and rate student work against each. The outcomes and rating data are stored in the Sakai database and can be aggregated and reported.

While we do not have funding right now to continue developing the idea, the basic premise has rung true with many other institutions and is likely to be further developed by Indiana University (hopefully with helpful input by the rest of the community) to be an integrated component of versions of OSP in future releases of the software.

In the interest of full disclosure, our own implementation of Goal Management in Sakai is NOT the production system that our assessment coordinator uses to collect data. Our implementation is a research project that lacks some critical features that would make it an acceptible substitute for the current production system. Discussions are in progress about the cost-benefit of building the necessary features and maintaining a Sakai instance as compared to continuing to use the current system.