The Living Schoolbook

In preparation for my last scheduled day here at the LSB, I have scraped my bits of this site and created a static archive of it on my own web site (I put enough into it that I want to take it with me). I have also set up my own personal blog there as well.

Did you ever see those Capital One commercials where a band of marauders advances on an innocent customer as he makes a purchase with his credit card? The marauders are stopped just before they strike when they realize that this guy was smart enough to choose a credit card that didn't have ridiculous interest rates. The slogan "What's in your wallet?" begs the watcher to make an either/or decision to get slaughtered by the marauders, or to go with their low interest card.

As I read Trent Batson's Campus Technology article, The ePortfolio Hijacked, I thought of hordes of accreditors descending on innocent students, ready to take control of their portfolios and destroy their opportunities to recast themselves as educated, enlightened members of society. He really makes it clear that an assessment system portfolio is not a "learning portfolio". He has drawn a line in the pedagogical sand and proclaimed that there are the assessment system portfolios and then there are REAL portfolios.

Here at Syracuse, the assessment system that is used as a tool to gather data for program review and accreditation needs is not readily available to students. Assessments are mapped to program proficiency standards and faculty rate student work on a 1-4 scale in a system that the students do not have access to. Most of the faculty do not have access to the database that stores that info. Data is collected through spreadsheets and fed into the assessment system by hand by just a few people.

An alternative approach for deploying an assessment system might open possibilities for reflection and formative feedback that we are currently missing. If the data was turned around so that students could see what ratings they received on each of the standards and why, it could stir up some really rich discussion between teachers and students. The Goal Management project that we started here was meant to make transparent the relationship between the program standards, classroom assignments and the data collection process that was being performed. Trent might say that this data only has an audience with deans, accreditors and department heads but I think that it may be the spark that starts students down the road to reflective thinking.

I'd like to think of an assessment system as a means to start a dialogue between teachers and learners around some common themes and a compliment to formative assessment and "assessment for learning", rather than a competitor to both. It is really how you design and use the system, not an inherent property of it. My guess is that there are ways to implement assessment systems that would encourage reflection and result in ideas that a student might take a lot of pride in. The key is in the teachers' ability to engage the student in conversation and seize the teachable moments that such a system makes possible.

The faculty and students have been "doing" portfolios in the Inclusive Elementary Special Education program since before I came here in 2000. They used to have a catered dinner at "Drumlins" and sit at round tables while 3 or 4 students would show their binder-based portfolio to a host teacher and a faculty member while nibbling on hors d'oevres and ham sandwiches at the end of the semester. The event was mostly celebratory. Even if some advice was dispensed during one of these sessions, it seemed like it was too late to me, a newcomer to all of this.

Several years later, as the school began experimenting with "e" portfolios in the form of Dreamweaver and Netscape Composer authored web sites, more time was spent teaching technical skills to the future elementary teachers so that they could showcase the highlights of the semester to their peers and teachers. As a member of a tech group that created and maintained a home made course management system called "Dialogue", which was designed around the idea that EVERY opportunity for assessment/review was also an opportunity to begin a thread of formative discussion, I thought the portfolio idea had a lot of potential to be a great learning opportunity.

The technical barrier to authoring one of these portfolios was substantial, even when we shifted gears and began starting students with a PowerPoint template, with stubbed out pages suggesting that the students address our emerging "School of Education Proficiencies" in their portfolios. How to resize images, scan documents that proved that they were "proficient" and embed video snippets that played on multiple platforms remained issues that were dealt with during the last few weeks at the tail end of the semester. The rest of the semester was spent doing the real work of the school, training students to be excellent teachers.

For the past two years we have been using the web based Open Source Portfolio software. While OSP has improved markedly in that short time, the other barriers remain. As such, our student's portfolios are usually not worked on (or read) by most of the teachers in the school until the end of the semester. In that regard, nothing has changed. It seems that coursework is coursework and the portfolio remains this "other" thing that gets tacked on the end of the semester.

I have this vision of faculty sitting down in the morning with a cup of coffee and reading their students' daily musings (just like I check a pile of blogs on a regular basis), writing back a few thoughts and using that information to inform the next week's lessons.

I wonder if e-portfolios are way too ambitious. A blog with a tagging feature, an few rss feeds and a class blog "aggregator" might be all we need to embed "portfolio thinking" into classes.

Disclaimer: For those of you who do not know me, I am not a teacher. I provide technical support for grant funded educational projects in the School of Education at Syracuse University. Take this perspective with that grain of salt.

Over the past three years of working with the Open Source Portfolio and Sakai communities, I have heard lots of people talk about how they plan to implement the portfolio tools. Some of the uses highlight the ability of the tools to create a standard presentation (like a resume or curriculum-vitae) by using a template. In a later entry, I’ll be walking you through the steps I would follow in order to create a set of forms, wizards and a template that will allow users to pop out a presentation like that. This entry isn’t about those types of uses. I am sure that everyone understands the purpose of a portfolio like that. I want to take a moment to talk about the types of portfolios that can be useful in education settings. This entry doesn't really talk about how to imlpement an accreditation portfolio. I want to set the stage first.

The outcomes based accreditation or program review portfolio

Some schools need a means to review student learning/progress/achievement of ALL of the students in their programs in a very comprehensive way. The prerequisite to a system like this is that the faculty has decided upon an assessment framework that will be applied to all of their students and all of their classes. A “portfolio” effort like this has several components.

Establishment of a framework for evaluation

In order to set this up, a set of outcomes must be agreed to by the faculty. This is a process that many schools apparently do not engage in very often. I haven’t heard of a faculty that spontaneously decides to engage in this process. They usually have more interesting things to do than debate the purpose of the programs in which they teach. Usually it is a mandate from an accrediting body that initiates the process. It was for our faculty.

Identification of key program assessments

A plan has to be designed for measuring the performance/achievement of the students in the program (and in so doing, the effectiveness of the program). A natural tool to use for this purpose is the assessments already in place in the program’s courses. Chances are, a subset or cross section of the work the students already do in their classes would provide the information needed to see how the students and the program are doing. By highlighting the assessments already in place in the program’s curriculum and matching them to the outcomes that they address we can aggregate student work aggregated into FACULTY DESIGNED portfolios that serve the purpose of the program.

Assessment of student work in light of the program outcomes

Once the a faculty member has identified an assessment (an assignment usually) in their course that is intended to assess (at least in part) one or more of the program outcomes, the student’s work will need to be assessed against each of those outcomes. Whether the instructor of the course performs that assessment while grading papers or a "program review panel" performs the assessment later on, the process requires standardized rating scales and rubrics to ensure that all of the faculty are on the same page when assessing the student work. Establishing standard rubrics that ensure inter-rater reliability is another huge task that takes place outside of the “system”. The axiom “garbage in – garbage out” applies here.

Analysis of the Data

It may surprise you to see the multiplicative effects that adding new assessments, outcomes and students to this structure has on the amount of data that there is to look at and analyze. A cohort of 100 students going through a program with 30 highlighted assessments each identified as addressing on average of 2 outcome each provides 6000 pieces of data for analysis. If assessment is continuously performed by instructors (rather than by a “panel” at the end of the semester) it might be helpful if both students and the faculty could have real time access to this assessment portfolio data to help identify problems and trends in student performance and hopefully correct the problem with relevant formative feedback, counseling, etc. If the data analysis is done at the end of the semester, the possibility of formative feedback may be forfeit and the focus on the data analysis would then be to get an overview of the performance of various aspects of the program in order to improve it.

Implementing Change

Once the data analysis is completed, some issues may be identified. The faculty may decide to change its rubrics to tune the assessment process, rephrase the program outcomes, restructure the program curriculum and/or pick different assessments for review. Accreditation bodies are particularly interested in this process for overall review and data driven decision-making.

Without going into details about the implimentation of a solution in Sakai and OSP, it should be clear enough that a mandate to engage in this level of program review requires changes in the organization of a school. In our school, a new full time position was created to manage the assessment process. The assessment coordinator is charged with setting up the assessment framework, collecting the data and running reports.

Goal Management

When the LSB started using OSP to address our program review portfolio needs, we quickly identified a gap between the tool set's capabilities and design philosophy. Most everything in OSP is geared towards making the process initiated by students. In the OSP matrix tool, the STUDENT gets to pick the items that meet the criteria of the matrix. While we were interested in any additional material that students felt was relevent to present as part of their own evidence of mastery of the program outcomes, we knew that asking them to resubmit all of the work that the faculty had identified was a process that was unreliable and unnecessary.

The Goal Management tool released as a “contrib” tool for the 2.4 release was an effort to provide a means for a user to articulate multiple sets of outcomes (goals) and to share those goals with a number of class worksites in Sakai. Instructors who create assignments in their classes can identify the outcomes that pertain to each assignment and rate student work against each. The outcomes and rating data are stored in the Sakai database and can be aggregated and reported.

While we do not have funding right now to continue developing the idea, the basic premise has rung true with many other institutions and is likely to be further developed by Indiana University (hopefully with helpful input by the rest of the community) to be an integrated component of versions of OSP in future releases of the software.

In the interest of full disclosure, our own implementation of Goal Management in Sakai is NOT the production system that our assessment coordinator uses to collect data. Our implementation is a research project that lacks some critical features that would make it an acceptible substitute for the current production system. Discussions are in progress about the cost-benefit of building the necessary features and maintaining a Sakai instance as compared to continuing to use the current system.

The Open Source Portfolio (http://www.osportfolio.org) is often difficult to explain to folks. There are some aspects to the design of the software that seem to be in contradiction with one another. There is a tension between the belief that portfolios are "student owned" and the technical reality that some of the tools are far too complex for students to use. As a result, implementers of OSP use the flexibility of the tools to attempt to craft an environment that meets the needs of all of the users. The simple fact that the experience is designed and implemented by users who may be two degrees from the users make this a challenging piece of software to work with.

I plan to write a bit more about this "suite" of tools in the coming weeks to help identify some ways that I think implementors can get productive fast, but read on for a brief intro to the tools of the Open Source Portfolio.

A quick model of what I thought I heard about the future of Goal Management.

The big things that are different here:

• Goal sets include both Goals and Levels. In essence a Goal Set actuallly will be a matrix. The illustration belies the multi-dimensional aspect that will result from a hierarchical set of goals.
• Program chairs or assessment coordinators will be able to "stub out" assessment placeholders and link them to cells (a goal and a level).
• Linking of activities (like assignments) can be semi-automated when faculty are prompted to identify their activity with an assessment placeholder.
• The assessment placeholder would allow the chair or corrdinator to present a consolidated set of fields that will be used to collect ratings. Each field will need to have rules that disseminate the input ratings up to the linked cells.

Last week I had the privilege of attending a brown bag lunch seminar given by Professor Emeritus Donald Ely on the subject of distance learning.

The discussion began with Dr. Ely putting an emphasis on assessment but seemed to diverge quickly to participant discussions about how it is difficult to create the virtual environment necessary to teach tacit skills like public speaking and medicine. Questions like, "How do I assess active listening skills over distance" were typical of the initial discussion.

It was noted that some LMS's seemed to have certain features that filled certain functional gaps and some of the participants made suggestions/observations that many tools that are widely used today (AIM, Skype, Google Docs) may help fill some perceived gaps in learning mangement tools such as Blackboard and WebCT.

It was a good discussion that was accompanied by the excited enthusiasm that is typical of sharing sessions about the new tools. Markedly absent from the entire discussion, however, was the realization that there is going to be a huge gap left as faculty and students move from a provisioned set of tools (their centrally managed, institutional courseware systems) to a distributed set of tools. Keep in mind that the accountability movement almost demands a centralized system from which we query about student, class and program performance.

It strikes me that next generation LMS's may be more like a mashup view of the "tool set du jour". Faculty and students will continue to find and use new tools that better suit their pedagogical/learning needs. The institution will want to gather and analyze data created in those tools for a variety of reasons (program evaluation, best practice identification, etc.) What is missing is the common language that these new tools will need to speak before we can use them. Sure Google Docs is great as a place for students to work collaboratively, but the institution needs a means to harvest that document for its own purposes. Student portfolios can be created on Facebook, but how does the college compare the student's portfolio over time? Someone may build a killer 3rd party assignment tool, but can we realistically consider it if the institutiuonal system can't interface with it?

I think that this is a strategic direction where a community like Sakai could weigh in heavily. Rather than emulating what we already see out there in the courseware market, let's distinguish ourselves and anticipate where the market is going.

I spoke this morning with John Gosney from IUPUI about their plans to move the Goal Management project forward. One of the functional areas that seems to need a little attention was the need to have multiple rubrics. A while ago I emailed the OSP portfolio list this bit:

Rubric needs GM-122

During our time in Ann Arbor we talked about the need for managed rubrics.

I don't know if anyone has any experience with other rubric management or creation tools out there...I know rubristar (http:// rubistar.4teachers.org) because of another project I was involved in a few years back. After revisiting the rubristar site and talking with Joe Shedd about his expectations, I thought I would bounce some ideas off the community about the requirements for GM-122 (http:// jira.sakaiproject.org/jira/browse/GM-122).

A rubric is usually represented by a matrix of goals/outcomes (usually shown on the left side) and a rating scale (shown across the top) and in each cell is some statement about performance indicators that would inform an evaluator how to choose a score for each outcome. I think that a common use of the OSP matrix has been to set up a structure where the goals are the "Criteria" and the "Scales" and statements are embedded in the evaluation forms in each cell of the matrix. The horizontal axis of the matrix is often thought of as a "time" or "stage" dimension.

When I think of this in terms of Goal Management, I see the linking of Goals with a Rating Scale as setting up a rubric structure. The current implementation provides a blank text area where the author of the assignment/datapoint/wizard page/matrix cell can write in plain text how they plan to score student work. This works ok if there is no programatic effort to organize how the rating scales are used, but the next step is obviously to support such an effort. I believe that there are a few such efforts that we are hearing that we would like OSP to support. Note, from here on out, when I say "rubric" I am using it to describe the descriptive text that would go in each cell of just one row of a "real" matrix (performance indicators as they relate to just one Goal).

Here are some of the rubric organization scenarios I have heard about in my conversations:

1. Managed assessments: Some rubrics are rather specific to (and must be tied to) a particular assessment item and must be approved by an "assessment coordinator" for educational QA purposes as part of a larger assessment system strategy. Changing the assessment/rubric in this case involves more than just the teacher. (Saginaw, Syracuse, RINET)
2. Generally reusable (but unchangeable) rubrics: Some rubrics may be general purpose rubrics that are NOT tied to an assessment, but the dissemination of these approved rubrics may be a strategy of an institution to push forward an agenda of best practice for assessment by providing a handy reference library of general purpose writing, mathematics and science rubrics (for example). While the choice whether or not to use one of these "off the shelf" rubrics (and which one) is left to the teacher, providing some information to the teacher about the schools expectations of its students at different stages (and perhaps suggesting an appropriate rubric for this grade level/stage of development) would make this service more valuable. (Indiana?)
3. Reusable rubric templates: Similar to the above, but the library of "off the shelf" rubrics are merely starting points. There is not a priority to ensure that everyone is doing assessment the exact same way. When a teacher uses one of these rubrics, they can easily edit the performance indicators to suit their needs and create a new rubric, just for their new assignment. (Rubristar approach)
4. Sharing of rubrics: This is a bottom up approach to establishing "best practice". As the teachers create their own rubrics against goals, they have the opportunity to publish them as part of the "reusable" library so other teachers can use/edit/republish them. (Someone?)

Do these make sense? Which efforts do you think we should place a priority on? Should they all be supported eventually?

I was asked if I could provide "just a paragraph about your view of both OSP/Sakai, the state of eportfolios in the country, and a sentence or two about where Syracuse with ePortfolios for the upcoming Academic Impressions: ePortfolios for Learning and Assessment :: Web Conference. Here is what I wrote...

The Open Source Portfolio community understands that portfolio systems are often implemented to fulfill a business need as a key component of a program or institution wide assessment system. Portfolio proponents have long lamented that this taints the portfolio system and inhibits its use as a platform for storytelling and making meaning out of a diverse set of ideas.

We also understand that accreditation boards and external agencies that advocate for standardized data collection and quantitative analysis of student performance are not likely to be receptive to the subjective feedback and guidance that should accompany student owned portfolios.

I think that we generally DISAGREE with the widely help belief that one system can not fulfill both the needs of the institution/program for standardized review/ evaluation AND the pedagogical needs for a student owned portfolio. Over the past two years we have come to recognize that by incorporating the institutional portfolio needs for standardization and data collection into the courseware tools in Sakai, we can free the portfolio tools for student reflection and presentation of their ideas to their peers, their teachers or professional contacts.

In the School of Education at Syracuse University we have been piloting the “Goal Aware” courseware tools in one of our programs. While the tools are functional, the use of the standardized data collection features has not yet been institutionally mandated and (not surprisingly) have only seen limited use. We require our students to publish a portfolio that addresses our “five proficiencies” by completing an OSP matrix column two or three times during their education here. My sense is that while some students find the experience valuable, the back loaded timing of the portfolio review each semester prevents the type of ongoing feedback loop that makes all of the difference. Many students find the additional work to create media rich portfolios technically challenging (given the compressed time frame that they have to complete the work) and also have a difficult time switching their mindset from “giving the right answer” to “providing evidence of learning” and “reflection” at the end of the semester.

I’d enjoy the opportunity to discuss our work with anyone interested with it. Feel free to share my contact info with your audience.

I wanted the recipient to know that I was "I am NOT faculty nor in a decision making role in our college. I manage the technical development, implementation and support of the system. My perspective comes from discussions with faculty and students in the course of doing my job. I tend to be critical of our local implementation while remaining VERY optimistic of what others could do with the Open Source Portfolio system."

A while back on the Sakai lists there was some talk about making Sakai a place that integrated with other, new Web 2.0 applications.

When I build and deploy Sakai, I often wonder to myself i Sakai is getting too big for its britches. Its great that anyone can build a tool for this platform, but that has negative consequences too. My sense is that the bulk of the resources are still devoted to development. Tools are released way too early (IMHO) without enough QA. Reviews like the one I just did for the OSP matrix tool are getting done after the tool is already out there being used.

When it comes to building a freeforn portfolio authoring tool, OSP hasn't put forth the resources to do it yet. I recently wrote the list about considering external authoring environments.

I was looking at Google Presentations tonight. It would make a pretty nice portfolio platform. For those of us looking for an easy to use freeform portfolio tool that allows users to author a simple presentation and share it with others, this is pretty attractive. It allows collaborative editing and chat by viewers of the portfolio (what a fun way to get feedback!). If you aren't interested in assessment tools in your portfolio authoring environment, it might be attractive.

If I remember what LaGuardia wanted to do, they wanted to be give out a "stub" portfolio starter that would get the portfolio creation process rolling. Google just let me import and then edit a PowerPoint file.

Wouldn't it be great if a student could "turn in" their URL for their Google "portfolio" (or a MySpace/FaceBook page, etc.) in OSP so that a portfolio evaluator could view and assess it?

I love the idea of separating the student owned authoring and portfolio storage space from the institutional system. If OSP could retrieve and save a copy of the student's portfolio, it would allow the institution to get a copy of the student's work while allowing the student to REALLY OWN their stuff.

With OSP, its the opposite. Students author their stuff in Sakai/OSP and then wonder, "How will I get my stuff out of there so I can use it after I leave here?" The IMS portfolio spec seems to assume that students will want to move their stuff from portfolio system to portfolio system. Export from one place -> Import somewhere else. I would prefer to author and publish external to all institutional systems and show them where MY stuff is.

I think OSP excels as a content management system with a workflow built in for assessment. Prompting students for certain types of information and presenting that information to a board of evaluators consistently is a process that needs to have an information system specifically for that. OSP forms, wizards and templates support that.

We haven't done very well at supporting freeform portfolios. I'm wondering if we should even be in that business.

Sean