I recently had the chance to read an article published by Educause called Measuring Student Experiences with Course Management Systems. I think its a great eye opener for anybody that is a user or maintains a CMS. Here is a summary I wrote on the article. At the end, I try to answer the questions that the article proposes to the reader. My answers are based off of my opinions and my experience as a student (used Blackboard for 7 years) and as an instruction technologist. If you are interested in reading the article yourself, please visit this link:
http://www.educause.edu/content.asp?page_id=666&ID=ERB0619&bhcp=1
You will need to sign-up to access the article.
Summary of Measuring Student Experiences with Course Management Systems
This article discusses how you can measure the use of a CMS (course management system) by faculty and students, and what features are most important the users. The first section discusses different methods to measure how a CMSs effects teaching and learning. One method is to record and look at CMS usage data through reports and usage logs (like the Tracking feature in Bb). Another method is to poll faculty on how they use the CMS, how it effects or enhances their teaching in the classroom, and to ask them what functions they use in the CMS to teach their subjects. The short side is that these methods can not measure learning. These methods can, however, measure topics such as student’s grades, how they interact with the CMS, and what functions interest the student in the CMS.
The second section discusses student experiences with CMS. Again, it discusses the use of log data to see how the student engages in the CMS. They mention an open source tool called CourseVIs. CourseVIs can transfer log data into graphic data (again, like Tracking feature in Bb). One study (Kvavik & Caruso, 2005) found that students visited the Syllabus more than any other feature in the CMS. Similar to students, faculty members (Morgan, 2003) accessed their Announcements the most, then their syllabus, then course documents.
Students say that having a reliable and available (24 hours x 7 days a week) CMS ranks high on their list of expectations. In one study, students commented on how having couses and its material on-line opened their selection of courses they could participate in (Kavavik & Caruso, 2005). Another study (Rivera & Rice, 2002) found that student were not satisfied with their on-line course because of technical issues, a big learning curve for the CMS, and the students themselves not having a thoro techincal background for the basic of computer technology.
Students highly valued these two features about their CMS: they could see their grades on test and assignmenst, and they could access sample test and quizzes via on-line. Online discussion boards were valued the least (Kavavik & Caruso, 2005). Another study (Burger, 2006) showed that student ranked syllabus, online readings, sample quizzes/tests, and turning in assignments online were most helpful; least helpful were logistics, chat, and online office hours.
Taking quizzes online has mixed reviews for students (depending on the course that is being taught online). Students that did not favor online quizzes where students that had to “show the work” or write paragraphs out for essays. If the online assessment tools didn’t fit the subject well, the students gave it a low review (Kavavik & Caruso, 2005).
Students only found discussion boards useful if the discussion was interesting and carried into the classroom or if they received a grade for it (such as participation) and wasn’t a waste of time. Students only valued the discussion if they felt they learned something from it or got something out of it (such as a higher grade).
In some studies, students felt that IT and CMSs improved their learning. When the student had a very positive experience with their CMS, they highly agreed that IT in courses improves learning. The top seven benefits cited by the author from students were:
1. access to sample quizzes and exams for learning purposes.
2. access to audio/video materials
3. sharing materials among students
4. online readings and links to other text-based course materials
5. online discussion board
6. taking exams and quizzes online for grading
7. getting assignments back from prof w/ comments and grade (Hanson, 2003)
A study (Rivera & Rice, 2002) investigated to see if IT really improves learning. 3 sections of the same business class where compared (one traditional-face2face, one hybrid, and one web based). The results demonstrated little difference in exam scores among sections. More research has to be done to come up with a more precise conclusion to this topic.
In conclusion, the following questions that the author proposes should be asked when a faculty member ventures into using CMS for their teaching (suggested by Parker, 2004):
-What is the purpose for offering this course?
-Do we know what we expect students to learn?
-Do we have the tech infrastructure to support our students? Is it up-to-date?
-How skilled are our course developers and instructors in the online environment?
-What tech assistance do we have available?
CMSs are, without a doubt, developing new features to use in the classroom. Institutions should be encourage explore these new and needed features as they become developed in a CMS (such as blogs, discussion boards, video). This research also shows the need by faculty and students to explore, in more depth, the existing features in the CMS. To explore these current features will, in turn, engage students more in the subject matter being taught via the CMS. The author suggests asking these key questions when evaulating your CMS (I also added my own interspersed answers to these questions outside of the blockquote):
-How can our institutions measure student use and satisfaction w/ CMSs?
Give end of the semester surveys via the instructor or by looking at data logs, help logs, and help calls via the help line.
-What can the usage of data of the CMS features tell us?
It can tell us what features faculty and students are comfortable with. It can also tell us what features users are not comfortable with and design training around the features that are not intuitive. It can also help us explore different solutions that we can add or supplement the CMSs.
-How can our institution increase CMS reliability and availability?
Help lines, on-line tutorials or forums for help.
-What are the most important considerations in faculty training in the use of CMSs?
Level of computer knowledge. Are faculty comfortable with the technology? Do they really want to need to teach using the CMS? How can they translate the basics of their course into a CMS environment?
-What can institutions learn from each other in seeking to improve student satisfaction with CMSs?
See what features work, what features need an easier user interface for students.
-What future feature developments in CMSs will contribute most to learning?
I believe video, audio, blogs and wikis will contribute most to learning in the future. Some of these features are now being introduced into CMSs. With Web 2.0 becoming a big part of students’ regular on-line life, their on-line lifestyle will start to effect their learning style in the classroom.
~Angel Brady
No Comments » | 
Inst. Technology, Reviews | Tagged: about, second_life | 
Permalink
Posted by Min
