There are a variety of tools, rubrics and processes available for analyzing quality in online learning. Dominant among these tools are the Quality Matters Rubric (V. 5) and process, Online Learning Consortium’s Quality Scorecard for the Administration of Online Programs, the Blackboard Exemplary Course Rubric, iNACOL’s National Standards for Quality Online Courses (V2,) eCampusAlberta’s Essential Quality Standards 2.0, and Universal Design for Learning Guidelines Version 2.0.
This lesson reviews those tools in more detail; a PDF printout of much (but not all) of this information can be found at: https://drive.google.com/file/d/0B0lwpuBcseSvVWpWb2xMRzI5aXc/view?usp=sharing.
Elements that Impact Quality
Much has been written about elements that impact effective learning in online environments, and what constitutes “quality.” All online courses are not the same, and a critical examination of pedagogy and design, as well as academic and other student supports, is needed to enable colleges and universities to effectively create systems and processes for instructional design and institutional support to enable students to have a quality learning experience online (Jaggars & Bailey, 2010). A variety of tools have emerged throughout the field, including those specifically focused on the instructional design of online courses (some of which also include guidelines for delivery), those focused on designing instruction for students with diverse needs, and those focused on, or including, elements of institutional strategy and student support services (Blackboard, n.d.; CAST, 2011; eCampusAlberta, 2013; iNACOL, 2011; Online Learning Consortium, n.d.; Quality Matters, 2014).
Within a course-specific context, Smith, Jaggars & Xu (2013) organized these into four primary themes: “(1) the extent to which the course interface is well organized and easy to navigate; (2) the clarity of learning objectives and performance standards; (3) the strength and diversity of interpersonal interaction; and (4) the extent to which technology is effectively used” (p. 7). A wide variety of evidence supports each of these components, however social engagement in the online environment was heavily supported in being a critical component to student success – both learner to faculty (e.g., Arbaugh, 2001; Picciano, 2002; Young, 2006) and learner to learner (e.g., Bangert, 2006; Matthew, Felvegi, & Callaway, 2009; Balaji & Chakrabarti, 2010) (as cited in Smith Jaggars & Xu, 2013, p. 10).
Quality tools, guidelines and processes used in the field include engagement that is learner to learner, learner to faculty member, and learner to content as a critical piece. However the other essential factors of course navigation, alignment of instructional components (i.e. objectives, assessments and activities,) the effective use of instructional technology, and academic and institutional supports and commitments are holistically viewed as impacting student success (Blackboard, n.d.; CAST, 2011; eCampusAlberta, 2013; iNACOL, 2011; Online Learning Consortium, n.d.; Quality Matters, 2014).
First, let’s take a look at Quality Matters. Quality Matters is a national nonprofit designed to improve the quality of online learning. They have both a quality tool (the Quality Matters Rubric,) and a peer review process by which courses can be submitted for certification based on 8 General Standards and 43 Specific Review Standards (Quality Matters, 2014). Quality Matters has over 900 subscribing institutions as members. Alignment is critical to the Quality Matters Rubric, whereby all instructional elements of the course (objectives, assessments, learning resources, activities and technology) work together to enable students to meet course outcomes (Quality Matters, 2014). Quality Matters also has a separate rubric for K-12 education.
The underlying principles of Quality Matters are 1) continuous, 2) centered (on research, student learning and quality,) 3) collegial, and 4) collaborative. Quality Matters is one of the leading quality tools for instructional design of online courses within higher education, with over 900 institutions subscribing. It has expanded its usage beyond higher education to include K-12, educational publishing, and continuing and professional education.
The Quality Matters rubric has 8 General Standards, each of which has Specific Review Standards. Core to the Quality Matters rubric is the concept of alignment, wherein “critical course components – Learning Objectives (2), Assessment and Measurement (3), Instructional Materials (4), Course Activities and Learner Interaction (5), and Course Technology (6) – work together to ensure students achieve desired learning outcomes.” The Quality Matters Rubric comes accompanied by the Annotations, which describe in more detail how the standards might look in action in an online or blended/hybrid course.
When a QM Review is conducted, the course either “Meets” or “Does Not Meet” the criteria at an 85% or better threshold. The course must meet all of the Essential Standards (which are 3 points), and enough of the Very Important and Important standards to have a percentage of 85% of the points overall.
Quality Matters include standards on accessibility, however it is important to note that their Specific Review Standards associated with accessibility are broader than the requirements set forth in Titles II and III, Section 504, and in Section 508 where it applies.
Watch this video overview of Quality Matters to gain some background information on the rubric and the process.
Blackboard Exemplary Course Program
The Blackboard Exemplary Course Program is a course review rubric that reviews broadly course design, interaction, assessment and learner support. It includes descriptive criteria by category, ranging from incomplete to exemplary. There are three ways in which the rubric can be used – as a self, peer, or Blackboard certified Achievement review (Blackboard, n.d.).
iNACOL’s National Standards for Quality Online Courses
iNACOL’s National Standards for Quality Online Courses (v2) is a rubric designed for the K-12 environment. As such there are elements of the rubric specific to state and national curricular standards which do not directly apply to higher education (though in fields that require additional accreditation by external bodies like nursing, etc., there is a great deal of parallels.) Originally iNACOL had adopted the Quality Online Course Standards created by the Southern Regional Education Board (SREB,) but in 2011 they created their own version (iNACOL, 2011). Their rubric ranges from “Absent” to “Very Satisfactory.” No descriptive criteria accompanies those levels. To address this, the Texas Education Agency’s Texas Virtual School Network (TxVSN) created an accompanying Course Review Scoring Rubric which iNACOL publishes with their created standards (iNACOL, 2011).
Essential Quality Standards 2.0 from eCampus Alberta
The Essential Quality Standards 2.0 from eCampusAlberta in Canada can be found online with an interactive scoring tool, literature review, and other supporting documents (eCampusAlberta, 2013). Version 2.0 was published in 2013 and contains seven categories including (1) Course Information Standards, (2) Organization Standards, (3) Pedagogy Standards, (4) Writing Standards, (5) Resource Standards, (6) Web Design Standards, and (7) Technology Standards. Interestingly, this rubric (which does provide descriptors for each of three levels of achievement labeled Essential, Excellent, and Exemplary. These standards incorporate course design elements with some Universal Design for Learning (UDL) elements, along with some student/institutional support elements.
Universal Design for Learning
Universal Design for Learning does not have a rubric, rather it has guidelines for the creation of online learning based on research in three primary learning networks grounded in cognitive science: (1) recognition networks, (2) strategic networks, and (3) affective networks. These networks translate to three core principles, with three guidelines under each. Their UDL Guidelines Version 2.0 contains rich descriptions and examples of what following each of those guidelines might look like in an online course (Cast, 2011).
UDL was created by researchers from the Center for Applied Special Technology[i]. UDL[ii] is an approach that minimizes barriers and maximizes learning for all students by utilizing the three primary brain networks: (1) recognition networks, (2) strategic networks, and (3) affective networks, or the “what, why, and how” of learning. “In learning environments…individual variability is the norm, not the exception. When curricula are designed to meet the needs of an imaginary “average”, they do not address the reality of learner variability.” Three primary domains are used based on cognitive science:
- Recognition: Present information and content in different ways
- Action and Expression: Differentiate the ways that students can express what they know
- Engagement: Stimulate interest and motivation for learning
The research evidence utilized in the creation of UDL Guidelines Version 2.0 was gathered in three stages: (1) constructing the framework from research in cognitive science, cognitive neuroscience, neuropsychology, and neuroscience, resulting in the three basic learning networks; (2) drilling down to create the categories of the 9 guidelines; (3) reviews of research to create recommendations for practice to reduce barriers in each of those 9 guidelines.
Principle I: Provide Multiple Means of Representation
- Guideline 1: Provide options for perception
- Guideline 2: Provide options for language, mathematical expressions, and symbols
- Guideline 3: Provide options for comprehension
Principle 2: Provide Multiple Means of Action and Expression
- Guideline 4: Provide options for physical action
- Guideline 5: Provide options for expression and communication
- Guideline 6: Provide options for executive functions
Principle 3: Provide Multiple Means of Engagement
- Guideline 7: Provide options for recruiting interest
- Guideline 8: Provide options for sustaining effort and persistence
- Guideline 9: Provide options for self-regulation
Learning and teaching is addressed. The UDL framework does not separate out specific support for online or blended/hybrid programs. It is less mechanical and more philosophical.https://youtu.be/bDvKnY0g6e4
It is important to note that Universal Design for Learning goes beyond the legal requirements for accessibility in online and blended/hybrid courses. As a design philosophy and set of guidelines, it is the most comprehensive and detailed. It is also highly supported by research that is correlated with each of the Guidelines. UDL provides checkpoints for each of the Guidelines.
Note that the language of “engagement” used in Principle 3 is focused on motivation and learner self-efficacy, rather than how “engagement” is typically used in learning vernacular as associated with how learners interact with content, with each other, and with instructors.
View this video on Universal Design for Learning.
Rubrics and Why to Use Them
While the above examples are all tools for designing quality courses, and conducting quality assurance for those courses, some internal tools are important within the course to ensure that learners are developing the skills and gaining the knowledge necessary to be successful in the course. This is the “proof” part of learning.
Rubrics enable the measurement of learner performance against criteria. They can be graded, but are not necessarily used as a mechanism for assigning grades or scores.
Jon Mueller describes this as:
A student’s aptitude on a task is determined by matching the student’s performance against a set of criteria to determine the degree to which the student’s performance meets the criteria for the task. To measure student performance against a pre-determined set of criteria, a rubric, or scoring scale, is typically created which contains the essential criteria for the task and appropriate levels of performance for each criterion.
Have you ever been in a class and gotten a paper back with a random number on it and some scribbled comments in the margins? Did it provide you with a comprehensive sense of where you met expectations, where you exceeded them, and if you needed to improve, what specifically you could do better next time? No? Not surprising. Rubrics enable learners to have a richer understanding of how to adjust their own learning as well as how to predict their own success by using rubrics as tools to prepare for assessments.
Review these examples of rubrics. Likely they are more specific than the typical information that you’re provided with about an assignment or assessment. Take some notes in your journal on this. How would rubrics improve your learning process? How would they improve the quality of a course or even an entire online program?
Sue Lieberman created this great SlideShare presentation (below,) which provides an overview of why and how you can use rubrics. Take notes as you go – you’ll be using those resources in your presentation for Instructional Design and Development.
View this Slideshare presentation by Sue Lieberman embedded below.
This overview of quality tools and process, as well as the importance of using rubrics and some information about how to design them well, should help you gain an understanding of why and how quality tools are important in online courses and programs.
Remember to keep taking notes in your journal – it will become your resource for when you create your learning objects and presentations, and I’ll be checking in regularly to help you along the way.
Blackboard. (n.d.). Exemplary course program rubric. Retrieved from http://www.blackboard.com/resources/catalyst-awards/bbexemplarycourserubric_rebrand_july2015.pd
CAST (2011). Universal Design for Learning Guidelines version 2.0. Wakefield, MA: Author.
eCampusAlberta. (2013). Essential quality standards 2.0. Retrieved from http://quality.ecampusalberta.ca/the-etoolkit
INACOL International Association for K-12 Online Learning. (2011). National standards for quality online courses version 2. Retrieved from http://www.inacol.org/resource/inacol-national-standards-for-quality-online-courses-v2/
Jaggars, S., & Bailey, T. R. (2010). Effectiveness of fully online courses for college students: Response to a Department of Education meta-analysis.
Online Learning Consortium. (n.d.). Quality scorecard for the administration of online programs version 2. Retrieved from http://onlinelearningconsortium.org/consult/quality-scorecard/
Quality Matters. (2014). Higher ed program rubric Quality Matters. Retrieved from https://www.qualitymatters.org/rubric
Smith, Jaggars, S., & Xu, D. (2013). Predicting online student outcomes from a measure of course quality (57). Retrieved from Community College Research Center, Teachers College, Columbia University website: http://ccrc.tc.columbia.edu/publications/predicting-online-student-outcomes-and-course-quality.html
Swan, K. (2003). Learning effectiveness online: What the research tells us. Elements of quality online education, practice and direction, 4, 13-47.