Resource Evaluation Rubrics

Introduction

Illinois is publishing resource evaluations using a derivative of the Achieve Open Education Resources Rubric and IOER's derivative of Achieve's EquIP CCSS Rubric to the Learning Registry. Achieve's EQuIP rubric is intended only for units and lessons, but is written in such a way that Illinois feels that it can be applied to more than just units and lessons. We are using a derivative of the Achieve OER rubric as well, because we are not publishing evaluations using Rubric I or Rubric VIII of the Achieve OER rubric. We are also using a different scale for the Achieve rubrics. Achieve's rubrics are on a 0-3 scale, we are using a 0-100 scale. Because of these changes, Illinois cannot say we are publishing with the Achieve rubrics, we must say that we are publishing with a derivative of those rubrics.

Achievement Standards Network (ASN) is hosting the dot notations for the Achieve rubrics. This information was made available to ASN using the Simple Knowledge Organization System (SKOS) schema in RDF/XML format by Southern Illinois University at Carbondale. Links to the rubrics' SKOS data are at http://www.achievementstandards.org/content/technical-documentation.

1. Common Core State Standards Organization

Common Core State Standards seem to be organized along the following lines:

• CCSS – the acronym for Common Core State Standards
• Subject – CCSS covers Math and English Language Arts/Literacy.
• Sub-subject (for example, Content, or Practice for Math, RL for ELA/Literacy – Reading Standards for Literature).
• Grade Level
• Standard Number

2. EQuIP Rubric Dot Notation Proposal for Individual Bulleted Items in the Rubric

The EQuIP Rubric is available at http://www.achieve.org/EQuIP/. The CCSS Rubric is a derivative of the EQuIP Rubric.

The CCSS Rubric has a dot notation similar to that used for CCSS. To make this work for other rubrics as they are published, we have done something like this:

• Acronym of Rubric – "EQR"
• Subject – subjects used are in quotes.
• "Math"
• ELA/Literacy (Grades 3-5) – "ELA"
• "ELA" (Grades 6-12)
• There is a rubric for K-2 ELA – "ELA".  Dot notation for this rubric is under development.
• Grade Level
• Math is always "K-12"
• ELA is divided into 2 grade ranges: "K-2" and "3-12".
• Dimension
• Dimension I: "AD"
• Dimension II: "KS"
• Dimension III: "IS"
• Dimension IV: "AS"
• Criteria Number within dimension – assign in order as they are shown within the dimensions.

For example, criteria #3 of Dimension 2 of the CCSS Math Rubric might be laid out as follows:

• EQR.Math.K-12.KS.3. criteria #6 of the Dimension 3 of the CCSS ELA-Literacy rubric for grades 3-5 and 6-12 might be laid out as follows:
• EQR.ELA.3-12.IS.6. Bullet #1 of Dimension 4 of the CCSS ELA-Literacy rubric for grades K-2 would be laid out as EQR.ELA-Literacy.K-2.AS.1.

2.1. CCSS Rubric Dot Notation Proposal for Dimension of the Rubric

To publish ratings at the dimension level, simply leave off the bullet number. Thus for the three dimensions given in the previous section, the dot notation for these would be as follows:

• EQR.Math.K-12.KS.3 becomes EQR.Math.K-12.KS.
• EQR.ELA-Literacy.3-12.IS.6 becomes EQR.ELA-Literacy.3-12.IS.
• EQR.ELA-Literacy.K-2.AS.1 becomes EQR.ELA-Literacy.K-2.AS.

3. Achieve OER Rubric

The Achieve OER Rubric is available at http://www.achieve.org/oer-rubrics. The Achieve OER rubric has a dot notation similar to that used for the EQuIP rubric:

• Acronym of Rubric - "AOER"
• Rubric Identifier - (identifier is in quotes)
  • Rubric I: "DA"
  • Rubric II: "ES"
  • Rubric III: "ST"
  • Rubric IV: "AS"
  • Rubric V: "TI"
  • Rubric VI: "IP"
  • Rubric VII: "DL"
  • Rubric VIII: "AA"
• Criteria Number within dimension - assign in order as they are shown within the dimensions. Applies to Rubric VIII only.

4. How ISLE intends to publish ratings to rubrics to the LR

ISLE intends to publish ratings to the EQuIP CCSS rubric and AOER rubrics to the LR only at the Dimension Level at this time. We will not be publishing ratings using dimension 1 of either rubric as we are publishing those to the standards alignment. We are not publishing ratings using dimension (Rubric) VIII of the AOER rubric at this time.

4.1 Dimension Level Ratings

Certain aspects of the rubric only apply if the resource is a Unit of Study. An example is the ELA-Literacy rubric for grades 3-12, Dimension 2, bullet 5, which only applies to units. ISLE will publish ratings for the dimensions on a 0-100 scale. If a resource is not tagged as a unit, ratings for dimension 2 will be published on the first four bullets. If a resource is later tagged as unit, ratings for dimension 2 will then include the last four bullets instead of the first four. Both will be scaled to a 0-100 scale so they will be normalized.

5. Paradata Samples

5.1 Ratings

5.1.1. Alignment to the Standard

A resource can be aligned to multiple standards. Alignments to standards are published as follows:Dimension 1 of the CCSS rubric is the alignment to the rigor of the CCSS, and should therefore be handled as an alignment to a standard. Therefore a rating on a scale of 0-100 for dimension 1 as applied to CCSS.Math.Content.K.G.A.1 would not be published using the dot notation of EQR.Math.K-12.AD, but would be published using the CCSS.Math.Content.K.G.A.1 notation.

{ 
	"activity": 
	{ 
		"actor": 
		{
			"description": ["9","10","English Language Arts"],  
			"objectType": "Educator"  },
			"verb": 
			{
				"action": "rated",
				"date": "2013-03-01/2013-03-31",
				"context": 
				{
					"id": "http://url/to/ISLE/page/",
					"description": "ISLE detail page",
					"objectType": "OER"
				},
				"measure": 
				{
					"sampleSize": 100,
					"scaleMin": 0,
					"scaleMax": 100,
					"value": 72.0 
				},
			},
		"object": {	"id": "http://URL/to/resource/" },
		"related": 
		[
			{ "objectType": "Common Core State Standard", "id": "CCSS.Math.Content.K.G.A.1", "content": "Describe objects in the environment using names of shapes, and describe the relative positions of these objects using terms such as above, below, beside, in front of, behind, and next to."  }
		]  
	} 
}

5.1.2. Dimensions of the CCSS Rubric

At present ISLE will publish ratings for the rubric at the dimension level, not at the criteria level. As part of the CCSS rubrics, if dimension 1 is not rated as a "2" or "3" ("Exemplar" or "Exemplar if Improved"), the rest of the rubric is skipped.

A rating of 72 on a scale of 0-100 for math, dimension 2 would be published as follows:

{
	"activity": 
	{
		"actor": 
		{
			"description": ["9","10","English Language Arts"],
			"objectType": "Educator"  
		},
		"verb": 
		{  
			"action": "rated",
			"date": "2013-03-01/2013-03-31",
			"context":
			{
				"id": "http://url/to/ISLE/page/",
				"description": "ISLE detail page",
				"objectType": "OER"
			},
			"measure": 
			{
				"sampleSize": 100,
				"scaleMin": 0,
				"scaleMax": 100,
				"value": 72.0
			},
		},
		"object": {	"id": "http://URL/to/resource/"	},  
		"related": 
		[
			{
				"objectType": "EQuIP Rubric",
				"id": "EQR.Math.K-12.KS",
				"content": "II. Key Areas of Focus in the CCSS"
			}
		]
	}
}

5.2 Comments

Comments will be available for each dimension of the rubric (not criteria within a dimension). Comments related to a rubric dimension will be published to the Learning Registry using the LR Paradata 1.0 schema as follows:

{
	"activity":
	{
		"actor": 
		{
			"description": ["9","10"],
			"objectType": "educator"
		},
		"verb": 
		{
			"action": "commented",
			"date": "2013-03-01",
			"comment": "Some comment goes here",
			"context": 
		{ 
			"id": "http://url/to/ISLE/page/",
			"description": "ISLE detail page",
			"objectType": "OER"
		}
	},   
	"related": 
	[
		{
			"objectType": "EQuIP Rubric",
			"id": "EQR.Math.K-12.KS",
			"content": "II. Key Areas of Focus in the CCSS"
		}
	],   
	"object": { "id": "http://URL/to/resource/" }
	} 
}

Appendix

A.1 Appendix A – Dot Notation of EQuIP Rubrics

A.1.1 Mathematics Rubric

ASN is hosting this rubric. It is available at http://purl.org/ASN/scheme/EQuIPMathK-12/.

Text	Dot Notation	Notes
I. Alignment to the Depth of the CCSS	EQR.Math.K-12.AD	Dimension 1
Targets a set of grade-level CCSS mathematics standard(s) to the full depth of the standards for teaching and learning	EQR.Math.K-12.AD.1
Standards for Mathematical Practice that are central to the lesson are identified, handled in a grade appropriate way, and well connected to the content being addressed.	EQR.Math.K-12.AD.2
Presents a balance of mathematical procedures and deeper conceptual understanding inherent in the CCSS.	EQR.Math.K12.AD.3
II. Key Shifts in the CCSS	EQR.Math.K-12.KS	Dimension 2
Focus: Lessons and units targeting the major work of the grade (at the standard and cluster level) provide an especially in-depth treatment, with especially high expectations. Lessons and units targeting supporting work of the grade (at the standard and cluster level) have visible connection to the major work of the grade and are sufficiently brief. Lessons and units do not hold students responsible for material from later grades.	EQR.Math.K-12.KS.1
Coherence: The content develops through reasoning about the new concepts on the basis of previous understandings.	EQR.Math.K-12.KS.2
Rigor: Requires students to engage with and demonstrate challenging mathematics with appropriate balance among the following: − Application: Provides opportunities for students to independently apply mathematical concepts in real world situations and problem solve with persistence, choosing and applying an appropriate model or strategy to new situations. − Conceptual Understanding: Develops students' understanding through brief conceptual problems and questions, multiple representations and opportunities for students to write and speak about their understanding. − Procedural Skill and Fluency: Expects, supports and provides guidelines for procedural skill and fluency with core calculations and mathematical procedures (when called for in the standards for the grade) to be performed quickly and accurately.	EQR.Math.K-12.KS.3
III. Instructional Supports	EQR.Math.K-12.IS
Includes clear and sufficient guidance to support teaching and learning of the targeted standards, including, when appropriate, the use of technology and media.	EQR.Math.K-12.IS.1
Uses and encourages precise and accurate mathematics, academic language, terminology and concrete or abstract representations (e.g., pictures, symbols, expressions, equations, graphics, models) in the discipline.	EQR.Math.K-12.IS.2
Engages students in productive struggle through relevant, thought-provoking questions, problems and tasks that stimulate interest and elicit mathematical thinking.	EQR.Math.K-12.IS.3
Addresses instructional expectations and is easy to understand and use	EQR.Math.K-12.IS.4
Provides appropriate level and type of scaffolding, differentiation, intervention, and support for a broad range of learners.	EQR.Math.K-12.IS.5
Supports diverse cultural and linguistic backgrounds, interests and styles.	EQR.Math.K-12.IS.6
Provides extra supports for students working below grade level.	EQR.Math.K-12.IS.7
Provides extensions for students with high interest or working above grade level.	EQR.Math.K-12.IS.8
Recommend and facilitate a mix of instructional approaches for a variety of learners such as using multiple representations (e.g., including models, using a range of questions, checking for understanding, flexible grouping, pair-share).	EQR.Math.K-12.IS.9	Applies to Units or longer lessons
Gradually remove supports, requiring students to demonstrate their mathematical understanding independently.	EQR.Math.K-12.IS.10	Applies to Units or longer lessons
Demonstrate an effective sequence and a progression of learning where the concepts or skills advance and deepen over time.	EQR.Math.K-12.IS.11	Applies to Units or longer lessons
Expect, support and provide guidelines for procedural skill and fluency with core calculations and mathematical procedures (when called for in the standards for the grade) to be performed quickly and accurately.	EQR.Math.K-12.IS.12	Applies to Units or longer lessons
IV. Assessment	EQR.Math.K-12.AS	Dimension 4
Is designed to elicit direct, observable evidence of the degree to which a student can independently demonstrate the targeted CCSS.	EQR.Math.K-12.AS.1
Assesses student proficiency using methods that are accessible and unbiased, including the use of grade-level language in student prompts.	EQR.Math.K-12.AS.2
Includes aligned rubrics, answer keys and scoring guidelines that provide sufficient guidance for interpreting student performance.	EQR.Math.K-12.AS.3
Use varied modes of curriculum embedded assessments that may include pre-, formative, summative and self-assessment measures.	EQR.Math.K-12.AS.4	Applies to Units or longer lessons

A.1.2. ELA/Literacy (Grades 3-5) and ELA (Grades 6-12)

ASN is hosting this rubric. It is available at http://purl.org/ASN/scheme/EQuIPELA3-12/.

Text	Dot Notation	Notes
I. Alignment to the Depth of the CCSS	EQR.ELA.3-12.AD	Dimension 1
Targets a set of grade-level CCSS ELA/Literacy standards.	EQR.ELA.3-12.AD.1
Includes a clear and explicit purpose for instruction.	EQR.ELA.3-12.AD.2
Selects text(s) that measure within the grade-level text complexity band and are of sufficient quality and scope for the stated purpose (e.g., presents vocabulary, syntax, text structures, levels of meaning/purpose, and other qualitative characteristics similar to CCSS grade-level exemplars in Appendices A & B).	EQR.ELA.3-12.AD.3
Integrate reading, writing, speaking and listening so that students apply and synthesize advancing literacy skills	EQR.ELA.3-12.AD.4	Applies to Units or longer lessons.
(Grades 3-5) Build students' content knowledge and their understanding of reading and writing in social studies, the arts, science or technical subjects through the coherent selection of texts.	EQR.ELA.3-5.AD.5	Applies to Units or longer lessons, grades 3-5 only. NOTE: the grade range for the dot notation of this criteria is 3-5, not 3-12.
II. Key Shifts in the CCSS	EQR.ELA.3-12.KS	Dimension 2
Reading Text Closely: Makes reading text(s) closely, examining textual evidence, and discerning deep meaning a central focus of instruction	EQR.ELA.3-12.KS.1
Text-Based Evidence: Facilitates rich and rigorous evidence-based discussions and writing about common texts through a sequence of specific, thought-provoking, and text-dependent questions (including, when applicable, questions about illustrations, charts, diagrams, audio/video, and media).	EQR.ELA.3-12.KS.2
Writing from Sources: Routinely expects that students draw evidence from texts to produce clear and coherent writing that informs, explains, or makes an argument in various written forms (e.g., notes, summaries, short responses, or formal essays).	EQR.ELA.3-12.KS.3
Academic Vocabulary: Focuses on building students' academic vocabulary in context throughout instruction.	EQR.ELA.3-12.KS.4
Increasing Text Complexity: Focus students on reading a progression of complex texts drawn from the grade-level band. Provide text centered learning that is sequenced, scaffolded and supported to advance students toward independent reading of complex texts at the CCR level.	EQR.ELA.3-12.KS.5	Applies to Units or longer lessons.
Building Disciplinary Knowledge: Provide opportunities for students to build knowledge about a topic or subject through analysis of a coherent selection of strategically sequenced, discipline-specific texts.	EQR.ELA.3-12.KS.6	Applies to Units or longer lessons.
Balance of Texts: Within a collection of grade-level units a balance of informational and literary texts is included according to guidelines in the CCSS (p. 5).	EQR.ELA.3-12.KS.7	Applies to Units or longer lessons.
Balance of Writing: Include a balance of on-demand and process writing (e.g., multiple drafts and revisions over time) and short, focused research projects, incorporating digital texts where appropriate.	EQR.ELA.3-12.KS.8	Applies to Units or longer lessons.
III. Instructional Supports	EQR.ELA.3-12.IS	Dimension 3
Cultivates student interest and engagement in reading, writing and speaking about texts.	EQR.ELA.3-12.IS.1
Addresses instructional expectations and is easy to understand and use.	EQR.ELA.3-12.IS.2
Provides all students with multiple opportunities to engage with text of appropriate complexity for the grade level; includes appropriate scaffolding so that students directly experience the complexity of the text.	EQR.ELA.3-12.IS.3
Focuses on challenging sections of text(s) and engages students in a productive struggle through discussion questions and other supports that build toward independence.	EQR.ELA.3-12.IS.4
Integrates appropriate supports in reading, writing, listening and speaking for students who are ELL, have disabilities, or read well below the grade level text band.	EQR.ELA.3-12.IS.5
Provides extensions and/or more advanced text for students who read well above the grade level text band.	EQR.ELA.3-12.IS.6
Include a progression of learning where concepts and skills advance and deepen over time.	EQR.ELA.3-12.IS.7	Applies to Units or longer lessons.
Gradually remove supports, requiring students to demonstrate their independent capacities.	EQR.ELA.3-12.IS.8	Applies to Units or longer lessons.
Provide for authentic learning, application of literacy skills, student directed inquiry, analysis, evaluation and/or reflection.	EQR.ELA.3-12.IS.9	Applies to Units or longer lessons.
Integrate targeted instruction in such areas as grammar and conventions, writing strategies, discussion rules and all aspects of foundational reading for grades 3-5.	EQR.ELA.3-5.IS.10	Applies to Units or longer lessons, for grades 3-5 only. NOTE: The grade range for this criteria is 3-5, not 3-12.
Include independent reading based on student choice and interest to build stamina, confidence and motivation; indicates how students are accountable for that reading.	EQR.ELA.3-12.IS.11	Applies to Units or longer lessons.
Use technology and media to deepen learning and draw attention to evidence and texts as appropriate.	EQR.ELA.3-12.IS.12	Applies to Units or longer lessons.
IV. Assessment	EQR.ELA.3-12.AS	Dimension 4
Elicits direct, observable evidence of the degree to which a student can independently demonstrate the major targeted grade-level CCSS standards with appropriately complex text(s).	EQR.ELA.3-12.AS.1
Assesses student proficiency using methods that are unbiased and accessible to all students.	EQR.ELA.3-12.AS.2
Includes aligned rubrics or assessment guidelines that provide sufficient guidance for interpreting student performance.	EQR.ELA.3-12.AS.3
Use varied modes of assessment, including a range of pre, formative, summative and self-assessment measures.	EQR.ELA.3-12.AS.4	Applies to Units or longer lessons.

A.1.3. ELA/Literacy (Grades K-2)

ASN is hosting this rubric. It is available at http://purl.org/ASN/scheme/EQuIPELAK-2/.

Text	Dot Notation	Notes
I. Alignment to the Depth of the CCSS	EQR.ELA.K-2.AD	IOER does not publish rubrics using this dimension.
Targets a set of K-2 ELA/Literacy CCSS for teaching and learning.	EQR.ELA.K-2.AD.1
Includes a clear and explicit purpose for instruction.	EQR.ELA.K-2.AD.2
Selects quality text(s) that align with the requirements outlined in the standards, presents characteristics similar to CCSS K-2 exemplars (Appendix B), and are of sufficient scope for the stated purpose.	EQR.ELA.K-2.AD.3
Provides opportunities for students to present ideas and information through writing and/or drawing and speaking experiences.	EQR.ELA.K-2.AD.4
Emphasize the explicit, systematic development of foundational literacy skills (concepts of print, phonological awareness, the alphabetic principle, high frequyency sight words, and phonics).	EQR.ELA.K-2.AD.5	Applies to units or longer lessons.
Regularly include specific fluency-building techniques supported by research (e.g., monitored partner reading, choral reading, repeated readings with text, following along in the text when teacher or other fluent reader is reading aloud, short timed practice that is slightly challenging to the reader).	EQR.ELA.K-2.AD.6	Applies to units or longer lessons.
Integrate reading, writing, speaking and listening so that students apply and synthesize advancing literacy skills.	EQR.ELA.K-2.AD.7	Applies to units or longer lessons.
Build students' content knowledge in social studies, the arts, science or technical subjects through a coherent sequence of texts and series of questions that build knowledge withyin the topic.	EQR.ELA.K-2.AD.8	Applies to units or longer lessons.
II. Key Shifts in the CCSS	EQR.ELA.K-2.KS
Reading Text Closely: Makes reading text(s) closely (including read alouds) a central focus of instruction and includes regular opportunities for students to ask and answer text-dependent questions.	EQR.ELA.K-2.KS.1
Text-Based Evidence: Facilitates rich text-based discussions and writing through specific, thought-provoking questions about common texts (including read aloutds, and when applicable, illustrations, audio/video and other media).	EQR.ELA.K-2.KS.2
Academic Vocabulary: Focuses on explicitly building students' academic vocabulary and concepts of syntax throughout instruction.	EQR.ELA>K-2.KS.3
Grade-Level Reading: Include a progression of texts as students learn to read (e.g., additional phonic patterns are introduced, increasing sentence length). Provides text-centered learning that is sequenced, scaffolded and supported to advance students toward independent grade-level reading.	EQR.ELA.K-2.KS.4	Applies to units or longer lessons.
Balance of Texts: Focus instruction equally on literary and informational texts as stipulated in the CCSS (p.5) and indicated by instructional time (may be more applicable across a year or several units).	EQR.ELA.K-2.KS.5	Applies to units or longer lessons.
Balance of Writing: Include prominent and varied writing opportunities for students that balance communicating thinking and answering questions with self-expression and exploration.	EQR.ELA.K-2.KS.6	Applies to units or longer lessons.
III. Instructional Supports.	EQR.ELA.K-2.IS
Cultivates student interest and engagement in reading, writing and speaking about texts.	EQR.ELA.K-2.IS.1
Addresses instructional expectations and is easy to understand and use for teachers (e.g., clear directions, sample proficient student responses, sections that build teacher understanding of the whys and how of the material).	EQR.ELA.K-2.IS.2
Integrates targeted instruction in multiple areas such as grammar and syntax, writing strategies, discussion rules and aspects of foundational reading.	EQR.ELA.K-2.IS.3
Provides substantial materials to support students who need more time and attention to achieve automaticity with decoding, phonemic awareness, fluency and/or vocabulary acquisition.	EQR.ELA.K-2.IS.4
Provides all students (including emergent and beginning readers) with extensive opportunities to engage with grade-level texts and read alouds that are at high levels of complexity including appropriate scaffolding so that students directly experience the complexity of text.	EQR.ELA.K-2.IS.5
Focuses on sections of rich text(s) (including read alouds) that present the greatest challenge; provides discussion questions and other supports to promote student engagement, understanding and progress toward independence.	EQR.ELA.K-2.IS.6
Integrates appropriate, extensive and easily implemented supports for students who are ELL, have disabilities and/or read or write below grade level.	EQR.ELA.K-2.IS.7
Provides extensions and/or more advanced text for students who read or write above grade level.	EQR.ELA.K-12.IS.8
Include a progression of learning where concepts, knowledge and skills advance and deepen over time (may be more applicable across the year or several units).	EQR.ELA.K-2.IS.9	Applies to units or longer lessons.
Gradually remove supports, allowing students to demonstrate their independent capacities (may be more applicable across the year or several units).	EQR.ELA.K-2.IS.10	Applies to units or longer lessons.
Provide for authentic learning, application of literacy skills and/or student-directed inquiry.	EQR.ELA.K-2.IS.11	Applies to units or longer lessons.
Indicate how students are accountable for independent engaged reading based on student choice and interest to build stamina, confidence and motivation (may be more applicable across the year or several units).	EQR.ELA.K-2.IS.12	Applies to units or longer lessons.
Use technology and media to deepen learning and draw attention to evidence and texts as appropriate.	EQR.ELA.K-2.IS.13	Applies to units or longer lessons.
IV. Assessment	EQR.ELA.K-2.AS
Elicits direct, observable evidence of the degree to which a student can independently demonstrate foundational skills and targeted grade level literacy CCSS (e.g., reading, writing, speaking and listining and/or language).	EQR.ELA.K-2.AS.1
Assesses student proficiency using methods that are unbiased and accessible to all students.	EQR.ELA.K-2.AS.2
Includes aligned rubrics or assessment guidelines that provide sufficient guidance for interpreting student performance and responding to areas where students are not yet meeting standards.	EQR.ELA.K-2.AS.3
Use varied modes of assessment, including a range of pre-, formative, summative and self-assessment measures.	EQR.ELA.K-2.AS.4	Applies to units or longer lessons.

A.2. Attributions and Licensing

This document is based on the EQuIP rubric, available at http://www.achieve.org/EQuIP/. This document is Copyright © 2013 by Southern Illinois University at Carbondale, and is licensed under the Creative Commons Attribution 3.0 Unported License, viewable at http://www.creativecommons.org/licenses/by/3.0/. If modified, please attribute Southern Illinois University at Carbondale and EQuIP, and re-title.

B.1 Appendix B - Dot Notation of Achieve OER Rubric

ASN hosts this rubric for us at http://purl.org/ASN/scheme/AOER/. The scoring guide is not shown here, but is available at http://www.achieve.org/files/AchieveOERRubrics.pdf. All rubrics with the exception of Rubric VIII use a 0-3 scale, plus N/A. Rubric VIII uses Yes, No, and N/A.

Text	Dot Notation	Notes
Rubric I: Degree of alignment to Standards This rubric is applied to learning objects that have suggested alignments to standards. It is used to rate the degree to which an individual object actually ​aligns to each proposed standard. The rubric was designed specifically for the Common Core State Standards, but can be used with any set of standards. Before the rubric can be applied, the assumption is that a user has proposed an alignment between the object and the selected standard(s). There are two major aspects of standards that are vital to a meaningful alignment review: content and performance expectations. It is important that the content addressed in the object matches the content addressed in each proposed standard. Evaluating the alignment of the performances required in both the object and the standard is equally essential and should be considered along with the content.	AOER.DA	IOER will not publish to the Learning Registry with this notation.
Rubric II: Quality of Explanation of the Subject Matter This rubric is applied to objects designed to explain subject matter. It is used to rate how thoroughly the subject matter is explained or otherwise revealed in the object. Teachers might use this object with a whole class, a small group, or an individual student. Students might use the object to self-tutor. For objects that are primarily intended for teacher use, the rubric is applied to the explanation of the subject matter not to the planning instructions for the teacher.	AOER.ES
Rubric III: Utility of Materials Designed to Support Teaching This rubric is applied to objects designed to support teachers in planning or presenting subject matter. The primary user would be a teacher. This rubric evaluates the potential utility of an object at the intended grade level for the majority of instructors.	AOER.ST
Rubric IV: Quality of Assessments. This rubric is applied to those objects designed to determine what a student knows before, during, or after a topic is taught. When many assessment items are included in one object, as is often the case, the rubric is applied to the entire set.	AOER.AS
Rubric V: Quality of Technology Interactivity This rubric is applied to objects designed with a technology-based interactive component. It is used to rate the degree and quality of the interactivity of that component. "Interactivity" is used broadly to mean that the object responds to the user, in other words, it behaves differently based on what the user does. This is not a rating for technology in general, but for technological interactivity. The rubric does not apply to interaction between students, but rather to how the technology responds to the individual user.	AOER.TI
Rubric VI: Quality of Instructional and Practice Exercises This rubric is applied to objects that contain exercises designed to provide an opportunity to practice and strengthen specific skills and knowledge. The purpose of these exercises is to deepen understanding of subject matter and to routinize foundational skills and procedures. When concepts and skills are introduced, providing a sufficient number of exercises to support skill acquisition is critical. However when integrating skills in complex tasks, the number of exercise problems is less important than their richness. These types of practice opportunities may include as few as one or two instructional exercises designed to provide practice applying specific concepts and/or skills. Sets of practice exercises are treated as a single object, with the rubric applied to an entire group.	AOER.IP
Rubric VII: Opportunities for Deeper Learning This rubric is applied to objects designed to engage learners in at least one of the following deeper learning skills, which can be applied across all content areas: Think critically and solve complex problems. Work collaboratively. Communicate effectively. Learn how to learn. Reason abstractly. Construct viable arguments and critique the reasoning of others. Apply discrete knowledge and skills to real-world situations. Construct, use, or analyze models.	AOER.DL
Rubric VIII: Assurance of Accessibility Standards This rubric is used to assure materials are accessible to all students, including students identified as blind, visually impaired or print disabled, and those students who may qualify under the Chafee Amendment to the U.S. 1931 Act to Provide Books to the Adult Blind as Amended. It was developed to assess compliance with U.S. standards and requirements, but could be adapted to accommodate differences in other sets of requirements internationally. Accessibility is critically important for all learners and should be considered in the design of all online materials. Identification of certain characteristics will assist in determining if materials will be fully accessible for all students. Assurance that materials are compliant with the standards, recommendations, and guidelines specified assists educators in the selection and use of accessible versions of materials that can be used with all students, including those with different kinds of challenges and assistive devices. The Assurance of Accessibility Standards Rubric does not ask reviewers to make a judgment on the degree of object quality. Instead, it requests that a determination (yes/no) of characteristics be made that, together with assurance of specific Standards, may determine the degree to which the materials are accessible. Only those who feel qualified to make judgments about an object's accessibility should use this rubric.	AOER.AA	IOER is not utilizing this rubric at this time.
Available in Tagged PDF Format	AOER.AA.1	Adobe maintains this standard.
Available in ePUB format	AOER.AA.2	International Digital Publishing Forum maintains this standard.
Accessible Course within an Open Learning Management System (LMS)	AOER.AA.3	Moodle maintains this standard.
Accessible Course within another Learning Management System (LMS)	AOER.AA.4	LMS Provider maintains this standard.
Available in an accessible media format and includes alternate text or subtitles.	AOER.AA.5	Provider or Publisher maintains this standard.
Includes alternative text (image)	AOER.AA.6	Provider or Publisher maintains this standard.
Includes captions and subtitles (video)	AOER.AA.7	Provider or Publisher maintains this standard.
Includes flash accessibility functions (SWF)	AOER.AA.8	Adobe maintains this standard.
Includes functionality that provide [sic] accessibility	AOER.AA.9	Provider or Publisher maintains this standard.
Complies with WC3 [sic] WCAG2 Recommendations for Web Pages	AOER.AA.10	Should be W3C not WC3. W3C maintains this standard.
Compliant with Section 508 of the Rehabilitation Act	AOER.AA.11	The US Government maintains this standard.
Available in National Accessible Instructional Materials Standard (NIMAS) format - Accessible XML	AOER.AA.12	NIMAS Center at CAST maintains this standard.
Complies with Video/Audio Cassette Production Standards	AOER.AA.13	ITA Standards maintains this standard.
Complies with DVD/DVD-ROM Production Standards	AOER.AA.14	DVD Forum maintains this standard.
Complies with Blue-ray [sic] Disk Production Standards	AOER.AA.15	It's actually Blu-ray, not Blue-ray. UDF 2.5 - Blu-Ray Disk Association maintains this standard.
Complies with NCAM Guidelines for Movies, Web, and Multimedia	AOER.AA.16	The Carl and Ruth Shapiro Family National Center for Accessible Media at Boston public broadcaster WGBH maintains this standard.

B.2 - Attributions and Licensing of Achieve OER Rubric

This document is based on the Achieve Rubrics for Evaluating Open Education Resources, available at http://www.achieve.org/files/AchieveOERRubrics.pdf and http://www.achieve.org/oer-rubrics/. The rubrics are licensed under the Creative Commons Attribution 3.0 Unported License.

This document is Copyright © 2014 by Southern Illinois University at Carbondale, and is licensed under the Creative Commons Attribution 3.0 Unported License. If modified, please attribute Southern Illinois University at Carbondale and Achieve Rubrics for Evaluating Open Education Resources, and re-title.

The Creative Commons Attribution 3.0 Unported License is viewable at http://www.creativecommons.org/licenses/by/3.0/

Paradata Decoded

1. Overview

There are four main parts to a basic paradata statement:

• actor
• verb
• object
• related

2. Actor

The actor is the description of the entity or persona that performed the activity. Actors contain an object type, and one or more description. An actor might be "An educator of grades 9, 10, 11, and 12." The object type would be "educator" and the descriptions would be 9, 10, 11, and 12.

3. Verb

The verb is either a string or an object. A verb might simply be "taught." So "an educator", "taught", "the lesson at some URL." A verb which is an object would contain an action (taught), a date (today), and a measure (how many times the teacher taught). There can be only one verb per paradata document. This precludes having a standard alignment (with an action of "aligned" or "matched") with a rating (degree of alignment), and (generally) precludes having a comment tied to a rating. Standards alignments and ratings would have an object rather than a string for the verb. This is necessary so the scale (minimum and maximum possible rating), number of ratings, and average rating can be included with the verb.

A verb can contain a context. A context typically is where the action was performed. An example context would be "the OER Commons detail page."

An example of a teacher bookmarking a resource on Delicious would have an actor of "Educator...", an action of "bookmarked", and a context of "Delicious"

4. Object

The object is what the paradata statement is about (for example, "http://URL/to/lesson/"). In the example of an educator bookmarking a resource on Delicious, the actor would be "Educator …", the action would be "bookmarked", the context would be "Delicious", and the object would be "http://URL/to/lesson/".

5. Related

Related is an array of objects. This is how you would tie a rating to a standard, thus showing degree of alignment, or how you would tie a rating to a rubric. At the present time, nobody is publishing a single rating record that ties a rating to a standard and a rubric for rating the standard. For example, OER Commons publishes multiple ratings. If a resource is aligned to three standards and six rubrics, they publish:

One rating for each of the three standards that shows how well the resource aligns to the standards.

One rating for each dimension of the rubric - in their case, "Quality of Explanation of the Subject Matter," "Utility of Materials Designed to Support Teaching," "Quality of Assessments", "Quality of Technological Interactivity," "Quality of Instructional and Practice Exercises," and "Opportunities for Deeper Learning."

LR Publishers such as OER Commons never ever publish a rating to both a rubric dimension that applies to a specific learning standard.

6. Comments

The LR supports comments about a resource. LR publishers (like OER Commons and CTE Online) publish comments about a resource. They do not publish comments about a rating. This is because ratings are published as a summary (e.g. "91 people rated this resource a 2.5 on a scale of 0-3). Because data incoming from the LR is summarized in this fashion, ISLE also summarizes ratings and displays them as an average rating, not as "50 people rated this a 1, 35 people rated it a 2, and 95 people rated it a 3. Publishing ratings as a summary precludes publishing comments tied to a rating.

Comments will be displayed in exactly one location on the page, and not as part of the rubric dimensions. Nothing prevents a user from commenting "I rated this a 1 for dimension 1 of the Tri-State Rubric because …"

7. Publishing Resource Ratings

Publishers routinely publish ratings about a resource in general. ISLE has taken the approach of showing Likes and Dislikes. At the present time, no LR publisher is publishing Likes and Dislikes to the LR. Instead they are publishing ratings on a numeric scale. Based on what is currently contained in the LR, ISLE will publish Likes and Dislikes as ratings. ISLE will publish two ratings summaries at a time, one for likes, and one for dislikes. This will allow others to consume our likes and dislikes and interpret them as ratings, or as likes and dislikes. Likes will be published with a rating of 3 on a scale of 0-3. Dislikes will be published with a rating of 0 on a scale of 0-3.

8. User Interface for Standards Alignment and Degree of Alignment

The LR paradata schema specification limits publishers to one verb per paradata statement. A single paradata statement cannot both align a resource to a standard, and rate the degree of alignment. This is done with two separate statements. The user interfaces at OER Commons and at ISLE treat standards alignment and rating a degree of alignment as two separate things, just as the LR does. In order to rate the degree of alignment to a standard, the resource must first be aligned to the standard. Therefore, for every degree of alignment rating of a resource to a standard in the LR, meta- or paradata aligning the resource to the standard should exist in the LR.

When an ISLE user aligns a new resource to a standard, metadata is included which aligns the resource to a standard. When the ISLE user aligns an already existing resource to a standard (such as an item published to the LR by NSDL), paradata will be published which aligns the resource to the standard.

Once the resource is aligned to the standard, ISLE users can rate the degree of alignment to the standard. These ratings will be periodically summarized and published to the LR.

9. ISLE Storing of Likes/Dislikes and Publishing Them to the LR

Individual likes and dislikes coming from our system will be stored in a different table from those that come in from the LR. This gives ISLE the flexibility to take a facebook-style approach to indicate to a user "You like this." It is also important from the standpoint of publishing likes and dislikes to the LR, as it is better to publish these in a summarized form rather than one paradata statement for each person rating. If 100 people like a resource, and they are published to the LR as individual paradata statements, that's a hundred statements that an LR consumer will have to consume, instead of consuming a single statement that contains a summary of the activity. ISLE will take the approach of publishing likes in one LR paradata statement, and dislikes in a second LR paradata statement. This will allow LR consumers to have the flexibility to interpret our likes/dislikes as ratings, or as likes/dislikes.

For example, suppose that between Feb. 1st and Feb. 28th, 2013, 100 people like resource A, and 50 people dislike it. We would publish this in the following fashion:

100 likes:

{ 
	"activity": 
	{
		"actor": 
		{  
			"description": ["9","10"],  
			"objectType": "educator"  
		},
		"verb": 
		{
			"action": "rated",
			"date": "2013-02-01/2013-02-28",
			"context": 
			{
				"id": "http://url/to/resource/Detail/Page",  
			},  
			"measure":
			{
				"sampleSize": 100,  
				"scaleMin": 0,  
				"scaleMax": 3,  
				"value": 3.0  
			},  
		},  
		"object": { "id": "http://url/to/resource/A"  },
		"content": "100 educators rated http://url/to/resource/A 3 on a scale from 0 to 3 during February 2013." 
	}
}

50 dislikes:

{ 
	"activity": 
	{
		"actor": 
		{  
			"description": ["9","10"],  
			"objectType": "educator"  
		},
		"verb": 
		{
			"action": "rated",
			"date": "2013-02-01/2013-02-28",
			"context": 
			{
				"id": "http://url/to/resource/Detail/Page",  
			},  
			"measure":
			{
				"sampleSize": 50,  
				"scaleMin": 0,  
				"scaleMax": 3,  
				"value": 0.0  
			},  
		},  
		"object": { "id": "http://url/to/resource/A"  },
		"content": "50 educators rated http://url/to/resource/A 0 on a scale from 0 to 3 during February 2013." 
	}
}

10. Statistics on Ratings tied to Standards, general ratings, and rubrics

As of 2/28/2013 there are 6,920 rating summary rows for 444,695 resources (so about 1.5% of all resources have a rating).

803 resources have a general rating tied to them (0.18%).

909 resources have a rating tied to an OER Rubric (0.2%). There are 3788 ratings for OER Rubrics (0.6%).

666 resources have a rating tied to at least one CCSS (0.14%). There are 1512 ratings for CCSS standards (0.3%).

And finally, there are 664 resources with a rating tied to both a CCSS and an OER Rubric.

Publishing Paradata

1. Overview

Paradata is often published as a summary of activity to the Learning Registry (LR). Publishing the paradata as a summary where possible makes it easier for those consuming the LR to extract, transform, and load the data into their own database for analysis and searching. Items for which summarization is useful include ratings (general ratings as well as degree of alignment to a standard and ratings based on a rubric), likes and dislikes, resource views, and favorite counts.

In some limited cases, Paradata items can be published individually. An example of this would be comments, which generally cannot be summarized. Another example would be standards alignment.

The Illinois Shared Learning Environment (ISLE) will publish Paradata to the LR on a periodic basis. The period may vary depending upon the type of paradata being published and whether or not it is paradata that can be summarized. Periods will include daily, weekly, and monthly publishing.

2. Views

An example of a record where 50 people have viewed a resource during the month of March 2013:

{
	"activity": 
	{
		"actor": "multiple users",
		"verb": 
		{
			"action": "viewed",
			"date": "2013-03-01/2013-03-31",
			"measure": 
			{
				"measureType": "count",
				"value": 50
			}
		},
		"object": {  "id": "http://url/to/resource/"  }
	}
}

3. Favorites

An example of a record where 50 people have favorited a resource (added it to a library) during the month of March 2013:

{
	"activity": 
	{
		"actor": "multiple users",
		"verb": 
		{
			"action": "favorited",
			"date": "2013-03-01/2013-03-31",
			"measure": 
			{
				"measureType": "count",
				"value": 50
			}
		},
		"object": {  "id": "http://url/to/resource/"  }
	}
}

4. Standards Alignment

A teacher aligned a resource to a common core standard (in this case CCSS.Math.Content.K.CC.A.1):

{
	"activity": 
	{
		"actor": "educator",
		"verb": 
		{
			"action": "aligned",
			"date": "2013-03-01"
		},
		"related": [ 
			{ "object": { "objectType": "Common Core Standard",  "id": "CCSS.Math.Content.K.CC.A.1"  }  } 
		],
		"object": { "id": "http://url/to/resource/"  }
	}
}

5. Tags

An example of a record where someone has tagged additional keywords to a resource:

{
	"activity": 
	{
		"actor":
		{
			"description": ["9","10"],
			"objectType": "educator"  
		},
		"verb": 
		{  
			"action": "tagged", 
			"date": "2013-03-01",
			"context": 
			{  
				"id": "http://some.isle.page/",  
				"description": "ISLE Detail Page",  
				"objectType": "OER"
			},
			"description": ["physics","chemistry"]  
		},
		"object": {  "id": "http://URL/to/resource"  }
	}
}

6. Comments

Each individual comment will need to be published separately. There's no good way to summarize these, but they will be published probably on a monthly basis, just as ratings are.

{
	"activity": 
	{
		"actor": "educator",
		"verb": 
		{
			"action": "commented",
			"date": "2013-03-01",
			"comment": "This resource is a good one for working with kids who have trouble reading."
		},
		"object": {  "id": "http://url/to/resource/"  }
	}
}

7. Assertions

7.1 Deleted Resources

{
	"activity":
	{
		"actor": "educator",
		"verb": 
		{
			"action": "deleted",
			"date": "2013-03-01"
		},
		"object": {  "id": "http://url/to/resource/"  }
	}
}

7.2 Resource is Superceded (has moved)

Organization A asserts that the URL of resource X supercedes the URL of resource Y (Y has moved to X):

{
	"activity": 
	{
		"verb": "superceded", 
		"object": "http://resourceurl/resourceX/", 
    "related": ["http://resourceurl/resourceY/"]
	}
}