Canvas MOOC Date Announced

David Cutler interviewed historian, Eric Foner, in the Atlantic magazine. The entire interview is worth reading. The article is cleverly titled – You Have to Know History to Actually Teach It. Cutler asked Dr. Foner: Do you have other specific advice for what teachers can do to more effectively instruct history students?
The first thing I would say is that we have to get away from the idea that any old person can teach history. A lot of the history teachers in this country are actually athletic coaches. I mention this in class, and students always say, “Oh yeah, Coach Smith, he taught my history course.” Why? Well, Coach Smith is the football coach, and in the spring he’s not doing much, and they say, “Well, put him in the history course, he can do that.” They wouldn’t put him in a French course, or a physics course. The number-one thing is, you have to know history to actually teach it. That seems like an obvious point, but sometimes it’s ignored in schools. Even more than that, I think it’s important that people who are teaching history do have training in history. A lot of times people have education degrees, which have not actually provided them with a lot of training in the subject.
HelpHistTchUnder Common Core, all teachers need to be writing teachers. Unfortunately many History/Social Studies teachers have not had significant instruction and/or practice in historical writing. Worse, very few teacher professional development seminars focus on this topic. As a first step toward becoming writing teachers, Social Studies teachers can increase student literacy skills by inspiring their students to interpret history through documents.
These new standards call for teachers to emphasize argumentative, explanatory/informative, and narrative writing into History/Social Studies. Many teachers are unsure how to respond to these new standards. Should teachers stop delivery of subject content to explicitly teach spelling, vocabulary, and sentence construction? Should professional learning communities (PLCs) devote a specific amount of time to writing instruction in each subject? How many writing projects should be delivered in each subject? These questions are unlikely to be answered by Coach Smith. However, free professional development is available to History teachers looking to improve their writing instruction. https://www.canvas.net/courses/helping-history-teachers-become-writing-teachers
This website will curate many of the readings and resources for the course over the next six months. We are excited to connect and collaborate with History teachers around the globe. So far we have confirmed the following Guest Lecturers: Dr. Chris Schunm (Pitt) Implementing Peer Review; Dr. Darren Reid (Coventry U) Sourcing & Contextualizing Primary Sources – modeling teacher thinkalouds; Dr. Sherri Colby (Texas A&M, Commerce) Historical Narratives; and Will Fitzhugh (The Concord Review) Teaching with Examples. We are actively reaching out to others in the Historical Writing community. Please feel free to comment or email us any suggestions. We hope to see you in class on January 12, 2015.

References
http://www.theatlantic.com/education/archive/2014/01/you-have-to-know-history-to-actually-teach-it/282957/2/

Which Tasks Improve Historical Reasoning?

Scholars do not know the influence of specific task structures on students’ writing or historical reasoning. Historical reasoning is defined as analyzing evidence, understanding the meaning of evidence, and using evidence to construct and explain historically plausible accounts of the past (p. 291).

Do all argumentative writing tasks provide students with the same opportunity to develop their historical thinking or writing? Are some ways of framing questions to promote historical thinking and writing better than others? Can the structure of a writing prompt influence student outcomes? This study suggests YES. The type of task explained 31% of the variance in the quality of students’ overall historical reasoning.

Much of the research on history writing has focused on how students draw on multiple sources in constructing essays. Some attention has been given to comparing argumentative writing in comparison with other genres such as narratives, summaries, and explanations. Almost zero research has been done on what we ask students to write and how that affects their thinking and writing.

In developing arguments, writing is often complicated by patterns of thinking and working with evidence. The use of evidence indicates aspects of disciplinary reasoning, including recognizing biases in sources, comparing evidence, situating evidence in its context, and taking into account different perspectives and multiple causes. Historical interpretations rely on the public display of evidence to substantiate claims – a claim cannot stand without evidence.

In defining approaches to historical texts, Sam Wineburg identified discipline-specific ways of reading and thinking. For historians, primary documents are regarded as excerpts of social interactions. They have to be reconstructed with context added to make the documents meaningful.

Providing writing prompts that require close reading and consideration of the author’s perspective supports historical thinking and greater understanding. So, perhaps the structure and focus of the writing prompt affects the quality of students’ historical reasoning? These researchers sought to find out how.

The authors created four reading and writing tasks using the same documents and randomly assigned one task to each student. Each task presented the same background information adapted from the social studies textbook. They worded each prompt differently to frame the issue of Cold War causes from a variety of historical angles. The situated prompt encouraged students to imagine they heard these speeches and write as though they were living in 1947. The sourcing prompt encouraged students to focus on the motivations of each author in making their respective speeches. The document analysis prompt encouraged students to identify similarities and differences in the documents. The causal prompt asked students why Churchill and Truman spoke out against the Soviet Union and communism directly. At the end of each prompt, the researchers asked students to write “M.E.A.L.” (main idea, evidence, analysis, and link to thesis) paragraphs.

The sourcing, document analysis, and causal prompts were associated with higher student scores. The situated prompt had the lowest mean score of all the tasks. Results indicated that the writing prompts centered on sourcing, corroboration of documents, and causation were more likely to focus attention on historical perspectives than prompts that asked students to imagine themselves as historical figures. Because so much of history relies on evidence-based thinking, prompts that focus students directly on sources may be more likely to promote historical reasoning.

Unfortunately, when most history teachers assign writing, the focus is on summarization. Because interpreting history relies on reconciling multiple sources of evidence, this focus inhibits historical reasoning. So, the key takeaway from this study is – how can high school history teachers create prompts that emphasize corroboration, sourcing, and causal analysis into their classroom practices?

Reference

Monte-Sano, C., & De La Paz, S. (2012). Using writing tasks to elicit adolescents’ historical reasoning. Journal of Literacy Research, 44(3), 273-299.

This week our discussion board will focus on developing prompts like these and integrating them with your lessons that are already in progress. Please describe the grade, level and subject you are teaching, the focus of your lesson, and identify an idea for an essay that could that emphasize corroboration, sourcing, and causal analysis. The community will chime in and suggest ideas for documents, scaffolding, and fine-tuning. Follow the rule, give one to get one.

Learn to Teach, Teach to Learn

by Dr. Strojny

Recently, Salon online magazine ran an article that touched on whether successful teaching is just a learned talent or a natural skill that some people have, and some don’t. The article touches on that author Elizabeth Green titles, The Myth of the Natural Born Teacher.”

http://www.salon.com/2014/08/09/secrets_of_amazing_teachers_what_both_sides_of_the_education_reform_debate_get_wrong_about_autonomy_and_accountability/

This type of myth-making—that some folks are just “born to teach,” and others will hopelessly bore (at best) or (at worst), mis-educate and harm the learning of our students, is surprisingly prevalent, Green asserts, on not only both sides of the political spectrum, but even in the Schools of Education of colleges and universities across the country. Yet considerable evidence exists, that, in fact, using particular skills and strategies result in increased levels of learning for students. This is fantastic news. Just as no one would want a “a natural born surgeon,” (as opposed to one who had spent years in medical school) and pro sports players don’t get paid the big bucks for potential, so too, teaching well is a learned skill, one that takes hard work, and willingness to adapt to new strategies.

For many teachers, the problem can be in sorting through the tremendous amounts of “noise” out there to find the best practices that will truly result in increased learning. In 2008, Dr. Monte-Sano compared two different types of teaching strategies for secondary Social Studies/History teachers. Specifically, Monte-Sano was looking for results the writing skills of history students based on type of instruction. In Monte-Sano’s work, both teachers engaged in a mix of lecture, regular reading and writing assignments and some use of textbook reading. (Monte-Sano, 2008). However, in other important ways, teacher practices differed. One teacher modeled active reading strategies, and encouraged students view history writing as an interpretive exercise, with the interpretation of historical documents as critical to creating meaningful and intelligent interpretations. The other teacher leaned towards teaching history more as serious of static factual events.

Monte-Sano found that those students whose teacher who relied more heavily on memorization of facts and assigned frequent essays, but provided limited feedback, had writing skills that stayed the same or even declined slightly over the course of the school year. However, students whose teacher gave consistent targeted feedback had students whose scores improved over the year. The teacher also engaged in traditional “English teacher” type reading comprehension and writing strategies, such as modeling, creating scaffolded opportunities for students to write, one-on-one conferencing and targeted feedback.

See: Monte-Sano, C. (2008). Qualities of effective writing instruction in history classrooms: A cross-case comparison of two teachers’ practices. American Educational Research Journal, 45 (4), 1045-1079. 

Nevertheless, many history teachers continue to maintain the first type of more “traditional” type of classroom, expending a lot of work on reading and frequent essay practices that may result in more limited in feedback for students. Why is this? In general, teachers work hard and most want to see the best possible results for their students.

Well, the teacher struggling to see gains in low-SES schools may be more amenable to trying something new, since they are can see clearly their student’s English language deficiencies. Many (certainly not all!) students in low-income schools may already be scoring at a low level on high stakes tests, and so teachers are willing to adopt a new strategy in an attempt to help their students.

In addition, students themselves are pre-disposed, as late as college, to view history as a single story, and less likely to view history through the complex lens of interpretative art (Monte-Sano, 2012). Students come in to class “expecting” to hear a story with names, dates, and events they have to memorize, and may be more challenged to adapt to class that offers a truly college preparatory vision of historical writing. If we ask students to write true historical interpretative essays, we are asking type of careful sifting of evidence and crafting of interpretation is some of the most sophisticated and complex writing students in all of their high school careers—and we are asking them to do it in History/Social Studies classes – not English classes! No wonder History and Social Studies teachers are feeling the heat.

So what types of strategies should History teachers focus on in teaching Historical writing to students? The research suggests they should focus on the same high-impact strategies English teachers use, but focused or adjusted appropriately for History/Social studies.

High Impact strategies for ALL students writing in History and Social Studies class include:

(1) Modeling reading strategies to help students navigate texts;

(2) Are taught to study history as evidence-based interpretation, as opposed to a single narrative;

(3) Are taught to read primary source documents, and to point to evidence from such sources to support their own ideas;

(4) Engage in frequent writing work, that includes not just formal essays, but informal writing opportunities;

(5) Receive targeted feedback that encourages them to look at History and Social Studies as a way of asking lots of questions, rather then memorizing a set series of answers.

These types of critical thinking skills are what will help our students to be successful readers and writers not only in high school, but in the collegiate and university setting and beyond. Just as true for us as teachers, as it is for our students: It’s not just WHO you are, but WHAT you do that will make a difference in student learning. Do we really want to teach our students any other lesson?

 Citations:

Monte-Sano, C. (2008). Qualities of effective writing instruction in history classrooms: A cross-case comparison of two teachers’ practices. American Educational Research Journal, 45 (4), 1045-1079. 

Monte-Sano, C. (2012). Toward disciplinary writing in history: Preparing the next generation. Perspectives on History, 50 (5). Available at: www.historians.org/perspectives/issues/2012/1205/

http://www.salon.com/2014/08/09/secrets_of_amazing_teachers_what_both_sides_of_the_education_reform_debate_get_wrong_about_autonomy_and_accountability/ (Found August 11, 2014)

Preview:

For next time, I will look at some of the highest-impact writing strategies for English teachers, and see how they can be adapted for use by Social Studies teachers. If you can’t wait to get started, have a sneak peak at this important study for English teachers: A meta-analysis of writing instruction for adolescent students.

Graham, Steve; Perin, Dolores Journal of Educational Psychology, Vol 99(3), Aug 2007, 445-476

Nurture vs Nature Ed Debate

by Dr Strojny

Recently, Salon online magazine ran an article that speculated whether successful teaching is a nurtured talent or a natural skill that some people have, and some don’t. The article was entitled “The Myth of the Natural Born Teacher” by Elizabeth Green.

http://www.salon.com/2014/08/09/secrets_of_amazing_teachers_what_both_sides_of_the_education_reform_debate_get_wrong_about_autonomy_and_accountability/

This type of myth-making—that some folks are just “born to teach,” and others will hopelessly bore (at best) or (at worst), harm the learning of our students, is surprisingly prevalent, Green asserts, on not only both sides of the political spectrum, but nationwide in schools of Education. Yet considerable evidence exists, that, in fact, using particular skills and strategies result in increased levels of learning for students. This is fantastic news. Just as no one would want a “natural born surgeon,” teaching well is a learned skill, one that takes hard work, and willingness to adopt new strategies.

For many teachers, the problem can be in sorting through the tremendous amounts of “noise” out there to find the best practices that will truly result in increased learning. In 2008, Monte-Sano compared two different types of teaching strategies for secondary Social Studies/History teachers. This research examined the writing skills of history students based on type of instruction. In this work, both teachers engaged in a mix of lecture, regular reading and writing assignments and some use of textbook reading. One teacher modeled active reading strategies, and encouraged students view history writing as an interpretive exercise, with the interpretation of historical documents as critical to creating meaningful and intelligent interpretations. The other teacher leaned toward teaching history as serious of static factual events.

Monte-Sano found that those students whose teacher who relied more heavily on memorization of facts and assigned frequent essays, but provided limited feedback, had writing skills that stayed the same or even declined slightly over the course of the school year. However, students whose teacher gave consistent targeted feedback had students whose scores improved over the year. The teacher also engaged in traditional “English teacher” type reading comprehension and writing strategies, such as modeling, creating scaffolded opportunities for students to write, one-on-one conferencing and targeted feedback.

In addition, students are pre-disposed to view history as a single story, and less likely to view history through the complex lens of interpretative art (Monte-Sano, 2012). Students come in to class “expecting” to hear a story with names, dates, and events they have to memorize, and may be more challenged to adapt to class that offers a truly college preparatory vision of historical writing. If we ask students to write interpretative essays, we are asking for a careful sifting of evidence and crafting of interpretation, which is some of the most complex writing required of students, and we are asking them to do it in History/Social Studies classes – not English classes! No wonder History and Social Studies teachers are feeling the heat.

So what types of strategies should History teachers focus on in teaching Historical writing to students? The research suggests they should focus on the same high-impact strategies English teachers use, but adjusted for History/Social Studies.

High Impact strategies for ALL students writing in History and Social Studies class include:

(1) Modeling reading strategies to help students navigate texts;

(2) Are taught to study history as evidence-based interpretation, as opposed to a single narrative;

(3) Are taught to read primary source documents, and to point to evidence from such sources to support their own ideas;

(4) Engage in frequent writing work, that includes not just formal essays, but informal writing opportunities;

(5) Receive targeted feedback that encourages them to look at History and Social Studies as a way of asking lots of questions, rather then memorizing a set series of answers.

These types of critical thinking skills are what will help our students to be successful readers and writers not only in high school, but in the  university setting and beyond. This is just as true for us as teachers, as it is for our students: It’s not just WHO you are, but WHAT you do that will make a difference in student learning. Do we really want to teach our students any other lesson?

 References

Monte-Sano, C. (2008). Qualities of effective writing instruction in history classrooms: A cross-case comparison of two teachers’ practices. American Educational Research Journal, 45 (4), 1045-1079.

Monte-Sano, C. (2012). Toward disciplinary writing in history: Preparing the next generation. Perspectives on History, 50 (5). Available at: www.historians.org/perspectives/issues/2012/1205/

http://www.salon.com/2014/08/09/secrets_of_amazing_teachers_what_both_sides_of_the_education_reform_debate_get_wrong_about_autonomy_and_accountability/ (Found August 11, 2014)

Preview:

For next time, I will look at some of the highest-impact writing strategies for English teachers, and see how they can be adapted for use by Social Studies teachers. If you can’t wait to get started, have a sneak peak at this important study for English teachers: A meta-analysis of writing instruction for adolescent students.

Graham, Steve; Perin, Dolores Journal of Educational Psychology, Vol 99(3), Aug 2007, 445-476

 

 

 

The Document Based Lesson

The Document Based Lesson (DBL) is the third section of Abby Reisman’s award-winning dissertation. Avishag (Abby) Reisman lectures at the Teachers College at Columbia University and is an Assistant Professor at Penn. She won the Larry Metcalf Exemplary Dissertation Award from the National Council for the Social Studies (NCSS) in 2011. This post attempts to summarize pp. 124-168. The entire dissertation is a fantastic read for History geeks.

This paper came out of a six-month intervention that was tested in five San Francisco schools. The Reading like a Historian program (RLH) created significant outcomes on student learning along four measures: (1) historical thinking, (2) factual knowledge, (3) general reasoning, and (4) reading comprehension. DBL requires students to engage in rigorous, open-ended historical investigations. The curriculum provided teachers with classroom-ready, sequenced lessons that were free. The activity sequence followed three distinct structures in the same order in each lesson: (a) establishment of background knowledge, (b) historical inquiry with multiple documents, and (c) discussion. DBL Fig 1 Each lesson begins with a review of background knowledge via lecture, video, or textbook questions. Students read between 2-5 primary documents that examine a historical question from several perspectives. Documents offer conflicting interpretations. They are sequenced to make students change their minds. These conflicting accounts forced students to evaluate the level of truthiness (thanks Steven Colbert) in the claims, consider the context, and rationalize their judgments. Finally, students participate in a whole-class discussion around the central question and were required to use evidence from the documents to substantiate their claims. Teachers remain active leaders of classroom activities. They rely on sequences to review students’ content knowledge and to redirect discussion to the documents. Key to the program’s success was how DBL embedded historical inquiry into familiar structures and rearranged them into a repeatable instructional order. Doc Based Lesson RLH simplified excerpts of primary source documents for presentation and focus. Each doc was presented in large font with lots of white space and no longer than 250 words. Then, RLH turned social studies teachers into reading instructors who delivered explicit strategy instruction. Students need to watch teachers repeatedly practice the strategies of disciplinary reading. The DBL curriculum chose four strategies used by expert historical readers: (1) sourcing (considering the document’s source and purpose), (2) contextualization (placing the document in a temporal and spatial context), (3) corroboration (comparing the accounts of multiple sources against each other), and (4) close-reading (considering an author’s use of language and word choice.

This lesson encompassed five activities in a 50-minute class period: video, lecture, teacher model, small group-work, and whole-class discussion. The activities all shared the goal of initiating students into the practices of historical inquiry. DBL mirrored reform efforts that have produced instructional change by providing extensive materials to support teacher change, clear and specific methods for instruction practice, and local facilitators whose job it is to coach teachers and ensure curricular fidelity. Teachers interested in adopting this model may get free lessons.

References

https://stacks.stanford.edu/file/druid:vv771bw4976/Reisman_Dissertation_ReadinglikeaHistorian-augmented.pdf

Goal-setting Strategies

The College and Career Readiness Anchor Standards in Writing demand that students are able to write arguments on discipline-specific content, developing claims and counterclaims, while establishing a formal tone and objective style citation. Using controversial, or debatable content, teaching students to write in the third person, and using their historical content knowledge to “qualify them as an expert” may provide motivation when teaching this type of writing. Explicit instruction on these skills and displaying data that informs students as to where they rank among their peers are essential components in setting goals and teaching students new to academic writing.

CR&Writing Stndrds

Rogers & Graham (2008) found goal setting for productivity effective in a meta-analysis of single subject writing interventions. For History teachers, who are not used to being writing teachers, goal-setting may help motivate students who aren’t used to writing in class. Depending on the skill level of the students in your class, goal-setting can center on word production, number of claims, rebuttals, argumentative strategies, document usage, or citations.

De La Paz (2005) compared 8th-grade students (N=70) in an integrated social studies and language arts unit designed to promote historical understanding and argumentative writing to a control group of students (N=62) who did not receive writing intervention or instruction. Results indicated the students who demonstrated mastery of the target strategies during instruction wrote historically more accurate and more persuasive essays regardless of their initial learning profile.

Similarly, De La Paz & Felton (2010) compared 11th grade students who learned a pre-writing strategy (N=81) for composing argumentative essays related to historical events to a control group (N=79) that read the same primary and secondary source document sets. They found that the essays written by students who received pre-writing instruction were longer, were rated as having significantly greater historical accuracy, were significantly more persuasive, and claims and rebuttals within each argument became more elaborated. The word count for the pre-writing instruction group increased from 195.32 to 327.86 , an average increase  of 132.54 words. Yet for the control group, word production only increased by 14.45 words.

This research suggests that writing instruction focused on goal setting strategies, argumentative claims and rebuttals, and historical accuracy may be effective when introducing common core writing tasks to students. Hence, instructional leaders should encourage teachers to design, develop, and analyze DBQs as formative assessments in common planning time, or department professional development.

Complex writing assignments, or DBQs, are essential for improving adolescent literacy (Fisher & Frey, 2007). DBQ units align with plans for increasing writing proficiency, critical thinking, and creating a college-going culture. DBQ’s can be designed to give students a preview of Advanced Placement curriculum. Increased use of DBQs should lead to greater English proficiency and help students avoid costly and demoralizing remedial coursework that has an adverse effect on college completion rates. DBQs can be jointly developed and graded by History and English teachers to ensure that students will meet the new Common Core standards for Writing. Students may be more motivated when they get credit in both classes for the same assignment. This credit should be given in multiple stages for planning, writing, and revising DBQs.

References

Blanton, L.L. (1986). Reshaping ESL students’ perceptions of writing. ELT Journal. 41(2), 112-118. DOI: 10.1093/elt/41.2.112

De La Paz, S. (2005). Effects of historical reasoning instruction and writing strategy mastery in culturally and academically diverse middle school classrooms. Journal of Educational Psychology, 97(2), 139-156.

De La Paz, S., & Felton, M. (2010). Reading and writing from multiple source documents in history: Effects of strategy instruction with low to average high school writers. Journal of Contemporary Educational Psychology, 35, 174-192.

Fisher, D., & Frey, N. (2007). Checking for understanding: Formative assessment techniques for your classroom. Alexandria, VA: Association for Supervision and Curriculum Development.

Rogers, L., & Graham, S. (2008). A meta-analysis of single subject design writing intervention research. Journal of Educational Psychology, 100(4), 879-906.

Validating Rubrics

When teachers, departments, and schools use writing from sources as formative assessments in History, protocols need to be followed before evaluating these assessments. Many departments or schools collaboratively grade these assessments during common planning time, or teacher professional development. This requires training evaluators in using validated rubrics before applying this knowledge to the analysis of student work. Teachers work together to identify exemplars that strongly correlate with the spectrum of work defined by the rubric.

Annotated LDC Rubric

Holistic scoring involves assigning a single score that indicates the overall quality of a text (Bang, 2012). Raters give one summary score based on their impression of a text without trying to evaluate a specific set of skills. Analytic scoring examines multiple aspects of writing (e.g., content, structure, mechanics, etc.) and assigns a score for each. This type of evaluation generates several scores useful for guiding instruction.

Broadly defined, reliability is the consistency with which an instrument/method produces measurements, while validity is the extent to which an instrument/method actually measures what it is meant to measure, or its accuracy. In testing writing rubrics, agreement rates are used to determine inter-rater reliability, where agreement is further defined as exact or adjacent scores. Exact agreement consensus rates need to be 70% or greater to be considered reliable (Stemler, 2004). Adjacent agreements within one score point should exceed 90% to indicate a good level of consistency (Jonsonn & Svingby, 2007).

I used Google Forms to have my students validate the above rubric from the Literacy Design Collaborative. I found the LDC rubric to be more student friendly than the rubric my District adapted from the Smarter Balanced consortium.

LDC Rubric Agreement Frequency

Jonsonn & Svingby (2007) analyzed 75 rubric validation studies and found (a) benchmarks are most likely to increase agreement, but they should be chosen with care since the scoring depends heavily on the benchmarks chosen to define the rubric; (b) agreement is improved by training, but training will probably never totally eliminate differences; (c) topic-specific rubrics are likely to produce more generalizable and dependable scores than generic rubrics; and (d) augmentation of the rating scale (for example so the raters can expand the number of levels using + or − signs) seems to improve certain aspects of inter-rater reliability, although not consensus agreements.

Validating a rubric with your class gives your students additional time to consider their historical writing. When they have to review more than one student’s writing, they establish a context for evaluating their own writing. Class discussions should identify exemplars of strong historical writing. Direct instruction should focus on improving examples of weak writing.  Rubric validation is a much-needed historical thinking exercise. Otherwise your students may develop  what educational psychologists call the Dunning-Kruger Effect.

The Dunning-Kruger Effect describes a cognitive bias in which people perform poorly on a task, but lack the meta-cognitive capacity to properly evaluate their performance. As a result, such people remain unaware of their incompetence and accordingly fail to take any self- improvement measures that might rid them of their incompetence.

References

Bang, H. J. (N.D.) Reliability of National Writing Project’s Analytic Writing Continuum Assessment System.

Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144.

Kruger, J. &  Dunning, D. (1999) Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, Vol 77(6), Dec 1999, 1121-1134.

Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4).