Validating Rubrics

When teachers, departments, and schools use writing from sources as formative assessments in History, protocols need to be followed before evaluating these assessments. Many departments or schools collaboratively grade these assessments during common planning time, or teacher professional development. This requires training evaluators in using validated rubrics before applying this knowledge to the analysis of student work. Teachers work together to identify exemplars that strongly correlate with the spectrum of work defined by the rubric.

Annotated LDC Rubric

Holistic scoring involves assigning a single score that indicates the overall quality of a text (Bang, 2012). Raters give one summary score based on their impression of a text without trying to evaluate a specific set of skills. Analytic scoring examines multiple aspects of writing (e.g., content, structure, mechanics, etc.) and assigns a score for each. This type of evaluation generates several scores useful for guiding instruction.

Broadly defined, reliability is the consistency with which an instrument/method produces measurements, while validity is the extent to which an instrument/method actually measures what it is meant to measure, or its accuracy. In testing writing rubrics, agreement rates are used to determine inter-rater reliability, where agreement is further defined as exact or adjacent scores. Exact agreement consensus rates need to be 70% or greater to be considered reliable (Stemler, 2004). Adjacent agreements within one score point should exceed 90% to indicate a good level of consistency (Jonsonn & Svingby, 2007).

I used Google Forms to have my students validate the above rubric from the Literacy Design Collaborative. I found the LDC rubric to be more student friendly than the rubric my District adapted from the Smarter Balanced consortium.

LDC Rubric Agreement Frequency

Jonsonn & Svingby (2007) analyzed 75 rubric validation studies and found (a) benchmarks are most likely to increase agreement, but they should be chosen with care since the scoring depends heavily on the benchmarks chosen to define the rubric; (b) agreement is improved by training, but training will probably never totally eliminate differences; (c) topic-specific rubrics are likely to produce more generalizable and dependable scores than generic rubrics; and (d) augmentation of the rating scale (for example so the raters can expand the number of levels using + or − signs) seems to improve certain aspects of inter-rater reliability, although not consensus agreements.

Validating a rubric with your class gives your students additional time to consider their historical writing. When they have to review more than one student’s writing, they establish a context for evaluating their own writing. Class discussions should identify exemplars of strong historical writing. Direct instruction should focus on improving examples of weak writing.  Rubric validation is a much-needed historical thinking exercise. Otherwise your students may develop  what educational psychologists call the Dunning-Kruger Effect.

The Dunning-Kruger Effect describes a cognitive bias in which people perform poorly on a task, but lack the meta-cognitive capacity to properly evaluate their performance. As a result, such people remain unaware of their incompetence and accordingly fail to take any self- improvement measures that might rid them of their incompetence.

References

Bang, H. J. (N.D.) Reliability of National Writing Project’s Analytic Writing Continuum Assessment System.

Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144.

Kruger, J. &  Dunning, D. (1999) Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, Vol 77(6), Dec 1999, 1121-1134.

Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4).

Creating Peer Review Systems

With the implementation of the Common Core State Standards, writing instruction will become distributed throughout the school. Writing from sources require students to respond to the ideas, events, facts, and arguments presented in texts they are assigned. Teachers can improve student literacy skills by increasing writing assignments, yet some teachers have expressed a reluctance to assign more frequent writing tasks because they fear it will increase their workload.

Peer Review

Implementing an effective peer review program with free online polling tools like surveymonkey, polleverywhere, and google forms can transfer the burden of grading from teachers to students. The grading process becomes a student-centered, learning by evaluation collaborative activity. O’Toole (2013) suggested peer assessment should be structured, with a learning design that includes “phases of activity, peer assessment, reviewing and reflecting” (p. 5). Brookhart (2013) recommended student-generated rubrics to allow for highly effective peer grading systems. Bardine and Fulton (2008) advocated using revision memos to have students explicitly address weaknesses in drafts and develop confidence in academic writing.

Peer review programs give students practice in developing the skills necessary to recognize effective thesis statements, use textual evidence, and refine arguments. Learning by evaluation significantly improves a student’s self-assessment abilities and lays the groundwork for self-improvement. Thus, learning by evaluation programs should focus on one or two aspects of effective writing, include student discussion to drive reflection about writing as an iterative process, and allow increased instructional time for student revision.

I polled English teachers at my school and found that 39% were confident in their ability to teach students how to write a thesis. After surveying our students, however, only 9% were confident in their ability to develop a thesis statement. This gap suggests teachers need to give students more practice in developing, identifying, and assessing thesis statements. Further, teachers can showcase student exemplars and improve weak thesis statements via thinkalouds. Once students gain more confidence and proficiency in writing thesis statements, teachers can move on and address other factors in effective academic writing, such as claims, rebuttals, argumentative strategies, document usage, and citations.

References

Bardine, B., & Fulton, A. (2008). Analyzing the benefits of revision memos during the writing and revision process. The Clearing House, 81(4), 149-154.

Brookhart, S. M. (2013). How to create and use rubrics for formative assessment and grading. Teacher Librarian, 40(4), 52.

O’Toole, R. (2013) Pedagogical strategies and technologies for peer assessment in Massively Open Online Courses (MOOCs). Discussion Paper. University of Warwick, Coventry, UK: University of Warwick. (Unpublished).

 

History Rewriter

The Common Core State Standards call for teachers to emphasize argumentative, explanatory/informative, and narrative writing into all subjects. Many teachers across the content areas are unsure how to respond to these new standards. Should teachers stop delivery of subject content to explicitly teach spelling, vocabulary, and sentence construction? Should professional learning communities (PLCs) devote a specific amount of time to writing instruction in each subject? How many writing projects should be delivered in each subject? Educators will struggle with these questions as they implement the Common Core writing and literacy standards, however, this website will present methods for how can teachers begin improving writing instruction in History-Social Science classes immediately.

HRW logo

There is concern that a majority of adolescents do not develop the competence in writing they need to be successful in school, the workplace, or their personal lives (Graham & Perin, 2007). Other researchers (Bissex & Bullock, 1987; Calkins, 1994; Graves, 1983;) have noted a connection between increased reading and writing and higher levels of academic achievement. Hence, Common Core and an increasing number of assessments, including the ACT, NY Regents Exam, and CRWA, employ writing-from-sources tasks that integrate reading and writing.

HSTRWTR

Under Common Core, all teachers need to be writing teachers. Unfortunately many History/Social Studies teachers have not had significant instruction and/or practice in historical writing. Worse, very few teacher professional development seminars focus on this topic. As a first step toward becoming writing teachers, Social Studies teachers can increase student literacy skills by incorporating writing from sources, or document-based questions (DBQs) as formative assessments. Goal-setting strategies may be well-suited for guiding and motivating students as the introduction of historical writing increases the rigor in your instructional program.