Evidence-Centered Assessment Design (ECD) and Design Patterns at AERA 2011

Symposium: How Design Patterns Address Three Pressing Challenges in Large-Scale Assessment

This symposium describes an assessment design methodology, design patterns, for improving the validity of inferences about students from performances in complex large-scale assessments. The presentations describe the framework of evidence-centered assessment design (ECD) and focus on the layer called domain modeling, where design patterns are used. Design patterns are a conceptual tool to guide assessment designers through the web of design decisions inherent in developing assessment tasks. Findings focus on the use of design patterns to address three challenges in large-scale assessment: (1) hard-to-assess standards in the domain of interest (e.g., inquiry science skills), (2) accessibility of tasks for diverse learners with widely varying perceptual and expressive capabilities, and (3) efficient and valid design and development of complex tasks.

1. The Critical Role of Design Patterns in Large-Scale Assessment (Mislevy, Haertel, Liu)
2. Design Patterns to Support Storyboards and Scenario-Based, Innovative Item Types (Snow, Fullkerson, Nichols, Feng)
3. How Design Patterns Integrate Universal Design for Learning (UDL) Into Assessments for Students With Disabilities (Murray, Cheng; Cameto, Haertel, DeBarger, Morrison)
4. Design Patterns Based on a Cognitive Perspective on Learning (Yarnall)

Professional Development Course: Designing Scenario-based Assessment Items Using an Evidence-Centered Design Framework
(Snow, Mislevy, Haertel, Feng, Fulkerson, Nichols)

The course introduces design patterns under the evidence-centered design framework as a high level tool to guide the design and development of scenario-based, innovative assessment items. The course includes lecture, discussion, and group activities. Participants will use design patterns to: (1) select a scenario topic; (2) identify the focal knowledge, skills and abilities to be assessed; (3) conceptualize sequential scenes and "item ideas"; and (4) identify sources of construct irrelevant variance that may contribute to students' poor performance and ways to reduce the variance through task design.

1. Agenda
2. Presentation Slide Deck
3. Resource Materials:
• Samuel Messick's Assessment Argument
• Narrative Structures
• Observational Investigation Design Pattern
• Example Standards and Benchmarks for Activities
• Observational Investigation Design Pattern - Horizontal Associations
• Project URLs
• Annotated Reading List for ECD
• Further Reading on the ECD Project
4. Activity Materials:
• Activity 1: Making Sense of Assessment Design Patterns
• Activity 2: Brainstorm and Specify a Storyboard Topic
• Activity 3: Draft Storyboard Scenes
• Activity 4: Sketch and Fine-Tune Item Ideas