Services

Handscoring

MI hand-scores tens of millions of student responses annually using our state-of-the-art Virtual Scoring Center (VSC). This secure, web-based platform integrates all aspects of the scoring process—from hiring and training to scoring and reporting—ensuring accuracy, consistency, and data security. The VSC includes three core systems:

• VSC Capture Acquires and decodes response data from paper-based tests.
• VSC Train: A secure platform that provides consistent training and practice opportunities for raters and scoring leaders.
• VSC Score: Manages user access, scoring activities, and generates real-time reports.

MI's comprehensive training process equips raters with the knowledge and skills necessary to apply scoring criteria consistently and accurately. Through VSC, raters first receive uniform, in-depth training using rubrics and anchor sets of responses to understand the scoring standards. They then practice scoring student responses, refining their skills while receiving feedback. Finally, raters must pass a qualification stage, ensuring they meet performance standards before participating in operational scoring. This structured approach ensures raters are thoroughly prepared to apply the scoring criteria as intended. Building on this foundation, MI's training methods are reinforced by automated quality assurance processes, ensuring that raters continue to apply scoring criteria consistently during live scoring while maintaining high inter-rater reliability. Our automated tools play a critical role in maintaining scoring quality by providing several layers of monitoring and feedback:

• Automated score resets: If scorer agreement on an item falls below quality expectations, recently scored responses are automatically reset and re-evaluated.
• Automated scorer reassignment: Scorers who fail to meet performance thresholds are automatically reassigned or removed from scoring, ensuring that only high-quality raters remain engaged.
• Automated feedback: MI is the only company offering scorer-specific automated feedback, providing raters with custom, real-time performance insights to correct issues immediately.

Automated mechanisms reduce the need for subjective human analysis of rater performance, allowing human intervention to take a supporting role. Alongside these automated tools, MI also employs traditional quality assurance methods, such as read-behinds (or backreads), where scoring leaders review a portion of scored responses to ensure accuracy and address discrepancies. Human review of rater performance data complements this process.

Our handscoring services include:

• Rangefinding proceedings,
• Scoring material development,
• Recruitment, training, and supervision of scoring personnel,
• Real-time monitoring and automated feedback to maintain performance.

In summary, MI's handscoring services combine advanced technology, rigorous training, and automated quality assurance processes to deliver accurate, reliable, and scalable scoring solutions. By integrating human expertise with automated oversight, we ensure the highest standards of performance across every stage of the scoring process.

Automated Scoring

MI's leadership in automated scoring is evident through our success in major public competitions such as the Hewlett Foundation's Automated Student Assessment Prize (ASAP) and the NAEP Automated Scoring Challenge. In the NAEP Automated Scoring Challenge, MI was one of four grand prize winners, producing the most accurate score predictions for Reading open-ended items while also demonstrating model interpretability and fairness across diverse student demographic groups. In ASAP Phase 1, MI achieved the most accurate predictions for student essays, surpassing two professional raters in reliability. Similarly, in ASAP Phase 2, MI led the field in scoring short, open-ended responses in English Language Arts and Science, achieving the highest levels of agreement with human raters.

At the core of our automated scoring solution is the Project Essay Grade (PEG) engine, originally developed by Ellis Page, the "father of automated essay scoring." MI acquired PEG in 2003 and has since expanded it by incorporating computational linguistics, machine learning, and natural language processing techniques. In addition to hundreds of handcrafted linguistic features, PEG uses advanced technologies such as kernel functions, Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), and even features derived from fine-tuned large language models (LLMs). These advancements enable PEG to capture local context, recognize patterns in writing, and manage complex linguistic relationships, resulting in the most accurate evaluation of student responses.

A key differentiator in MI's automated scoring approach is the way we train the PEG engine. When selecting the training sample, we exclusively use responses scored by our most accurate raters, rather than all responses scored by the larger population of raters. This practice, unique in the industry, provides strong validity evidence and ensures the highest quality of automated scores.

Hybrid Scoring

MI offers a best-in-class hybrid scoring approach that seamlessly integrates human expertise with advanced automated scoring technologies. This approach leverages the strengths of both methods to meet or exceed human scoring standards in quality, accuracy, and reliability.

Our hybrid system incorporates several features designed to improve score accuracy and validity compared to human-only scoring. These include a confidence measure for each response, dynamic assessments of scorer accuracy, and intelligent response routing. Low-confidence responses—those that are difficult to score accurately as they reflect characteristics of multiple score points—are automatically flagged and routed to human raters for further review, following a human-in-the-loop approach. This ensures that responses requiring additional scrutiny receive the appropriate level of attention, leading to more accurate scoring outcomes.

MI has successfully implemented hybrid scoring solutions throughout the United States, consistently delivering reliable, timely, and actionable data. By combining human judgment with machine precision, our hybrid scoring model ensures that even the most challenging responses are evaluated with the highest degree of accuracy.