Help

Exam processing procedure

Item discrimination report

Item Response

The "Item Response" table displays a summary of each item in the exam

• Question
  • The question number
• Option (Score)
  • The response. The first row for a question always corresponds to the correct response. It is labelled as "Overall"
  • The number in the brackets indicate the weight assigned to a response
• Difficulty
  • The proportion of students who selected this response for this item. E.g. 0.9215 = 92.15% of students selected this response
• Standard Deviation
  • Standard deviation for this response (the square of this is the item variance)
• Discrimination
  • A correlation between the item score and the total test score
  • Also known as "item total correlation"
  • Positive item-total correlations mean high-scoring examinees tend to get this item correct, and low scoring examinees tend to get this item wrong.
  • Higher values indicate a larger amount of discrimination
  • Values near 0 indicate little to no discrimination
  • Negative values indicate a problem – this means low scoring examinees tend to get the item correct – you may need to check your answer key is correct.
  • To assist with visualising the table, items in red have a discrimination of less than 0.1, items in orange have a discrimination between 0.1 and 0.3, and items in green have a discrimination > 0.3

Reliablity Analysis

Reliability refers to the reproducibility of test scores. If reliability is high (closer to 1), test scores from one test should be similar to another test that was intended to measure the same thing.
The table below contains a range of reliability measures. Multiple measures are reported to account for different sources of measurement error.
Reliability values range from 0 to 1. A large reliability value indicates that variance in the observed score (exam score) is due to the true score (what was intended to be measured). A low reliability value (near 0) indicates most of the observed score variance is due to measurement error.
SEM (Standard Error of Measurement) quantifies the amount of uncertainty with the exam score. This is the difference between the score a student obtains on a test (an observed score) and the hypothetical "true score" that the same student would obtain if a test could measure the student's achievement level with perfect accuracy. A student will have an exam score within one SEM of the true score about 68 percent of the time and within two SEMs of the true score about 95 percent of the time.

Reliability If Item Deleted

This table describes the result of the reliability measures assuming the item listed has been removed from the exam. An increase to the reliability would suggest that the item is not consistent with the other items in the exam. For example, removing Q2 appears to increase the reliability of the exam as a whole.

Conditional Standard Error Of Measurement

The Conditional Standard Error of Measurement (CSEM) is the SEM at a specific score level.

Reference:

Meyer, J. Patrick. Applied Measurement with JMetrik. Routledge, 2014.

The fine print

This system is based on the open source paper form processing system, queXF, also developed by ACSPRI. It replaces the need for specialised mark recognition hardware, and related printing, such as Scantron devices. Multiple choice examination papers can be printed in black and white or colour using regular printers, and scanned in PDF format using regular scanners. Exam participants can fill in the sheet with a pen or dark pencil. Once the papers are uploaded, the queXF mark recognition software attempts to automatically process the papers. If there are any results it cannot be 100% satisfied with, these are sent to verification staff who identify the selected response. Once all papers for an examination are processed, the bulk results are checked for any errors or unexpected results, and then sent to the examination creator via a one-time link secure download. In addition to the students score in CSV/spreadsheet format, raw results are provided (each response written by the student) along with a report containing item discrimination and reliability analyses.