Score Aggregation Matrix: Normalize Reviewer Scores Free & Online

Reviewers do not use the same scale, even when they share one. This free tool normalizes reviewer scores with z-scores, so a harsh reviewer and a generous reviewer count fairly in editorial decisions.

What the tool does

Reviewers do not use the same scale, even when they share one. One reviewer’s 7 is another’s 9, and a paper with scores of 9, 8, and 5 may be better than a paper with 8, 8, and 8, or worse, depending on who gave the 5. This tool normalises reviewer scores with z-scores: it subtracts the group mean from each score and divides by the spread, so every score is expressed as distance from the average. Harsh and generous reviewers then count more fairly, and the table shows raw scores beside normalised ones.

How to use it

Paste at least two scores, separated by commas or line breaks, in the order of your reviewers. Use one scale for all of them, for example 1 to 10, and do not mix scales. Press the button and read the mean raw score first, then the z-scores. A positive z-score means above the group average, a negative one means below it, and the size tells you by how many standard deviations. Compare papers by their mean z-score when different reviewer panels scored them.

Worked example

Take scores of 7, 8, 5, and 9. The mean is 7.25 and the spread is about 1.48. The normalised scores are roughly minus 0.17, plus 0.51, minus 1.52, and plus 1.18. The story is now clear: the 5 is a real outlier, more than one and a half standard deviations below the group, while the 7 and 8 sit close to the middle. An editor reading only the mean, 7.25, would miss the disagreement entirely. The normalised table shows it at a glance.

Reading the results honestly

Normalisation helps comparison, but it also hides information. A z-score of zero on a panel that scored 9, 9, and 9 means something very different from zero on a panel that scored 3, 3, and 3, because the raw level disappears. Always report the raw mean alongside the normalised result, and never normalise scores from different scales or different questions together. With only two reviewers, a z-score can only tell you who scored higher, which you already knew.

Honest limits

This tool does plain arithmetic on the numbers you paste. It does not weight reviewers by expertise, does not adjust for reviewer bias beyond the group spread, and cannot handle missing scores or text grades such as accept or minor revision unless you convert them to numbers first. With very small panels the spread is unstable, so treat the output as a discussion aid for an editorial meeting, not as an automatic accept or reject rule.

Practical tips for editorial meetings

Bring both numbers to the meeting: the raw mean, which shows the overall level, and the normalised table, which shows the disagreement. When one reviewer is a strong outlier, discuss that review first, because the disagreement usually points to a real question about the paper rather than a scoring quirk. Never average scores across different questions, such as novelty and presentation, into one number without saying so, since the blend hides exactly what needs discussion. For borderline papers, ask whether a fourth reviewer would help more than another round of arithmetic; often the honest answer is yes. Record the scores and the decision together, so a later appeal can be answered from the record instead of from memory. Transparency here protects everyone: authors see a fair process, reviewers see their scores handled carefully, and editors avoid decisions that cannot be explained afterwards.

Embed this tool on your website — free

Copy the code below to add this working tool to your own page. A small credit link stays with it — that’s all we ask.