CopCo RCS

Reading Comprehension Skill (CopCo)

Test

Model	Unseen Reader RMSE	Unseen Text RMSE	Unseen Text and Reader RMSE	Average RMSE	Unseen Reader MAE	Unseen Text MAE	Unseen Text and Reader MAE	Average MAE	Unseen Reader R²	Unseen Text R²	Unseen Text and Reader R²	Average R²
Majority Class / Chance	2.66 ± 0.1	2.67 ± 0.0	2.56 ± 0.0	2.66 ± 0.1	2.19 ± 0.1	2.2 ± 0.0	2.13 ± 0.1	2.18 ± 0.0	-0.05 ± 0.0	-0.05 ± 0.0	-0.09 ± 0.0	-0.03 ± 0.0
Reading Speed	2.69 ± 0.1	2.68 ± 0.1	2.6 ± 0.0	2.68 ± 0.1	2.21 ± 0.1	2.24 ± 0.0	2.18 ± 0.1	2.22 ± 0.0	-0.07 ± 0.0	-0.06 ± 0.0	-0.12 ± 0.0	-0.05 ± 0.0
Text-Only Roberta	2.65 ± 0.1	2.63 ± 0.0	2.61 ± 0.1	2.64 ± 0.0	2.22 ± 0.1	2.19 ± 0.1	2.23 ± 0.1	2.21 ± 0.1	-0.04 ± 0.0	-0.01 ± 0.0	-0.13 ± 0.1	-0.02 ± 0.0
Logistic Regression [meziere2023using]	2.74 ± 0.1	2.75 ± 0.0	2.7 ± 0.0	2.74 ± 0.1	2.26 ± 0.1	2.3 ± 0.0	2.3 ± 0.1	2.28 ± 0.0	-0.11 ± 0.0	-0.12 ± 0.0	-0.21 ± 0.0	-0.1 ± 0.0
SVM [hollenstein2023zuco]	2.89 ± 0.1	2.63 ± 0.1	2.74 ± 0.1	2.76 ± 0.1	2.28 ± 0.0	2.1 ± 0.1	2.17 ± 0.1	2.18 ± 0.0	-0.28 ± 0.2	-0.03 ± 0.1	-0.27 ± 0.1	-0.12 ± 0.1
Random Forest [makowski2024detection]	2.76 ± 0.1	2.22 ± 0.1	2.54 ± 0.1	2.51 ± 0.1	2.2 ± 0.1	1.75 ± 0.0	2.08 ± 0.1	1.99 ± 0.1	-0.14 ± 0.1	0.27 ± 0.1	-0.07 ± 0.1	0.08 ± 0.0
AhnRNN [ahn2020towards]	2.64 ± 0.1	2.62 ± 0.0	2.58 ± 0.1	2.63 ± 0.0	2.21 ± 0.1	2.19 ± 0.0	2.19 ± 0.1	2.2 ± 0.0	-0.03 ± 0.0	-0.0 ± 0.0	-0.1 ± 0.1	-0.01 ± 0.0
AhnCNN [ahn2020towards]	2.64 ± 0.1	2.61 ± 0.0	2.58 ± 0.1	2.63 ± 0.0	2.22 ± 0.1	2.19 ± 0.0	2.19 ± 0.1	2.2 ± 0.0	-0.03 ± 0.0	-0.0 ± 0.0	-0.11 ± 0.1	-0.01 ± 0.0
BEyeLSTM [reich_inferring_2022]	2.64 ± 0.1	2.62 ± 0.0	2.59 ± 0.1	2.63 ± 0.0	2.22 ± 0.1	2.19 ± 0.0	2.19 ± 0.1	2.21 ± 0.0	-0.04 ± 0.0	-0.01 ± 0.0	-0.11 ± 0.1	-0.01 ± 0.0
PLM-AS [Yang2023PLMASPL]	2.66 ± 0.1	2.62 ± 0.0	2.6 ± 0.1	2.64 ± 0.0	2.21 ± 0.1	2.19 ± 0.1	2.19 ± 0.1	2.2 ± 0.0	-0.05 ± 0.0	-0.01 ± 0.0	-0.12 ± 0.1	-0.02 ± 0.0
PLM-AS-RM [haller2022eye]	2.69 ± 0.1	2.65 ± 0.0	2.6 ± 0.1	2.67 ± 0.0	2.23 ± 0.1	2.23 ± 0.0	2.19 ± 0.1	2.23 ± 0.1	-0.07 ± 0.0	-0.03 ± 0.0	-0.12 ± 0.0	-0.04 ± 0.0
RoBERTEye-W [Shubi2024Finegrained]	2.67 ± 0.1	2.63 ± 0.0	2.6 ± 0.1	2.65 ± 0.0	2.24 ± 0.1	2.2 ± 0.0	2.2 ± 0.1	2.22 ± 0.0	-0.06 ± 0.0	-0.02 ± 0.0	-0.12 ± 0.0	-0.03 ± 0.0
RoBERTEye-F [Shubi2024Finegrained]	2.67 ± 0.1	2.64 ± 0.1	2.65 ± 0.1	2.66 ± 0.1	2.24 ± 0.1	2.2 ± 0.1	2.25 ± 0.1	2.23 ± 0.1	-0.06 ± 0.0	-0.02 ± 0.0	-0.16 ± 0.1	-0.04 ± 0.0
MAG-Eye [Shubi2024Finegrained]	2.65 ± 0.1	2.63 ± 0.0	2.59 ± 0.1	2.64 ± 0.0	2.22 ± 0.1	2.2 ± 0.1	2.21 ± 0.1	2.21 ± 0.0	-0.04 ± 0.0	-0.01 ± 0.0	-0.11 ± 0.1	-0.02 ± 0.0
PostFusion-Eye [Shubi2024Finegrained]	2.9 ± 0.1	2.67 ± 0.1	2.77 ± 0.1	2.79 ± 0.1	2.4 ± 0.1	2.23 ± 0.1	2.37 ± 0.1	2.33 ± 0.1	-0.25 ± 0.1	-0.05 ± 0.0	-0.27 ± 0.0	-0.14 ± 0.0

Validation

Model	Unseen Reader RMSE	Unseen Text RMSE	Unseen Text and Reader RMSE	Average RMSE	Unseen Reader MAE	Unseen Text MAE	Unseen Text and Reader MAE	Average MAE	Unseen Reader R²	Unseen Text R²	Unseen Text and Reader R²	Average R²
Majority Class / Chance	2.82 ± 0.1	2.48 ± 0.0	2.58 ± 0.1	2.65 ± 0.0	2.36 ± 0.1	1.98 ± 0.0	2.11 ± 0.1	2.17 ± 0.1	-0.06 ± 0.0	-0.08 ± 0.1	-0.1 ± 0.0	-0.03 ± 0.0
Reading Speed	2.82 ± 0.1	2.44 ± 0.0	2.62 ± 0.1	2.66 ± 0.0	2.38 ± 0.1	1.95 ± 0.1	2.17 ± 0.1	2.19 ± 0.1	-0.07 ± 0.0	-0.04 ± 0.0	-0.14 ± 0.1	-0.03 ± 0.0
Text-Only Roberta	2.77 ± 0.1	2.46 ± 0.1	2.55 ± 0.1	2.62 ± 0.1	2.38 ± 0.1	1.99 ± 0.0	2.17 ± 0.1	2.2 ± 0.0	-0.02 ± 0.0	-0.05 ± 0.0	-0.07 ± 0.0	-0.0 ± 0.0
Logistic Regression [meziere2023using]	3.02 ± 0.1	2.49 ± 0.0	2.71 ± 0.1	2.78 ± 0.1	2.58 ± 0.1	1.98 ± 0.0	2.3 ± 0.1	2.3 ± 0.0	-0.23 ± 0.1	-0.08 ± 0.1	-0.22 ± 0.1	-0.13 ± 0.1
SVM [hollenstein2023zuco]	2.97 ± 0.1	2.31 ± 0.1	2.63 ± 0.1	2.68 ± 0.1	2.47 ± 0.1	1.71 ± 0.0	2.1 ± 0.1	2.12 ± 0.1	-0.19 ± 0.1	0.05 ± 0.1	-0.15 ± 0.1	-0.05 ± 0.0
Random Forest [makowski2024detection]	2.86 ± 0.1	1.82 ± 0.1	2.78 ± 0.2	2.52 ± 0.1	2.34 ± 0.1	1.38 ± 0.1	2.25 ± 0.2	1.97 ± 0.1	-0.1 ± 0.0	0.41 ± 0.1	-0.29 ± 0.1	0.07 ± 0.0
AhnRNN [ahn2020towards]	2.76 ± 0.1	2.46 ± 0.1	2.55 ± 0.1	2.62 ± 0.1	2.37 ± 0.1	2.01 ± 0.1	2.16 ± 0.1	2.2 ± 0.0	-0.01 ± 0.0	-0.05 ± 0.0	-0.08 ± 0.0	-0.0 ± 0.0
AhnCNN [ahn2020towards]	2.76 ± 0.1	2.46 ± 0.1	2.55 ± 0.1	2.62 ± 0.1	2.38 ± 0.1	2.0 ± 0.1	2.16 ± 0.1	2.2 ± 0.0	-0.01 ± 0.0	-0.05 ± 0.0	-0.07 ± 0.0	0.0 ± 0.0
BEyeLSTM [reich_inferring_2022]	2.76 ± 0.1	2.45 ± 0.1	2.56 ± 0.1	2.62 ± 0.1	2.38 ± 0.1	2.0 ± 0.1	2.17 ± 0.1	2.2 ± 0.1	-0.01 ± 0.0	-0.05 ± 0.0	-0.08 ± 0.1	-0.0 ± 0.0
PLM-AS [Yang2023PLMASPL]	2.76 ± 0.1	2.46 ± 0.1	2.54 ± 0.1	2.62 ± 0.1	2.37 ± 0.1	2.0 ± 0.1	2.13 ± 0.1	2.19 ± 0.0	-0.02 ± 0.0	-0.05 ± 0.0	-0.07 ± 0.0	-0.0 ± 0.0
PLM-AS-RM [haller2022eye]	2.79 ± 0.1	2.47 ± 0.1	2.63 ± 0.1	2.65 ± 0.0	2.39 ± 0.1	2.0 ± 0.1	2.22 ± 0.1	2.22 ± 0.1	-0.04 ± 0.0	-0.06 ± 0.0	-0.15 ± 0.1	-0.03 ± 0.0
RoBERTEye-W [Shubi2024Finegrained]	2.77 ± 0.1	2.44 ± 0.0	2.55 ± 0.1	2.61 ± 0.1	2.37 ± 0.1	1.98 ± 0.0	2.15 ± 0.1	2.18 ± 0.0	-0.02 ± 0.0	-0.04 ± 0.0	-0.07 ± 0.0	0.0 ± 0.0
RoBERTEye-F [Shubi2024Finegrained]	2.77 ± 0.1	2.45 ± 0.0	2.54 ± 0.1	2.62 ± 0.1	2.4 ± 0.1	1.98 ± 0.0	2.15 ± 0.1	2.2 ± 0.0	-0.03 ± 0.0	-0.05 ± 0.0	-0.07 ± 0.1	-0.0 ± 0.0
MAG-Eye [Shubi2024Finegrained]	2.76 ± 0.1	2.46 ± 0.1	2.55 ± 0.1	2.62 ± 0.1	2.37 ± 0.1	1.99 ± 0.1	2.18 ± 0.1	2.2 ± 0.0	-0.01 ± 0.0	-0.05 ± 0.0	-0.08 ± 0.0	0.0 ± 0.0
PostFusion-Eye [Shubi2024Finegrained]	2.86 ± 0.2	2.52 ± 0.1	2.66 ± 0.1	2.71 ± 0.1	2.49 ± 0.1	2.06 ± 0.1	2.34 ± 0.1	2.3 ± 0.1	-0.08 ± 0.0	-0.11 ± 0.1	-0.17 ± 0.1	-0.07 ± 0.0