MECOL2 LEX
Vocabulary Knowledge (MECOL2)
Test
| Model | Unseen Reader RMSE | Unseen Text RMSE | Unseen Text and Reader RMSE | Average RMSE | Unseen Reader MAE | Unseen Text MAE | Unseen Text and Reader MAE | Average MAE | Unseen Reader R² | Unseen Text R² | Unseen Text and Reader R² | Average R² |
|---|
| Majority Class / Chance | 12.47 ± 0.1 | 12.46 ± 0.0 | 12.39 ± 0.1 | 12.45 ± 0.1 | 10.41 ± 0.1 | 10.4 ± 0.0 | 10.32 ± 0.1 | 10.4 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 |
| Reading Speed | 28.98 ± 14.3 | 28.92 ± 14.3 | 28.97 ± 14.3 | 28.95 ± 14.3 | 27.18 ± 14.5 | 27.14 ± 14.5 | 27.14 ± 14.6 | 27.15 ± 14.5 | -9.86 ± 8.5 | -9.58 ± 8.3 | -10.06 ± 8.7 | -9.76 ± 8.5 |
| Text-Only Roberta | 12.47 ± 0.1 | 12.46 ± 0.0 | 12.39 ± 0.1 | 12.45 ± 0.1 | 10.41 ± 0.1 | 10.41 ± 0.0 | 10.32 ± 0.1 | 10.4 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 |
| Logistic Regression [meziere2023using] | 10.37 ± 0.1 | 10.31 ± 0.1 | 10.32 ± 0.2 | 10.34 ± 0.0 | 8.35 ± 0.1 | 8.31 ± 0.1 | 8.31 ± 0.2 | 8.32 ± 0.0 | 0.31 ± 0.0 | 0.31 ± 0.0 | 0.3 ± 0.0 | 0.31 ± 0.0 |
| SVM [hollenstein2023zuco] | 10.98 ± 0.1 | 10.94 ± 0.1 | 10.84 ± 0.1 | 10.94 ± 0.0 | 8.78 ± 0.0 | 8.76 ± 0.0 | 8.68 ± 0.1 | 8.76 ± 0.0 | 0.22 ± 0.0 | 0.23 ± 0.0 | 0.23 ± 0.0 | 0.23 ± 0.0 |
| Random Forest [makowski2024detection] | 10.2 ± 0.1 | 9.93 ± 0.0 | 10.24 ± 0.2 | 10.09 ± 0.0 | 8.2 ± 0.1 | 7.95 ± 0.0 | 8.26 ± 0.2 | 8.1 ± 0.0 | 0.33 ± 0.0 | 0.36 ± 0.0 | 0.31 ± 0.0 | 0.34 ± 0.0 |
| AhnRNN [ahn2020towards] | 12.48 ± 0.1 | 12.46 ± 0.0 | 12.4 ± 0.2 | 12.46 ± 0.1 | 10.42 ± 0.1 | 10.4 ± 0.0 | 10.32 ± 0.1 | 10.4 ± 0.1 | -0.0 ± 0.0 | 0.0 ± 0.0 | -0.01 ± 0.0 | -0.0 ± 0.0 |
| AhnCNN [ahn2020towards] | 12.28 ± 0.1 | 12.3 ± 0.0 | 12.2 ± 0.1 | 12.27 ± 0.0 | 10.25 ± 0.1 | 10.27 ± 0.0 | 10.16 ± 0.1 | 10.24 ± 0.0 | 0.03 ± 0.0 | 0.03 ± 0.0 | 0.03 ± 0.0 | 0.03 ± 0.0 |
| BEyeLSTM [reich_inferring_2022] | 12.65 ± 0.3 | 12.42 ± 0.1 | 13.42 ± 0.9 | 12.68 ± 0.3 | 10.38 ± 0.2 | 10.33 ± 0.1 | 10.41 ± 0.3 | 10.36 ± 0.1 | -0.03 ± 0.0 | 0.01 ± 0.0 | -0.19 ± 0.1 | -0.04 ± 0.0 |
| PLM-AS [Yang2023PLMASPL] | 12.48 ± 0.1 | 12.46 ± 0.0 | 12.4 ± 0.2 | 12.46 ± 0.1 | 10.42 ± 0.1 | 10.41 ± 0.0 | 10.32 ± 0.1 | 10.4 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.01 ± 0.0 | -0.0 ± 0.0 |
| PLM-AS-RM [haller2022eye] | 31.8 ± 0.3 | 31.81 ± 0.1 | 31.69 ± 0.4 | 31.79 ± 0.2 | 29.28 ± 0.3 | 29.28 ± 0.2 | 29.2 ± 0.4 | 29.27 ± 0.2 | -5.53 ± 0.1 | -5.52 ± 0.1 | -5.57 ± 0.2 | -5.52 ± 0.1 |
| RoBERTEye-W [Shubi2024Finegrained] | 11.11 ± 0.2 | 11.2 ± 0.1 | 11.16 ± 0.2 | 11.16 ± 0.1 | 9.15 ± 0.1 | 9.21 ± 0.1 | 9.2 ± 0.2 | 9.18 ± 0.0 | 0.2 ± 0.0 | 0.19 ± 0.0 | 0.19 ± 0.0 | 0.2 ± 0.0 |
| RoBERTEye-F [Shubi2024Finegrained] | 11.28 ± 0.3 | 11.18 ± 0.2 | 11.32 ± 0.3 | 11.25 ± 0.2 | 9.28 ± 0.2 | 9.19 ± 0.1 | 9.32 ± 0.3 | 9.25 ± 0.1 | 0.18 ± 0.0 | 0.19 ± 0.0 | 0.16 ± 0.0 | 0.18 ± 0.0 |
| MAG-Eye [Shubi2024Finegrained] | 12.47 ± 0.1 | 12.46 ± 0.0 | 12.39 ± 0.1 | 12.46 ± 0.1 | 10.42 ± 0.1 | 10.41 ± 0.0 | 10.32 ± 0.1 | 10.4 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 |
| PostFusion-Eye [Shubi2024Finegrained] | 13.44 ± 0.4 | 13.99 ± 0.8 | 15.26 ± 1.2 | 14.0 ± 0.4 | 10.63 ± 0.3 | 11.17 ± 0.7 | 11.92 ± 1.0 | 11.04 ± 0.4 | -0.17 ± 0.0 | -0.28 ± 0.1 | -0.54 ± 0.2 | -0.27 ± 0.1 |
Validation
| Model | Unseen Reader RMSE | Unseen Text RMSE | Unseen Text and Reader RMSE | Average RMSE | Unseen Reader MAE | Unseen Text MAE | Unseen Text and Reader MAE | Average MAE | Unseen Reader R² | Unseen Text R² | Unseen Text and Reader R² | Average R² |
|---|
| Majority Class / Chance | 12.49 ± 0.1 | 12.45 ± 0.1 | 12.39 ± 0.1 | 12.45 ± 0.1 | 10.42 ± 0.1 | 10.4 ± 0.1 | 10.31 ± 0.1 | 10.39 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 |
| Reading Speed | 28.76 ± 14.2 | 29.0 ± 14.4 | 28.61 ± 14.2 | 28.82 ± 14.2 | 26.97 ± 14.4 | 27.22 ± 14.6 | 26.83 ± 14.4 | 27.04 ± 14.5 | -9.09 ± 7.9 | -9.63 ± 8.3 | -9.07 ± 7.9 | -9.28 ± 8.0 |
| Text-Only Roberta | 12.49 ± 0.1 | 12.45 ± 0.1 | 12.38 ± 0.1 | 12.45 ± 0.1 | 10.42 ± 0.1 | 10.4 ± 0.1 | 10.31 ± 0.1 | 10.39 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 |
| Logistic Regression [meziere2023using] | 10.37 ± 0.2 | 10.27 ± 0.1 | 10.32 ± 0.1 | 10.32 ± 0.1 | 8.35 ± 0.1 | 8.29 ± 0.1 | 8.31 ± 0.1 | 8.32 ± 0.1 | 0.31 ± 0.0 | 0.32 ± 0.0 | 0.3 ± 0.0 | 0.31 ± 0.0 |
| SVM [hollenstein2023zuco] | 10.95 ± 0.1 | 10.9 ± 0.0 | 10.79 ± 0.1 | 10.9 ± 0.1 | 8.77 ± 0.1 | 8.74 ± 0.0 | 8.63 ± 0.1 | 8.73 ± 0.0 | 0.23 ± 0.0 | 0.23 ± 0.0 | 0.24 ± 0.0 | 0.23 ± 0.0 |
| Random Forest [makowski2024detection] | 10.19 ± 0.1 | 9.71 ± 0.0 | 10.26 ± 0.1 | 10.01 ± 0.1 | 8.16 ± 0.1 | 7.79 ± 0.0 | 8.32 ± 0.2 | 8.04 ± 0.0 | 0.33 ± 0.0 | 0.39 ± 0.0 | 0.31 ± 0.0 | 0.35 ± 0.0 |
| AhnRNN [ahn2020towards] | 12.48 ± 0.1 | 12.45 ± 0.1 | 12.37 ± 0.1 | 12.45 ± 0.1 | 10.41 ± 0.1 | 10.41 ± 0.1 | 10.3 ± 0.1 | 10.39 ± 0.1 | 0.0 ± 0.0 | -0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
| AhnCNN [ahn2020towards] | 12.33 ± 0.1 | 12.27 ± 0.1 | 12.21 ± 0.1 | 12.28 ± 0.1 | 10.28 ± 0.1 | 10.25 ± 0.1 | 10.17 ± 0.2 | 10.25 ± 0.1 | 0.02 ± 0.0 | 0.03 ± 0.0 | 0.02 ± 0.0 | 0.03 ± 0.0 |
| BEyeLSTM [reich_inferring_2022] | 12.64 ± 0.4 | 12.33 ± 0.1 | 12.55 ± 0.4 | 12.51 ± 0.3 | 10.4 ± 0.2 | 10.3 ± 0.1 | 10.29 ± 0.2 | 10.34 ± 0.1 | -0.03 ± 0.0 | 0.02 ± 0.0 | -0.03 ± 0.1 | -0.01 ± 0.0 |
| PLM-AS [Yang2023PLMASPL] | 12.48 ± 0.1 | 12.45 ± 0.1 | 12.37 ± 0.1 | 12.45 ± 0.1 | 10.41 ± 0.1 | 10.41 ± 0.1 | 10.3 ± 0.1 | 10.39 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 |
| PLM-AS-RM [haller2022eye] | 31.81 ± 0.3 | 31.78 ± 0.0 | 31.7 ± 0.4 | 31.78 ± 0.2 | 29.27 ± 0.3 | 29.25 ± 0.1 | 29.2 ± 0.4 | 29.25 ± 0.2 | -5.5 ± 0.2 | -5.52 ± 0.1 | -5.58 ± 0.2 | -5.52 ± 0.1 |
| RoBERTEye-W [Shubi2024Finegrained] | 11.05 ± 0.1 | 11.11 ± 0.1 | 11.16 ± 0.1 | 11.1 ± 0.0 | 9.1 ± 0.0 | 9.11 ± 0.1 | 9.22 ± 0.2 | 9.13 ± 0.0 | 0.22 ± 0.0 | 0.2 ± 0.0 | 0.18 ± 0.0 | 0.21 ± 0.0 |
| RoBERTEye-F [Shubi2024Finegrained] | 11.17 ± 0.1 | 10.98 ± 0.0 | 11.03 ± 0.1 | 11.07 ± 0.0 | 9.2 ± 0.1 | 9.04 ± 0.0 | 9.11 ± 0.1 | 9.12 ± 0.0 | 0.2 ± 0.0 | 0.22 ± 0.0 | 0.2 ± 0.0 | 0.21 ± 0.0 |
| MAG-Eye [Shubi2024Finegrained] | 12.49 ± 0.1 | 12.45 ± 0.1 | 12.38 ± 0.1 | 12.45 ± 0.1 | 10.42 ± 0.1 | 10.4 ± 0.1 | 10.31 ± 0.1 | 10.39 ± 0.1 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 | -0.0 ± 0.0 |
| PostFusion-Eye [Shubi2024Finegrained] | 13.01 ± 0.4 | 13.72 ± 0.9 | 13.15 ± 0.3 | 13.35 ± 0.5 | 10.37 ± 0.3 | 11.0 ± 0.8 | 10.2 ± 0.1 | 10.59 ± 0.4 | -0.09 ± 0.1 | -0.23 ± 0.2 | -0.13 ± 0.1 | -0.16 ± 0.1 |