CatBench
Catalan Linguistic Benchmarks
11/3/2025, 2:49:38 PM
5 models
Success rate by model
Percentage of correct answers per model
openai/gpt-5
60.38%
Total
53
Correct
32
Accuracy
60.38%
| Id | Ok | Out |
|---|---|---|
| subj_present_001 | ✓ | cregui |
| subj_present_002 | ✓ | sapigueu |
| subj_present_003 | ✓ | vinguin |
anthropic/claude-4.5-sonnet
75.47%
Total
53
Correct
40
Accuracy
75.47%
| Id | Ok | Out |
|---|---|---|
| subj_present_001 | ✓ | cregui |
| subj_present_002 | ✓ | sapigueu |
| subj_present_003 | ✓ | vinguin |
amazon/nova-premier-v1
69.81%
Total
53
Correct
37
Accuracy
69.81%
| Id | Ok | Out |
|---|---|---|
| subj_present_001 | ✓ | cregui |
| subj_present_002 | ✗ | sàpigueu |
| subj_present_003 | ✓ | vinguin |
google/gemini-2.5-pro
66.04%
Total
53
Correct
35
Accuracy
66.04%
| Id | Ok | Out |
|---|---|---|
| subj_present_001 | ✓ | cregui |
| subj_present_002 | ✓ | sapigueu |
| subj_present_003 | ✓ | vinguin |
x-ai/grok-4
26.42%
Total
53
Correct
14
Accuracy
26.42%
| Id | Ok | Out |
|---|---|---|
| subj_present_001 | ✓ | cregui |
| subj_present_002 | ✗ | sàpigueu |
| subj_present_003 | ✓ | vinguin |