LogicSkills: A Structured Benchmark for Formal Reasoning in Large Language Models

Large language models perform well on many logical reasoning benchmarks, but it remains unclear which core logical skills they truly master. To address this, we introduce LogicSkills, a benchmark that isolates three fundamental logical skills: (i) $\mathit{\text{formal}}\mathit{\text{symbolization}}\text{unicode}x2014$translating premises into first-order logic; (ii) $\mathit{\text{countermodel}}\mathit{\text{construction}}\text{unicode}x2014$showing that an argument is logically invalid by constructing a finite countermodel; and (iii) $\mathit{\text{validity}}\mathit{\text{assessment}}\text{unicode}x2014$determining whether a conclusion follows from a set of premises. Items are drawn from the two-variable fragment of first-order logic without identity and are presented in both English and a Carrollian nonce-word language. All instances are solver-verified with Z3 for correctness and non-triviality. Across conventional instruction-tuned LLMs, performance is high on $\mathit{\text{validity}}\mathit{\text{assessment}}$ but substantially lower on $\mathit{\text{formal}}\mathit{\text{symbolization}}$ and $\mathit{\text{countermodel}}\mathit{\text{construction}}$, highlighting that high task-level accuracy can mask weaknesses in core logical skills. In contrast, recent reasoning-tuned models perform strongly across all three tasks, suggesting a more systematic logical skill profile.