What Would Count as Evidence of Syntax in LLMs? Center-Embedding Diagnostics and Attractor Interference
What would count as evidence that large language models use syntactic structure during generation rather than producing output that merely looks well-formed? The current debate contrasts stochastic parrots with emergent human-like syntax; I argue this dichotomy does not exhaust the possibilities, distinguishing human-theoretic syntax, alien syntax, and fragmentary versions of each. I then present a behavioral test built on doubly center-embedded sentences:
[The mayorₙ₁ [the reportersₙ₂ [the investigatorₙ₃ questionsᵥ₃]interview/sᵥ₂] is late.]
whose rarity in corpora limits memorization and whose structure puts hierarchical agreement rules and nearest-noun heuristics into direct competition: at the middle verb (V₂), the nearest noun (N₃) is not the controller (N₂). Using 6,000 minimal pairs from 2,000 sentence families, I evaluate eleven open-weight autoregressive base models (124M–72B parameters) on agreement at three sites per sentence (N₁, N₂, N₃). Larger models show substantial accuracy drops at V₂ under attractor interference, and mismatch at N₁, which is neither the controller nor the nearest noun, interferes at least as strongly as N₃ mismatch in several models. Isolated agreement success is weak evidence of syntax use; interference profiles discriminate more finely, and structured failure may indicate syntactic rules unlike, though functionally analogous to, human-theoretic syntax.
Computation, Cognitive Science, and Cognition
Executive Conference Room