From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

Self-generated In-Context Learning (ICL) enables LLMs to synthesize their own few-shot demonstrations, often outperforming manual prompts. However, existing methods filter demonstrations by observation, assuming a rationale is high-quality if it leads to the correct output. We show such filtering suffers from $\mathit{\text{observational}}\mathit{\text{equivalence}}$, whereby both faithful causal rationales and spurious post-hoc rationales can yield same outputs and are statistically indistinguishable from observation alone. Inspired by independent causal mechanisms, we argue that faithful rationales imply $\mathit{\text{transfer}}\mathit{\text{invariance}}$, in which causal rationales should remain effective across logically similar samples. We instantiate this criterion as $\mathbf{\text{E}}$xecutable $\mathbf{\text{C}}$ausal $\mathbf{\text{I}}$ntervention for $\mathbf{\text{D}}$emonstrations $\mathbf{\text{D}}$istillation (ECI2D), which abstracts language rationales into executable programs and applies the causal interventions ($do$-operator) on logically similar samples, retaining only those demonstrations that imply transfer invariance. Experiments on math, logical and commonsense reasoning benchmarks demonstrate that ECI2D consistently outperforms observation-based ICL while retaining far fewer demonstrations, with gains exceeding 7 points as the test distribution shifts.