AcademicEval: Live Long-Context LLM Benchmark

Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose $\text{textsc}AcademicEval$, a live benchmark for evaluating LLMs over long-context generation tasks. $\text{textsc}AcademicEval$ adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, $\mathit{\text{i.e.}}$, $\text{textsc}Title$, $\text{textsc}Abstract$, $\text{textsc}Introduction$, and $\text{textsc}RelatedWork$, which cover a wide range of abstraction levels and require no manual labeling. Moreover, $\text{textsc}AcademicEval$ integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, $\text{textsc}AcademicEval$ features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on $\text{textsc}AcademicEval$, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs’ long-context modeling capabilities. Code is available at https://github.com/ulab-uiuc/AcademicEval