PsychEval: A Multi-Session and Multi-Therapy Benchmark for High-Realism AI Psychological Counselor

To develop a reliable AI for psychological assessment, we introduce $\mathtt{\text{PsychEval}}$, a multi-session, multi-therapy, and highly realistic benchmark designed to address three key challenges: $\mathbf{\text{1)}}\mathbf{\text{Can}}\mathbf{\text{we}}\mathbf{\text{train}}\mathbf{\text{a}}\mathbf{\text{highly}}\mathbf{\text{realistic}}\mathbf{\text{AI}}\mathbf{\text{counselor?}}$ Realistic counseling is a longitudinal task requiring sustained memory and dynamic goal tracking. We propose a multi-session benchmark (spanning 6-10 sessions across three distinct stages) that demands critical capabilities such as memory continuity, adaptive reasoning, and longitudinal planning. The dataset is annotated with extensive professional skills, comprising over 677 meta-skills and 4577 atomic skills. $\mathbf{\text{2)}}\mathbf{\text{How}}\mathbf{\text{to}}\mathbf{\text{train}}\mathbf{\text{a}}\mathbf{\text{multi-therapy}}\mathbf{\text{AI}}\mathbf{\text{counselor?}}$ While existing models often focus on a single therapy, complex cases frequently require flexible strategies among various therapies. We construct a diverse dataset covering five therapeutic modalities (Psychodynamic, Behaviorism, CBT, Humanistic Existentialist, and Postmodernist) alongside an integrative therapy with a unified three-stage clinical framework across six core psychological topics. $\mathbf{\text{3)}}\mathbf{\text{How}}\mathbf{\text{to}}\mathbf{\text{systematically}}\mathbf{\text{evaluate}}\mathbf{\text{an}}\mathbf{\text{AI}}\mathbf{\text{counselor?}}$ We establish a holistic evaluation framework with 18 therapy-specific and therapy-shared metrics across Client-Level and Counselor-Level dimensions. To support this, we also construct over 2,000 diverse client profiles. Extensive experimental analysis fully validates the superior quality and clinical fidelity of our dataset. Crucially, $\mathtt{\text{PsychEval}}$ transcends static benchmarking to serve as a high-fidelity reinforcement learning environment that enables the self-evolutionary training of clinically responsible and adaptive AI counselors.