How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
2026 IEEE Spoken Language Technology Workshop (SLT) , 2026
Abstract
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.
BibTeX
@inproceedings{lu2026auditory,
title = {How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation},
author = {Lu, Ke-Han and Fu, Szu-Wei and Yang, Chao-Han Huck and Chen, Zhehuai and Huang, Sung-Feng and Yang, Chih-Kai and Lin, Yi-Cheng and Hsiao, Chi-Yuan and Ren, Wenze and Hu, En-Pei and Huang, Yu-Han and Cheng, An-Yu and Chiang, Cheng-Han and Tsao, Yu and Wang, Yu-Chiang Frank and Lee, Hung-yi},
booktitle = {2026 IEEE Spoken Language Technology Workshop (SLT)},
year = {2026},
}