01 — Overview
Large audio-language models (LALMs) struggle to parse paralinguistics (e.g., emotion, pace, pitch) in speech that combines intense emotion with complex acoustic environments. Strengthening this ability calls for training data that reflects such conditions. However, existing corpora rely heavily on studio-recorded speech, offering limited emotional and acoustic diversity. To address this gap, we introduce TVPara, a paralinguistic captioning dataset built from TV series. Speech in TV series often carries diverse emotional expression and co-occurs with scene-specific background sound, providing expressive emotion and complex acoustic environments at scale. Constructed via LALM pre-annotation and human correction, TVPara includes two tasks: Single-Utterance (~492h) and Multi-Utterance (~213h) Paralinguistic Captioning. Fine-tuning open-source LALMs on TVPara yields an average relative improvement of 31.26% in paralinguistic understanding under complex acoustic environments. The gains further generalize to an unseen TV series and to cleaner, cross-domain speech.
02 — Method
(a) Single-Utterance Paralinguistic Captioning describes the paralinguistic attributes of a single utterance, including emotion, pace, pitch, and volume. (b) Multi-Utterance Paralinguistic Captioning describes a sequence of adjacent utterances in order, with emphasis on how these attributes change over time (e.g., from irony to excitement).
Emotion word clouds of (a) TVPara and (b) VoxCeleb. VoxCeleb is dominated by neutral and calm states, whereas TVPara covers many expressive emotions (e.g., frustration, sarcasm, anger).
(a) Single-Utterance Annotation: each clip is pre-annotated by an LALM from the audio and a prompt, the caption is polished by an LLM, and it is then corrected by human annotators, who remove hallucinated content (e.g., emotions that the audio does not support) and add omitted content. The corrected data are used to fine-tune the pre-annotation model from Mp to Mf, which then pre-annotates the remaining audio. (b) Multi-Utterance Construction: a clip covering N adjacent utterances is cut from the source recording, and the corrected captions of the single-utterance clips within it are merged by an LLM into one caption in time order.
During inference, the audio and user prompt are processed by the LALM to generate a paralinguistic caption. For evaluation, an LLM extracts atomic attributes (emotion, pace, pitch, volume) from both the generated caption and the ground-truth label. An LLM evaluator then compares these aligned attributes, assigning similarity scores on a discrete scale from 0 to 4.
03 — Listen
Prior corpora are mostly recorded in clean acoustic conditions. In TVPara, expressive, intense emotion co-occurs with background sound such as music, sound effects, and scene transitions.
The same opening line, heard two ways: first in isolation, then inside its full three-utterance exchange.
Read alone, the line is simply ironic. Inside the conversation, the caption tracks how the delivery moves across four attributes (emotion, pace, pitch, and volume), while shift words mark where the tone turns from one utterance to the next.
"No kidding? Am I asking for ransom?"
The speaker maintains a moderate volume and pace, allowing the words to resonate, while the tone ascends towards the end of each sentence, adding a layer of irony to the delivery.
1) "No kidding? Am I asking for ransom?"
2) "Well, you always said she was crazy. Now I feel your pain. Anyway, I just thought you should know, huh? Take care."
3) "Okay, enough kidding around."
The speaker delivers with a moderate volume and pace, utilizing a rising tone for irony. This shifts to a high pitch and rapid tempo, maintaining a moderate volume while enhancing engagement. Finally, excitement peaks with a cheerful, high-pitched voice and a fast pace at a high volume, radiating enthusiasm. The overall delivery effectively blends irony, liveliness, and excitement.
The same captioning task, answered by the zero-shot model and by the model fine-tuned on TVPara, each compared with the ground truth.
The base model falls back on vague adjectives or a fixed Neutral / None / Medium template. After fine-tuning, the model pins down emotion, pitch, volume, and pace, and narrates how the tone shifts across sentences, landing much closer to the ground truth.
"Ellie, you can't call me! The cops are all over my ass because they think I helped you escape."
The speaker's voice carries a high pitch, brimming with a palpable urgency. Their tone is rapid, and their demeanor appears tense, effectively communicating a deep sense of desperation. The volume is moderate.
The speaker's voice is high-pitched and emotional, with a lot of stress on certain words. They speak quickly and with a lot of intensity.
The speaker's voice, characterized by its high pitch and rapid tempo, conveys a sense of urgency and desperation, delivered at a moderate volume.
1) "No kidding? Am I asking for ransom?"
2) "Well, you always said she was crazy. Now I feel your pain. Anyway, I just thought you should know, huh? Take care."
3) "Okay, enough kidding around."
The speaker delivers with a moderate volume and pace, utilizing a rising tone for irony. This shifts to a high pitch and rapid tempo, maintaining a moderate volume while enhancing engagement. Finally, excitement peaks with a cheerful, high-pitched voice and a fast pace at a high volume, radiating enthusiasm.
1. 'No kidding, am I asking for ransom? You always said she was crazy. Anyway, I just thought you should know, take care.' Emotion: Neutral; Stress: None; Intonation: Neutral; Speed: Medium. 2. 'Okay, enough kidding around.' Emotion: Neutral; Stress: None; Intonation: Neutral; Speed: Medium.
Initially, a sarcastic and mocking tone is used, characterized by a high-pitched, rapid voice and moderate volume. This shifts to a calm, steady voice with low volume and gentle upward intonation, conveying empathy. Finally, a cheerful and relaxed tone is adopted, maintaining a moderate pace and volume, with rising intonation at the end of sentences.
04 — Provenance
TVPara is built from 19 English-language TV series covering sitcoms, dramas, crime series, animation, and documentaries. Among popular series, we prioritized shows with richer scene diversity and more expressive character portrayals. The audio is clipped from the shows' official YouTube channels; only the paralinguistic annotations, source video links, and clip timestamps will be released.