NeurIPS 2026 · Evaluations and Datasets Track · Poster
1Tsinghua University 2Shanghai Artificial Intelligence Laboratory
*Equal contribution †Corresponding author
Paper Abstract
Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence.
We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains: robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate, under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings.
We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.
Benchmark Scope
MMTA collects tri-modal tasks with diverse cross-modal complementarity patterns across nine real-world domains.
Static Visual Evidence
Each example pairs visual context with an aligned time series and a concrete target output.
PixelRec
Given an item cover, title metadata, and daily interaction counts, forecast the next six daily values.
MIMIC
Use chest X-ray images, a radiology report, and vital signs from a lookback window to answer a discharge question.
S&P 500
Forecast future prices from a candlestick image, recent news summaries, and historical close values.
Terra
Forecast climate variables for a 1 degree by 1 degree grid cell using static geophysical layers and 100 days of multivariate history.
Temporal Visual Evidence
Video segments are paired with synchronized numeric traces and task-specific question formats.
AgiBot
Determine the anomaly category from a robot shelf-manipulation video, step sequence, and arm-position sensor traces.
CoVLA
Infer why the ego vehicle suddenly braked from a dashcam segment and aligned vehicle state signals.
CUHK-X HAR
Classify a human activity using a short video segment and wearable IMU streams from chest, arms, and legs.
Future Factories
Judge whether a rocket assembly cycle proceeds normally or contains a missing-part anomaly.
HoloAssist
Decide whether all fine-grained steps inside a coarse-grained egocentric action are correct.
Provenance
Source links for each subset.
Reference
If you use MMTA or TimeOmni-v in your research, please cite our paper.
@inproceedings{zhang2026mmta,
title={{MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision}},
author={Zhang, Ziyang and Li, Shenyi and Wang, Yilin and Cui, Ziyun and Zhou, Bowen and Wu, Wen and Zhang, Chao},
booktitle={Proc. NeurIPS},
address={Sydney},
year={2026}
}