MMF-TSP: A Multimodal Fusion Network for Time Series Prediction Using Textual and Numerical Data

Abstract

This paper introduces an innovative multimodal fusion network architecture, namely MMF-TSP (Multimodal Fusion for Time Series Prediction). The architecture consists of four key components: textual data encoding, textual feature fusion, numerical data encoding, and multimodal feature fusion. Advanced techniques such as BERT for text processing, Temporal Convolutional Networks (TCNs) for handling numerical sequences, a global attention mechanism, and jump connections are employed to facilitate effective feature extraction and integration.The experimental results, using the electricity demand sequence dataset from California, USA, demonstrate the superiority of the proposed model. Compared with the BERT + LSTM model, our MMF-TSP model reduces RMSE by 6.32% (from 1012 to 948), improves R by 2.47% (from 0.851 to 0.872), and improves R - Squared by 4.97% (from 0.851 to 0.760), and reduces MAPE by 6.67% (from 0.045 to 0.042). On additional datasets including traffic flow data from New York City and weather forecasting coupled with energy consumption data from the UK, the MMF-TSP model also shows advantages. For example, in the New York City traffic flow data, compared to BERT+LSTM, it reduces RMSE by 4.8% (from 1040 to 990), increases R by 1.3% (from 0.821 to 0.834), and improves R - Squared by 2.1% (from 0.674 to 0.695), and decreases MAPE by 6.25% (from 0.048 to 0.045). This architecture thus presents a promising new tool and platform for the deep analysis and broad application of multimodal data.