Abstract
In scientific machine learning (SciML), a key challenge is learning unknown, evolving physical processes and making predictions across spatio-temporal scales. For example, in real-world manufacturing such as additive manufacturing, users adjust known machine settings while unknown environmental parameters fluctuate simultaneously. To make reliable predictions, a model should capture long-range spatio-temporal interactions from data and also adapt to new and unknown environments; traditional ML models excel at the first task but often lack physical interpretability and struggle to generalize under varying conditions.
To tackle these challenges, we propose the Attention-based Spatio-temporal Neural Operator (ASNO), a novel architecture that combines separable attention mechanisms for spatial and temporal interactions and adapts to unseen physical parameters. Inspired by the backward differentiation formula, ASNO learns a transformer for temporal prediction and extrapolation and an attention-based neural operator for handling varying external loads, enhancing interpretability by isolating historical-state contributions and external forces. This enables the discovery of underlying physical laws and generalizability to unseen physical environments. Empirical results on SciML benchmarks demonstrate that ASNO outperforms existing models.
Keywords: spatio-temporal neural operator, uncertainty quantification, scientific machine learning, out-of-distribution generalization, interpretability, attention mechanism, PDE modeling