Speaker
Description
Autoregressive transformers are increasingly applied outside the language domain that motivated them, but the tokenization step transfers poorly. Scientific data are already numerical, often partly discrete, and typically live in high-dimensional spaces where each token carries several features. The standard workaround, compressing feature vectors into a single token via a learned codebook, introduces reconstruction loss and a vocabulary that grows multiplicatively with resolution, inflating the embedding and unembedding layers until training becomes prohibitive.
We introduce SPADE (SPlit And Delay Embeddings), which embeds each feature of a token independently and staggers the resulting streams along the sequence with progressively increasing delays. The vocabulary then scales additively rather than multiplicatively, while intra-token correlations are recovered by the ordinary causal self-attention mechanism: each feature is predicted at its own sequence position, conditioned on the features already emitted for the same object. No auxiliary decoder or quantization stage is required.
We demonstrate SPADE on point-cloud calorimeter shower generation in the highly granular ILD electromagnetic calorimeter. SPADE is competitive with the state-of-the-art flow-matching model AllShowers on photon showers and substantially outperforms its VQ-VAE-based predecessor OmniJet-$\alpha_\text{C}$. Against a joint-vocabulary baseline at the finest granularity studied, SPADE uses 74× fewer parameters and converges 6.9× faster in GPU hours, while better reproducing observables sensitive to energy–position correlations.
Because the mechanism assumes only that tokens carry multiple features, discrete or continuous, it offers a route to LLM-style pretraining on high-dimensional sensor data across fundamental physics.