Speaker
Description
Tokenization plays a central role in modern foundation models, especially those relying on next-token prediction, and has been extensively studied in domains such as language and vision. In high-energy physics, however, the adaptation of tokenization to continuous jet data from collider experiments remains at an early stage, with few studies on tokenizer performance, comparable evaluation metrics, or generalization across datasets. This is particularly pressing given that jet data are produced by different generators and in various data-taking conditions, raising the question of whether a tokenizer trained on one domain transfers to another.
When benchmarking standard k-means, product-quantized, and neural (autoencoder-based) tokenizers on four jet datasets (RODEM, JetClass, JetSet, Aspen) using codebook utilization, perplexity, and reconstruction error as comparable evaluation metrics, we encounter domain shift leading to performance degradation. We address this issue by proposing an invariant tokenization strategy, built on a jet-radius-normalized feature space rather than detector-specific raw kinematics. This approach can generalize across jets from different generators, experimental setups and data-tracking conditions.