publications

2024

1. arXiv

   xLSTM: Extended Long Short-Term Memory

   Maximilian Beck*, Korbinian Pöppel* , Markus Spanring , and 6 more authors

   In Arxiv , 2024

   Abs arXiv Bib PDF

   In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.

   @inproceedings{beck2024xlstm,
     title = {xLSTM: Extended Long Short-Term Memory},
     author = {Beck*, Maximilian and P{\"o}ppel*, Korbinian and Spanring, Markus and Auer, Andreas and Prudnikova, Oleksandra and Kopp, Michael and Klambauer, G{\"u}nter and Brandstetter, Johannes and Hochreiter, Sepp},
     booktitle = {Arxiv },
     url = {https://arxiv.org/abs/2405.04517},
     year = {2024},
   }

2023

1. ICLR23

   Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by Aggregation

   Marius-Constantin Dinu , Markus Holzleitner , Maximilian Beck, and 7 more authors

   In The Eleventh International Conference on Learning Representations , 2023

   Abs arXiv Bib PDF Code Poster

   We study the problem of choosing algorithm hyper-parameters in unsupervised domain adaptation, i.e., with labeled data in a source domain and unlabeled data in a target domain, drawn from a different input distribution. We follow the strategy to compute several models using different hyper-parameters, and, to subsequently compute a linear aggregation of the models. While several heuristics exist that follow this strategy, methods are still missing that rely on thorough theories for bounding the target error. In this turn, we propose a method that extends weighted least squares to vector-valued functions, e.g., deep neural networks. We show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. We also perform a large scale empirical comparative study on several datasets, including text, images, electroencephalogram, body sensor signals and signals from mobile phones. Our method outperforms deep embedded validation (DEV) and importance weighted validation (IWV) on all datasets, setting a new state-of-the-art performance for solving parameter choice issues in unsupervised domain adaptation with theoretical error guarantees. We further study several competitive heuristics, all outperforming IWV and DEV on at least five datasets. However, our method outperforms each heuristic on at least five of seven datasets.

   @inproceedings{dinu2023aggregationmethod,
     title = {Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by Aggregation},
     author = {Dinu, Marius-Constantin and Holzleitner, Markus and Beck, Maximilian and Nguyen, Hoan Duc and Huber, Andrea and Eghbal-zadeh, Hamid and Moser, Bernhard A. and Pereverzyev, Sergei and Hochreiter, Sepp and Zellinger, Werner},
     booktitle = {The Eleventh International Conference on Learning Representations },
     year = {2023},
     url = {https://openreview.net/forum?id=M95oDwJXayG},
   }

2022

1. 

   Few-Shot Learning by Dimensionality Reduction in Gradient Space

   Martin Gauch , Maximilian Beck, Thomas Adler , and 10 more authors

   In The Conference on Lifelong Learning Agents , 2022

   Abs arXiv Bib PDF Blog Code Poster

   We introduce SubGD, a novel few-shot learning method which is based on the recent finding that stochastic gradient descent updates tend to live in a low-dimensional parameter subspace. In experimental and theoretical analyses, we show that models confined to a suitable predefined subspace generalize well for few-shot learning. A suitable subspace fulfills three criteria across the given tasks: it (a) allows to reduce the training error by gradient flow, (b) leads to models that generalize well, and (c) can be identified by stochastic gradient descent. SubGD identifies these subspaces from an eigendecomposition of the auto-correlation matrix of update directions across different tasks. Demonstrably, we can identify low-dimensional suitable subspaces for few-shot learning of dynamical systems, which have varying properties described by one or few parameters of the analytical system description. Such systems are ubiquitous among real-world applications in science and engineering. We experimentally corroborate the advantages of SubGD on three distinct dynamical systems problem settings, significantly outperforming popular few-shot learning methods both in terms of sample efficiency and performance.

   @inproceedings{gauch2022subgd,
     title = {Few-Shot Learning by Dimensionality Reduction in Gradient Space},
     author = {Gauch, Martin and Beck, Maximilian and Adler, Thomas and Kotsur, Dmytro and Fiel, Stefan and Eghbal-zadeh, Hamid and Brandstetter, Johannes and Kofler, Johannes and Holzleitner, Markus and Zellinger, Werner and Klotz, Daniel and Hochreiter, Sepp and Lehner, Sebastian},
     booktitle = {The Conference on Lifelong Learning Agents},
     year = {2022},
     editor = {Chandar, Sarath and Pascanu, Razvan and Precup, Doina},
     url = {https://proceedings.mlr.press/v199/gauch22a.html},
   }