05 Fakultät Informatik, Elektrotechnik und Informationstechnik

Permanent URI for this collectionhttps://elib.uni-stuttgart.de/handle/11682/6

Browse

Filters

20241

Settings

Institute: search.filters.UBSInstitut.Institut für Künstliche IntelligenzAuthor: search.filters.author.Hagnberger, Jan

Search Results

Now showing 1 - 1 of 1
  • Thumbnail Image
    ItemOpen Access
    Linear transformers for solving parametric partial differential equations
    (2024) Hagnberger, Jan
    The simulation of physical phenomena relies on solving Partial Differential Equations (PDEs), and Machine Learning models have increasingly addressed this task in recent years. PDEs often involve parameters influencing their evolution, prompting the development of models that consider these parameters as additional input. These parameter-conditioned models aim to generalize across different PDE parameters, replacing the need for multiple models trained on specific ones. Transformer models have been achieving great success in Natural Language Processing (NLP), Speech Processing, and even in domains such as Computer Vision. Due to their ability to effectively model long-range dependencies in sequential data, their field of application is steadily increasing. Calculating attention via Scaled Dot-Product Attention in Vanilla Transformers is computationally expensive and scales quadratically with the input length. This leads to a bottleneck for very long sequences. To address this challenge, Linear Transformers have been introduced, substituting the Scaled Dot-Product Attention to achieve linear time and space complexity. Consequently, Linear Transformers have shown promising potential for processing very long sequences efficiently. We investigate two approaches of utilizing Linear Transformers for solving PDEs and their associated problems. Moreover, we conduct a comprehensive comparison between our proposed transformer-based models and state-of-the-art models for solving parametric PDEs. The evaluation criteria include accuracy for short and long rollouts, memory consumption, and inference times. The results demonstrate that our proposed models perform competitively with the current state-of-the-art models, providing an efficient solution for PDE solving.