Zpracování velmi dlouhých sekvencí neuronovými sítěmi

• Název práce v češtině: Zpracování velmi dlouhých sekvencí neuronovými sítěmi
• Název v anglickém jazyce: Processing Very Long Sequences with Neural Networks
• Akademický rok vypsání: 2021/2022
• Typ práce: disertační práce
• Jazyk práce: čeština
• Ústav: Katedra teoretické informatiky a matematické logiky (32-KTIML)
• Vedoucí / školitel: prof. RNDr. Ing. Martin Holeňa, CSc.
• Řešitel: skrytý - zadáno a potvrzeno stud. odd.
• Datum přihlášení: 02.08.2022
• Datum zadání: 02.08.2022
• Datum potvrzení stud. oddělením: 03.10.2022
• Konzultanti: doc. RNDr. Ondřej Bojar, Ph.D.

Zásady pro vypracování

The key neural-learning-based approach to sequence processing tasks,
such as machine translation, text summarization, question answering,
or speech recognition, are now transformer-based architectures. Their
power relies on the attention mechanism, which allows to discover
long-term dependencies via pairwise matching, and their generality on
the query-key-value framework. However, this comes at a great
cost. The pairwise comparison framework induces a quadratic complexity
both in time and space, which significantly hinders the model
applicability. An inability to practically handle long sequences
either precludes the use of thransfomrers in some domains
(e.g. protein sequence modeling) or impedes their performance in tasks
where a longer input can be beneficial (text and image modeling and
generation).

The purpose of the proposed doctoral research is to study the
possibilities to make transformers more efficient in processing long
sequences. This can include, for example, methods based on kernelized
attention formulation, the performer architecture of linear
complexity, but also other paradigmas, such as structured state
spaces. Because the topic of the proposed research belongs to the
overlap between machine learning and natural language processing, it
will be co-supervised by supervisors from both areas.

Seznam odborné literatury

Dodá vedoucí práce.