The encoder-decoder architecture was the original transformer architecture that was proposed by Vaswani et al (2017). An example is T5. These models are now largely legacy architectures except for some specific applications like translation as they have been supplanted by encoder models for understanding (i.e., using embeddings) and increasingly decoder-only models for understanding and generation.
It is also worth noting that encoder models are sometimes still preferred over decoders for embeddings because of their transparency and their bidirectional attention allows each token to attend to the full context in both directions which produces richer embeddings. However, the largest decoder models can match or exceed encoder performance even on understanding tasks although they are not as transparent.
One area where encoder-decoder bi-directional and cross directional attention has been found to outperform decoder only approaches is scoring free text. Later in this book, we show how you can fine-tune a T5 encoder-decoder model to score free text based on the autoregressive multi-trait scoring model. In this demo we fine-tuned T5 base to score that Hewlett Foundation’s Automated Essay Scoring (AES) data set available on Kaggle.
Next page
Five criteria for choosing a language model in AI psychometrics
Last page
Decoder architectures explained: GPT-2 reconstruction
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
