What is a Transformer Model?
Generative AI Fundamentals Part 3: https://youtu.be/WJVkwFBe2rY?si=Jgh31TpzxNejwbOm If you have looked up the full-form of ChatGPT, you know that GPT stands for Generative Pre-Trained Transformer. What is a Transformer Model? A transformer model is a neural network that learns context and thus meaning by tracking relationships in sequential data like the words in this sentence. It was first introduced in a 2017 research paper by Google titled “Attention is All You Need”. At the time, it was showcased as a translation model but its applications in text generation have become exceedingly popular. How does a Transformer Model work? Transformer models apply an evolving set of mathematical techniques, called attention or self-attention, to detect subtle ways even distant data elements in a series influence and depend on each other. The power of the transformer architecture lies in its ability to learn the relevance and context of all of the words in a sentence. Not just for each word to its neighbor, but to every other word in a sentence. The model can then apply attention weights to those relationships and learn the relevance of each word to each other word, no matter where they are in the input. Transformer Models provide a step-change in performance over RNNs or Recurrent Neural Networks by parallelizing the process of text generation which was previously conducted sequentially, and was therefore more limiting. This paved the way for the Generative AI revolution we are experiencing today. Stay tuned for the next video where we will dive deeper into pre-training and fine-tuning LLMs.
