{"id":156,"date":"2025-02-28T12:18:50","date_gmt":"2025-02-28T12:18:50","guid":{"rendered":"https:\/\/ai.munishkaushik.com\/?p=156"},"modified":"2025-02-28T12:18:52","modified_gmt":"2025-02-28T12:18:52","slug":"one-hot-encoding-dot-product-and-matrices-multiplication-the-basics-of-transformers","status":"publish","type":"post","link":"https:\/\/ai.munishkaushik.com\/index.php\/2025\/02\/28\/one-hot-encoding-dot-product-and-matrices-multiplication-the-basics-of-transformers\/","title":{"rendered":"One-Hot Encoding, Dot Product and Matrices multiplication: The Basics of Transformers"},"content":{"rendered":"\n

Introduction<\/strong><\/p>\n\n\n\n

In the world of natural language processing (NLP), everything begins with words. However, computers don\u2019t understand words directly – they need numbers. Our first task is to convert words into numerical representations so that we can perform mathematical operations on them. This is especially important when building systems like voice-activated assistants, where we need to transform sequences of sounds into sequences of words.<\/p>\n\n\n\n

To achieve this, we start by defining a Vocabulary<\/strong>, which is the set of symbols (or words) we\u2019ll be working with. For simplicity, let\u2019s assume we\u2019re working with English, which has tens of thousands of words, plus additional terms specific to technology or other domains. This could result in a vocabulary size of nearly a hundred thousand words.<\/p>\n\n\n\n

One straightforward way to convert words into numbers is to assign each word a unique integer. For example, in a small vocabulary consisting of the words [basics<\/em>, of<\/em>, transformers<\/em>]. we could assign basics<\/em> = 1, transformers<\/em> = 2, and of<\/em> = 3.<\/p>\n\n\n\n

Then, a sentence like “Basics of transformers” would be represented as the sequence [1, 3, 2].<\/p>\n\n\n\n

This is called label encoding. While this method is valid, it has some challenges primarily being giving false magnitude interpretation as these numbers have a natural ordering (e.g., 0 < 1 < 2).<\/p>\n\n\n\n

One hot encoding: Efficient encoding<\/strong><\/p>\n\n\n\n

There\u2019s a more efficient way to represent words numerically, known as one-hot encoding<\/strong>. In one-hot encoding, each word is represented as a vector (a one-dimensional array) with a length equal to the size of the vocabulary. The vector is mostly zeros, except for a single “1” at the position corresponding to the word\u2019s index. For example, in our small vocabulary:<\/p>\n\n\n\n