{"id":895,"date":"2023-12-03T08:10:34","date_gmt":"2023-12-03T08:10:34","guid":{"rendered":"https:\/\/tbekk.com\/devstream\/?p=895"},"modified":"2023-12-03T08:10:34","modified_gmt":"2023-12-03T08:10:34","slug":"how-large-language-models-work","status":"publish","type":"post","link":"https:\/\/tbekk.com\/devstream\/2023\/12\/03\/how-large-language-models-work\/","title":{"rendered":"How Large Language Models work"},"content":{"rendered":"\n<h2 class=\"wp-block-heading has-normal-font-size\" id=\"531b\"><em>From zero to ChatGPT<\/em><\/h2>\n\n\n\n<hr class=\"wp-block-separator has-text-color has-light-gray-color has-alpha-channel-opacity has-light-gray-background-color has-background is-style-wide\"\/>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><em>Link:<\/em><\/strong> <a href=\"https:\/\/medium.com\/data-science-at-microsoft\/how-large-language-models-work-91c362f5b78f\"><em>Data Science at Microsoft<\/em><\/a><\/li>\n\n\n\n<li><em><strong>Author:<\/strong><\/em> <a href=\"https:\/\/medium.com\/@andreas.stoeffelbauer?source=post_page-----91c362f5b78f--------------------------------\"><em>Andreas St\u00f6ffelbauer<\/em><\/a><\/li>\n\n\n\n<li><em><strong>Publication date:<\/strong><\/em> <em>Oct 24, 2023<\/em><\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-text-color has-light-gray-color has-alpha-channel-opacity has-light-gray-background-color has-background is-style-wide\"\/>\n\n\n\n<p id=\"b0c6\">Thanks to Large Language Models (or LLMs for short), Artificial Intelligence has now caught the attention of pretty much everyone. ChatGPT, possibly the most famous LLM, has immediately skyrocketed in popularity due to the fact that natural language is such a, well, natural interface that has made the recent breakthroughs in Artificial Intelligence accessible to everyone. Nevertheless, how LLMs work is still less commonly understood, unless you are a Data Scientist or in another AI-related role. In this article, I will try to change that.<\/p>\n\n\n\n<p id=\"3e79\">Admittedly, that\u2019s an ambitious goal. After all, the powerful LLMs we have today are a culmination of decades of research in AI. Unfortunately, most articles covering them are one of two kinds: They are either very technical and assume a lot of prior knowledge, or they are so trivial that you don\u2019t end up knowing more than before.<\/p>\n\n\n\n<p id=\"9581\">This article is meant to strike a balance between these two approaches. Or actually let me rephrase that, it\u2019s meant to take you from zero all the way through to how LLMs are trained and why they work so impressively well. We\u2019ll do this by picking up just all the relevant pieces along the way.<\/p>\n\n\n\n<p id=\"d6d1\">This is not going to be a deep dive into all the nitty-gritty details, so we\u2019ll rely on intuition here rather than on math, and on visuals as much as possible. But as you\u2019ll see, while certainly being a very complex topic in the details, the main mechanisms underlying LLMs are very intuitive, and that alone will get us very far here.<\/p>\n\n\n\n<p id=\"ce83\">This article should also help you get more out of using LLMs like ChatGPT. In fact, we will learn some of the neat tricks that you can apply to increase the chances of a useful response. Or as Andrei Karparthy, a well-known AI researcher and engineer, recently and pointedly said: \u201cEnglish is the hottest new programming language.\u201d<\/p>\n\n\n\n<p id=\"a15a\">But first, let\u2019s try to understand where LLMs fit in the world of Artificial Intelligence.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*a8LZZFns5L2VNZIUl6UasA.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The field of Artificial Intelligence in layers.<\/figcaption><\/figure>\n\n\n\n<p id=\"b6ab\">The field of AI is often visualized in layers:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Artificial Intelligence<\/strong>&nbsp;(AI) is very a broad term, but generally it deals with intelligent machines.<\/li>\n\n\n\n<li><strong>Machine Learning<\/strong>&nbsp;(ML) is a subfield of AI that specifically focuses on pattern recognition in data. As you can imagine, once you recoginze a pattern, you can apply that pattern to new observations. That\u2019s the essence of the idea, but we will get to that in just a bit.<\/li>\n\n\n\n<li><strong>Deep Learning<\/strong>&nbsp;is the field within ML that is focused on unstructured data, which includes text and images. It relies on artificial neural networks, a method that is (loosely) inspired by the human brain.<\/li>\n\n\n\n<li><strong>Large Language Models<\/strong>&nbsp;(LLMs) deal with text specifically, and that will be the focus of this article.<\/li>\n<\/ul>\n\n\n\n<p id=\"a55e\">As we go, we\u2019ll pick up the relevant pieces from each of those layers. We\u2019ll skip only the most outer one, Artificial Intelligence (as it is too general anyway) and head straight into what is Machine Learning.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*6Qjq2mrNOUSMMFOURVewDw.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Machine Learning. Level: Beginner.<\/figcaption><\/figure>\n\n\n\n<p id=\"9eea\">The goal of Machine Learning is to discover patterns in data. Or more specifically, a pattern that describes the relationship between an input and an outcome. This is best explained using an example.<\/p>\n\n\n\n<p id=\"dc71\">Let\u2019s say we would like to distinguish between two of my favorite genres of music:&nbsp;<a href=\"https:\/\/www.youtube.com\/watch?v=QXQQAsIhHMw\" rel=\"noreferrer noopener\" target=\"_blank\">reggaeton<\/a>&nbsp;and&nbsp;<a href=\"https:\/\/www.youtube.com\/watch?v=T4kAYIC5Xzo\" rel=\"noreferrer noopener\" target=\"_blank\">R&amp;B<\/a>. If you are not familiar with those genres, here\u2019s a very quick intro that will help us understand the task. Reggaeton is a Latin urban genre known for its lively beats and danceable rhythms, while R&amp;B (Rhythm and Blues) is a genre rooted in African-American musical traditions, characterized by soulful vocals and a mix of upbeat and slower-paced songs.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*l2ttjCcoTpbr-QmxXNZWBw.gif\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Machine Learning in practice. Predicting music genre is an example of a classification problem.<\/figcaption><\/figure>\n\n\n\n<p id=\"e5d7\">Suppose we have 20 songs. We know each song\u2019s tempo and energy, two metrics that can be simply measured or computed for any song. In addition, we\u2019ve labeled them with a genre, either reggaeton or R&amp;B. When we visualize the data, we can see that high energy, high tempo songs are primarily reggaeton while lower tempo, lower energy songs are mostly R&amp;B, which makes sense.<\/p>\n\n\n\n<p id=\"c0a7\">However, we want to avoid having to label the genre by hand all the time because it\u2019s time consuming and not scalable. Instead, we can learn the relationship between the song metrics (tempo, energy) and genre and then make predictions using only the readily available metrics.<\/p>\n\n\n\n<p id=\"ce64\">In Machine Learning terms, we say that this is a classification problem, because the outcome variable (the genre) can only take on one of a fixed set of classes\/labels \u2014 here reggaeton and R&amp;B. This is in contrast to a regression problem, where the outcome is a continuous value (e.g., a temperature or a distance).<\/p>\n\n\n\n<p id=\"30f8\">We can now \u201ctrain\u201d a Machine Learning model (or \u201cclassifier\u201d) using our labeled dataset, i.e., using a set of songs for which we do know the genre. Visually speaking, what the training of the model does here is that it finds the line that best separates the two classes.<\/p>\n\n\n\n<p id=\"2be4\">How is that useful? Well, now that we know this line, for any new song we can make a prediction about whether it\u2019s a reggaeton or an R&amp;B song, depending on which side of the line the song falls on. All we need is the tempo and energy, which we assumed is more easily available. That is much simpler and scalable than have a human assign the genre for each and every song.<\/p>\n\n\n\n<p id=\"5f0e\">Additionally, as you can imagine, the further away from the line, the more certain we can be about being correct. Therefore, we can often also make a statement on how confident we are that a prediction is correct based on the distance from the line. For example, for our new low-energy, low-tempo song we might be 98 percent certain that this is an R&amp;B song, with a two percent likelihood that it\u2019s actually reggaeton.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*2THD_Jr2Z0ANagf4WWMsgQ.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">In reality, things are often much more complex.<\/figcaption><\/figure>\n\n\n\n<p id=\"833b\">But of course, reality is often more complex than that.<\/p>\n\n\n\n<p id=\"8fac\">The best boundary to separate the classes may not be linear. In other words, the relationship between the inputs and the outcome can be more complex. It may be curved as in the image above, or even many times more complex than that.<\/p>\n\n\n\n<p id=\"3b90\">Reality is typically more complex in another way too. Rather than only two inputs as in our example, we often have tens, hundreds, or even thousands of input variables. In addition, we often have more than two classes. And all classes can depend on all these inputs through an incredibly complex, non-linear relationship.<\/p>\n\n\n\n<p id=\"6773\">Even with our example, we know that in reality there are more than two genres, and we need many more metrics other than tempo and energy. The relationship among them is probably not so simple either.<\/p>\n\n\n\n<p id=\"3fea\">What I mainly want you to take away is this: The more complex the relationship between input and output, the more complex and powerful is the Machine Learning model we need in order to learn that relationship. Usually, the complexity increases with the number of inputs and the number of classes.<\/p>\n\n\n\n<p id=\"f992\">In addition to that, we also need more data as well. You will see why this is important in just a bit.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*S7S78GODPPL2mMBTGgYFGg.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Image classification example.<\/figcaption><\/figure>\n\n\n\n<p id=\"6138\">Let\u2019s move on to a slightly different problem now, but one for which we will simply try to apply our mental model from before. In our new problem we have as input an image, for example, this image of a cute cat in a bag (because examples with cats are always the best).<\/p>\n\n\n\n<p id=\"be97\">As for our outcome, let\u2019s say this time that we have three possible labels: tiger, cat, and fox. If you need some motivation for this task, let\u2019s say we may want to protect a herd of sheep and sound an alarm if we see a tiger but not if we see a cat or a fox.<\/p>\n\n\n\n<p id=\"fa4c\">We already know this is again a classification task because the output can only take on one of a few fixed classes. Therefore, just like before, we could simply use some available labeled data (i.e., images with assigned class labels) and train a Machine Learning model.<\/p>\n\n\n\n<p id=\"c157\">However, it\u2019s not quite obvious as to exactly how we would process a visual input, as a computer can process only numeric inputs. Our song metrics energy and tempo were numeric, of course. And fortunately, images are just numeric inputs too as they consist of pixels. They have a height, a width, and three channels (red, green, and blue). So in theory, we could directly feed the pixels into a Machine Learning model (ignore for now that there is a spatial element here, which we haven\u2019t dealt with before).<\/p>\n\n\n\n<p id=\"02a5\">However, now we are facing two problems. First, even a small, low-quality 224&#215;224 image consists of more than 150,000 pixels (224x224x3). Remember, we were speaking about a maximum of hundreds of input variables (rarely more than a thousand), but now we suddenly have at least 150,000.<\/p>\n\n\n\n<p id=\"1bcf\">Second, if you think about the relationship between the raw pixels and the class label, it\u2019s incredibly complex, at least from an ML perspective that is. Our human brains have the amazing ability to generally distinguish among tigers, foxes, and cats quite easily. However, if you saw the 150,000 pixels one by one, you would have no idea what the image contains. But this is exactly how a Machine Learning model sees them, so it needs to learn from scratch the mapping or relationship between those raw pixels and the image label, which is not a trivial task.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*FMGxDQ2NouQYuwqKOlTHqA.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Sentiment classification example.<\/figcaption><\/figure>\n\n\n\n<p id=\"6d01\">Let\u2019s consider another type of input-output relationship that is extremely complex \u2014 the relationship between a sentence and its sentiment. By sentiment we typically mean the emotion that a sentence conveys, here positive or negative.<\/p>\n\n\n\n<p id=\"7114\">Let\u2019s formalize the problem setup again: As the input here we have a sequence of words, i.e., a sentence, and the sentiment is our outcome variable. As before, this is a classification task, this time with two possible labels, i.e., positive or negative.<\/p>\n\n\n\n<p id=\"b79f\">As with the images example discussed earlier, as humans we understand this relationship naturally, but can we teach a Machine Learning model to do the same?<\/p>\n\n\n\n<p id=\"0815\">Before answering that, it\u2019s again not obvious at the start how words can be turned into numeric inputs for a Machine Learning model. In fact, this is a level or two more complicated than what we saw with images, which as we saw are essentially already numeric. This is not the case with words. We won\u2019t go into details here, but what you need to know is that every word can be turned into a word embedding.<\/p>\n\n\n\n<p id=\"037e\">In short, a word embedding represents the word\u2019s semantic and syntactic meaning, often within a specific context. These embeddings can be obtained as part of training the Machine Learning model, or by means of a separate training procedure. Usually, word embeddings consist of between tens and thousands of variables, per word that is.<\/p>\n\n\n\n<p id=\"7955\">To summarize, what to take away from here is that we can take a sentence and turn it into a sequence of numeric inputs, i.e., the word embeddings, which contain semantic and syntactic meaning. This can then be fed into a Machine Learning model. (Again, if you\u2019re observant you may notice that there is a new sequential dimension that is unlike our examples from before, but we will ignore this one here too.)<\/p>\n\n\n\n<p id=\"ddde\">Great, but now we face the same challenges as with the visual input. As you can imagine, with a long sentence (or paragraph or even a whole document), we can quickly reach a very large number of inputs because of the large size of the word embeddings.<\/p>\n\n\n\n<p id=\"fc98\">The second problem is the relationship between language and its sentiment, which is complex \u2014 very complex. Just think of a sentence like \u201cThat was a great fall\u201d and all the ways it can be interpreted (not to mention sarcastically).<\/p>\n\n\n\n<p id=\"9a1d\">What we need is an extremely powerful Machine Learning model, and lots of data. That\u2019s where Deep Learning comes in.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*ASocEkv7llQwid-SBRohlQ.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Deep Learning. Level: Advanced.<\/figcaption><\/figure>\n\n\n\n<p id=\"4a95\">We already took a major step toward understanding LLMs by going through the basics of Machine Learning and the motivations behind the use of more powerful models, and now we\u2019ll take another big step by introducing Deep Learning.<\/p>\n\n\n\n<p id=\"13fd\">We talked about the fact that if the relationship between an input and output is very complex, as well as if the number of input or output variables is large (and both are the case for our image and language examples from before), we need more flexible, powerful models. A linear model or anything close to that will simply fail to solve these kinds of visual or sentiment classification tasks.<\/p>\n\n\n\n<p id=\"d8ad\">This is where neural networks come in.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*NJktkZVJbpG0nvGuU2AI5Q.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Neural Networks are the most powerful Machine Learning models we have today.<\/figcaption><\/figure>\n\n\n\n<p id=\"c5bf\">Neural networks are powerful Machine Learning models that allow arbitrarily complex relationships to be modeled. They are the engine that enables learning such complex relationships at massive scale.<\/p>\n\n\n\n<p id=\"d19c\">In fact, neural networks are loosely inspired by the brain, although the actual similarities are debatable. Their basic architecture is relatively simple. They consist of a sequence of layers of connected \u201cneurons\u201d that an input signal passes through in order to predict the outcome variable. You can think of them as multiple layers of linear regression stacked together, with the addition of non-linearities in between, which allows the neural network to model highly non-linear relationships.<\/p>\n\n\n\n<p id=\"6a0b\">Neural networks are often many layers deep (hence the name Deep Learning), which means they can be extremely large. ChatGPT, for example, is based on a neural network consisting of 176 billion neurons, which is more than the approximate 100 billion neurons in a human brain.<\/p>\n\n\n\n<p id=\"6b74\">So, from here on we will assume a neural network as our Machine Learning model, and take into account that we have also learned how to process images and text.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*PMxQpoVjON8AUhd9cA9T4A.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Large Language Models. Level: Expert.<\/figcaption><\/figure>\n\n\n\n<p id=\"671c\">Finally, we can start talking about Large Language Models, and this is where things get really interesting. If you have made it this far, you should have all the knowledge to also understand LLMs.<\/p>\n\n\n\n<p id=\"3157\">What\u2019s a good way to start? Probably by explaining what&nbsp;<em>Large Language Model<\/em>&nbsp;actually means. We already know what large means, in this case it simply refers to the number of neurons, also called parameters, in the neural network. There is no clear number for what constitutes a Large Language Model, but you may want to consider everything above 1 billion neurons as large.<\/p>\n\n\n\n<p id=\"061c\">With that established, what\u2019s a \u201clanguage model\u201d? Let\u2019s discuss this next \u2014 and just know that in a bit, we\u2019ll also get to learn what the GPT in ChatGPT stands for. But one step at a time.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*M3n8sKITIwtcSXHfGd2dOA.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Language modeling is learning to predict the next word.<\/figcaption><\/figure>\n\n\n\n<p id=\"1907\">Let\u2019s take the following idea and frame it as a Machine Learning problem: What is the next word in a given sequence of words, i.e., in a sentence or paragraph? In other words, we simply want to learn how to predict the next word at any time. From earlier in this article we\u2019ve learned everything we need to frame that as a Machine Learning problem. In fact, the task is not unlike the sentiment classification we saw earlier.<\/p>\n\n\n\n<p id=\"11b3\">As in that example, the input to the neural network is a sequence of words, but now, the outcome is simply the next word. Again, this is just a classification task. The only difference is that instead of only two or a few classes, we now have as many classes as there are words \u2014 let\u2019s say around 50,000. This is what language modeling is about \u2014 learning to predict the next word.<\/p>\n\n\n\n<p id=\"498d\">Okay, so that\u2019s orders of magnitude more complex than the binary sentiment classification, as you can imagine. But now that we also know about neural networks and their sheer power, the only response to that concern is really \u201cwhy not?\u201d<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p id=\"3eb2\"><strong>Quick disclaimer<\/strong>: Of course, we are simplifying many things here (as is done throughout the article). In reality things are a little more complex, but that shouldn\u2019t hold us back from understanding the main mechanics, which is why we simplify and leave out some of the details.<\/p>\n<\/blockquote>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*dKWfZr1R8R55eIPvCV-PzQ.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Massive amounts of traning data can be created relatively easily.<\/figcaption><\/figure>\n\n\n\n<p id=\"3af8\">We know the task, and now we need data to train the neural network. It\u2019s actually not difficult to create a lot of data for our \u201cnext word prediction\u201d task. There\u2019s an abundance of text on the internet, in books, in research papers, and more. And we can easily create a massive dataset from all of this. We don\u2019t even need to label the data, because the next word itself is the label, that\u2019s why this is also called&nbsp;<em>self-supervised learning<\/em>.<\/p>\n\n\n\n<p id=\"eee3\">The image above shows how this is done. Just a single sequence can be turned into multiple sequences for training. And we have lots of such sequences. Importantly, we do this for many short and long sequences (some up to thousands of words) so that in every context we learn what the next word should be.<\/p>\n\n\n\n<p id=\"bdc7\">To summarize, all we are doing here is to train a neural network (the LLM) to predict the next word in a given sequence of words, no matter if that sequence is long or short, in German or in English or in any other language, whether it\u2019s a tweet or a mathematical formula, a poem or a snippet of code. All of those are sequences that we will find in the training data.<\/p>\n\n\n\n<p id=\"4e62\">If we have a large enough neural network as well as enough data, the LLM becomes really good at predicting the next word. Will it be perfect? No, of course not, since there are often multiple words that can follow a sequence. But it will become good at selecting one of the appropriate words that are syntactically and semantically appropriate.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*faLf-OAINgRAyMyCLyZLvg.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">We can perform natural language generation by predicting one word at a time.<\/figcaption><\/figure>\n\n\n\n<p id=\"86c1\">Now that we can predict one word, we can feed the extended sequence back into the LLM and predict another word, and so on. In other words, using our trained LLM, we can now generate text, not just a single word. This is why LLMs are an example of what we call Generative AI. We have just taught the LLM to speak, so to say, one word at a time.<\/p>\n\n\n\n<p id=\"3f8b\">There\u2019s one more detail to this that I think is important to understand. We don\u2019t necessarily always have to predict the most likely word. We can instead sample from, say, the five most likely words at a given time. As a result, we may get some more creativity from the LLM. Some LLMs actually allow you to choose how deterministic or creative you want the output to be. This is also why in ChatGPT, which uses such a sampling strategy, you typically do not get the same answer when you regenerate a response.<\/p>\n\n\n\n<p id=\"1fab\">Speaking of ChatGPT, you could ask yourself now why it\u2019s not called ChatLLM. As it turns out, language modeling is not the end of the story \u2014 in fact it\u2019s just the beginning. So what does the GPT in ChatGPT stand for?<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*KBtpzU-6fYjyhyYmj4APIw.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">GPT = Generative Pre-trained Transformer.<\/figcaption><\/figure>\n\n\n\n<p id=\"c9b5\">We have actually just learned what the G stands for, namely \u201cgenerative\u201d \u2014 meaning that it was trained on a language generation pretext, which we have discussed. But what about the P and the T?<\/p>\n\n\n\n<p id=\"53cc\">We\u2019ll gloss over the T here, which stands for \u201ctransformer\u201d \u2014 not the one from the movies (sorry), but one that\u2019s simply the type of neural network architecture that is being used. This shouldn\u2019t really bother us here, but if you are curious and you only want to know its main strength, it\u2019s that the transformer architecture works so well because it can focus its attention on the parts of the input sequence that are most relevant at any time. You could argue that this is similar to how humans work. We, too, need to focus our attention on what\u2019s most relevant to the task and ignore the rest.<\/p>\n\n\n\n<p id=\"c877\">Now to the P, which stands for \u201cpre-training\u201d. We discuss next why we suddenly start speaking about pre-training and not just training any longer.<\/p>\n\n\n\n<p id=\"0122\">The reason is that Large Language Models like ChatGPT are actually trained in phases.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*aNf8qJHyrd8zGE199E387g.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Phases of LLM training: (1) Pre-Training, (2) Instruction Fine-Tuning, (3) Reinforcement from Human Feedback (RLHF).<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"e985\">Pre-training<\/h2>\n\n\n\n<p id=\"d302\">The first stage is pre-training, which is exactly what we\u2019ve gone through just now. This stage requires massive amounts of data to learn to predict the next word. In that phase, the model learns not only to master the grammar and syntax of language, but it also acquires a great deal of knowledge about the world, and even some other emerging abilities that we will speak about later.<\/p>\n\n\n\n<p id=\"35bc\">But now I have a couple of questions for you: First, what might be the problem with this kind of pre-training? Well, there are certainly a few, but the one I am trying to point to here has to do with what the LLM has really learned.<\/p>\n\n\n\n<p id=\"0e49\">Namely, it has learned mainly to ramble on about a topic. It may even be doing an incredibly good job, but what it doesn\u2019t do is respond well to the kind of inputs you would generally want to give an AI, such as a question or an instruction. The problem is that this model has not learned to be, and so is not behaving as, an assistant.<\/p>\n\n\n\n<p id=\"bf49\">For example, if you ask a pre-trained LLM \u201cWhat is your fist name?\u201d it may respond with \u201cWhat is your last name?\u201d simply because this is the kind of data it has seen during pre-training, as in many empty forms, for example. It\u2019s only trying to complete the input sequence.<\/p>\n\n\n\n<p id=\"99f5\">It doesn\u2019t do well with following instructions simply because this kind of language structure, i.e., instruction followed by a response, is not very commonly seen in the training data. Maybe Quora or StackOverflow would be the closest representation of this sort of structure.<\/p>\n\n\n\n<p id=\"2afc\">At this stage, we say that the LLM is not aligned with human intentions. Alignment is an important topic for LLMs, and we\u2019ll learn how we can fix this to a large extent, because as it turns out, those pre-trained LLMs are actually quite steerable. So even though initially they don\u2019t respond well to instructions, they can be taught to do so.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"6c61\">Instruction fine-tuning and RLHF<\/h2>\n\n\n\n<p id=\"f847\">This is where instruction tuning comes in. We take the pre-trained LLM with its current abilities and do essentially what we did before \u2014 i.e., learn to predict one word at a time \u2014 but now we do this using only high-quality instruction and response pairs as our training data.<\/p>\n\n\n\n<p id=\"b36c\">That way, the model un-learns to simply be a text completer and learns to become a helpful assistant that follows instructions and responds in a way that is aligned with the user\u2019s intention. The size of this instruction dataset is typically a lot smaller than the pre-training set. This is because the high-quality instruction-response pairs are much more expensive to create as they are typically sourced from humans. This is very different from the inexpensive self-supervised labels we used in pre-training. This is why this stage is also called&nbsp;<em>supervised instruction fine-tuning<\/em>.<\/p>\n\n\n\n<p id=\"6bd1\">There is also a third stage that some LLMs like ChatGPT go through, which is reinforcement learning from human feedback (RLHF). We won\u2019t go into details here, but the purpose is similar to instruction fine-tuning. RLHF also helps alignment and ensures that the LLM\u2019s output reflects human values and preferences. There is some early research that indicates that this stage is critical for reaching or surpassing human-level performance. In fact, combining the fields of reinforcement learning and language modeling is being shown to be especially promising and is likely to lead to some massive improvements over the LLMs we currently have.<\/p>\n\n\n\n<p id=\"e710\">So now let\u2019s test our understanding on some common use cases.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*Aer25cNHgj0pDicfQ9o0sA.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Examples to test our understanding of LLMs.<\/figcaption><\/figure>\n\n\n\n<p id=\"baba\">First,&nbsp;<strong>why can an LLM perform summarization<\/strong>&nbsp;of a longer piece of text? (If you didn\u2019t already know, it does a really great job. Just paste in a document and ask it to summarize it.)<\/p>\n\n\n\n<p id=\"a6eb\">To understand why, we need to think about the training data. As it so happens, people often make summarizations \u2014 on the internet, in research papers, books, and more. As a result, an LLM trained on that data learns how to do that too. It learns to attend to the main points and compress them into a short text.<\/p>\n\n\n\n<p id=\"4cf6\">Note that when a summary is generated, the full text is part of the input sequence of the LLM. This is similar to, say, a research paper that has a conclusion while the full text appears just before.<\/p>\n\n\n\n<p id=\"513d\">As a result, that skill has probably been learned during pre-training already, although surely instruction fine-tuning helped improve that skill even further. We can assume that this phase included some summarization examples too.<\/p>\n\n\n\n<p id=\"8d0c\">Second,&nbsp;<strong>why can a LLM answer common knowledge questions<\/strong>?<\/p>\n\n\n\n<p id=\"6b8a\">As mentioned, the ability to act as an assistant and respond appropriately is due to instruction fine-tuning and RLHF. But all (or most of) the knowledge to answer questions itself was already acquired during pre-training.<\/p>\n\n\n\n<p id=\"3b3f\">Of course, that now raises another big question:&nbsp;<strong>What if the LLM doesn\u2019t know the answer<\/strong>? Unfortunately, it may just make one up in that case. To understand why, we need to think about the training data again, and the training objective.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*FEXnWmjZ5lkIVjw8xLC60A.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">LLMs suffer from hallucinations, but this can be mitigated by providing additional context.<\/figcaption><\/figure>\n\n\n\n<p id=\"9343\">You might have heard about the term \u201challucination\u201d in the context of LLMs, which refers to the phenomenon of LLMs making up facts when they shouldn\u2019t.<\/p>\n\n\n\n<p id=\"5d9d\">Why does that happen? Well, the LLM learns only to generate text, not factually true text. Nothing in its training gives the model any indicator of the truth or reliability of any of the training data. However, that is not even the main issue here, it\u2019s that generally text out there on the internet and in books sounds confident, so the LLM of course learns to sound that way, too, even if it is wrong. In this way, an LLM has little indication of uncertainty.<\/p>\n\n\n\n<p id=\"1cab\">That being said, this is an active area of research, from which we can expect that LLMs will be less prone to hallucinations over time. For example, during instruction tuning we can try and teach the LLM to abstain from hallucinating to some extent, but only time will tell whether we can fully solve this issue.<\/p>\n\n\n\n<p id=\"e4d0\">You may be surprised that we can actually try to solve this problem here together right now. We have the knowledge we need to figure out a solution that at least partially helps and is already used widely today.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*PnhT-x8jxSXpta0FCMxmgA.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Bing chat is an example of a search-based LLM workflow.<\/figcaption><\/figure>\n\n\n\n<p id=\"4380\">Suppose that you ask the LLM the following question: Who is the current president of Colombia? There\u2019s a good chance an LLM may respond with the wrong name. This could be because of two reasons:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The first is what we have already brought up: The LLM may just hallucinate and simply respond with a wrong or even fake name.<\/li>\n\n\n\n<li>The second one I will mention only in passing: LLMs are trained only on data up to a certain cut-off date, and that can be as early as last year. Because of that, the LLM cannot even know the current president with certainty, because things could have changed since the data was created.<\/li>\n<\/ul>\n\n\n\n<p id=\"81d9\">So how can we solve both these problems? The answer lies in providing the model some relevant context. The rationale here is that everything that\u2019s in the LLM\u2019s input sequence is readily available for it to process, while any implicit knowledge it has acquired in pre-training is more difficult and precarious for it to retrieve.<\/p>\n\n\n\n<p id=\"af03\">Suppose we were to include the Wikipedia article on Colombia\u2019s political history as context for the LLM. In that case it would much more likely to answer correctly because it can simply extract the name from the context (given that it is up to date and includes the current president of course).<\/p>\n\n\n\n<p id=\"7c4f\">In the image above you can see what a typical prompt for an LLM with additional context may look like. (By the way, prompt is just another name for the instructions we give to an LLM, i.e., the instructions form the input sequence.)<\/p>\n\n\n\n<p id=\"c8f2\">This process is called grounding the LLM in the context, or in the real world if you like, rather than allowing it to generate freely.<\/p>\n\n\n\n<p id=\"fb32\">And that\u2019s exactly how Bing Chat and other search-based LLMs work. They first extract relevant context from the web using a search engine and then pass all that information to the LLM, alongside the user\u2019s initial question. See the illustration above for a visual of how this is accomplished.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*V7m6-AFcychP0RxYvp9kzA.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Back to the AI Magic. Level: Unicorn.<\/figcaption><\/figure>\n\n\n\n<p id=\"f4d8\">We\u2019ve now reached a point where you pretty much understand the main mechanisms of the state-of-the art LLMs (as of the second half of 2023, anyway).<\/p>\n\n\n\n<p id=\"fdfa\">You may be thinking \u201cthis is actually not that magical\u201d because all that is happening is the predicting of words, one at a time. It\u2019s pure statistics, after all. Or is it?<\/p>\n\n\n\n<p id=\"973d\">Let\u2019s back up a bit. The magical part of all this is how remarkably well it works. In fact, everyone, even the researchers at OpenAI, were surprised at how far this sort of language modeling can go. One of the key drivers in the last few years has simply been the massive scaling up of neural networks and data sets, which has caused performance to increase along with them. For example, GPT-4, reportedly a model with more than one trillion parameters in total, can pass the bar exam or AP Biology with a score in the top 10 percent of test takers.<\/p>\n\n\n\n<p id=\"1622\">Surprisingly, those large LLMs even show certain&nbsp;<strong>emerging abilities<\/strong>, i.e., abilities to solve tasks and to do things that they were not explicitly trained to do.<\/p>\n\n\n\n<p id=\"aa35\">In this last part of the article, we\u2019ll discuss some of these emerging abilities and I\u2019ll show you some tricks for how you can use them to solve problems.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*BEV_ok87UZ5B9j5tIN7eUw.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">LLMs can solve entirely new tasks in a zero-shot manner.<\/figcaption><\/figure>\n\n\n\n<p id=\"2ffb\">A ubiquitous emerging ability is, just as the name itself suggests, that LLMs can perform entirely new tasks that they haven\u2019t encountered in training, which is called zero-shot. All it takes is some instructions on how to solve the task.<\/p>\n\n\n\n<p id=\"42cd\">To illustrate this ability with a silly example, you can ask an LLM to translate a sentence from German to English while responding only with words that start with \u201cf\u201d.<\/p>\n\n\n\n<p id=\"7ebd\">For instance, when asked to translate a sentence using only words that start with \u201cf\u201d, an LLM translated \u201cDie Katze schl\u00e4ft gerne in der Box\u201d (which is German and literally means \u201cThe cat likes to sleep in the box\u201d) with \u201cFeline friend finds fluffy fortress\u201d, which is a pretty cool translation, I think.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*w6Nhrd0Pf5OA03AIYS7Fag.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">LLMs, just like humans, can benefit from providing them with examples or demonstrations.<\/figcaption><\/figure>\n\n\n\n<p id=\"fe7f\">For more complex tasks, you may quickly realize that zero-shot prompting often requires very detailed instructions, and even then, performance is often far from perfect.<\/p>\n\n\n\n<p id=\"7519\">To make another connection to human intelligence, if someone tells you to perform a new task, you would probably ask for some examples or demonstrations of how the task is performed. LLMs can benefit from the same.<\/p>\n\n\n\n<p id=\"6123\">As an example, let\u2019s say you want a model to translate different currency amounts into a common format. You could describe what you want in details or just give a brief instruction and some example demonstrations. The image above shows a sample task.<\/p>\n\n\n\n<p id=\"c7a8\">Using this prompt, the model should do well on the last example, which is \u201cSteak: 24.99 USD\u201d, and respond with $24.99.<\/p>\n\n\n\n<p id=\"4045\">Note how we simply left out the solution to the last example. Remember that an LLM is still a text-completer at heart, so keep a consistent structure. You should almost force the model to respond with just what you want, as we did in the example above.<\/p>\n\n\n\n<p id=\"f9a2\">To summarize, a general tip is to provide some examples if the LLM is struggling with the task in a zero-shot manner. You will find that often helps the LLM understand the task, making the performance typically better and more reliable.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*tY162Kdt8Mgi4EOvdmFHkQ.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Chain-of-thought provides LLMs a working memory, which can improve their performance substantially, especially on more complex tasks.<\/figcaption><\/figure>\n\n\n\n<p id=\"2cb1\">Another interesting ability of LLMs is also reminiscent of human intelligence. It is especially useful if the task is more complex and requires multiple steps of reasoning to solve.<\/p>\n\n\n\n<p id=\"2124\">Let\u2019s say I ask you \u201cWho won the World Cup in the year before Lionel Messi was born?\u201d What would you do? You would probably solve this step by step by writing down any intermediate solutions needed in order to arrive at the correct answer. And that\u2019s exactly what LLMs can do too.<\/p>\n\n\n\n<p id=\"2428\">It has been found that simply telling an LLM to \u201cthink step by step\u201d can increase its performance substantially in many tasks.<\/p>\n\n\n\n<p id=\"45f1\">Why does this work? We know everything we need to answer this. The problem is that this kind of unusual composite knowledge is probably not directly in the LLM\u2019s internal memory. However, all the individual facts might be, like Messi\u2019s birthday, and the winners of various World Cups.<\/p>\n\n\n\n<p id=\"d79b\">Allowing the LLM to build up to the final answer helps because it gives the model time to think out loud \u2014 a working memory so to say \u2014 and to solve the simpler sub-problems before giving the final answer.<\/p>\n\n\n\n<p id=\"15ae\">The key here is to remember that everything to the left of a to-be-generated word is context that the model can rely on. So, as shown in the image above, by the time the model says \u201cArgentina\u201d, Messi\u2019s birthday and the year of the Word Cup we inquired about are already in the LLM\u2019s working memory, which makes it easier to answer correctly.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"49b3\">Conclusion<\/h1>\n\n\n\n<p id=\"ba1f\">Before I wrap things up, I want to answer a question I asked earlier in the article. Is the LLM really just predicting the next word or is there more to it? Some researchers are arguing for the latter, saying that to become so good at next-word-prediction in any context, the LLM must actually have acquired a compressed understanding of the world internally. Not, as others argue, that the model has simply learned to memorize and copy patterns seen during training, with no actual understanding of language, the world, or anything else.<\/p>\n\n\n\n<p id=\"0f79\">There is probably no clear right or wrong between those two sides at this point; it may just be a different way of looking at the same thing. Clearly these LLMs are proving to be very useful and show impressive knowledge and reasoning capabilities, and maybe even show some sparks of general intelligence. But whether or to what extent that resembles human intelligence is still to be determined, and so is how much further language modeling can improve the state of the art.<\/p>\n\n\n\n<p id=\"0a48\">I hope that this article helps you understand LLMs and the current craze that is surrounding them, so that you can form your own opinion about AI\u2019s potentials and risks. It\u2019s not only up to AI researchers and data scientists to decide how AI is used to benefit the world; everyone should be able to have a say. This is why I wanted to write an article that doesn\u2019t require a lot of background knowledge.<\/p>\n\n\n\n<p id=\"0f8d\">If you made it through this article, I think you pretty much know how some the state-of-the-art LLMs work (as of Autumn 2023), at least at a high level.<\/p>\n\n\n\n<p id=\"4c81\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>From zero to ChatGPT Thanks to Large Language Models (or LLMs for short), Artificial Intelligence has now caught the attention of pretty much everyone. ChatGPT, possibly the most famous LLM,&#8230; <a class=\"read-more-link\" href=\"https:\/\/tbekk.com\/devstream\/2023\/12\/03\/how-large-language-models-work\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51,203],"tags":[215],"class_list":["post-895","post","type-post","status-publish","format-standard","hentry","category-article","category-llm","tag-llm"],"_links":{"self":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/895","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/comments?post=895"}],"version-history":[{"count":1,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/895\/revisions"}],"predecessor-version":[{"id":896,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/895\/revisions\/896"}],"wp:attachment":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/media?parent=895"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/categories?post=895"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/tags?post=895"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}