\n\n\n\n
Getting a handle on temperature, Top-k, Top-p, frequency, and presence penalties can be a bit of a challenge, especially when you\u2019re just starting out with LLM hyperparameters. Terms like \u201cTop-k\u201d and \u201cpresence penalty\u201d can feel a bit overwhelming at first.<\/p>\n\n\n\n
When you look up \u201cTop-k,\u201d you might find a definition like: \u201cTop-k sampling limits the model\u2019s selection of the next word to only the top-k most probable options, based on their predicted probabilities.\u201d That\u2019s a lot to take in! But how does this actually help when you\u2019re working on prompt engineering?<\/p>\n\n\n\n
If you\u2019re anything like me and learn best with visuals, let\u2019s break these down together and make these concepts easy to understand once and for all.<\/p>\n\n\n\n
LLMs under the hood<\/h1>\n\n\n\n
Before we dive into LLM hyperparameters, let\u2019s do a quick thought experiment. Imagine hearing the phrase \u201cA cup of \u2026\u201d. Most of us would expect the next word to be something like \u201ccoffee\u201d (or \u201ctea\u201d if you\u2019re a tea person!) You probably wouldn\u2019t think of \u201cstars\u201d or \u201ccourage\u201d right away.<\/p>\n\n\n\n
What\u2019s happening here is that we\u2019re instinctively predicting<\/em> the most likely words to follow \u201cA cup of \u2026\u201d, with \u201ccoffee\u201d being a much higher likelihood than \u201cstars\u201d.<\/p>\n\n\n\n This is similar to how LLMs work \u2014 they calculate the probabilities of possible next words and choose one based on those probabilities.<\/p>\n\n\n\n So on a high level, the hyperparameters are ways to tune how we select the next probable words<\/em>.<\/p>\n\n\n\n Let\u2019s start with the most common hyperparameter:<\/p>\n\n\n\n Temperature controls the randomness of the models\u2019 output. A lower temperature makes the output more deterministic, favoring more likely words, while a higher temperature allows for more creativity by considering less likely words.<\/p>\n\n\n\n In our \u201cA cup of\u2026\u201d example, setting the temperature to 0 makes the model favor the most likely word, which is \u201ccoffee\u201d.<\/p>\n\n\n\n As temperature increases, the sampling probabilities between different words start to even out, prompting the model to generate highly unusal or unexpected outputs.<\/p>\n\n\n\n Note, that setting the temperature to 0 still doesn\u2019t make the model completely deterministic, though it gets very close.<\/p>\n\n\n\n Use cases<\/strong><\/p>\n\n\n\n Max tokens define the maximum number of tokens (which can be words or parts of words) the model can generate in its responses. Tokens are the smallest units of text that a model processes.<\/p>\n\n\n\n Relationship between tokens and words:<\/strong><\/p>\n\n\n\n Use cases<\/strong><\/p>\n\n\n\n Top-k sampling restricts the model from selecting from the top k<\/em> most likely next words. By narrowing the choices, it helps reduce the chances of generating irrelevant or nonsensical outputs.<\/p>\n\n\n\n In the diagram below, if we set k<\/em> to 2, the model will only consider the two most likely next words \u2014 in this case, \u2018coffee\u2019 and \u2018courage.\u2019 These two words are then resampled, with their probabilities adjusted to sum to 1, ensuring one of them is chosen.<\/p>\n\n\n\n Use cases:<\/strong><\/p>\n\n\n\n Top-p sampling selects the smallest set of words whose combined probability exceeds a threshold p<\/em> (e.g., 0.9), allowing for a more context-sensitive choice of words.<\/p>\n\n\n\n In the diagram below, we start with the most probable word, \u2018coffee,\u2019 which has a probability of 0.6. Since this is less than our threshold of p<\/em> = 0.9, we add the next word, \u2018courage,\u2019 with a probability of 0.2. Together, these give us a total probability of 0.8, which is still below 0.9. Finally, we consider the word \u2018dreams\u2019 with a probability of 0.13, bringing the total to 0.93, which exceeds 0.9. At this point, we stop, having selected the first two most probable words.<\/p>\n\n\n\n Use cases:<\/strong><\/p>\n\n\n\n A frequency penalty reduces the likelihood of the model repeating the same word within the text, promoting diversity and minimizing redundancy in the output. By applying this penalty, the model is encouraged to introduce new words instead of reusing ones that have already appeared.<\/p>\n\n\n\n The frequency penalty is calculated using the formula:<\/p>\n\n\n\n Adjusted probability = initial probability \/ (1 + frequency penalty * count of appearance)<\/strong><\/p>\n\n\n\n For example, let\u2019s say that the word \u201csun\u201d has a probability of 0.5, and it has already appeared twice in the text. If we set the frequency penalty to 1, the adjusted probability for \u201csun\u201d would be:<\/p>\n\n\n\n Adjusted probability = 0.5 \/ (1 + 1 * 2) = 0.5 \/ 3 = 0.16<\/p>\n\n\n\n Use cases:<\/strong><\/p>\n\n\n\n The presence penalty is similar to the frequency penalty but with one key difference: it penalizes the model for reusing any word or phrase that has already been mentioned, regardless of how often it appears<\/em>.<\/p>\n\n\n\n In other words, repeating the word 2 times is as bad as repeating it 20 times.<\/p>\n\n\n\n The formula for adjusting the probability with a presence penalty is: Let\u2019s revisit the earlier example with the word \u201csun\u201d. Instead of multiplying the penalty by the frequency of how many times \u201csun\u201d has appeared, we simply check whether it has appeared at all \u2014 in this case, it has, so we count it as 1.<\/p>\n\n\n\n If we set the presence penalty to 1, the adjusted probability would be:<\/p>\n\n\n\n Adjusted probability = 0.5 \/ (1 + 1 * 1) = 0.5 \/ 2 = 0.25<\/p>\n\n\n\n This reduction makes it less likely for the model to choose \u201csun\u201d again, encouraging the use of new words or phrases, even if \u201csun\u201d has only appeared once in the text.<\/p>\n\n\n\n Use cases:<\/strong><\/p>\n\n\n\n Now that we\u2019ve gone over the basics, let\u2019s dive into how frequency and presence penalties are often used together. Just a heads-up, though \u2014 they\u2019re powerful tools, but it\u2019s important to use them with a bit of caution to get the best results.<\/p>\n\n\n\n When to use them:<\/strong><\/p>\n\n\n\n When to not use them<\/strong><\/p>\n\n\n\n By now, you should have a clearer picture of how temperature, Top-k, Top-p, frequency, and presence penalties work together to shape the output of your language model. And if it still feels a bit tricky, that\u2019s totally okay \u2014 these concepts can take some time to fully click. Just keep experimenting and exploring, and you\u2019ll get the hang of it before you know it.<\/p>\n\n\n\n If you find visual content like this helpful and want more, we\u2019d love to see you in our Discord community.<\/a> It\u2019s a space where we share ideas, help each other out, and learn together.<\/p>\n","protected":false},"excerpt":{"rendered":" Understand temperature, Top-k, Top-p, Frequency & Precense Penalty once and for all. Getting a handle on temperature, Top-k, Top-p, frequency, and presence penalties can be a bit of a challenge,… Read more »<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51,203],"tags":[464,215,199],"class_list":["post-940","post","type-post","status-publish","format-standard","hentry","category-article","category-llm","tag-hyperparameters","tag-llm","tag-prompt-engineering"],"_links":{"self":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/940","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/comments?post=940"}],"version-history":[{"count":3,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/940\/revisions"}],"predecessor-version":[{"id":943,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/940\/revisions\/943"}],"wp:attachment":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/media?parent=940"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/categories?post=940"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/tags?post=940"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Temperature<\/h1>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*XHNwK9UBbrvVe8_Idcu7EA.png\")
\n
Max Tokens<\/h1>\n\n\n\n
\n
\n
Top-k<\/h1>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*374nBkpu2vH81XxbKD96Bg.png\")
\n
Top-p<\/h1>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*VqKeizk-ODwudYjnQrO3VA.png\")
\n
Frequency Penalty<\/h1>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*Z1VeRHdraE9AXrJTCoWlcw.png\")
\n
Presence Penalty<\/h1>\n\n\n\n
Adjusted probability = initial probability \/ (1 + presence penalty * presence)<\/strong><\/p>\n\n\n\n![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*BoVtpFQKzSPbMAWAnaml3A.png\")
\n
Frequency and Presence Penalties often go hand-in-hand<\/h1>\n\n\n\n
\n
\n