{"id":935,"date":"2024-07-16T10:16:51","date_gmt":"2024-07-16T10:16:51","guid":{"rendered":"https:\/\/tbekk.com\/devstream\/?p=935"},"modified":"2024-07-16T10:16:51","modified_gmt":"2024-07-16T10:16:51","slug":"the-llm-triangle-principles-to-architect-reliable-ai-apps","status":"publish","type":"post","link":"https:\/\/tbekk.com\/devstream\/2024\/07\/16\/the-llm-triangle-principles-to-architect-reliable-ai-apps\/","title":{"rendered":"The LLM Triangle Principles to Architect Reliable AI Apps"},"content":{"rendered":"\n<p class=\"has-medium-gray-color has-text-color\"><em>Software design principles for thoughtfully designing reliable, high-performing LLM applications. A framework to bridge the gap between potential and production-grade performance.<\/em><\/p>\n\n\n\n<hr class=\"wp-block-separator has-text-color has-medium-gray-color has-alpha-channel-opacity has-medium-gray-background-color has-background is-style-wide\"\/>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em><strong>Link:<\/strong><\/em> <a href=\"https:\/\/towardsdatascience.com\/the-llm-triangle-principles-to-architect-reliable-ai-apps-d3753dd8542e\"><em>towardsdatascience.com<\/em><\/a><\/li>\n\n\n\n<li><em><strong>Author:<\/strong><\/em> <a href=\"https:\/\/medium.com\/@almogbaku?source=post_page-----d3753dd8542e--------------------------------\"><em>Almog Baku<\/em><\/a><\/li>\n\n\n\n<li><em><strong>Publication date:<\/strong><\/em> <em>July 16, 2024<\/em><\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-text-color has-medium-gray-color has-alpha-channel-opacity has-medium-gray-background-color has-background is-style-wide\"\/>\n\n\n\n<p id=\"3ceb\">Large Language Models (LLMs) hold immense potential, but developing reliable production-grade applications remains challenging. After building dozens of LLM systems, I\u2019ve distilled the formula for success into four fundamental principles that any team can apply.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p id=\"c385\">\u201cLLM-Native apps are 10% sophisticated model, and 90% experimenting data-driven engineering work.\u201d<\/p>\n<\/blockquote>\n\n\n\n<p id=\"f678\">Building production-ready LLM applications requires&nbsp;<em>careful engineering practices<\/em>. When users cannot interact&nbsp;<em>directly&nbsp;<\/em>with the LLM, the prompt must be meticulously composed to cover all nuances, as&nbsp;<em>iterative user feedback may be unavailable<\/em>.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"4528\">Introducing the LLM Triangle Principles<\/h1>\n\n\n\n<p id=\"77cf\">The LLM Triangle Principles encapsulate the essential guidelines for building effective LLM-native apps. They provide a solid conceptual framework, guide developers in constructing robust and reliable LLM-native applications, and offer direction and support.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1242\/1*7uKE_KWvRRyxCNdNbEP4xg.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">An optimal LLM Usage is achieved by optimizing the three prominent principles through the lens of the SOP. (Image by author)<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"609d\">The Key Apices<\/h2>\n\n\n\n<p id=\"6de0\">The LLM Triangle Principles introduces four programming principles to help you design and build LLM-Native apps.<\/p>\n\n\n\n<p id=\"b48f\">The first principle is the&nbsp;<em>Standard Operating Procedure (<\/em><strong><em>SOP<\/em><\/strong><em>).<\/em>&nbsp;The SOP guides the three apices of our triangle:&nbsp;<strong><em>Model<\/em><\/strong>,&nbsp;<strong><em>Engineering Techniques<\/em>,<\/strong>&nbsp;and&nbsp;<strong><em>Contextual Data<\/em><\/strong>.<\/p>\n\n\n\n<p id=\"88bd\">Optimizing the three apices principles&nbsp;<em>through the lens<\/em>&nbsp;of the&nbsp;<strong>SOP is the key<\/strong>&nbsp;<strong>to ensuring a high-performing<\/strong>&nbsp;LLM-native app.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"be9a\">1. Standard Operating Procedure (SOP)<\/h1>\n\n\n\n<p id=\"30e1\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Standard_operating_procedure\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>S<\/strong>tandard&nbsp;<strong>O<\/strong>perating&nbsp;<strong>P<\/strong>rocedure (SOP)<\/a>&nbsp;is a well-known terminology in the industrial world. It\u2019s a set of step-by-step instructions compiled by large organizations to help their workers carry out routine operations while maintaining high-quality and similar results each time. This practically turns inexperienced or low-skilled workers into experts by writing detailed instructions.<\/p>\n\n\n\n<p id=\"3228\">The LLM Triangle Principles&nbsp;<em>borrows<\/em>&nbsp;the SOP paradigm and encourages you to&nbsp;<strong>consider the model an inexperienced\/unskilled worker<\/strong>. We can ensure higher-quality results by \u201cteaching\u201d the model how an expert would perform this task.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:656\/1*sUQp9VQhXd51MnoKlrz--g.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The SOP&nbsp;<strong><em>guiding<\/em><\/strong>&nbsp;principle. (image by author)<\/figcaption><\/figure>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p id=\"1932\">\u201cWithout an SOP, even the most powerful LLM will fail to deliver consistently high-quality results.\u201d<\/p>\n<\/blockquote>\n\n\n\n<p id=\"ca13\">When thinking about the&nbsp;<strong>SOP&nbsp;<em>guiding<\/em>&nbsp;principle<\/strong>, we should identify what techniques will help us implement the SOP most effectively.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"1210\">1.1. Cognitive modeling<\/h2>\n\n\n\n<p id=\"5325\">To create an SOP, we need to take our best-performing workers (domain experts), model how they think and work to achieve the same results, and write down everything they do.<\/p>\n\n\n\n<p id=\"8173\">After editing and formalizing it, we\u2019ll have detailed instructions to help every inexperienced or low-skilled worker succeed and yield excellent work.<\/p>\n\n\n\n<p id=\"b2cf\">Like humans, it\u2019s essential to&nbsp;<em>reduce the cognitive load<\/em>&nbsp;of the task by simplifying or splitting it. Following a simple step-by-step instruction is more straightforward than a lengthy, complex procedure.<\/p>\n\n\n\n<p id=\"f8c9\">During this process, we identify the hidden&nbsp;<a href=\"https:\/\/en.wikipedia.org\/wiki\/Implicit_cognition\" rel=\"noreferrer noopener\" target=\"_blank\"><em>implicit cognition<\/em><\/a><em>&nbsp;\u201cjumps\u201d<\/em>&nbsp;\u2014 the small, unconscious steps experts take that significantly impact the outcome. These subtle, unconscious, often unspoken assumptions or decisions can substantially affect the final result.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*lfpXTu4ll6fYg3w7t1Q2Ng.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">An example of an \u201cimplicit cognition jump.\u201d (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"2d83\">For example, let\u2019s say we want to model an SQL analyst. We\u2019ll start by interviewing them and ask them a few questions, such as:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What do you do when you are asked to analyze a business problem?<\/li>\n\n\n\n<li>How do you make sure your solution meets the request?<\/li>\n\n\n\n<li>&lt;reflecting the process as we understand to the interviewee&gt;<\/li>\n\n\n\n<li>Does this accurately capture your process? &lt;getting corrections&gt;<\/li>\n\n\n\n<li>Etc.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*mYfmcCM2cg-QL2ItqlKLsQ.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">An example of the cognitive process that the analyst does and how to model it. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"59f4\">The implicit cognition process takes many shapes and forms; a typical example is a \u201cdomain-specific definition.\u201d For example, \u201cbestseller\u201d might be a prominent term for our domain expert, but not for everyone else.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*zTWkVPNzER_ikUo8HQkPzw.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Expanding the implicit cognition process in our SQL analyst example. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"acb4\">Eventually, we\u2019ll have a full SOP \u201crecipe\u201d that allows us to emulate our top-performing analyst.<\/p>\n\n\n\n<p id=\"12d5\">When mapping out these complex processes, it can be helpful to visualize them as a graph. This is especially helpful when the process is nuanced and involves many steps, conditions, and splits.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1310\/1*fSG-kH2fkxmQTTHp4X1P-Q.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The \u201cSQL Analyst SOP\u201d includes all the required technical steps, visualized as a graph. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"b92d\">Our final solution should mimic the steps defined in the SOP. In this stage, try to ignore the implementation\u2014later, you can implement it across one or many steps\/chains throughout our solution.<\/p>\n\n\n\n<p id=\"c1f8\">Unlike the rest of the principles, the cognitive modeling (SOP writing) is the&nbsp;<em>only standalone process<\/em>. It\u2019s highly recommended that you model your process before writing code. That being said, while implementing it, you might go back and change it based on new insights or understandings you gained.<\/p>\n\n\n\n<p id=\"ede6\">Now that we understand the importance of creating a well-defined SOP, that guides our&nbsp;<em>business understanding<\/em>&nbsp;of the problem, let\u2019s explore how we can effectively implement it using various engineering techniques.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"3221\">2. Engineering Techniques<\/h1>\n\n\n\n<p id=\"da94\"><a href=\"https:\/\/www.promptingguide.ai\/\" rel=\"noreferrer noopener\" target=\"_blank\">Engineering Techniques<\/a>&nbsp;help you practically implement your SOP and get the most out of the model. When thinking about the&nbsp;<strong>Engineering Techniques principle<\/strong>, we should consider what tools(techniques) in our toolbox can help us implement and shape our SOP and assist the model in communicating well with us.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:656\/1*mqGzEs5ycZtfNhNgFguchQ.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The Engineering Techniques principle. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"efea\">Some engineering techniques are only implemented in the prompt layer, while many require a software layer to be effective, and some combine both layers.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*MSlZsx0nzNC87I6Oq0Gq4w.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Engineering Techniques Layers. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"ebb8\">While many small nuances and techniques are discovered daily, I\u2019ll cover two primary techniques: workflow\/chains and agents.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"64b7\">2.1. LLM-Native architectures (aka flow engineering or chains)<\/h2>\n\n\n\n<p id=\"0a92\">The LLM-Native Architecture describes the agentic flow your app is going through to yield the task\u2019s result.<\/p>\n\n\n\n<p id=\"28a2\">Each step in our flow is a standalone process that must occur to achieve our task. Some steps will be performed simply by deterministic code; for some, we will use an LLM (agent).<\/p>\n\n\n\n<p id=\"63b7\">To do that, we can reflect on the Standard Operating Procedure (SOP) we drew and think:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Which SOP steps should we glue together to the same agent? And what steps should we split as different agents?<\/li>\n\n\n\n<li>What SOP steps should be executed in a standalone manner (but they might be fed with information from previous steps)?<\/li>\n\n\n\n<li>What SOP steps can we perform in a deterministic code?<\/li>\n\n\n\n<li>Etc.<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1062\/1*_NdNLbgGoLhy4q2wNHGFcg.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">An LLM-Native Architecture example for \u201cWikipedia writer\u201d based on a given SOP. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"9668\">Before navigating to the next step in our architecture\/graph, we should define its key properties:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Inputs and outputs<\/strong>&nbsp;\u2014 What is the signature of this step? What is required before we can take an action? (this can also serve as an output format for an agent)<\/li>\n\n\n\n<li><strong>Quality assurances\u2014<\/strong>What makes the response \u201cgood enough\u201d? Are there cases that require human intervention in the loop? What kinds of assertions can we configure?<\/li>\n\n\n\n<li><strong>Autonomous level&nbsp;<\/strong>\u2014 How much control do we need over the result\u2019s quality? What range of use cases can this stage handle? In other words, how much can we trust the model to work independently at this point?<\/li>\n\n\n\n<li><strong>Triggers&nbsp;<\/strong>\u2014 What is the next step? What defines the next step?<\/li>\n\n\n\n<li><strong>Non-functional&nbsp;<\/strong>\u2014 What\u2019s the required latency? Do we need special business monitoring here?<\/li>\n\n\n\n<li><strong>Failover control&nbsp;<\/strong>\u2014 What kind of failures(systematic and agentic) can occur? What are our fallbacks?<\/li>\n\n\n\n<li><strong>State management<\/strong>&nbsp;\u2014 Do we need a special state management mechanism? How do we retrieve\/save states (define the indexing key)? Do we need persistence storage? What are the different usages of this state(e.g., cache, logging, etc.)?<\/li>\n\n\n\n<li>Etc.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"e1f6\">2.2. What are agents?<\/h2>\n\n\n\n<p id=\"9328\">An LLM agent is a standalone component of an LLM-Native architecture that involves calling an LLM.<\/p>\n\n\n\n<p id=\"aa98\">It\u2019s an instance of LLM usage with the prompt containing the context. Not all agents are equal \u2014 Some will use \u201ctools,\u201d some won\u2019t; some might be used \u201cjust once\u201d in the flow, while others can be called recursively or multiple times, carrying the previous input and outputs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"71ad\">2.2.1. Agents with tools<\/h2>\n\n\n\n<p id=\"e3ac\">Some LLM agents can use \u201ctools\u201d \u2014 predefined functions for tasks like calculations or web searches. The agent outputs instructions specifying the tool and input, which the application executes, returning the result to the agent.<\/p>\n\n\n\n<p id=\"66ef\">To understand the concept, let\u2019s look at a simple prompt implementation for tool calling. This can work even with models not natively trained to call tools:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">You are an assistant with access to these tools:<br><br>- calculate(expression: str) -&gt; str - calculate a mathematical expression<br>- search(query: str) -&gt; str - search for an item in the inventory<br><br>Given an input, Respond with a YAML with keys: `func`(str) and `arguments`(map) or `message`(str).Given input<\/pre>\n\n\n\n<p id=\"4379\">It\u2019s important to distinguish between agents with tools (hence&nbsp;<em>autonomous agents<\/em>) and agents whose output can lead to performing an action:<br>Autonomous agents are&nbsp;<em>given the right<\/em>&nbsp;to&nbsp;<strong>decide<\/strong>&nbsp;if they should act and with what action. In contrast, a (nonautonomous) agent simply \u201cprocesses\u201d our request(e.g., classification), and based on this process, our deterministic code performs an action, and the model has zero control over that.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*AfwTUBzLnX7Vk1Wzs9o8Vg.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">An autonomous agent VS agent that triggers an action. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"d6a2\">As we increase the agent\u2019s autonomy in planning and executing tasks, we enhance its decision-making capabilities but potentially reduce control over output quality. Although this might look like a magical solution to make it more \u201csmart\u201d or \u201cadvanced,\u201d it comes with the cost of losing control over the quality.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1282\/1*LW3sI8MA_H0DHgUjJYvyTw.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The tradeoffs of an autonomous agent. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"c426\">Beware the allure of fully autonomous agents. While their architecture might look appealing and simpler, using it for everything (or as the initial PoC) might be very deceiving from the \u201creal production\u201d cases. Autonomous agents are hard to debug and unpredictable(response with unstable quality), which makes them unusable for production.<\/p>\n\n\n\n<p id=\"4669\">Currently, agents (without implicit guidance) are not very good at planning complex processes and usually skip essential steps. For example, in our \u201cWikipedia writer\u201d use-case, they\u2019ll just start writing and skip the systematic process. This makes agents (and autonomous agents especially) only as good as the model, or more accurately \u2014 only as good as the data they were trained on relative to your task.<\/p>\n\n\n\n<p id=\"8558\">Instead of giving the agent (or a swarm of agents) the liberty to do everything end-to-end, try to hedge their task to a specific region of your flow\/SOP that requires this kind of agility or creativity. This can yield higher-quality results because you can enjoy both worlds.<\/p>\n\n\n\n<p id=\"3bb4\">An excellent example is&nbsp;<a href=\"https:\/\/www.codium.ai\/blog\/alphacodium-state-of-the-art-code-generation-for-code-contests\/\" rel=\"noreferrer noopener\" target=\"_blank\">AlphaCodium<\/a>: By combining a structured flow with different agents (including a novel agent that iteratively writes and tests code), they increased GPT-4 accuracy (pass@5) on CodeContests from 19% to 44%.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*lMdogwmv1N9YurJPM8Pidg.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">AlphaCodium\u2019s LLM Architecture. (Image by the curtesy&nbsp;<a href=\"https:\/\/www.codium.ai\/\" rel=\"noreferrer noopener\" target=\"_blank\">Codium.ai<\/a>)<\/figcaption><\/figure>\n\n\n\n<p id=\"a2a1\">While engineering techniques lay the groundwork for implementing our SOP and optimizing LLM-native applications, we must also carefully consider another critical component of the LLM Triangle: the model itself.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"d33d\">3. Model<\/h1>\n\n\n\n<p id=\"df82\">The model we choose is a critical component of our project\u2019s success\u2014a large one (such as GPT-4 or Claude Opus) might yield better results but be quite costly at scale, while a smaller model might be less \u201csmart\u201d but help with the budget. When thinking about the<strong>&nbsp;Model principle<\/strong>, we should aim to identify our constraints and goals and what kind of model can help us fulfill them.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:656\/1*GAMy4zwftAOOgPdkmhg4AA.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The Model principle. (Image by author)<\/figcaption><\/figure>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p id=\"6c60\">\u201cNot all LLMs are created equal. Match the model to the mission.\u201d<\/p>\n<\/blockquote>\n\n\n\n<p id=\"96e5\">The truth is that we don\u2019t always need the largest model; it depends on the task. To find the right match, we must have an&nbsp;<a rel=\"noreferrer noopener\" target=\"_blank\" href=\"https:\/\/towardsdatascience.com\/building-llm-apps-a-clear-step-by-step-guide-1fe1e6ef60fd\">experimental process<\/a>&nbsp;and try multiple variations of our solution.<\/p>\n\n\n\n<p id=\"0ce0\">It helps to look at our \u201cinexperienced worker\u201d analogy \u2014 a very \u201csmart\u201d worker with many academic credentials probably will succeed in some tasks easily. Still, they might be overqualified for the job, and hiring a \u201ccheaper\u201d candidate will be much more cost-effective.<\/p>\n\n\n\n<p id=\"1b2f\">When considering a model, we should define and compare solutions based on the tradeoffs we are willing to take:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Task Complexity<\/strong>&nbsp;\u2014 Simpler tasks (such as summarization) are easier to complete with smaller models, while reasoning usually requires larger models.<\/li>\n\n\n\n<li><strong>Inference infrastructure<\/strong>&nbsp;\u2014 Should it run on the cloud or edge devices? The model size might impact a small phone, but it can be tolerated for cloud-serving.<\/li>\n\n\n\n<li><strong>Pricing<\/strong>&nbsp;\u2014 What price can we tolerate? Is it cost-effective considering the business impact and predicated usage?<\/li>\n\n\n\n<li><strong>Latency<\/strong>&nbsp;\u2014 As the model grows larger, the latency grows as well.<\/li>\n\n\n\n<li><strong>Labeled data<\/strong>&nbsp;\u2014 Do we have data we can use immediately to enrich the model with examples or relevant information that is not trained upon?<\/li>\n<\/ul>\n\n\n\n<p id=\"b5ba\">In many cases, until you have the \u201cin-house expertise,\u201d it helps to pay a little extra for an experienced worker \u2014 the same applies to LLMs.<\/p>\n\n\n\n<p id=\"d250\">If you don\u2019t have&nbsp;<em>labeled data<\/em>, start with a stronger (larger) model,&nbsp;<em>collect data<\/em>, and then utilize it to empower a model using a few-shot or fine-tuning.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"6f93\">3.1. Fine-tuning a model<\/h2>\n\n\n\n<p id=\"c4d9\">There are a few aspects that you must consider before resorting to fine-tune a model:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Privacy<\/strong>&nbsp;\u2014 Your data might include pieces of private information that must be kept from the model. You must anonymize your data to avoid legal liabilities if your data contains private information.<\/li>\n\n\n\n<li><strong>Laws, Compliance, and Data Rights&nbsp;<\/strong>\u2014 Some legal questions can be raised when training a model. For example, the OpenAI terms-of-use policy prevents you from training a model without OpenAI using generated responses. Another typical example is complying with the GDPR\u2019s laws, which require a \u201cright for revocation,\u201d where a user can require the company to remove information from the system. This raises legal questions about whether the model should be retrained or not.<\/li>\n\n\n\n<li><strong>Updating latency \u2014&nbsp;<\/strong>The latency or data cutoff is much higher when training a model. Unlike embedding the new information via the context (see \u201c4. Contextual Data\u201d section below), which provides immediate latency, training the model is a long process that takes time. Due to that, models are retrained less often.<\/li>\n\n\n\n<li><strong>Development and operation \u2014&nbsp;<\/strong>Implementing a reproducible, scalable, and monitored fine-tuning pipeline is essential while continuously evaluating the results\u2019 performance. This complex process requires constant maintenance.<\/li>\n\n\n\n<li><strong>Cost \u2014&nbsp;<\/strong>Retraining is considered expensive due to its complexity and the highly intensive resources(GPUs) required per training.<\/li>\n<\/ul>\n\n\n\n<p id=\"cc9c\">The ability of LLMs to act as&nbsp;<em>in-context learners<\/em>&nbsp;and the fact that the newer models support a much larger context window simplify our implementation dramatically and can provide excellent results even without fine-tuning. Due to the complexity of fine-tuning, using it as a last resort or skipping it entirely is recommended.<\/p>\n\n\n\n<p id=\"4487\">Conversely, fine-tuning models for specific tasks (e.g., structured JSON output) or domain-specific language can be highly efficient. A small, task-specific model can be highly effective and much cheaper in inference than large LLMs. Choose your solution wisely, and assess all the relevant considerations before escalating to LLM training.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p id=\"34dc\">\u201cEven the most powerful model requires relevant and well-structured contextual data to shine.\u201d<\/p>\n<\/blockquote>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"7b49\">4. Contextual Data<\/h1>\n\n\n\n<p id=\"efe8\"><strong><em>LLMs are in-context learners.<\/em><\/strong>&nbsp;That means that by providing task-specific information, the LLM agent can help us to perform it&nbsp;<em>without<\/em>&nbsp;special training or fine-tuning. This enables us to \u201cteach\u201d new knowledge or skills easily. When thinking about the&nbsp;<strong>Contextual Data principle<\/strong>, we should aim to organize and model the available data and how to compose it within our prompt.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:656\/1*i3TmpMkZNNB2uZM_zCB5og.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The Contextual Data principle. (Image by author)<\/figcaption><\/figure>\n\n\n\n<p id=\"2430\">To compose our context, we include the relevant (contextual) information within the prompt we send to the LLM. There are two kinds of contexts we can use:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Embedded contexts<\/strong>&nbsp;\u2014 embedded information pieces provided as part of the prompt.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">You are the helpful assistant of &lt;name&gt;, a &lt;role&gt; at &lt;company&gt;<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Attachment contexts<\/strong>&nbsp;\u2014 A list of information pieces glues by the beginning\/end of the prompt<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">Summarize the provided emails while keeping a friendly tone.<br>---<br><br>&lt;email_0&gt;<br>&lt;email_1&gt;<\/pre>\n\n\n\n<p id=\"7151\">Contexts are usually implemented using a \u201cprompt template\u201d (such as&nbsp;<a href=\"https:\/\/jinja.palletsprojects.com\/en\/3.1.x\/\" rel=\"noreferrer noopener\" target=\"_blank\">jinja2<\/a>&nbsp;or&nbsp;<a href=\"https:\/\/mustache.github.io\/\" rel=\"noreferrer noopener\" target=\"_blank\">mustache<\/a>&nbsp;or simply native&nbsp;<a href=\"https:\/\/docs.python.org\/3\/reference\/lexical_analysis.html#formatted-string-literals\" rel=\"noreferrer noopener\" target=\"_blank\">formatting literal strings<\/a>); this way, we can compose them elegantly while keeping the essence of our prompt:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"># Embedded context with an attachment context<br>prompt = f\"\"\"<br>You are the helpful assistant of {name}. {name} is a {role} at {company}.<br><br>Help me write a {tone} response to the attached email.<br>Always sign your email with:<br>{signature}<br><br>---<br><br>{email}<br>\"\"\"<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"3d0a\">4.1. Few-shot learning<\/h2>\n\n\n\n<p id=\"0d84\">Few-shot learning is a powerful way to \u201cteach\u201d LLMs by example without requiring extensive fine-tuning. Providing a few representative examples in the prompt can guide the model in understanding the desired format, style, or task.<\/p>\n\n\n\n<p id=\"de0c\">For instance, if we want the LLM to generate email responses, we could include a few examples of well-written responses in the prompt. This helps the model learn the preferred structure and tone.<\/p>\n\n\n\n<p id=\"a859\">We can use diverse examples to help the model catch different corner cases or nuances and learn from them. Therefore, it\u2019s essential to include a variety of examples that cover a range of scenarios your application might encounter.<\/p>\n\n\n\n<p id=\"11c0\">As your application grows, you may consider implementing \u201c<a href=\"https:\/\/arxiv.org\/abs\/1804.09458\" rel=\"noreferrer noopener\" target=\"_blank\">Dynamic few-shot<\/a>,\u201d which involves programmatically selecting the most relevant examples for each input. While it increases your implementation complexity, it ensures the model receives the most appropriate guidance for each case, significantly improving performance across a wide range of tasks without costly fine-tuning.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"76a4\">4.2. Retrieval Augmented Generation<\/h2>\n\n\n\n<p id=\"8be5\"><a href=\"https:\/\/www.promptingguide.ai\/techniques\/rag\" rel=\"noreferrer noopener\" target=\"_blank\">Retrieval Augmented Generation (RAG)<\/a>&nbsp;is a technique for retrieving relevant documents for additional context before generating a response. It\u2019s like giving the LLM a quick peek at specific reference material to help inform its answer. This keeps responses current and factual without needing to retrain the model.<\/p>\n\n\n\n<p id=\"caa8\">For instance, on a support chatbot application, RAG could pull relevant help-desk wiki pages to inform the LLM\u2019s answers.<\/p>\n\n\n\n<p id=\"00d3\">This approach helps LLMs&nbsp;<em>stay current<\/em>&nbsp;and&nbsp;<em>reduces hallucinations<\/em>&nbsp;by grounding responses in retrieved facts. RAG is particularly handy for tasks that require updated or specialized knowledge without retraining the entire model.<\/p>\n\n\n\n<p id=\"1ca0\">For example, suppose we are building a support chat for our product. In that case, we can use RAG to retrieve a relevant document from our helpdesk wiki, then provide it to an LLM agent and ask it to compose an answer based on the question and provide a document.<\/p>\n\n\n\n<p id=\"e9d3\">There are three key pieces to look at while implementing RAG:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Retrieval mechanism<\/strong>&nbsp;\u2014 While the traditional implementation of RAG involves retrieving a relevant document using a vector similarity search, sometimes it\u2019s better or cheaper to use simpler methods such as keyword-based search (like&nbsp;<a href=\"https:\/\/en.wikipedia.org\/wiki\/Okapi_BM25\" rel=\"noreferrer noopener\" target=\"_blank\">BM-25<\/a>).<\/li>\n\n\n\n<li><strong>Indexed data structure \u2014<\/strong>Indexing the entire document naively, without preprocessing, may limit the effectiveness of the retrieval process. Sometimes, we want to add a data preparation step, such as preparing a list of questions and answers based on the document.<\/li>\n\n\n\n<li><strong>Metadata\u2014<\/strong>Storing relevant metadata allows for more efficient referencing and filtering of information (e.g., narrowing down wiki pages to only those related to the user\u2019s specific product inquiry). This extra data layer streamlines the retrieval process.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"3733\">4.3. Providing relevant context<\/h2>\n\n\n\n<p id=\"2277\">The context information relevant to your agent can vary. Although it may seem beneficial, providing the model (like the \u201cunskilled worker\u201d) with too much information can be overwhelming and irrelevant to the task. Theoretically, this causes the model to learn irrelevant information (or token connections), which can lead to confusion and&nbsp;<a href=\"https:\/\/en.wikipedia.org\/wiki\/Hallucination_(artificial_intelligence)\" rel=\"noreferrer noopener\" target=\"_blank\">hallucinations<\/a>.<\/p>\n\n\n\n<p id=\"e4b3\">When Gemini 1.5 was released and introduced as an LLM that could process up to 10M tokens, some practitioners questioned whether the context was still an issue. While it\u2019s a fantastic accomplishment, especially for some use cases (such as chat with PDFs), it\u2019s still limited, especially when reasoning over various documents.<\/p>\n\n\n\n<p id=\"4b3b\">Compacting the prompt and providing the LLM agent with only relevant information is crucial. This reduces the processing power the model invests in irrelevant tokens, improves the quality, optimizes the latency, and reduces the cost.<\/p>\n\n\n\n<p id=\"46dd\">There are many tricks to improve the relevancy of the provided context, most of which relate to how you store and catalog your data.<br>For RAG applications, it\u2019s handy to add a data preparation that shapes the information you store (e.g., questions and answers based on the document, then providing the LLM agent only with the answer; this way, the agent gets a summarized and shorter context), and use re-ranking algorithms on top of the retrieved documents to refine the results.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p id=\"9055\">\u201cData fuels the engine of LLM-native applications. A strategic design of contextual data unlocks their true potential.\u201d<\/p>\n<\/blockquote>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"9957\">Conclusion and Implications<\/h1>\n\n\n\n<p id=\"0420\">The LLM Triangle Principles provide a structured approach to developing high-quality LLM-native applications, addressing the gap between LLMs\u2019 enormous potential and real-world implementation challenges. Developers can create more reliable and effective LLM-powered solutions by focusing on three key principles\u2014the&nbsp;<strong>Model<\/strong>,&nbsp;<strong>Engineering Techniques<\/strong>, and&nbsp;<strong>Contextual Data<\/strong>\u2014all guided by a well-defined&nbsp;<strong>SOP<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:656\/1*WtuBoAkh-HXOMXAjXtT_OQ.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">The LLM Triangle Principles. (Image by author)<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"5200\">Key takeaways<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Start with a clear SOP<\/strong>: Model your expert\u2019s cognitive process to create a step-by-step guide for your LLM application. Use it as a guide while thinking of the other principles.<\/li>\n\n\n\n<li><strong>Choose the right model<\/strong>: Balance capabilities with cost, and consider starting with larger models before potentially moving to smaller, fine-tuned ones.<\/li>\n\n\n\n<li><strong>Leverage engineering techniques<\/strong>: Implement LLM-native architectures and use agents strategically to optimize performance and maintain control. Experiment with different prompt techniques to find the most effective prompt for your case.<\/li>\n\n\n\n<li><strong>Provide relevant context<\/strong>: Use in-context learning, including RAG, when appropriate, but be cautious of overwhelming the model with irrelevant information.<\/li>\n\n\n\n<li><strong>Iterate and experiment<\/strong>: Finding the right solution often requires testing and refining your work. I recommend reading and implementing the&nbsp;<a rel=\"noreferrer noopener\" target=\"_blank\" href=\"https:\/\/towardsdatascience.com\/building-llm-apps-a-clear-step-by-step-guide-1fe1e6ef60fd\">\u201cBuilding LLM Apps: A Clear Step-By-Step Guide\u201d<\/a>&nbsp;tips for a detailed LLM-Native development process guide.<\/li>\n<\/ol>\n\n\n\n<p id=\"8735\">By applying the LLM Triangle Principles, organizations can move beyond a simple proof-of-concept and develop robust, production-ready LLM applications that truly harness the power of this transformative technology.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Software design principles for thoughtfully designing reliable, high-performing LLM applications. A framework to bridge the gap between potential and production-grade performance. Large Language Models (LLMs) hold immense potential, but developing&#8230; <a class=\"read-more-link\" href=\"https:\/\/tbekk.com\/devstream\/2024\/07\/16\/the-llm-triangle-principles-to-architect-reliable-ai-apps\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[181,51,203],"tags":[27,215,462,461],"class_list":["post-935","post","type-post","status-publish","format-standard","hentry","category-ai-2","category-article","category-llm","tag-ai","tag-llm","tag-software-design","tag-software-dev"],"_links":{"self":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/935","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/comments?post=935"}],"version-history":[{"count":1,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/935\/revisions"}],"predecessor-version":[{"id":936,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/935\/revisions\/936"}],"wp:attachment":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/media?parent=935"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/categories?post=935"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/tags?post=935"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}