\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*LfUQuhGdsGQ8oCZ6pkbJng.png\")
<\/figure>\n\n\n\nOver the last few months, we have looked into around 100 agents<\/a> with various use cases, studied SDKs and frameworks<\/strong> for agents<\/a>, and discussed challenges faced by agents<\/strong> with founders of Cognosys<\/a>, Aomni<\/a>, Superagent<\/a>, Sweep<\/a>, and more.<\/p>\n\n\n\n Here is what we learned about products built on top of agents, their challenges, standardization, and the future.<\/p>\n\n\n\n There is still some ambiguity in the terms like \u201cagents\u201d, \u201cAI agents\u201d, \u201cautonomous agents\u201d, or \u201cLLM agents\u201d.<\/p>\n\n\n\n We define an agent (using interchangeably with the other variations) similarly to Shawn Wang, aka \u201cSwyx\u201d (founder of smol ai<\/a>), Matt Schlicht<\/a> (CEO of Octane AI), and mainly Lilian Weng<\/a> from OpenAI.<\/p>\n\n\n\n AI agents possess three main capabilities.<\/p>\n\n\n\n These qualities distinct agents from semi or non-autonomous LLM-powered apps<\/strong>. When compared with \u201cmainstream\u201d automation \u2014 where you set up a range of triggers based on data or system states and configure what happens next \u2014 AI agents can work in unpredictable environments where there\u2019s a lot of new information.<\/p>\n\n\n\n Defining agents correctly may not be needed soon, as the trend is moving from popular standalone agents, often trying to solve a broad variety of problems<\/strong> at the expense of quality, to agents being just an unmentioned part of a bigger product<\/strong>.<\/p>\n\n\n\n Companies work on agent-powered assistants as an additional feature in existing products<\/strong>. Examples include Hyperwrite AI\u2019s Otherside<\/a>, which serves as a personal assistant for daily tasks, MultiOn<\/a>, a personal life assistant, and Deepnote\u2019s AI Copilot<\/a>.<\/p>\n\n\n\n We see an increase in the complexity of the agents-centered projects. Sweep<\/a>, for instance, is an open-source GitHub assistant with a significant amount of code built around the AI agent. Another example is Grit.io<\/a> \u2014 a tool for automated code migrations and dependency upgrades.<\/p>\n\n\n\n The main incentives for enterprises to use agents are saving costs and money<\/strong>. However, they are still hesitant towards agents until they become more reliable.<\/p>\n\n\n\n \u201cFor enterprise customers, we are talking at least ~99.9% reliability,\u201d thinks David Zhang, the founder of Aomni Agent.<\/p>\n\n\n\n The end users have high standards for fast software, while LLM-powered agents sometimes run slow. Sully Omar, the CEO of Cognosys, comments<\/a>: \u201cIn traditional SW engineering, around 200 milliseconds is already considered slow. For agents and LLM apps, latency is a big issue, with LLM calls taking more than 30 seconds.\u201d<\/p>\n\n\n\n In general, developers of agents currently struggle with testing, evaluating, debugging, latency, and monitoring. One particular example of a common problem is identifying at what step their agent broke and why.<\/p>\n\n\n\n Another big question that runs through the entire AI industry is that of privacy, security, and data retention policy.<\/p>\n\n\n\n Agent developers differ in the paradigms they choose for solving the said challenges.<\/p>\n\n\n\n They either build on top of existing tools, create their own internal solutions, or adopt some of the products built specifically for agents<\/strong>, many still in an early stage or in alpha\/beta version.<\/p>\n\n\n\n David Zhang, the founder of Aomni, points out how a lot of agent developers try to reinvent the wheel<\/strong><\/a> <\/strong>with new frameworks and SDKs, instead of building on top of existing technology.<\/p>\n\n\n\n Developers chose solutions for equivalents of agents\u2019 problems in traditional software, e.g.<\/p>\n\n\n\n The traditional software solutions still fail for very agent-specific challenges given by the nature of LLMs. One example is debugging agents, which is essentially playing around with prompts, and the lack of an agent equivalent of real-time debugging<\/a>.<\/p>\n\n\n\n We have met with developers of agents like Grit<\/a> or Sweep<\/a>, who are either building their completely custom infrastructure <\/strong>or trying to use existing technologies to at least somehow fit their agent use-case. As mentioned by Swyx<\/a>, the infrastructure complement to multi-agent systems is agent clouds<\/strong>. E2B has built AI playgrounds<\/strong><\/a>, sandboxed cloud environments for agents or AI apps<\/strong>, that are especially useful for the coding use-case of agents.<\/p>\n\n\n\n There are more projects tailored for AI agents or LLM apps, most often frameworks for building, monitoring, and analytics.<\/p>\n\n\n\n As we\u2019re moving closer and closer to more advanced agents, the community is having discussions<\/strong><\/a> about establishing a common \u201cframework\u201d<\/strong> to help the agent ecosystem grow faster and simplify the work.<\/p>\n\n\n\n Particular questions include how to design realistic benchmarks<\/strong> for better evaluation of agents\u2019 performance, and also to incorporate safety considerations<\/strong>.<\/p>\n\n\n\n The benchmarking effor<\/strong>t<\/a> (a benchmarking tool for Agent Evals<\/a>) by AutoGPT<\/strong> originates from a need to truly understand the agent\u2019s ongoing processes and to determine whether the modifications made to an agent genuinely enhance its performance.<\/p>\n\n\n\n The biggest challenges with designing the agents\u2019 benchmarks are cost, time, and choosing the most optimal design of tests<\/strong>. There is a tradeoff between the diversity and uniqueness of the testing environment versus realism and naturality.<\/p>\n\n\n\n \u201cIf an agent fails a simple test, it won\u2019t pass the more difficult ones. Part of the challenge is hence structuring tests in the correct order\u201d said Silen Naihin, an R&D lead at AutoGPT, in the X space about agents benchmarking<\/a>.<\/p>\n\n\n\n Other benchmarking efforts:<\/p>\n\n\n\n The Agent Protocol<\/strong><\/a>, adopted in the AutoGPT benchmarks<\/a>, is a tech stack agnostic way to standardize and hence benchmark and compare AI agents<\/strong>.<\/p>\n\n\n\n It is an OpenAPI specification v3-based protocol \u2014 a list of endpoints<\/strong>, which the agent should expose with predefined response models, and defines an interface for interacting with your agent. Developers of LLM apps, such as AutoGPT, LemonAI, or BabyAGI <\/a>are currently adopting the protocol.<\/p>\n\n\n\n The protocol serves as a single communication interface with agents<\/strong>, making it also easier to develop developer tools that work with agents out of the box.<\/p>\n\n\n\n The hype where people experimented with the first open-source agent projects<\/strong> like AutoGPT<\/a> or BabyAGI<\/a> is starting to gradually calm down. End users are now looking to solve specific problems<\/strong>.<\/p>\n\n\n\n Agent use cases are being narrowed down to achieve perfection in one specific role<\/strong>. Today\u2019s most common use cases are coding, personal daily tasks, or research<\/strong><\/a>.<\/p>\n\n\n\n The future of software will likely include apps powered by dozens of \u201csmall\u201d AI agents serving specific purposes<\/strong> and interacting with each other. Agents will need their own secure cloud space to seamlessly communicate and conduct their tasks with autonomy.<\/p>\n\n\n\n We may expect a further shift towards a vertical market, for example, one app with different underlying agents <\/strong>designed for code writing, code debugging, code migration, e-mail communication, calendar planning, and task management.<\/p>\n\n\n\n To increase the ratio of returning users, developers focus on showcasing real tangible results<\/strong> and use cases instead of over-explaining how the agent works and why people should use it.<\/p>\n\n\n\n Sully Omar, the founder of Cognosys AI, enhances, how users care about tangible results, rather than underlying technology. \u201cFor example, offering users different models is redundant if they do not understand which is the most suitable for their needs.\u201d<\/a><\/p>\n\n\n\n A famous example of avoiding description of the technology itself<\/strong> is Apple, not mentioning \u201cAI\u201d at all<\/a> during an important presentation, or not mentioning \u201cmetaverse\u201d because \u201cthe average person doesn\u2019t know what it means<\/a>\u201d.<\/p>\n\n\n\n Agents still have a long way to enterprise-level reliability<\/strong>. There are still challenges to overcome with agent-specific SDKs, frameworks, and tools<\/a>. The biggest ones are debugging, monitoring, deployment, and benchmarking of agents. The Agent Protocol is one of the efforts to standardize agents and improve their communication and benchmarking.<\/p>\n\n\n\n The space switches from agents as a standalone code to \u201cagent as a feature\u201d<\/strong>, being part of a more complex product. Agent developers are focusing on more narrow use cases and learning to communicate better with end users.<\/p>\n\n\n\n The most common use cases of agent technology are coding, personal assistance with daily tasks, and search. We see that the future of software includes autonomous LLM agents.<\/p>\n\n\n\n For trying out autonomous agents, check out the overview of popular AI agents<\/a>.<\/p>\n\n\n\n Over the last few months, we have looked into around 100 agents with various use cases, studied SDKs and frameworks for agents, and discussed challenges faced by agents with founders of Cognosys, Aomni, Superagent, Sweep, and more. Here is what… Read more »<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[181,51],"tags":[27,331,330],"class_list":["post-892","post","type-post","status-publish","format-standard","hentry","category-ai-2","category-article","tag-ai","tag-ai-agents","tag-state-of-the-art"],"_links":{"self":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/892","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/comments?post=892"}],"version-history":[{"count":1,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/892\/revisions"}],"predecessor-version":[{"id":893,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/posts\/892\/revisions\/893"}],"wp:attachment":[{"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/media?parent=892"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/categories?post=892"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tbekk.com\/devstream\/wp-json\/wp\/v2\/tags?post=892"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}1. The space lacks consensus on the definition of an AI agent<\/h1>\n\n\n\n
\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*Kh2CDnBXx-ThG0iu.png\")
2. Agents switch from a standalone product to an \u201cinvisible\u201d feature<\/h1>\n\n\n\n
3. Agents still have a long way to enterprise-level reliability<\/h1>\n\n\n\n
4. Agents are in need of specific SDKs and frameworks<\/h1>\n\n\n\n
Existing \u201ctraditional software\u201d solutions<\/h2>\n\n\n\n
\n
Agent-specific solutions<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*D2T6WQHCedWhCs2x.png\")
5. The community is looking for standards for autonomous agents<\/h1>\n\n\n\n
Benchmarking<\/h1>\n\n\n\n
\n
The Agent Protocol<\/h1>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*28Ein570kAZ8HW9w.png\")
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:692\/0*-AK5gR4mboRkPiLt.png\")
6. Agents are moving in the vertical direction<\/h1>\n\n\n\n
Communication with end users<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*J8jpe6KnR7c9v8mt.png\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*dDzLDIww7HnabBb3.png\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*jZtesdI1vTMeUeqN.png\")
Conclusion<\/h1>\n\n\n\n
![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*EDt_Z9-495ogoUjO.png\")