I have been building and operating Agentic AI Systems for the past few years and the same patterns keep emerging. 👇

𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗗𝗿𝗶𝘃𝗲𝗻 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 is the most reliable way to be successful in building your 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 and continue improving them - here is my template.

Let’s zoom in:

𝟭. Define a problem you want to solve: is GenAI even needed?

𝟮. Build a Prototype: figure out if the solution is feasible.

𝟯. Define Performance Metrics: you must have output metrics defined for how you will measure success of your application.

𝟰. Define Evals: split the above into smaller input metrics that can move the key metrics forward. Decompose them into tasks that could be automated and move the given input metrics. Define Evals for each. Store the Evals in your Observability Platform.

ℹ️ Steps 𝟭. - 𝟰. are where AI Product Managers can help, but can also be handled by AI Engineers.

𝟱. Build a PoC: it can be simple (excel sheet) or more complex (user facing UI). Regardless of what it is, expose it to the users for feedback as soon as possible.

𝟲. Instrument your application: gather traces and human feedback and store it in an Observability Platform next to previously stored Evals.

𝟳. Run Evals on traced data: traces contain inputs and outputs of your application, run evals on top of them.

𝟴. Analyse Failing Evals and negative user feedback: this data is gold as it specifically pinpoints where the Agentic System needs improvement.

𝟵. Use data from the previous step to improve your application - prompt engineer, improve AI system topology, finetune models etc. Make sure that the changes move Evals into the right direction.

𝟭𝟬. Build and expose the improved application to the users.

𝟭𝟭. Monitor the application in production: this comes out of the box - you have implemented evaluations and traces for development purposes, they can be reused for monitoring. Configure specific alerting thresholds and enjoy the peace of mind.

✅ 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗼𝗳 𝘆𝗼𝘂𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻:

➡️ Run steps 𝟲. - 𝟭𝟬. to continuously improve and evolve your application.

➡️ As you build up in complexity, new requirements can be added to the same application, this includes running steps 𝟭. - 𝟱. and attaching the new logic as routes to your Agentic System.

➡️ You start off with a simple Chatbot and add a route that can classify user intent to take action (e.g. add items to a shopping cart).

Learn all of the practices of Eval Driven Development Hands-on in my End-to-end AI Engineering Bootcamp:

🎁 Grab your 15% discount by applying code KICKOFF15 at the check-out.

What is your experience in evolving Agentic Systems? Let me know in the comments 👇