# Evaluating Agentic AI: DeepEval, RAGAS & TruLens Frameworks Compared

In this episode of Memriq Inference Digest - Leadership Edition, we unpack the critical frameworks for evaluating large language models embedded in agentic AI systems. Leaders navigating AI strategy will learn how DeepEval, RAGAS, and TruLens provide complementary approaches to ensure AI agents perform reliably from development through production.

In this episode:

- Discover how DeepEval’s 50+ metrics enable comprehensive multi-step agent testing and CI/CD integration

- Explore RAGAS’s revolutionary synthetic test generation using knowledge graphs to accelerate retrieval evaluation by 90%

- Understand TruLens’s production monitoring capabilities powered by Snowflake integration and the RAG Triad framework

- Compare strategic strengths, limitations, and ideal use cases for each evaluation framework

- Hear real-world examples across industries showing how these tools improve AI reliability and speed

- Learn practical steps for leaders to adopt and combine these frameworks to maximize ROI and minimize risk

Key Tools & Technologies Mentioned:

- DeepEval

- RAGAS

- TruLens

- Retrieval Augmented Generation (RAG)

- Snowflake

- OpenTelemetry

Timestamps:

0:00 Intro & Why LLM Evaluation Matters

3:30 DeepEval’s Metrics & CI/CD Integration

6:50 RAGAS & Synthetic Test Generation

10:30 TruLens & Production Monitoring

13:40 Comparing Frameworks Head-to-Head

16:00 Real-World Use Cases & Industry Examples

18:30 Strategic Recommendations for Leaders

20:00 Closing & Resources

Resources:

- Book: "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

Discover how the Model Context Protocol (MCP) is revolutionizing AI system integration by simplifying complex connections between AI models and external tools. This episode breaks down the technical and strategic impact of MCP, its rapid adoption by industry giants, and what it means for your AI strategy.

In this episode:

- Understand the M×N integration problem and how MCP reduces it to M+N, enabling seamless interoperability

- Explore the core components and architecture of MCP, including security features and protocol design

- Compare MCP with other AI integration methods like OpenAI Function Calling and LangChain

- Hear real-world results from companies like Block, Atlassian, and Twilio leveraging MCP to boost efficiency

- Discuss the current challenges and risks, including security vulnerabilities and operational overhead

- Get practical adoption advice and leadership insights to future-proof your AI investments

Key tools & technologies mentioned:

- Model Context Protocol (MCP)

- OpenAI Function Calling

- LangChain

- OAuth 2.1 with PKCE

- JSON-RPC 2.0

- MCP SDKs (TypeScript, Python, C#, Go, Java, Kotlin)

Timestamps:

0:00 - Introduction to MCP and why it matters

3:30 - The M×N integration problem solved by MCP

6:00 - Why MCP adoption is accelerating now

8:15 - MCP architecture and core building blocks

11:00 - Comparing MCP with alternative integration approaches

13:30 - How MCP works under the hood

16:00 - Business impact and real-world case studies

18:30 - Security challenges and operational risks

21:00 - Practical advice for MCP adoption

23:30 - Final thoughts and strategic takeaways

Resources:

• "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
• This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

In this episode of Memriq Inference Digest - Leadership Edition, we explore how Retrieval-Augmented Generation (RAG) systems maintain quality and trust at scale through advanced evaluation methods. Join Morgan, Casey, and special guest Keith Bourne as they unpack the game-changing RAGAS framework and the emerging practice of reference-free evaluation that enables AI to self-verify without costly human labeling.

In this episode:

- Understand the limitations of traditional evaluation metrics and why RAG demands new approaches

- Discover how RAGAS breaks down AI answers into atomic fact checks using large language models

- Hear insights from Keith Bourne’s interview with Shahul Es, co-founder of RAGAS

- Compare popular evaluation tools: RAGAS, DeepEval, and TruLens, and learn when to use each

- Explore real-world enterprise adoption and integration strategies

- Discuss challenges like LLM bias, domain expertise gaps, and multi-hop reasoning evaluation

Key tools and technologies mentioned:

- RAGAS (Retrieval Augmented Generation Assessment System)

- DeepEval

- TruLens

- LangSmith

- LlamaIndex

- LangFuse

- Arize Phoenix

Timestamps:

0:00 - Introduction and episode overview

2:30 - What is Retrieval-Augmented Generation (RAG)?

5:15 - Why traditional metrics fall short for RAG evaluation

7:45 - RAGAS framework and reference-free evaluation explained

11:00 - Interview highlights with Shahul Es, CTO of RAGAS

13:30 - Comparing RAGAS, DeepEval, and TruLens tools

16:00 - Enterprise use cases and integration patterns

18:30 - Challenges and limitations of LLM self-evaluation

20:00 - Closing thoughts and next steps

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Visit Memriq AI at https://Memriq.ai for more AI engineering deep-dives, guides, and research breakdowns

Thanks for tuning in to Memriq AI Inference Digest - Leadership Edition. Stay ahead in AI leadership by integrating continuous evaluation into your AI product strategy.

Is agent engineering the next big AI discipline or a repackaged buzzword? In this episode, we cut through the hype to explore what agent engineering really means for business leaders navigating AI adoption. From market growth and real-world impact to the critical role of AI memory and the evolving tool landscape, we provide a clear-eyed view to help you make strategic decisions.

In this episode:

- The paradox of booming agent engineering markets despite high AI failure rates

- Why agent engineering is emerging now and what business problems it solves

- The essential role of AI memory systems and knowledge graphs for real impact

- Comparing agent engineering frameworks and when to hire agent engineers vs ML engineers

- Real-world success stories and measurable business payoffs

- Risks, challenges, and open problems leaders must manage

Key tools and technologies mentioned: LangChain, LangMem, Mem0, Zep, Memobase, Microsoft AutoGen, Semantic Kernel, CrewAI, OpenAI GPT-4, Anthropic Claude, Google Gemini, Pinecone, Weaviate, Chroma, DeepEval, LangSmith

Timestamps:

00:00 – Introduction & Why Agent Engineering Matters

03:45 – Market Overview & The Paradox of AI Agent Performance

07:30 – Why Now: Technology and Talent Trends Driving Adoption

11:15 – The Big Picture: Managing AI Unpredictability

14:00 – The Memory Imperative: Transforming AI Agents

17:00 – Knowledge Graphs & Domain Expertise

19:30 – Framework Landscape & When to Hire Agent Engineers

22:45 – How Agent Engineering Works: A Simplified View

26:00 – Real-World Payoffs & Business Impact

29:15 – Reality Check: Risks and Limitations

32:30 – Agent Engineering In the Wild: Industry Use Cases

35:00 – Tech Battle: Agent Engineers vs ML Engineers

38:00 – Toolbox for Leaders: Strategic Considerations

41:00 – Book Spotlight & Sponsor Message

43:00 – Open Problems & Future Outlook

45:00 – Final Words & Closing Remarks

Resources:

• "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
• This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

Is your AI initiative falling short despite the hype? The root cause often lies not in the AI technology itself but in how your architecture handles the Natural Language Understanding (NLU) layer. In this episode, we explore why treating AI as a bolt-on feature leads to failure and what leadership must do to embrace the fundamental paradigm shift required for success.

In this episode, you'll learn:

- Why legacy deterministic web app architectures break when faced with conversational AI

- The critical role of the NLU layer as the "brain" driving dynamic, user-led interactions

- How multi-intent queries, partial understanding, and fallback strategies redefine system design

- The importance of AI-centric orchestration bridging probabilistic AI reasoning with deterministic backend execution

- Practical architectural patterns like the 99-intents fallback and context management to improve reliability

- How to turn unsupported user requests into upsell and engagement opportunities

Key tools and technologies mentioned include Large Language Models (LLMs), function-calling APIs, AI orchestration layers, and frameworks from thought leaders like Keith Bourne, Ivan Westerhof, and Sunil Ramlochan.

Timestamps:

0:00 - Introduction & Why AI Projects Fail

3:30 - The NLU Paradigm Shift Explained

7:15 - User Perspective vs. System Reality

10:20 - Handling Multi-Intent & Partial Understanding

13:10 - Architecting Fallbacks & Out-of-Scope Requests

16:00 - Business Impact & ROI of Robust NLU Architectures

18:30 - Closing Thoughts & Leadership Takeaways

Resources:

• "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
• This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

Unlock how AI is evolving beyond static models into adaptive experts with integrated memories. In the previous 3 episodes, we secretly built up what amounts to a 4-part series on agentic memory. This is the final piece of that 4-part series that pulls it ALL together.

In this episode, we unpack Chapter 19 of Keith Bourne's 'Unlocking Data with Generative AI and RAG,' exploring how advanced Retrieval-Augmented Generation (RAG) leverages episodic, semantic, and procedural memory types to create continuously learning AI agents that drive business value.

This also concludes our book series, highlighting ALL of the chapters of the 2nd edition of "Unlocking Data with Generative AI and RAG" by Keith Bourne. If you want to dive even deeper into these topics and even try out extensive code labs, search for 'Keith Bourne' on Amazon and grab the 2nd edition today!

In this episode:

- What advanced RAG with complete memory integration means for AI strategy

- The role of LangMem and the CoALA Agent Framework in adaptive learning

- Comparing learning algorithms: prompt_memory, gradient, and metaprompt

- Real-world applications across finance, healthcare, education, and customer service

- Key risks and challenges in deploying continuously learning AI

- Practical leadership advice for scaling and monitoring adaptive AI systems

Key tools & technologies mentioned:

- LangMem memory management system

- CoALA Agent Framework

- Learning algorithms: prompt_memory, gradient, metaprompt

Timestamps:

0:00 – Introduction and episode overview

2:15 – The promise of advanced RAG with memory integration

5:30 – Why continuous learning matters now

8:00 – Core architecture: Episodic, Semantic, Procedural memories

11:00 – Learning algorithms head-to-head

14:00 – Under the hood: How memories and feedback loops work

16:30 – Real-world use cases and business impact

18:30 – Risks, challenges, and leadership considerations

20:00 – Closing thoughts and next steps

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Visit Memriq.ai for AI insights, guides, and tools

Thanks for tuning in to Memriq Inference Digest - Leadership Edition.

Unlock how procedural memory transforms Retrieval-Augmented Generation (RAG) systems from static responders into autonomous, self-improving AI agents. Join hosts Morgan and Casey with special guest Keith Bourne as they unpack the concepts behind LangMem and explore why this innovation is a game-changer for business leaders.

In this episode:

- Understand what procedural memory means in AI and why it matters now

- Explore how LangMem uses hierarchical scopes and feedback loops to enable continuous learning

- Discuss real-world applications in finance, healthcare, and customer service

- Compare procedural memory with traditional and memory-enhanced RAG approaches

- Learn about risks, governance, and success metrics critical for deployment

- Hear practical leadership tips for adopting procedural memory-enabled AI

Key tools & technologies mentioned:

- LangMem procedural memory system

- LangChain AI orchestration framework

- CoALA modular architecture

- OpenAI's GPT models

Timestamps:

0:00 - Introduction and episode overview

2:30 - What is procedural memory and why it’s a breakthrough

5:45 - The self-healing AI concept and LangMem’s hierarchical design

9:15 - Comparing procedural memory with traditional RAG systems

12:00 - How LangMem works under the hood: feedback loops and success metrics

15:30 - Real-world use cases and business impact

18:00 - Challenges, risks, and governance best practices

19:45 - Final thoughts and next steps for leaders

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Visit Memriq.ai for more AI insights, tools, and resources

Unlock how RAG-based agentic memory is transforming AI from forgetful chatbots into intelligent assistants that remember and adapt. In this episode, we break down the core concepts from Chapter 17 of Keith Bourne’s “Unlocking Data with Generative AI and RAG,” exploring why memory-enabled AI is a game changer for customer experience and operational efficiency.

In this episode, you’ll learn:

- What agentic memory means in AI and why it matters for leadership strategy

- The difference between episodic and semantic memory and how they combine

- Key tools like CoALA, LangChain, and ChromaDB that enable memory-enabled AI

- Real-world applications driving business value across industries

- The trade-offs and governance challenges leaders must consider

- Actionable tips for adopting RAG-based memory systems today

Key tools and technologies: CoALA, LangChain, ChromaDB, GPT-4, vector embeddings

Timestamps:

00:00 – Introduction and overview

02:30 – The AI memory revolution: episodic and semantic memory explained

07:15 – Why now: Technology advances driving adoption

10:00 – Comparing memory approaches: stateless vs episodic vs combined

13:30 – Under the hood: architecture and workflow orchestration

16:00 – Real-world impact and business benefits

18:00 – Risks, challenges, and governance

19:30 – Practical leadership takeaways and closing

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Memriq.ai – Tools and resources for AI practitioners and leaders

Thanks for listening to Memriq Inference Digest - Leadership Edition.