Instacart Tackles ‘Brownie Recipe Problem’ with Advanced AI Solutions

Large Language Models (LLMs) are making strides in reasoning capabilities, but they often struggle to deliver contextually relevant results in real-time scenarios, particularly in grocery delivery services such as Instacart. At a recent event, Instacart’s Chief Technology Officer, Anirban Kundu, highlighted this challenge, referring to it as the “brownie recipe problem.” Simply asking an LLM for a brownie recipe is insufficient; the system must understand the user’s local preferences and the availability of ingredients in their area to provide useful assistance.

To create a seamless shopping experience, LLMs must account for various factors, including the user’s dietary preferences, such as choosing between organic and regular eggs, and ensuring that the selected items can be delivered without spoilage. For Instacart, the goal is to balance quick response times with accurate contextual understanding, ideally delivering results in under one second. Kundu remarked, “If reasoning itself takes 15 seconds, and if every interaction is that slow, you’re gonna lose the user.”

Understanding User Intent and Market Availability

Kundu emphasized the necessity for LLMs to navigate both a “world of reasoning” and a “world of state,” which includes real-time product availability and user preferences. This complex process cannot merely rely on a vast repository of a user’s purchase history, as it would lead to an unmanageable system. Instead, Instacart employs a method where data is processed through a foundational model tasked with intent recognition and product categorization.

Subsequently, this information is transferred to smaller language models (SLMs) that focus on catalog context and semantic understanding. For instance, if a shopper wishes to buy healthy snacks for children, the SLM must evaluate what constitutes a healthy option and identify products that appeal to an eight-year-old. If certain items are out of stock, the model must recommend suitable alternatives.

Kundu noted that such substitutions are critical, particularly for a company like Instacart, which experiences a significant number of cases where requested products are unavailable. He stated that they encounter “over double digit cases” of product unavailability in local markets.

Logistical considerations also play a crucial role in this process. Products like ice cream require specific handling to prevent melting, while frozen vegetables need to remain at consistent temperatures. The model must factor in deliverability time to ensure that products arrive in good condition.

Adopting a Modular Approach to AI Integration

Instacart is exploring the use of AI agents, discovering that a modular approach is more effective than relying on a single, monolithic system. This strategy aligns with the Unix philosophy of utilizing smaller, specialized tools, which allows for better adaptability in managing various payment systems and their different failure modes. Kundu explained, “Having to build all of that within a single environment was very unwieldy.”

To facilitate communication between their AI agents and third-party platforms, Instacart has integrated with OpenAI’s Model Context Protocol (MCP) and Google’s Universal Commerce Protocol (UCP). These systems enable AI agents to interact directly with merchant systems, although the integration process is not without complications. Kundu pointed out that the challenges often stem from how reliably different integrations function and how well users understand them.

The implementation of MCP and UCP has been approached in “very different” ways, with significant attention given to failure modes and latency issues. Kundu remarked, “The response times and understandings of both of those services are very, very different. I would say we spend probably two-thirds of the time fixing those error cases.”

As Instacart continues to refine its AI capabilities, addressing the “brownie recipe problem” remains a priority. The company’s commitment to enhancing user experience through sophisticated technology underscores the evolving landscape of grocery delivery in a digital-first world.