Building Agentic Memory with Databricks Lakebase

Meta description: Databricks announced a fully managed Lakebase beta for agentic memory in late September 2026, moving from Unity Catalog to Lakebase as its managed store path.

Tags: #AgenticMemory #DatabricksLakebase #AIInfrastructure #ProductionAgents

Featured image: abstract friendly illustration of layered memory stores, session threads, user identities, and search retrieval, with no real people, no logos, and no text.

For agent developers and platform teams building assistants that need to remember users, Databricks Lakebase managed agent memory is the news: Databricks announced Fully Managed Lakebase (Beta) in late September 2026, moving its managed memory offering from the legacy Unity Catalog path to a Lakebase service. LLMs are stateless, so agents need external memory to recall facts and context from prior conversations. Databricks says the new service handles storage, indexing, and search; Self Managed Lakebase and Fully Managed Unity Catalog remain legacy.

From Postgres schemas to a managed store

Most agentic platforms offer memory services, including AWS Agentcore Memory and GCP Vertex AI Memory Bank. Databricks' suite changed significantly as of Sep 2026. Self Managed Lakebase (Legacy) uses a Postgres database as a self-managed memory store: builders define and own the schema, have fine-grained permissions, and store memory fields as traditional SQL data types. Fully Managed Unity Catalog (Legacy) is a Databricks-managed store aligned with Agentcore Memory and Vertex Memory Bank, with Databricks running infrastructure, isolating each user's memories, and applying Unity Catalog governance and access controls.

Fully Managed Lakebase (Beta), announced in late September, is the newest Databricks managed memory store. Previously teams had to manually define a traditional Postgres schema to store memory in Lakebase postgres. Now they set a few inputs and Databricks defines the memory store container, similar to AWS and Google managed services.

Identifiers decide isolation

Memory organization determines isolation and retrieval efficiency. The key identifiers are session ID, thread ID, user ID, and actor ID. Definitions vary by framework, but short-term memory is scoped to individual sessions while long-term memory is scoped to individual users.

A session ID represents one single user interaction, the equivalent of one sitting in front of an app; opening the app gives a new session ID, and returning tomorrow gives another. Sessions usually map to threads, often thread_id = session_id. Thread ID is used in LangGraph, whose checkpointer uses it to save and reload conversation state. Reusing the same thread ID lets the agent replay running history of messages and tool calls; a new thread ID starts blank. Session and thread often line up one to one.

User ID is an identity concept requiring an Identity Provider; it identifies a person and persists information across every thread and session. Thread/session IDs split memory horizontally by conversation, while user IDs split it vertically by owner. A thread-scoped short-term field cannot carry information across conversations; long-term memory keys on the user.

Inside a Databricks managed memory store, each entry has free-form text, a retrieval-helping description, and three locating fields: actor_id + session_id + path. actor_id is the user, session_id is the source session, and path is a custom file-like name identifying the container.

Three memory layers

Agentic memory has three layers. The most ephemeral is the in-memory checkpointer, or in-process session persistence. Many frameworks expose it. LangGraph's checkpointer snapshots the graph's state after every step, meaning each node execution. AWS Strands uses Sessions Management, and Claude Agents SDK utilizes session transcript. These snapshots preserve message and tool-call history so an agent can pause, resume, retry a failed step, or rewind and reuse the same identifier: thread_id in LangGraph, session_id in Strands and Claude Agent SDK. The data lives in RAM, fine for local development but untenable for production; a durable backend survives restarts.

Short-term memory, in Databricks Managed Sessions, is the agent's in-process state within one conversation persisted to an external durable store rather than held only in RAM. It is scoped to a single thread_id/session_id, exactly what an in-memory checkpointer captures. Frameworks typically persist in-memory sessions to external short-term memory.

Long-term memory, in Databricks Managed Memory, is keyed by user identity rather than conversation. It lets facts outlive any one conversation by extracting information, preferences, and past decisions under the user's identity, so a new thread tomorrow can recall them. Short-term memory remembers this conversation; long-term memory remembers this user.

To scope memory to a user, the system needs an Identity Provider such as Okta or AWS Cognito; the user authenticates, and the resulting OAuth / on behalf of (OBO) token carries a verified end-user identity.

Extraction and retrieval need design work

Writing memory can be a simple call. The memory_store.add call may use actor_id="user-123", path="/preferences/communication.md", content="Prefers email over phone. Timezone: PST. Enterprise subscription.", and description="User 123 communication preferences". The harder task is deciding what relevant information should be persisted: agents should extract long-term-relevant facts and ignore irrelevant small talk or one-off facts.

As released so far, Databricks managed memory ships no built-in extraction strategies; teams define custom strategies. AWS Agentcore Memory does include built-in strategies: Semantic pulls durable facts; Summary stores a session-scoped summary; User Preferences captures how the user likes things; Episodic, the most complex, records specific past events. In Agentcore, each strategy is an extraction prompt run over the raw event stream. Episodic memory captures what happened in an interaction, such as a decision made or a problem solved or approached, so the agent can recall it later.

With Databricks, developers implement each strategy. One approach is a tool call: give the agent a custom save_memory tool and let it decide when something is worth keeping, using standard tool frontmatter, descriptions, and skills. The tool's description and schema, including content, description, and path, tell the model when and how to call it; a skill can add guidance. Another is a distill step: a separate LLM call or system prompt over the entire interaction to pull durable facts and write them to the store.

Retrieval happens when a new interaction launches. The agent should pull relevant memory to generate personalized responses. Databricks uses natural language search: memory_store.list with actor_id="user-123" and path_prefix="/preferences/", then memory_store.search with actor_id="user-123", query="communication preferences", and limit=10. Databricks docs indicate this is “BM25 full-text relevance and returns a top-N set (up to 100 entries).” It is not standard vector similarity search in the same way AWS Agentcore memory utilizes. Because retrieved items are injected into the model's context window, prudent retrieval is required. Best practice is to run list first, return all memory entries, and search through those to determine entries needed for the current interaction.

What comes next

The switch from Unity Catalog to Lakebase changes operational assumptions. Managed Lakebase reduces schema work but shifts some control to Databricks; self-managed Postgres remains available for teams needing database-level ownership. The absence of built-in extraction strategies means memory quality depends on developer-defined tools, prompts, or distillation steps. Teams should decide which identifier owns each fact: thread/session for conversation state, user/actor for durable preferences. They should also monitor context-window cost, because top-N retrieval can return up to 100 entries.

Databricks' memory roadmap moved from a fully self-managed Lakebase Postgres DB to managed agentic memory using Unity Catalog, and as of Sep 2026 the managed offering switched from Unity Catalog to Lakebase. The underlying article was originally published in Towards AI on Medium.