In Let's Ask Ilya Episode 151, can AI forget without performing worse?
Issue #151 covers AI memory, tool recovery, unlearning, feedback, and topology: what systems should keep, restore, or forget without harm.
Key Takeaways
- AI memory should be treated as an active selection problem: recency, relevance, and recurring patterns can help older context become useful again, while over-deletion can harm reasoning.
- Machine unlearning is difficult to verify because models may stop producing unwanted content without actually removing the underlying knowledge, and forgetting can damage downstream decisions if useful history is removed.
- Recovering from failed tool calls often depends more on the information returned by the runtime than on the model’s raw capability; useful error messages, validation, limits, and defaults can enable correction.
- Tool topology shapes how much control remains with the agent versus the application, from simple tool calls to parallel, sequential, and graph-based workflows with branching and approvals.
- Post-training and LLM judging require reliable feedback signals and bias controls, since judges can favor their own models and production outcomes need careful evaluation.
Meta description: Learn AI Together #151 covers AI memory, tool recovery, post-training feedback, tool topology, and machine unlearning in a practical weekly update.
Tags: AI agent memory, tool recovery, machine unlearning, post-training feedback, LLM judges
Featured image: abstract, friendly illustration of AI memory, tool recovery, and machine unlearning; no real people, no logos, no text.
For developers working on AI agent memory, tool recovery, and machine unlearning, Learn AI Together issue #151, published this week, turns a set of experiments, community projects, and product updates into one practical question: how should AI systems decide what to keep, what to restore, and what to forget? The issue's central finding is that memory is not just storage. Systems accumulate conversation history, retrieved facts, user preferences, tool results, training feedback, and instructions. Keeping all of that can be expensive or noisy, but deleting too much can remove context that becomes useful again, and a failed tool call may contain exactly the information an agent needs to repair its next attempt.
Memory, forgetting, and the risk of damaging reasoning
The issue's article of the week, Hayanan's "Tidal Attention: A Moon-Cycle-Inspired Mechanism for Long-Context Memory," proposes a memory-selection method that combines recency, semantic relevance, and recurring patterns. The goal is to let older information return when it becomes useful again. The article links the idea to RetNet, Mamba, LongNet, Infini-attention, and Titans, then tests the policy with a simple cache simulation. In that simulation, the tidal policy beat a least-recently-used cache, though it did not match frequency-based caching. The author also describes an evaluation plan with ablations to check whether recurrence truly improves long-context memory.
Hayanan's "Machines That Forget on Purpose" adds the harder side of the problem. A model can stop producing memorized content without actually deleting the underlying knowledge, which makes machine unlearning difficult to evaluate. The article reviews TOFU, selective disentanglement, and R-TOFU, and points to research showing that some forgetting methods can also damage a model's ability to reason from newly provided evidence. A separate 240-case policy-retrieval experiment found that removing outdated records improved decisions, while removing useful history made them worse. From these results, the article separates forgetting into three layers: stored evidence, model behavior, and downstream decisions. Each layer needs to be tested independently.
Tool recovery depends on what the runtime returns
The article "Make Your AI Agent Recover When a Tool Call Fails" by Satsawat Natakarnkitkul, also known as Net, shows why failed calls should not be treated as dead ends. In a ticket-handling example, the OpenAI Agents SDK returned an error without telling the model which priority codes the server accepted. The mistake was recoverable, but the missing information prevented correction. The author tested eight frameworks and six models, then implemented six recovery patterns: useful error messages, sanitized exceptions, stored-value validation, turn and budget limits, repeated-call detection, and pinned defaults. When the approved error information was added, both hosted models corrected every failed ticket call. The result points to a practical lesson: recovery often depends less on the model's raw ability and more on the information the harness returns.
"Designing Tool Topologies for AI Agents" by Deepak Kumar moves from single calls to the overall shape of tool use. The article argues that the way an agent is connected to tools determines which decisions the model controls and which decisions the application controls. It uses a money-transfer example to compare four patterns: individual tool calls, parallel execution with asyncio, sequential chains with LangChain's LCEL, and LangGraph workflows that include branching, merging, and human approval. The code examples show how each pattern becomes more useful as workflow complexity grows.
Feedback design, judge bias, and prompt maintenance
Vikram Kharvi's "LLM Post-Training: Feedback Design and Algorithm Selection" starts from a simple rule: post-training works only when feedback reliably separates better behavior from worse behavior. The article maps SFT, DPO, KTO, PPO, and GRPO to the training data and feedback each method needs, then explains the differences between RLHF, RLAIF, and RLVR. It also covers rubric-based rewards, feedback collected from production traces, dataset versioning, and a formula for comparing training cost with expected gains. The evaluation guidance includes held-out tests, regression slices, and matched baselines to check whether optimization actually transferred to real workloads.
The What's AI section adds a caution about LLM judges. The team built ToneBench and then tested whether the judges behind it could be trusted. Every model writes five drafts for each of ten tasks because the same model can produce very different results from the same prompt. Four judges score every draft across tone, substance, structure, hooks, slop, and other dimensions. When the judges themselves were measured, the results showed clear lab preferences. On the October 26 board, GPT-6.1 Sol gave OpenAI drafts 4.1 more points than the other LLM judges' scores would predict, and Claude Opus 5.5 gave Anthropic drafts a smaller boost. The full article explains the benchmark setup, why multiple drafts are used, how one script is scored, how the judges were calibrated against human ratings, and how the results could guide writing now. It also shares a practical idea under testing: generate several cheap drafts with a smaller model, then use a stronger judge to select and merge the best parts instead of paying for one expensive generation. The full explanation is also available as a video on YouTube.
Louis-François Bouchard, Towards AI Co-founder and Head of Community, contributes an AI tip from building Brown, the writing agent in the Agent Engineering course. Brown began with one prompt containing all writing rules. That was workable for one format. Once it had to write articles, video scripts, and social posts for different writers, every edit became riskier: a rule added for scripts could affect articles, and changing one writer's voice could touch instructions unrelated to voice. The solution was to split the prompt by how often each part needed to change. Four profile files stay fixed across tasks: mechanics, structure, terms, and tone. Two are loaded dynamically: the format profile and the writer's voice. Adding a video format now means changing one file. Adding a new writer means adding one voice profile. The shared writing rules stay untouched. The tip is to map instructions into stable and variable groups, pull out one variable group, run the same evaluation set before and after, and keep the split only if quality holds while the system becomes easier to change.
Infrastructure and community projects around AI memory
The issue is brought to readers by MongoDB, which frames persistent memory as a decision problem. A user preference may need to last across sessions, while a temporary plan or intermediate result may only matter until the task is finished. Keeping everything adds noise, and deleting too much makes the agent forget useful context. MongoDB's new Vercel AI SDK integration adds multi-tiered memory in Atlas, allowing each memory type to follow its own retention policy. It can also automatically create the vector indexes needed for retrieval. That lets builders separate short-lived context from facts and workflows the agent may need later. The integration is open source, supports AI SDK v6 and v7, and is available on npm.
The Learn AI Together Discord community is flooding with collaboration opportunities. Builders excited to dive into applied AI, want a study partner, or want a partner for a passion project can join the collaboration channel; the section shares cool opportunities every week.
The featured community post comes from __pop1, who built AI-Atlas, a project that evaluates public GitHub repositories to show what developers have actually built across machine learning, computer vision, natural language processing, LLMs, and AI agents. Each project is scored with a seven-part rubric covering technical depth, originality, completeness, reproducibility, and other engineering qualities, with the evaluation linked back to the source code. Developers can claim their profiles and receive a step-by-step roadmap for improving each project. The tool has already mapped 218 developers and analyzed 2,216 open-source AI repositories. A user can map a GitHub profile for free without creating an account. People working in machine learning research, open-source AI, or developer tooling are invited to share feedback with the creator in the thread.
The collaboration section highlights three open calls. Mtnmcg is working on an alternative machine learning architecture that outperforms transformers and is looking for people who can review it, share feedback, and help with the project; those interested can contact him in the thread. Bee_protego is building a tool that helps identify which words in AI prompts actually matter and is looking for people with an AI background to test and improve it; those who think they can help can reach out in the thread. Carlos1256 is studying toward becoming a machine learning engineer and focusing on software engineering fundamentals, and needs a study partner with the same goal; others can connect with them in the thread. The issue also includes the meme of the week, shared by jacekde.
The practical impact for builders is that memory, tool errors, and post-training feedback are not separate chores. They are linked by the same question: what information should influence the next decision? A retrieval system needs retention policies, a tool harness needs useful error signals, a training pipeline needs feedback that survives evaluation, and a judge needs calibration. Learn AI Together #151: Can AI Forget Without Getting Worse? was originally published in Towards AI on Medium, where readers continue the conversation by highlighting and responding to the story. Teams interested in publishing with Towards AI can check the guidelines and sign up; work is published to the network if it meets the outlet's editorial policies and standards.

Comments (0)