How to integrate an LLM with memory: store the facts yourself

LLMs reason very well. Their memory is poor.

Ask an AI assistant about something you mentioned earlier in a long dialogue — and it might get confused, mix up details, or just start to drift, inventing facts. The longer the context, the less predictable the answers. You start probing it, but facts drift further and further away. You’d think huge context windows would help, but no! And if you build a project that includes an LLM, this becomes dangerous — agree, when a support assistant starts glitching, it doesn’t do your reputation any favors. So what to do?

The solution is simple: don’t trust the model’s memory. Store the facts yourself.

How it works

User data — profile, history, any structured facts — live in a database. Yes, a boring SQL or NoSQL database. When a request comes in, the necessary data are retrieved and sent into the context window together with the prompt. The model always sees exactly what it needs — no more, no less.

What this gives

Full control over the context. You decide what the model knows at each step. No need to worry whether it will remember something from the session before last, mix up users, or give an answer that contradicts what was said earlier.

The same applies to conversation history. Instead of feeding the model the entire chat, store messages in your database and load only the last few — that’s enough for the model to understand the conversation context. If the task gets more complex, you can go further and set up RAG search over the history. But honestly, for most cases “the last N messages from the table” deliver 90% of the result — without a dissertation on vector databases.

And a nice bonus: you send only relevant data. Fewer tokens per request — a noticeable saving at any scale.

The main principle

An LLM is an engine for reasoning, not a data storage system. Once you separate these two responsibilities, the model starts working noticeably better. Not because the model changed — but because it works with quality input.

👉 Garbage in, garbage out. Clean structured data in — surprisingly good AI out.