The real secret sauce for getting AI to actually work for you isn’t what you say to it. It’s what you let it see.
Story time. Let’s grab a coffee and break this down.
I remember our team needed an AI agent to help parse some complex graph data models and map out our automated compliance workflows.
So, we did what we all do. Wrote a massive, intricate prompt, threw in a backstory, step-by-step instructions, three standard operating procedures, and a dozen examples.
Hit enter, expecting magic.
The result? Utter garbage.
The LLM didn’t just hallucinate; it completely forgot the core constraints we gave it in paragraph two.
The team spent the next hour tweaking words — changing “act as a data analyst” to “you are an expert business analyst” — hoping to unlock the secret code.
The prompt was never the problem.
Welcome to context engineering.
The big shift: why your prompts keep breaking
We aren’t just building simple chatbots anymore. We are building systems, low-code automations, and AI agents that run in continuous loops.
When you try to cram every single rule, operating procedure, and data point into one giant prompt, the AI gets overwhelmed. Neo4j describes this as “context rot”: every piece of text you feed an AI takes up its attention, and when the context window gets too big and cluttered, the model loses focus, gets distracted, and forgets the important stuff — just like a human would.
Think of it this way:
Context > Prompts
- Prompt engineering is about what you say. It’s like handing a chef a very detailed recipe.
- Context engineering is about what the model sees. It’s like organizing the chef’s kitchen. It’s making sure the exact ingredients they need are placed on the counter right before they chop them, and clearing away the trash so they don’t get distracted.
How to think like a context engineer
So how do we actually do this? Anthropic’s own engineering team has written directly about this, and practitioners like Addy Osmani and Akshay Pachaar have been making the same case from different angles. Here’s the thought process worth borrowing when an AI needs to tackle a heavy lift without losing its mind.
1. Stop dumping. Start curating.
It’s tempting to dump entire project strategy documents into the chat window. The better approach is to isolate the data — give the AI only the exact puzzle pieces it needs for the specific step it’s working on right now. Make every single token count.
2. Fetch “just-in-time” information
Instead of loading the AI up with historical data it might need, set up dynamic retrieval. Let the AI fetch what it needs, exactly when it needs it. As Clara Chong has written, well-managed context is what lets an agent reason and stay coherent across long, complex tasks — left unchecked, it grows, drifts, and eventually collapses under its own weight. Performance isn’t about how much context you give a model; it’s about how precisely you shape it.
3. Compress the past
If the AI and you have been going back and forth for 20 messages, the context window is getting bloated with old ideas. Periodically ask the model to summarize progress into a single paragraph, start a fresh chat, and use that summary as the new starting point. Clean kitchen, clear mind.
The bottom line: manage the memory, not just the words
We need to stop obsessing over the perfect phrasing. The AI understands English just fine. Where it struggles is managing its own working memory over time.
Context engineering is the definitive next step for anyone looking to build serious digital tools, automations, or just get a reliable output from a language model without wanting to throw their laptop out a window.
Next time you find yourself tweaking a prompt for the fifth time, stop typing. Zoom out. Look at the kitchen you’ve built for your AI.
Is it full of garbage? Clean it up. Give it only what it needs.
Sources and further reading
- Anthropic: Effective context engineering for AI agents
- Neo4j: What is context engineering in AI agents?
- Addy Osmani (via O’Reilly Radar): Context Engineering — Bringing Engineering Discipline to Prompts
- Akshay Pachaar, Daily Dose of Data Science: writing on context engineering and LLMOps
- Clara Chong, Towards Data Science: Context Engineering for AI Agents u2014 A Deep Dive
Related reading: Your AI Policy Is Not Your AI Inventory: Finding Shadow AI · From Process Maps to AI Opportunity Maps
