I Built a Second Brain That Actually Works
I kept losing context. So I made the AI maintain it for me
Last week Andrej Karpathy posted a gist about using an LLM to maintain a personal wiki. It went viral. 17 million views and counting.
I read it and thought: that’s exactly what I’ve been doing!
For the past several months I've been running my knowledge system this way. Not as an experiment. As actual infrastructure. By the time he posted, I had over 2,300 files in it.
The problem I was trying to solve was simple: I kept losing context.
Not because I wasn’t taking notes. I was. Notion, Obsidian, random docs, voice memos, Apple Notes. The usual graveyard. The problem was that none of it was connected to my AI tools. Every time I opened a new chat, I had to re-explain myself. Who I am, what I’m working on, what I’ve already decided, what I don’t want to do again.
I’m a software developer by background, always doing too many things at once. Context is expensive for me. Rebuilding it every conversation is a real cost.
So I built a system where the AI maintains the context for me.
The structure is simple. A directory of markdown files. Every file has a small header: type, summary, tags, status, date. The AI uses these to find what’s relevant without reading everything. It files new things, answers questions, tracks decisions, flags what’s gone stale.
The shift that made it work: I stopped using the AI to query the system and started using it to maintain the system. That sounds like a small thing. It’s not.
It notices when something I captured three months ago is relevant to what I’m asking today. It keeps a log of what it learned about how I work. After a few months it stopped feeling like a tool I was using and started feeling like someone who knows me.
“It stopped feeling like a tool I was using and started feeling like someone who knows me”
Also, the whole thing is just markdown files. Nothing to install, nothing running. When I wanted to start fresh at one point, I just had a new conversation and described what I needed. It rebuilt the structure in one session. I didn’t lose anything because there was nothing to lose, just text.
At some point the volume got high enough that the AI was doing cognitive work I would have had to do myself. Not writing things for me. Remembering things for me. Making connections. Pulling up the right thing before I made a decision.
When Karpathy posted, I watched people start building their own versions from scratch. Same idea, different implementations. And I thought: I’ve already built this. I’ve been using it under real conditions for months. I should just share what I have.
So I cleaned it up, stripped out the personal stuff, wrote down how it works, and released it as a template.
It’s called llm-context-base.
On top of what Karpathy described, it ships with:
Metadata standard — every file has a consistent header so the AI can find things without reading everything
Inbox protocol — capture first, organize later
Training period — starts chatty, learns your patterns, then goes quiet
Lint system — flags stale content, orphan pages, and schema drift automatically
Templates — common document types pre-built so you don’t start from blank
Works with Claude Code, Cursor, Copilot, Windsurf, whatever you use.
The training period is the thing that took the longest to get right. For the first few weeks, the AI asks questions. How you name files, what tags you use, where things should go. It's learning your patterns. After about a month it stops asking and just does the right thing. That transition is the whole point.
It’s a starting point. The structure changes as you use it.
I’m going to keep writing about where this goes. There’s an interesting next step where a mature wiki becomes something other agents can use to act on your behalf. But that’s a later post.
For now: clone the repo, fill in the About You section, and start talking to it.
The system will figure out the rest.
llm-context-base on GitHub — open source, Apache 2.0.
P.S. — While writing this post, the system added a blog post template, a visual planning protocol for all future drafts, and a voice analysis dashboard that breaks down how I write. I didn't plan any of that. It just happened because I was talking about it.