Andrej Karpathy's LLM Wiki: Build a Self-Updating AI Second Brain with Obsidian in 1 Hour
Karpathy's LLM Wiki spec is the blueprint. Add Obsidian, Codex automations, and a CRM layer to get a second brain that actually surfaces what you saved.
Andrej Karpathy Designed the Spec. Here’s How to Build the System.
Andrej Karpathy published a GitHub page describing exactly how he thinks an LLM-native knowledge base should work — and it’s the most useful piece of software architecture documentation most people have never read.
The document is called the LLM Wiki. It’s not a product. It’s a spec. A set of principles for how to structure a personal knowledge base so that a language model can actually use it, rather than just search it. Karpathy’s insight is that most second brain systems fail not because they lack content, but because they lack structure that an LLM can navigate. You can have ten thousand notes and still get worse answers than someone with fifty well-organized ones.
If you build from Andrej Karpathy’s LLM Wiki GitHub spec as your architectural foundation, you get something qualitatively different from a RAG pipeline or a vector search over your notes. You get a wiki that an agent can read, reason over, and update — the same way a human researcher would.
This post is about what that spec actually says, why it works, and how to extend it into something you’ll use every day.
What Karpathy Got Right That Everyone Else Gets Wrong
Most knowledge management systems are write-only. You clip articles, paste transcripts, dump PDFs. The information goes in and never comes back out in a useful form. The problem isn’t retrieval — it’s that the stored content was never processed into a form that’s easy to reason over.
Karpathy’s spec solves this with a simple architectural decision: separate raw source material from synthesized knowledge. Raw files go into /raw. An agent reads them, extracts concepts, entities, and ideas, and writes structured wiki pages into /wiki. The original source is preserved but the working knowledge lives in a different layer.
This matters because LLMs are much better at reasoning over a well-structured 500-word wiki page than over a raw 8,000-word YouTube transcript. The synthesis step isn’t just compression — it’s translation into a format the model can actually use.
The second key decision is the agents.md file. This is a plain-text prompt file that governs all agent behavior. It’s not code. It’s not a config file. It’s just instructions, written in natural language, that tell any agent how to process new files, how to answer queries, how to update the index. You can edit it directly in Obsidian. If you want the agent to do something new, you add a line.
This is the part most people miss when they try to build systems like this. They reach for databases, vector stores, elaborate APIs. Karpathy’s spec uses markdown files and a text document. The simplicity isn’t a limitation — it’s what makes the system durable and hackable.
The Architecture, Piece by Piece
The folder structure is minimal by design. You need: /raw for unprocessed source files, /raw/processed for files after ingestion, /wiki for AI-generated pages, /journal for dated entries, /crm for contact records, and three root files: agents.md, index.md, and log.md.
That’s the whole thing. No database. No embeddings. No infrastructure.
The index.md file is the catalog — a running list of everything in the wiki with links to individual pages. The log.md file records every operation: when a file was processed, when a query was answered, when the wiki was updated. These two files give any agent entering the system a complete picture of what’s there and what’s happened.
The Obsidian Web Clipper Chrome extension handles ingestion. Install it, point it at your vault, set the note location to /raw, and it will pull full article content and complete YouTube transcripts directly into your vault with a single click. The front matter it generates includes source title, source URL, and creation date. You can extend it — for instance, adding an instruction to agents.md to extract the YouTube channel name and add it to the source file’s front matter, which is useful when you’re tracking which creators you’re learning from.
When the agent processes a raw file, it does several things in sequence: reads the source, creates or updates wiki pages for the concepts and entities it finds, cross-links those wiki pages back to the original source file, updates index.md, and appends an entry to log.md. Then it moves the source file from /raw to /raw/processed. The processed folder is how you know what’s been ingested. The raw folder is how you know what’s waiting.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
One of the more interesting behaviors in the spec: when you ask the wiki a question, it doesn’t just answer. It checks whether the synthesized answer represents a new concept worth preserving, and if so, writes a new wiki page for it and logs the query. The wiki grows from questions, not just from saved content. Ask it about motivation strategies and you’ll find a new page in /wiki the next time you open Obsidian. This is the compounding behavior that makes the system more useful over time rather than just larger.
The Journal and CRM Extensions
The base Karpathy spec is a wiki. The extensions that make it a daily-use system are the journal and the CRM.
The journal trigger is simple: start a chat with the prefix journal. The agent treats everything that follows as a journal entry, writes it to a dated markdown file in /journal, and responds — but the response is grounded in the wiki. Not in what a generic LLM would say, but in what you’ve actually saved. If you journal about struggling with video ideas, the agent checks the wiki for relevant saved content and cites specific pages by date. The demo in the source video is instructive: a journal entry about YouTube title anxiety produced a response that cited specific saved videos about creator strategy, framing the anxiety in terms of the concepts those videos introduced. That’s not a chatbot response. That’s a thinking partner who’s read your notes.
The CRM trigger is even simpler: start a message with add to CRM followed by a person’s name and whatever details you have. The agent creates a named markdown file in /crm — the file name is the person’s name — and populates it with whatever you provided: how you met, contact details, context from your conversations. The /crm/index.md file maintains an alphabetical list with short bios. Later, you can ask “where did I meet [person]?” and get an answer drawn from the record.
What makes this work is that the journal, CRM, and wiki are all in the same vault. When you journal about a problem, the agent can pull from wiki pages, past journal entries, and CRM records simultaneously. The connections are implicit in the file structure, not in a graph database or a vector index.
Automation and the Hourly Loop
The manual version of this system — open your agent, tell it to process the raw folder — works, but it creates friction. The automation version removes that friction entirely.
In Codex, the automations tab lets you set recurring tasks. Create one called “process second brain raw files,” point it at your second brain project folder, set it to run hourly, and give it a single instruction: if there are any unprocessed files in the raw directory, process them. Set the model to GPT-4.5 on high reasoning, or whatever the strongest model you have access to is. That’s the whole automation.
Now the loop is: you clip something with the Obsidian Web Clipper, it lands in /raw, and within the hour it’s been processed into wiki pages, cross-linked, indexed, and moved to /raw/processed. You never have to think about it.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
The GitHub backup step is worth adding. Create a private repository, connect it to Codex, and add a second instruction to the automation: after processing, commit and push the current state to the main branch. Your entire second brain — every wiki page, every journal entry, every CRM record — is version-controlled and backed up automatically every hour.
This is also where the architecture’s simplicity pays off. Because everything is markdown files in a directory, any agent can work with it. The system built in Codex works identically if you point Claude Code at the same vault directory. You’re not locked into a specific tool. The vault is the system; the agent is interchangeable. For teams building more complex agent workflows, platforms like MindStudio handle this kind of multi-agent orchestration — 200+ models, 1,000+ integrations, visual workflow composition — but for a personal knowledge system, the markdown-native approach keeps the surface area small and the system auditable.
Why Markdown Is the Right Primitive
There’s a deeper reason this architecture works that isn’t obvious from the outside.
Markdown is human-readable and machine-readable simultaneously. When you open Obsidian and look at your wiki, you’re reading the same files the agent reads. There’s no translation layer, no database schema to understand, no API to query. The graph view in Obsidian — which shows the interconnection of wiki nodes as a visual network — is just a rendering of the link structure in the markdown files themselves.
This means the system is fully auditable. If the agent does something unexpected, you can open the file and see exactly what it wrote. If you want to change how it behaves, you open agents.md and edit the instructions. The entire system is legible to a human who can read plain text.
It also means the system is composable. The same vault that Codex processes can be read by Claude Code, or by any other agent that can access a file system. The self-evolving Claude Code memory system built with Obsidian and hooks uses the same underlying principle: the vault is the persistent state, and the agent is the thing that updates it. The agent is stateless; the vault is stateful.
This is a different mental model from most AI tooling, which tends to treat the AI as the source of truth and the outputs as ephemeral. Here, the vault is the source of truth and the AI is a processing layer. That inversion is what makes the system durable. Models will change. The markdown files will still be there.
The same logic applies when thinking about how applications get built from structured specifications. Remy takes this further in a different domain: you write an annotated markdown spec — prose carrying intent, annotations carrying precision — and it compiles into a complete TypeScript backend, SQLite database, auth, and deployment. The spec is the source of truth; the generated code is derived output. The underlying principle is the same as Karpathy’s wiki: keep the human-readable artifact primary, let the machine-generated artifact be downstream of it.
What This System Actually Changes
The standard critique of second brain systems is that they’re elaborate procrastination. You spend more time organizing notes than using them. Karpathy’s architecture addresses this directly by making the organization automatic and making retrieval conversational.
Coding agents automate the 5%. Remy runs the 95%.
The bottleneck was never typing the code. It was knowing what to build.
You don’t curate the wiki. The agent curates it. You don’t search the wiki. You ask it questions. You don’t connect the wiki to your journal. The agent does that when it responds to your entries. The human’s job is to clip things that seem interesting and to journal honestly. The system handles the rest.
The comparison between Karpathy’s LLM wiki approach and RAG is worth understanding here. RAG retrieves chunks of raw text and passes them to the model. The wiki approach pre-synthesizes content into structured pages, which means the model is reasoning over cleaner, denser information. For a personal knowledge base — hundreds of sources rather than millions — the wiki approach is almost always better. The synthesis step is what makes the difference.
What you end up with, after a few weeks of clipping and journaling, is something that behaves less like a search engine and more like a research assistant who’s been reading your notes. It knows what you’ve been interested in. It knows what you’ve been struggling with. It can connect a journal entry about a current problem to a video you saved three weeks ago that’s directly relevant. That connection doesn’t happen because of clever retrieval — it happens because the wiki was built to surface it.
The AI second brain built with Claude Code and Obsidian follows the same architectural logic, and the personal AI memory approaches like OpenBrain explore what happens when you want that memory to persist across different tools and contexts. But for a single-person system that you want to start using today, Karpathy’s spec is the right starting point. It’s minimal, it’s legible, and it compounds.
The graph view in Obsidian, when you’ve been running the system for a few weeks, looks like a brain. Nodes for concepts, edges for connections, clusters forming around the topics you keep returning to. That’s not a metaphor. That’s what’s actually in the files.
Start with the spec. Add the journal. The rest follows.
