How to Build an AI Second Brain Knowledge Base with Automated Hourly Processing

The Problem With Your Notes

You capture everything. Meeting notes, article highlights, random ideas at 11pm, voice memos you never transcribe. Months later, you can’t find any of it, and even when you do, you’ve lost the context that made it useful in the first place.

This is the core failure of most personal knowledge systems. The capture layer works. The retrieval layer doesn’t. And nothing connects the two.

An AI second brain knowledge base solves this by adding an automated processing layer between capture and retrieval. Instead of you manually tagging, linking, and summarizing notes, a scheduled workflow does it for you — every hour, quietly, in the background. By the time you need an insight, it’s already been extracted, linked to related ideas, and made searchable.

This guide walks through how to build exactly that using Claude Code and Obsidian, with a clear look at how to automate the processing pipeline so it runs without your involvement.

What an AI Second Brain Actually Does

The “second brain” concept, popularized by productivity researcher Tiago Forte, describes an external system for capturing, organizing, and retrieving information. The original version is manual — you read something, highlight it, tag it, file it.

The AI version automates the middle steps.

Here’s the basic flow:

Capture — You drop raw notes, URLs, voice transcripts, or highlights into an inbox
Process — An AI agent reads those inputs, extracts key ideas, creates summaries, adds metadata, and links to related notes
Surface — When you need something, you query the system and get back synthesized insights, not just raw text

Remy doesn't write the code. It manages the agents who do.

AGENTS ASSIGNED TO THIS BUILD

Remy

Product Manager Agent

Leading

Design

Engineer

Deploy

Remy runs the project. The specialists do the work. You work with the PM, not the implementers.

The critical addition is the scheduling layer. Instead of processing notes manually — which most people never do consistently — the system runs on a timer. Every hour, or every night, or on whatever cadence fits your workflow, the agent wakes up and processes whatever’s sitting in the inbox.

This is what makes it actually work in practice.

What You Need Before You Start

Tools and Prerequisites

Before writing any code or configuring any agent, get these in place:

Obsidian — A local-first markdown note-taking app. Your knowledge base lives here as plain text files in a vault (a folder on your machine or synced to a cloud service).
Claude Code — Anthropic’s agentic coding assistant, capable of reading and writing files, running shell commands, and executing multi-step tasks autonomously.
A folder structure — You need at minimum an inbox folder (raw captures), a processed folder (notes the agent has worked on), and a concepts folder (extracted ideas linked across notes).
Obsidian Sync or a local path Claude Code can access — The agent needs read/write access to your vault.

Optional but useful:

Obsidian Templater plugin — For consistent note structure
Dataview plugin — For querying your vault like a database
A task scheduler — cron on macOS/Linux, Task Scheduler on Windows, or a cloud-based trigger

Folder Structure

Set up your vault like this before building the agent:

/vault
  /inbox          ← raw, unprocessed captures
  /processed      ← agent has touched these
  /concepts       ← extracted atomic ideas
  /projects       ← project-specific notes
  /archive        ← older processed notes
  /meta           ← agent logs, config

This structure gives the agent clear separation between what needs processing and what’s already been handled. It also prevents the agent from re-processing notes it’s already touched.

Step 1: Set Up the Capture Layer

The capture layer is where everything enters the system. The goal is zero friction — you should be able to add a note in under ten seconds.

Option A: Plain Text Drop

The simplest approach. Create a shortcut or hotkey that opens a new markdown file in your /inbox folder. Write whatever you need, save it, and forget it. The agent handles the rest.

On macOS, you can use Automator or Shortcuts to create a “new inbox note” action tied to a keyboard shortcut. On iOS, the Shortcuts app can append to a specific Obsidian file via the Obsidian URL scheme.

Option B: Append to a Daily Capture File

Instead of one file per note, maintain a single daily capture file in /inbox. Every fleeting thought, URL, or fragment gets appended to that file with a timestamp. The agent processes the whole file at the end of the day.

This works well for voice-to-text workflows — speak a note into your phone, have it transcribed and appended automatically.

Option C: Web Clipper Integration

Obsidian has a web clipper browser extension that saves article content directly into your vault. Configure it to drop everything into /inbox so the agent picks it up automatically.

Regardless of which method you use, add a YAML frontmatter block to each capture note with at minimum:

---
status: unprocessed
captured: 2025-01-15T09:30:00
source: manual
---

The status: unprocessed field is what the agent scans for when it wakes up.

Step 2: Build the Processing Agent with Claude Code

This is the core of the system. You’re going to write a Claude Code agent that:

Scans /inbox for notes with status: unprocessed
Reads each note
Sends it to Claude for analysis
Updates the note with a summary, extracted concepts, related notes, and tags
Moves the note to /processed
Creates or updates concept notes in /concepts

Writing the Agent Script

Create a file called process_inbox.py (or use Claude Code’s built-in scripting). Here’s the logic in pseudocode — Claude Code can help you turn this into working code for your specific setup:

1. List all .md files in /inbox
2. For each file:
   a. Parse YAML frontmatter
   b. If status == "unprocessed":
      - Read full file content
      - Send to Claude with processing prompt
      - Parse Claude's response
      - Update frontmatter: status = "processed", add summary, tags, concepts
      - Extract concept list from response
      - For each concept, check if /concepts/[concept].md exists
        - If yes: append a backlink and brief mention
        - If no: create new concept note with initial definition
      - Move file to /processed/YYYY-MM/[original-filename]
3. Write processing log to /meta/agent-log.md

The Processing Prompt

The prompt you send to Claude with each note is the most important part. Here’s a solid starting template:

You are processing a note for a personal knowledge base. 

Note content:
[NOTE CONTENT]

Return a JSON object with:
- summary: 2-3 sentence summary of the core idea
- key_concepts: array of 3-8 core concepts mentioned (noun phrases, not tags)
- atomic_ideas: array of standalone insights that could stand alone as their own notes
- related_topics: array of broader topics this connects to
- suggested_tags: array of simple lowercase tags
- action_items: any tasks or follow-ups mentioned (empty array if none)
- questions_raised: interesting questions this note opens up

Be specific and concrete. Avoid vague generalizations.

Adjust this based on what you actually want the system to extract. If you mostly capture technical research, weight it toward methodology and evidence. If you capture business ideas, add fields for market opportunity and risks.

Handling the Response

Claude will return a structured JSON block. Your script parses it and:

Appends the summary and tags to the note’s YAML frontmatter
Adds a ## AI Processing section at the bottom of the note with the extracted content
Creates concept notes as needed

Keep the original note content untouched above the processing section. The agent adds to your notes, it doesn’t replace them.

Step 3: Schedule the Hourly Processing Run

A knowledge base that requires you to manually trigger processing will fail. The whole point is automation — it should run whether you think about it or not.

Using Cron (macOS/Linux)

Open your terminal and edit your crontab:

crontab -e

Add a line to run the script every hour:

0 * * * * /usr/bin/python3 /path/to/your/vault/meta/process_inbox.py >> /path/to/vault/meta/cron-log.txt 2>&1

This triggers at the top of every hour, runs the agent, and logs output to a file you can check if something breaks.

Using a Cloud-Based Scheduler

If you want the processing to happen even when your computer is off, you have a few options:

GitHub Actions — Push your vault to a private GitHub repo and use scheduled Actions to run the agent on GitHub’s servers
A cloud VM — A small VPS running your script on a cron schedule
Workflow automation platforms — Services that can trigger webhooks or scripts on a schedule

The trade-off is that cloud-based approaches require your vault to sync somewhere (iCloud, Dropbox, or a private repo), which some people prefer to avoid for privacy reasons.

Idempotency Check

Make sure your agent is idempotent — running it twice on the same note shouldn’t cause problems. The status: unprocessed check handles this, but also add a guard for the ## AI Processing section: if it already exists in a file, skip it.

Step 4: Build the Insight Retrieval Layer

Processing notes is only useful if you can retrieve what the system has learned. There are two retrieval modes worth building.

Query-Based Retrieval

Create a second script — query_knowledge.py — that accepts a plain-language question and searches your vault for relevant content.

The basic approach:

Accept a question as input
Use Claude to identify key concepts from the question
Search your /processed and /concepts folders for notes containing those concepts
Send the matching notes to Claude with a synthesis prompt
Return a narrative answer with citations (note filenames)

For better results, use embedding-based search rather than keyword search. Tools like ChromaDB or simple FAISS indexes let you embed your notes as vectors and retrieve semantically similar ones. Claude Code can help you set this up.

Daily Digest

Set up a second scheduled job that runs once a day (or week) and generates a digest:

What new concepts did the agent extract today?
What ideas appeared in multiple notes this week?
What action items are outstanding?
What questions did your notes raise that haven’t been answered?

This digest lands in a new note in /processed each morning. It’s the equivalent of a morning briefing generated from everything you captured the day before.

Step 5: Link Emerging Concepts Automatically

One of the most valuable things the agent can do is surface connections between notes you captured weeks apart.

Building the Concept Graph

Each time the agent processes a note and extracts concepts, it updates the relevant concept note in /concepts. Over time, each concept note accumulates backlinks to every note that mentioned it — creating an automatically-maintained knowledge graph.

You can visualize this directly in Obsidian using the built-in Graph View. What starts as a disconnected mess of notes gradually becomes a web of linked ideas, maintained without any manual effort.

Cross-Note Connection Prompts

Add a step to the processing pipeline that asks Claude a simple question after processing each note:

Given this note and its concepts, which of the following existing 
concept notes are most relevant? [List of concept note titles]

Return the top 3 and a one-sentence explanation of the connection.

Append those connections to the note as a ## Related Concepts section with links. Over time this becomes one of the most useful parts of the system — seeing how a new idea connects to something you thought about six months ago.

Where MindStudio Fits Into This Workflow

The Claude Code + Obsidian setup described above is powerful, but it requires some technical comfort — writing Python scripts, editing crontabs, managing file paths. If you want to build or extend this kind of automated knowledge pipeline without that overhead, MindStudio is worth looking at.

MindStudio lets you build scheduled background agents visually, without code. You can create a workflow that triggers hourly, reads from a connected data source (Google Drive, Notion, Airtable, email), runs an AI processing step, and writes results back — all through a drag-and-drop interface with 200+ AI models available and 1,000+ pre-built integrations.

For a second brain use case specifically, you could build a MindStudio agent that:

Pulls new notes from a shared Notion database or Google Doc your phone writes to
Runs the same extraction and summarization logic described above
Writes structured output back to Notion, Airtable, or a connected knowledge base
Sends you a daily digest via email or Slack

The autonomous background agents in MindStudio handle the scheduling and infrastructure layer — no cron jobs, no servers, no maintenance. You define the logic, and the platform handles the execution.

This is especially useful if you want to extend the second brain to team workflows, where multiple people are capturing and the system needs to aggregate and synthesize across sources. MindStudio’s integrations make it straightforward to build that layer on top of tools your team already uses.

You can try it free at mindstudio.ai.

Common Mistakes and How to Avoid Them

Over-Engineering the Prompt

It’s tempting to write a very long, complex prompt that tries to extract every possible insight from each note. In practice, simpler prompts produce more consistent, parseable output. Start minimal and add fields as you find yourself actually using them.

Not Handling Failures Gracefully

API calls fail. Files get locked. Your vault sync hiccups. Build error handling into the script from the start — if processing a note fails, log the error and skip it rather than crashing the whole run. The agent will catch it on the next hourly pass.

Processing Too Frequently

Hourly works for most people. Running every five minutes is overkill, burns API credits, and creates a lot of noise in your logs. Start with hourly and adjust based on how quickly your inbox fills up.

Neglecting the Concept Layer

Most people focus on processing individual notes and forget to maintain the concept notes. The concept graph is where the long-term value lives — it’s what turns a pile of processed notes into actual connected knowledge. Make sure the agent is creating and updating concept notes consistently.

No Review Loop

The agent can get things wrong. A summary might miss the point. A concept might be over-extracted. Build in a weekly 15-minute review where you scan the agent log and spot-check a few processed notes. This also helps you refine the prompt over time.

FAQ

What is an AI second brain knowledge base?

An AI second brain knowledge base is a personal information system where an AI agent automatically processes your notes — extracting key ideas, creating summaries, tagging content, and linking related concepts — so you can retrieve useful insights on demand without manual organization. The “second brain” metaphor refers to an external system that stores and connects knowledge in ways that complement your biological memory.

Do I need to know how to code to build this?

Catch up on Hermes — free 60-minute live workshop

The Claude Code + Obsidian approach described here requires basic Python familiarity and comfort with the command line. If you’re comfortable editing a script and setting up a cron job, you can build it. If you want a no-code alternative, platforms like MindStudio let you build equivalent automated workflows visually without writing any code.

How much does automated note processing cost?

Costs depend on API usage. Claude Haiku (Anthropic’s fastest, cheapest model) costs roughly $0.25 per million input tokens. A typical 500-word note runs about 650 tokens. If you process 20 notes a day, that’s around $0.003 per day — well under a dollar a month. Even at higher volumes, this is inexpensive. Using Claude Sonnet or Opus will cost more but produce significantly better extractions.

Can this work with tools other than Obsidian?

Yes. The core concepts apply to any plain-text or API-accessible knowledge base. Notion, Logseq, Roam Research, and even a simple folder of markdown files work as the storage layer. The key requirement is that your processing agent can read and write to wherever your notes live. Notion’s API, for example, makes it straightforward to build the same inbox-process-archive pattern using Notion databases instead of Obsidian files.

How does automated hourly processing compare to manual review?

Manual review requires discipline and consistent time. Most people don’t do it. Automated hourly processing works passively — you capture notes throughout the day and the system handles extraction and linking whether or not you ever open your knowledge base app. The trade-off is that automated processing can miss nuance or context that you would catch manually. The best systems combine automated processing with lightweight human review rather than replacing one with the other entirely.

What’s the best way to query an AI knowledge base?

For smaller vaults (under ~500 notes), keyword search over AI-extracted tags and concept notes works well. For larger vaults, embedding-based semantic search significantly outperforms keyword matching — you can ask questions in plain language and get back relevant notes even if they don’t share exact words with your query. Several open-source vector databases (ChromaDB, Qdrant, FAISS) can be embedded in a local Python script and don’t require cloud infrastructure.

Key Takeaways

The gap in most knowledge systems isn’t capture — it’s automated processing between capture and retrieval.
An hourly scheduled agent that reads your inbox, extracts ideas, and builds a concept graph solves this without requiring manual effort.
Claude Code handles the reasoning and file operations; Obsidian handles the storage; cron handles the scheduling.
The concept layer — where extracted ideas accumulate backlinks over time — is where the long-term value compounds.
If you want to skip the scripting and infrastructure work, MindStudio’s scheduled background agents handle the same logic visually and connect to the tools you already use.

If you’re ready to build automated knowledge workflows without writing boilerplate infrastructure code, MindStudio is a fast way to get started — most workflows take under an hour to build and deploy.