Two-Phase Pipelines, Persistent Memory, and Why RSS Still Wins
Building Personal AI Infrastructure in Five Hours
I spent five hours this morning with Claude rebuilding pieces of my personal automation stack. Not asking it questions. Not generating marketing copy. Engineering infrastructure: debugging silent failures and getting production fixes to a self-hosted ops platform.
Here’s what the session actually looked like, and the architectural patterns that emerged.
The Problem: A Newsletter Brief That Wouldn’t Update
I have a scheduled task that reads Nate B. Jones’ Substack newsletter every morning, generates a BLUF (Bottom Line Up Front) summary, and emails it to me.
If you’re not already following Nate, you should be. His YouTube channel and Substack are consistently among the most technically substantive resources on applied AI I’ve found. Not hype or product demos, but rigorous thinking about how these systems actually work and where they break down. The man does his homework.
The scheduled BLUF task stopped picking up new articles. Every morning I got the same stale summary from March 2nd, even though new posts were publishing daily.
Root cause: The task was scraping the Substack homepage, which serves cached, paywall-filtered content. The hero article on the rendered page was nine days old. The actual latest article only appeared in the RSS feed.
The fix: Switch from homepage scraping to RSS feed discovery (`/feed` endpoint). RSS returned 20+ articles in chronological order with today’s article first. The homepage returned two articles, neither of which was current.
Takeaway: In 2026, RSS is still the most reliable machine-readable interface for content discovery. Homepages are optimized for humans and ad platforms, not for automation pipelines.
The Pattern: Two-Phase Generate-Then-Send
The newsletter task also had a delivery problem. It was using a Gmail MCP connector that doesn’t exist in the scheduled task execution context. Emails were silently not sending.
This led to adopting a pattern I’d already proven with my morning sleep report: two-phase generate-then-send.
Phase 1 (Generate) runs at 8:30 AM:
Discovers the latest article via RSS
Validates the publication date (today = proceed, yesterday + Monday = proceed, 2+ days old = write a “no new article” placeholder)
Generates the BLUF summary
Saves to a temp file
Runs a validation script (file exists, size > 200 bytes, correct header format)
Phase 2 (Send) runs at 9:00 AM:
Preflight checks: file exists, age < 3 hours, size > 100 bytes, header validation
If all checks pass, sends via SMTP using a Python script with smtplib
If the file contains a “no new article” placeholder, it still sends (notifying me of the absence rather than staying silent)
The 30-minute gap between phases gives Phase 1 time to complete and provides a natural retry window. The preflight checks in Phase 2 catch any upstream failures before they reach an inbox.
Why SMTP instead of an API connector: MCP connectors (Gmail, Google Calendar, etc.) are available during interactive sessions but not during scheduled task execution. This is a hard architectural constraint. Every email-sending task in the stack now uses direct SMTP via Python scripts with stored credentials. No exceptions.
The Silent Failure: Calendar Events That Never Appeared
My morning sleep report generates an optimal exercise window from Eight Sleep data and creates a “Morning Walk” event on Google Calendar. The report was arriving every morning. The calendar event was not.
Root cause: Same as the email problem. The task was calling `gcal_create_event` through an MCP connector that doesn’t exist in the scheduled execution context. The task had a graceful fallback (”if gcal_create_event fails, log the error and continue”), which meant it failed silently every single day.
The fix: Build a standalone Python script (`gcal_create_walk.py`) that calls the Google Calendar API directly via OAuth2. The script handles its own token refresh, accepts CLI arguments for timing parameters and sleep score, and creates the event with a 10-minute popup reminder.
The OAuth2 journey required creating a Google Cloud project, configuring an OAuth consent screen, generating client credentials, adding myself as a test user, and running a one-time browser consent flow to generate a refresh token. The app was then published to Production to avoid the 7-day token expiry that Testing mode imposes.
Takeaway: Any automation that depends on interactive-session-only integrations will silently fail when moved to scheduled execution. The fix is always the same: replace the managed connector with a direct API call using stored credentials.
The Constraint That Governs Everything
This session produced a HIGH-severity architectural constraint that now applies across the entire stack:
All scheduled email tasks MUST use SMTP script pattern (smtplib + email_config.json). Never Gmail MCP. MCP connectors are unavailable in scheduled task context.
This constraint is stored in a persistent constraint library that loads at the start of every session. It prevents future sessions from re-introducing the same failure pattern.
Building the Memory Layer
The constraint above didn’t get stored in a text file or a sticky note. It went into a purpose-built SQLite database that serves as Claude’s persistent memory across sessions.
The system started as a notes database for tracking continuity in a fiction writing project: character decisions and craft patterns from session to session. But that architecture was too narrow. Infrastructure work, health tracking, career development, game modding. None of those had a place to land.
So this session expanded the MCP server from 27 tools to 32, adding two entirely new subsystems.
The Constraint Library encodes rejection patterns: the things Claude keeps getting wrong. Each constraint has a domain (writing, code, infrastructure, etc.), an optional scope (global or project-specific), a pattern describing the anti-pattern, a correction describing what to do instead, and a hit counter that increments every time the same mistake recurs. High-hit-count constraints surface first at session init. They’re the “institutional memory of no.”
The Affinity Library is the mirror: the patterns that work well. Same structure, but tracking positive patterns instead of failures. When a particular approach lands well (”present three options with tradeoffs before committing to one”), it gets logged with its domain and scope and accumulates hits over time. These are the “institutional memory of yes.”
The concept draws directly from Nate B. Jones’ framework: every rejection is a knowledge creation event. If the knowledge evaporates at the end of the conversation, you’re paying the same correction tax in every session. Encoding it in a queryable, bumpable, retirable database means each correction only happens once.
Two Architectures for the Same Problem
Nate’s “Open Brain” architecture tackles the same core problem (AI amnesia between sessions) but makes different tradeoffs. His system uses PostgreSQL with semantic embeddings, hosted in the cloud and queried by multiple AI tools through a shared MCP layer. It’s horizontal: one brain, many clients, automatic capture from sources like Slack, with weekly synthesis prompts surfacing patterns. Infrastructure cost: about $0.10–$0.30/month.
My system went vertical instead. SQLite, entirely local, purpose-built for one AI client with deep lifecycle integration. No semantic embeddings; structured fields, domain scoping, severity ranking, and FTS5 full-text search instead.
Constraints and affinities aren’t just stored; they’re loaded at session init, auto-detected during conversation, bumped on recurrence, and retired when they go stale. The tradeoff is portability for depth. I can’t query this from ChatGPT, but Claude doesn’t just read the memory; it actively maintains it as part of the session workflow. Infrastructure cost: zero. It runs on my machine.
Neither approach is wrong. If you’re working across multiple AI tools and need shared context, Nate’s cloud-hosted horizontal model is the right call. If you’re deep in one tool’s ecosystem and want the memory system woven into the session lifecycle itself, the local vertical model compounds faster.
Both tables use FTS5 full-text search with sync triggers. Both support scoping: a constraint tagged @sims4mod won’t fire during infrastructure work, and a global constraint with no scope applies everywhere.
The session also expanded the notes system itself: five new scopes (ai-learning, health, career, homelab, sims4mod) and four new categories (INSIGHT, REFERENCE, WORKFLOW, SESSION) joining the existing novel-writing taxonomy. The server went from a fiction-writing tool to a general-purpose cross-session context engine.
The architectural insight: LLMs don’t need more data. They need structured, queryable, scoped memory that loads the right context at the right time. A flat document with everything in it wastes tokens and dilutes signal. A database with domains, scopes, severity levels, and hit counts means the most important patterns surface first and stale ones get retired automatically.
Operational Documentation as Code
Every change made during the session was reflected in a suite of self-hosted HTML documentation deployed to IIS via a batch script. Not wikis. Not Notion pages. Version-controlled HTML that lives alongside the code it documents.
The stack has three documentation layers, each serving a different consumer.
Operations Runbook (elf-operations.html) is the “how it runs” layer: task inventory with cron schedules, the two-phase pipeline architecture, external API dependencies, a recipient directory, and troubleshooting decision trees. When something breaks at 6 AM, this is where you start.
Workflow Reference (vot-workflow.html) is the “how to use it” layer: every slash command in the plugin system, the init-to-close session lifecycle, prompt chaining patterns, and the relationship between skills and prompt files.
Data Architecture (vot-data-architecture.html) is the “where it lives” layer: every database table and FK-PK relationship across the canon and notes databases, the constraint and affinity schemas and the MCP tool surface area.
The governing principle is simple: it’s not done if it isn’t documented. A pipeline fix that doesn’t update the runbook is a future outage waiting to happen.
Critically, the documentation layer serves two consumers. It’s a reference for me, but it’s equally a reference for Claude. When a new session starts and Claude needs to understand the constraint schema or how the two-phase pipeline works, it reads the same HTML documentation a human would. The documentation isn’t just for the human operator. It’s part of the AI’s context loading pipeline.
All three documents update in the same session where the code changes. Documentation drift (where the docs describe last month’s architecture) doesn’t happen because the same AI session that builds the feature also updates the docs. The deployment is a single `Push-Wwwroot.bat` call that syncs all HTML files to the web server.
What Five Hours of AI-Assisted Infrastructure Work Actually Looks Like
The compound effect matters more than any individual fix. Each piece of the stack (the SMTP pattern, the two-phase pipeline, the constraint and affinity libraries, the scoped notes, the ops runbook) makes the next session faster because the decisions are already encoded and the documentation is already current. The next time Claude starts a session, it loads the patterns that worked and the mistakes to avoid before writing a single line of code. That’s the real output of this session: not the bug fixes, but the infrastructure that prevents the next round of bugs.
This wasn’t a conversation. It was a pair-programming session that produced:
RSS-based article discovery replacing a broken homepage scraper
An SMTP email sender matching an existing reference implementation’s patterns
An OAuth2 calendar script with automatic token refresh
Browser automation through Google Cloud Console for OAuth consent setup
An MCP server expansion from 27 to 32 tools, adding constraint and affinity subsystems
19 seed entries pre-populated across both memory tables from existing project files
Three layers of operational documentation updated to reflect every change
Init and close session protocols packaged into reusable plugin commands
Over a dozen files. 640 new lines in the MCP server. Two silent failures fixed. One architectural constraint encoded. Two persistent memory tables seeded. All documented, all deployed.
What patterns have you found for giving AI agents persistent, structured memory? I’m particularly interested in approaches that go beyond flat context documents into queryable, scoped systems, where the AI doesn’t just read its memory, but actively maintains it. Reply if you want to compare notes.
You may also like: