10 posts tagged with "Second Memory"

I'm building Second Memory—a personal memory app where users save entries and ask natural-language questions later. They often remember meaning, not the exact words they wrote. For example, they might have saved:

• "I forgot my umbrella and got soaked in the rain."
• "I lifted weights at the gym after work."
• "I caught up with John over coffee yesterday."
• "I stayed up late debugging my application."

Later they ask "Who did I meet recently?" The best match is I caught up with John over coffee yesterday.—but keyword search can miss it when "meet" and "caught up" differ. Below: semantic search with a local embedding model (no API key), cosine distance, ranking memories in Python, then storing and querying vectors with pgvector.

Authentication is essential to Second-Memory, but it is not the core product—capture and recall are. Building login, sessions, and social OAuth in-house would add security and ongoing operations without advancing that goal, and another password is friction most users do not need. I chose Firebase Authentication with social sign-in preferred, an internal user table keyed by firebaseUid, and backend verification on every API call. Below: build vs buy, what in-house auth actually involves, the end-to-end flow, and why authorization still lives in my services.

The monorepo post ended with a practical question: how do I manage builds, dependencies, task execution, and caching across all these projects? Second-Memory needs shared API types between web, mobile, and backend, but not a tangle of manual file: paths and one-off root scripts. Workspaces define packages and link local dependencies from the root; as the graph grows, task orchestration decides what to run and in what order. I added Turborepo for task graphs, affected tasks, and caching without adopting a full Nx-style platform. Below: why monorepo and workspaces fit this product, two problems filters alone do not solve, and how Turborepo compares to workspaces-only and Nx.

After choosing microservices, the next question was how should I organise the codebase? Second-Memory has several deployable apps and services—web, mobile, memory, ask, embedding worker—plus shared packages, all one product. I chose a monorepo because as a solo developer I valued easy shared API types, cross-project changes, and one local checkout more than repository-level isolation. Below: monorepo vs polyrepo on everyday tasks, when each fits, what deserves a shared package, CI/CD trade-offs, and the build-tooling question that followed.

The initial architecture had three main backend responsibilities:

• Memory Service
• Ask Service
• Embedding Worker

Behind that sat a simpler option: why not one backend? A monolith would be easier for a small app—and probably enough for V1. I still split Second-Memory into services for clear boundaries between memory and AI interaction, room to practice queues, workers, retrieval, and LLM integration as a real system, and independent scaling when load differs across capture, embed, and ask. Below: how those boundaries map to services, what a monolith would have saved, and the trade-off I accepted.

The previous post covered why I chose pgvector for semantic search. The next question was when should a memory be embedded? At first it looks simple—create memory, generate embedding, save embedding—but that ties memory creation to the embedding provider. If the provider is slow or unavailable, saving a memory can fail or stall. I separated those concerns with the outbox pattern, a queue, and an embedding worker so the API stays fast and durable while vectors catch up asynchronously. Below: each component, eventual consistency, failure handling, and the full flow.

If a user later asks:

“What did I write about performance problems?”

keyword search is not always enough. The memory might say:

“The database becomes slow when thousands of users query it at the same time.”

There may be no exact match between the question and the memory. Second-Memory uses semantic search—embeddings and similarity—and for V1 I kept that in PostgreSQL + pgvector inside the Memory Service instead of adding separate vector infrastructure. Below: text to vectors, why pgvector for now, and the path from question to LLM context.

I had the product definition, user stories, and backlog—now it was time to design the system. I sketched a first architecture for Second-Memory—web and mobile through a BFF, Memory and Ask services, an LLM for recall—and the principles I wanted to carry through: a BFF so clients are not tied to internal layout, data ownership so databases stay inside the service that owns them, and DDD bounded contexts to separate the memory domain from AI interaction. Below is the diagram, then how each choice shows up in the design.

I had the idea for Second-Memory, but it was still just an idea—too vague to start thinking about architecture or implementation. I wanted to develop it further before building, so I put on the Product Owner hat and followed the kind of process I had experienced in professional work.

I wanted to learn how to build AI systems, so I started looking for a project. I was using ChatGPT like everyone else: I'd type something, and it would answer—that's what makes a chatbot useful. But sometimes I don't want an answer, advice, or another question. I just want to write down what I'm thinking: a thought, an idea, something that happened today, a feeling, something I'm not even sure is important yet—like talking to myself.