BrainHQ & Second Brain Platform Documentation

Overview

The platform consists of two main repositories built by Misty Interactive Studios Inc.:

1. BrainHQ: The admin console for managing clients, handling data uploads, and orchestrating server-side training jobs.
2. Brains: The open-source standalone Next.js web app (client OS) that serves the visual knowledge-base and query interface.

System Architecture

1. BrainHQ (Admin Console)

• Purpose: Central hub for onboarding clients and preparing their data.
• Features:
  • Tracks client second-brain projects.
  • Accepts uploads of .mbox, Outlook, docs, notes, images, and archives.
  • Limits uploads to 1 GB by default (BRAINHQ_MAX_UPLOAD_BYTES).
  • Creates training job folders and manifest.json.
  • Generates Vercel deployments directly from the portal via VERCEL_TOKEN.
  • Derives per-client credentials using ADMIN_V2_CLIENT_CREDENTIAL_MASTER.
• API:
  • POST /api/deployments/create for deploying client apps.

2. Second Brain Client (Brains)

• Purpose: The user-facing application deployed at <client-slug>.brainhq.zip.
• Features:
  • Next.js App Router frontend.
  • Premium dark 3D graph canvas using Three.js and @react-three/fiber.
  • Vector search and RAG (Retrieval-Augmented Generation) query flow with citations.
  • skills.md style instruction editor.
• AI Integration:
  • Models: Uses gpt-5.5 for chat/synthesis and text-embedding-3-small for embeddings.
  • Vector DB: Pluggable adapter supporting pgvector, Qdrant, Milvus, Weaviate, LanceDB, FAISS, and HNSW.
  • OpenAI: Uses OpenAI Vector Stores for corpus storage and OpenAI Responses API for grounded answers.

Ingestion Pipeline

The private offline ingestion job performs the following steps:

1. Parsing: Reads Gmail .mbox files and extracts messages.
2. Chunking & Embedding: Splits text into chunks and generates deterministic embeddings using local embeddings or OpenAI API.
3. Extraction: Extracts sent-email training examples and style profile data.
4. Index Creation: Writes a private JSON index (visual/style brain index).
5. Output: Produces an OPENAI_VECTOR_STORE_ID and OPENAI_BRAIN_INDEX_FILE_ID.

Example Ingestion Command:

OPENAI_API_KEY=... npm run ingest:tong-guan -- \
  --source "/Volumes/External 1/Second Brain Clients/Tong Dats" \
  --output data/private/brain-index.json \
  --vector-store \
  --graph-node-limit 3000

Local Development Setup

BrainHQ

cd BrainHQ
npm install
npm run dev
# Runs on http://localhost:3000

Brains (Client App)

cd brains
npm install
cp .env.example .env.local
npm run dev
# Runs on http://localhost:3000

Required Env Vars for Brains: OPENAI_API_KEY, OPENAI_CHAT_MODEL, BRAIN_INDEX_PATH or BRAIN_INDEX_URL.