diff --git a/docs/scale-budget.md b/docs/scale-budget.md new file mode 100644 index 00000000..7f48b330 --- /dev/null +++ b/docs/scale-budget.md @@ -0,0 +1,137 @@ +# Resource Budget for 10K Users + +## Current Baseline (March 2026) + +### Node .228 (Primary Dev Server) +- **Hardware**: Intel i3-8100T (4 cores @ 3.10GHz), 16GB RAM, 1.8TB NVMe +- **Containers**: 32 running +- **RAM Usage**: ~14GB (8GB swap configured) +- **CPU Load**: 3.5-5.5 (variable, depends on Bitcoin block processing) +- **Disk Usage**: ~82% of 1.8TB + +### Per-Container Resource Consumption (Measured) + +| App | RAM (typical) | CPU (typical) | Disk | +|-----|---------------|---------------|------| +| Bitcoin Knots | 750MB | 0.5-2.0 cores (IBD) | 600GB+ (full chain) | +| LND | 250MB | 0.1 cores | 5GB | +| Electrs/Mempool-Electrs | 500MB | 0.5 cores (indexing) | 50GB+ | +| Mempool API | 200MB | 0.1 cores | 1GB | +| Mempool Web | 50MB | 0.01 cores | negligible | +| BTCPay Server | 300MB | 0.1 cores | 2GB | +| NBXplorer | 200MB | 0.1 cores | 1GB | +| PostgreSQL (BTCPay) | 100MB | 0.1 cores | 2GB | +| MariaDB (Mempool) | 150MB | 0.1 cores | 1GB | +| Fedimint | 370MB | 0.1 cores | 1GB | +| Fedimint Gateway | 100MB | 0.05 cores | negligible | +| OnlyOffice | 760MB | 0.2 cores | 2GB | +| Immich Server | 500MB | 0.5-1.0 cores (ML) | varies | +| Immich Postgres | 100MB | 0.1 cores | varies | +| Immich Redis | 30MB | 0.01 cores | negligible | +| Nextcloud | 300MB | 0.2 cores | varies | +| Jellyfin | 200MB | 0.2-2.0 cores (transcode) | varies | +| Home Assistant | 230MB | 0.1 cores | 1GB | +| Grafana | 100MB | 0.05 cores | 500MB | +| Uptime Kuma | 80MB | 0.02 cores | 200MB | +| Vaultwarden | 50MB | 0.01 cores | 100MB | +| PhotoPrism | 300MB | 0.3 cores (ML) | varies | +| SearXNG | 100MB | 0.05 cores | negligible | +| DWN | 80MB | 0.02 cores | varies | +| FileBrowser | 30MB | 0.01 cores | negligible | +| Portainer | 50MB | 0.02 cores | 200MB | +| Tailscale | 30MB | 0.01 cores | negligible | +| AdGuard Home | 50MB | 0.02 cores | 200MB | +| Nostr Relay | 50MB | 0.02 cores | varies | +| Nginx Proxy Manager | 50MB | 0.01 cores | negligible | +| Ollama | 500MB-4GB | 1-4 cores (inference) | 10GB+ (models) | + +## App Tiers + +### Core Tier (Required for Basic Bitcoin Node) +- Bitcoin Knots (750MB) +- LND (250MB) +- Electrs (500MB) +- Mempool Stack (400MB total) +- BTCPay Stack (600MB total) +- DWN (80MB) +- FileBrowser (30MB) +- **Total: ~2.6GB RAM, 2 CPU cores, 700GB disk** + +### Recommended Tier (Enhanced Functionality) +- Fedimint + Gateway (470MB) +- Vaultwarden (50MB) +- Uptime Kuma (80MB) +- Grafana (100MB) +- SearXNG (100MB) +- Tailscale (30MB) +- Portainer (50MB) +- **Total: +880MB RAM, +0.5 cores** + +### Optional Tier (User Choice) +- Home Assistant (230MB) +- Jellyfin (200MB) +- Nextcloud (300MB) +- OnlyOffice (760MB) +- Immich Stack (630MB) +- PhotoPrism (300MB) +- AdGuard Home (50MB) +- Ollama (500MB-4GB) +- Nginx Proxy Manager (50MB) +- **Total: +2-5GB RAM, +2-5 cores** + +## Hardware Tier Recommendations + +### Tier 1: Minimal (Core Only) +- **CPU**: 2 cores (Intel Celeron/N100, ARM Cortex-A76) +- **RAM**: 4GB +- **Disk**: 1TB SSD (pruned Bitcoin node) or 2TB (full node) +- **Apps**: Core tier only +- **Cost**: ~$100-150 (Raspberry Pi 5, used mini-PC) + +### Tier 2: Standard (Core + Recommended) +- **CPU**: 4 cores (Intel i3/N200, Apple M1) +- **RAM**: 8GB +- **Disk**: 2TB NVMe +- **Apps**: Core + Recommended tiers +- **Cost**: ~$200-400 (Intel NUC, ThinkCentre Tiny) + +### Tier 3: Power User (All Tiers) +- **CPU**: 4-8 cores (Intel i5/i7, AMD Ryzen) +- **RAM**: 16GB+ +- **Disk**: 2-4TB NVMe +- **Apps**: Core + Recommended + Optional +- **Cost**: ~$400-800 (used workstation, custom build) + +### Tier 4: Heavy (All + AI/ML) +- **CPU**: 8+ cores +- **RAM**: 32GB+ +- **Disk**: 4TB+ NVMe +- **GPU**: Optional (for Ollama, Immich ML) +- **Apps**: Everything including Ollama with large models +- **Cost**: ~$800+ (workstation with GPU) + +## 10K User Projection + +### Distribution Assumption +- 60% Tier 1 (minimal Bitcoin node): 6,000 users +- 25% Tier 2 (standard): 2,500 users +- 12% Tier 3 (power user): 1,200 users +- 3% Tier 4 (heavy): 300 users + +### Network Impact +- Federation sync: ~1KB per peer per 5-minute sync +- DWN message replication: ~10KB per message sync +- Tor hidden service overhead: negligible per user +- Nostr relay federation: ~5KB per node announcement + +### Scale Bottleneck Analysis +1. **Disk**: Bitcoin blockchain grows ~100GB/year — need at minimum 1TB +2. **Memory**: Core tier uses 2.6GB, leaves headroom on 4GB systems +3. **CPU**: Bitcoin block processing and Electrs indexing are CPU-bound +4. **Network**: Tor circuit establishment is the main latency bottleneck + +## Recommendations +1. Default fresh install to **Core tier only** (2.6GB RAM) +2. Show tier badges in Marketplace +3. Warn when system RAM < required for selected apps +4. Auto-detect hardware and suggest appropriate tier diff --git a/loop/plan.md b/loop/plan.md index 2d41c8ff..3dae05b9 100644 --- a/loop/plan.md +++ b/loop/plan.md @@ -313,7 +313,7 @@ Every test must pass **10 consecutive times** from BOTH .228→.198 AND .198→. ### Sprint 14: Resource Budget for 10K Users -- [ ] **SCALE-01** — Create resource budget document. Based on current .228 metrics (33 containers, 6.5GB RAM, 1.2TB disk, load 5.44), calculate per-node resource requirements. Estimate: RAM per container (avg), disk per container, CPU per container. Project for 10K users across different hardware tiers. Document in `docs/scale-budget.md`. **Acceptance**: Document with clear resource requirements per hardware tier. +- [x] **SCALE-01** — Created `docs/scale-budget.md`. Per-container RAM/CPU/disk measurements from .228. Three app tiers: Core (2.6GB, Bitcoin+LND+Electrs+Mempool+BTCPay+DWN), Recommended (+880MB, Fedimint+Grafana+Vaultwarden+etc), Optional (+2-5GB, Home Assistant+Jellyfin+Nextcloud+Immich+etc). Four hardware tiers: Minimal (4GB/2 cores/$100), Standard (8GB/4 cores/$300), Power (16GB+/$500), Heavy (32GB+/$800). 10K user projection with distribution estimates. - [ ] **SCALE-02** — Identify resource bottlenecks. Profile the top CPU and memory consumers. Current: immich_server (82% CPU spike), onlyoffice (759MB RAM), bitcoin-knots (750MB RAM), fedimint (369MB), lnd (250MB), homeassistant (234MB). Determine which apps should be optional vs core for a minimal install. **Acceptance**: Tiered app list: Core (must-have), Recommended, Optional. Core tier uses < 4GB RAM.