archy/.cursor/rules/Development-Workflow.md

Archipelago Development Workflow

Overview

Archipelago is a Bitcoin Node OS that users install from a bootable USB. We develop on a live development server, then package that server's state into an auto-installer ISO.

Target Experience (Like Other Bitcoin Nodes)

Users interact with Archipelago like Umbrel, Start9, RaspiBlitz:

1. Flash ISO to USB
2. Boot from USB → Auto-installer runs
3. Installer detects internal disk and installs Archipelago
4. Remove USB, reboot
5. Access web UI at http:// (port 80, served by Nginx)
6. Manage Bitcoin, Lightning, apps through web interface

Development Workflow

1. Development Server (Primary Development Environment)

Server: archipelago@192.168.1.228 Purpose: Live development and testing environment

This is where ALL development happens:

• Backend changes: /usr/local/bin/archipelago (Rust binary)
• Frontend changes: /opt/archipelago/web-ui (Vue.js, served by Nginx on port 80)
• Backend API: localhost:5678 (proxied by Nginx)
• System configs: Nginx, systemd services, etc.
• Container apps: Podman containers for Bitcoin, LND, etc.

CRITICAL: This is the AUTHORITATIVE source. The ISO must capture THIS server's exact state.

2. Build Process (Snapshot → ISO)

Goal: Create an auto-installer ISO that installs the EXACT state of the dev server

Process:

1. Snapshot the dev server (192.168.1.228):

   • Capture current backend binary (/usr/local/bin/archipelago)
   • Capture current frontend files (/opt/archipelago/web-ui)
   • When DEV_SERVER is set: capture container images from the live server so the ISO prepackages current apps
   • Capture system configs (Nginx, systemd, etc.)
   • Capture app manifests and configs

2. Package into bootable ISO:

   • Base: Debian Live (minimal installer environment)
   • Includes: Pre-built rootfs with all Archipelago components
   • Auto-installer script detects internal disk and installs system

3. Result: Bootable ISO that users can flash to USB

3. ISO Flash & Install (End User Experience)

User steps:

1. Flash archipelago-installer-x86_64.iso to USB
2. Boot from USB
3. Press Enter at "Install Archipelago" prompt
4. Installer automatically:
   • Detects internal disk (NVMe/SSD)
   • Creates partitions (EFI + Root)
   • Installs Archipelago system
   • Installs GRUB bootloader
   • Shows "INSTALLATION COMPLETE" with Web UI URL
5. Remove USB and reboot
6. Access Web UI at http://<IP>

4. Deployment Targets

• Development Server: 192.168.1.228 (always up to date)
• Test Devices:
  • Dell OptiPlex (current test device)
  • Start9 Server Pure (Intel i7, NVMe)
  • HP ProDesk 400 G4 DM
• Production: Any x86_64 device with NVMe/SSD

Architecture

Frontend (Web UI)

• Framework: Vue.js 3 + Vite
• Build Output: web/dist/neode-ui/ (NOT neode-ui/dist/)
• Deployment: Copied to /opt/archipelago/web-ui on dev server
• Served By: Nginx on port 80
• API Proxy: Nginx proxies /rpc/, /ws/, /health to localhost:5678

Backend (API Server)

• Language: Rust
• Binary Location: /usr/local/bin/archipelago
• Bind Address: 0.0.0.0:5678
• Systemd Service: archipelago.service
• Managed By: systemd (auto-start on boot)

System Integration

• OS: Debian 12 (Bookworm)
• Web Server: Nginx (port 80)
• Container Runtime: Podman (rootless)
• Apps: Bitcoin Core, LND, BTCPay, Nostr relays, etc.

Build Scripts

build-auto-installer-iso.sh (CORRECT SCRIPT)

Creates a bootable auto-installer ISO (like the working build from this morning).

Features:

• Pre-built rootfs (no network needed during install)
• Auto-detects internal disk
• One-button installation
• Boots directly to web UI after install
• Pre-bundles container images (Bitcoin, LND, etc.)

Usage:

cd image-recipe
sudo bash build-auto-installer-iso.sh

IMPORTANT: Must capture LIVE SERVER state, not build from source.

build-debian-iso.sh (DEPRECATED)

Creates a live system ISO (boots into a live environment, doesn't install). DO NOT USE - This was causing the boot-to-prompt issue.

Deployment to Dev Server

Dev server access

• Host: archipelago@192.168.1.228
• Password: archipelago — use this for deployment. For non-interactive sync/deploy from scripts or the agent, use: sshpass -p "archipelago" (e.g. sshpass -p "archipelago" rsync ... or prepend it to ssh/rsync when running ./scripts/deploy-to-target.sh or equivalent).
• Build approach: We build directly on the server by SSHing in and running cargo build --release there. Do not build the backend on macOS and copy the binary.

⚠️ CRITICAL: Backend Compilation Architecture

NEVER compile the Rust backend on macOS and deploy to Linux!

The dev server (192.168.1.228) is x86_64 Linux (Debian 12). Binaries compiled on macOS (even with cross-compilation) can cause "Exec format error" due to:

• Different architecture (macOS ARM64/Intel vs Linux x86_64)
• Different libc (macOS vs glibc)
• Different system call interfaces

ALWAYS build the backend directly on the Linux dev server.

Deployment Procedures

1. Backend (MUST build on Linux — use rsync then build on server):

   # From project root. Sync source to server (exclude local target/.git).
   sshpass -p "archipelago" rsync -avz --exclude target --exclude .git -e "ssh -o StrictHostKeyChecking=no" \
     core/ archipelago@192.168.1.228:~/archy/core/
   
   # Build on server and deploy binary
   sshpass -p "archipelago" ssh -o StrictHostKeyChecking=no archipelago@192.168.1.228 \
     'source ~/.cargo/env && cd ~/archy/core/archipelago && cargo build --release && \
      sudo systemctl stop archipelago && \
      sudo cp ~/archy/core/target/release/archipelago /usr/local/bin/ && \
      sudo systemctl start archipelago'
   

   Do not build the binary on macOS and copy it; always rsync source and build on the server.

2. Frontend (can build locally):

   # Build locally (macOS is fine for frontend)
   cd neode-ui
   npm run build
   
   # Deploy to server
   rsync -avz ../web/dist/neode-ui/ archipelago@192.168.1.228:/tmp/neode-ui-build/
   ssh archipelago@192.168.1.228 'sudo rm -rf /opt/archipelago/web-ui/* && sudo cp -r /tmp/neode-ui-build/* /opt/archipelago/web-ui/ && sudo chown -R www-data:www-data /opt/archipelago/web-ui'
   

3. Container Images (Docker/Podman):

   # Build locally and push to server
   cd docker/<app-name>
   podman build -t localhost/<app-name>:latest .
   podman save localhost/<app-name>:latest | ssh archipelago@192.168.1.228 'podman load'
   

CRITICAL RULES

🚨 NEVER VIOLATE THESE

1. ALWAYS deploy to the live development server (192.168.1.228) for testing
2. After every change: sync and build on the live server. When you finish implementing a feature or fix, run the deploy script so the live server has the latest code. Command: ./scripts/deploy-to-target.sh --live (from project root). If SSH is not available in the current environment, tell the user to run it locally. Do not skip this step. App UIs (e.g. docker/lnd-ui/, docker/bitcoin-ui/) are served by their own containers; the deploy script rebuilds the LND UI image and restarts its container so changes to the LND UI are visible after deploy.
3. 🔴 NEVER EVER compile the Rust backend on macOS and deploy to Linux
   • Dev server is x86_64 Linux (Debian 12)
   • Always build backend ON the Linux server using source ~/.cargo/env && cargo build --release
   • macOS binaries will cause "Exec format error" and break the system
   • Frontend (Vue.js) CAN be built on macOS - it's just HTML/CSS/JS
4. The ISO must capture the CURRENT STATE of the dev server, not build from source
5. Frontend build output is in web/dist/neode-ui/, NOT neode-ui/dist/
6. Nginx serves on port 80 and proxies backend on localhost:5678
7. App icons are in neode-ui/public/assets/img/app-icons/
8. The auto-installer ISO is the ONLY way to deploy - no live systems

Testing Checklist

Before creating ISO:

• Backend running on dev server (curl http://192.168.1.228:5678/health)
• Frontend accessible (curl http://192.168.1.228/)
• Web UI shows correct apps and icons
• API calls working (check browser console)
• All systemd services enabled and running

After flashing ISO:

• ISO boots to installer menu
• Auto-installer detects internal disk
• Installation completes without errors
• System reboots and shows Web UI URL
• Web UI accessible at http://<IP>
• Backend API responding
• Apps visible in marketplace

Common Issues

Issue: ISO boots to prompt instead of auto-starting

• Cause: Using build-debian-iso.sh (live system) instead of build-auto-installer-iso.sh
• Fix: Use correct auto-installer script

Issue: macOS backend binary on Linux server ("Exec format error")

• Cause: Compiling Rust backend on macOS and copying to Linux server
• Symptom: systemd service fails with "status=203/EXEC" and "Failed to execute: Exec format error"
• Why it happens: Different architectures and system ABIs between macOS and Linux
• Fix: ALWAYS build the backend ON the Linux server:

  ssh archipelago@192.168.1.228
  cd ~/archy/core/archipelago
  source ~/.cargo/env
  cargo build --release
  sudo systemctl stop archipelago
  sudo cp ../target/release/archipelago /usr/local/bin/
  sudo systemctl start archipelago
  

• Prevention: Never use local cargo build for deployment - always build on target system

Issue: Frontend not updating on server

• Cause: Building to wrong output directory or not deploying to correct Nginx root
• Fix: Build to web/dist/neode-ui/, deploy to /opt/archipelago/web-ui

Issue: ISO doesn't have latest changes

• Cause: Building from source instead of capturing live server state
• Fix: Modify build script to snapshot dev server, not compile from scratch

Next Steps

• Fix build-auto-installer-iso.sh to capture live server state
• Create snapshot script for dev server
• Document container image bundling process
• Create automated testing framework
• Set up CI/CD for ISO builds