archy/.cursor/rules/Development-Workflow.md

# Archipelago Development Workflow

## Overview

Archipelago is a Bitcoin Node OS that users install from a bootable USB. We develop on a live development server, then package that server's state into an auto-installer ISO.

## Target Experience (Like Other Bitcoin Nodes)

Users interact with Archipelago like **Umbrel**, **Start9**, **RaspiBlitz**:
1. Flash ISO to USB
2. Boot from USB → Auto-installer runs
3. Installer detects internal disk and installs Archipelago
4. Remove USB, reboot
5. **Access web UI at http://<IP>** (port 80, served by Nginx)
6. Manage Bitcoin, Lightning, apps through web interface

## Development Workflow

### 1. Development Server (Primary Development Environment)

**Server**: `archipelago@192.168.1.228`
**Purpose**: Live development and testing environment

This is where ALL development happens:
- Backend changes: `/usr/local/bin/archipelago` (Rust binary)
- Frontend changes: `/opt/archipelago/web-ui` (Vue.js, served by Nginx on port 80)
- Backend API: `localhost:5678` (proxied by Nginx)
- System configs: Nginx, systemd services, etc.
- Container apps: Podman containers for Bitcoin, LND, etc.

**CRITICAL**: This is the AUTHORITATIVE source. The ISO must capture THIS server's exact state.

### 2. Build Process (Snapshot → ISO)

**Goal**: Create an auto-installer ISO that installs the EXACT state of the dev server

**Process**:
1. **Snapshot the dev server** (192.168.1.228):
   - Capture current backend binary (`/usr/local/bin/archipelago`)
   - Capture current frontend files (`/opt/archipelago/web-ui`)
   - When `DEV_SERVER` is set: capture container images from the live server so the ISO prepackages current apps
   - Capture system configs (Nginx, systemd, etc.)
   - Capture app manifests and configs
   
2. **Package into bootable ISO**:
   - Base: Debian Live (minimal installer environment)
   - Includes: Pre-built rootfs with all Archipelago components
   - Auto-installer script detects internal disk and installs system
   
3. **Result**: Bootable ISO that users can flash to USB

### 3. ISO Flash & Install (End User Experience)

**User steps**:
1. Flash `archipelago-installer-x86_64.iso` to USB
2. Boot from USB
3. Press Enter at "Install Archipelago" prompt
4. Installer automatically:
   - Detects internal disk (NVMe/SSD)
   - Creates partitions (EFI + Root)
   - Installs Archipelago system
   - Installs GRUB bootloader
   - Shows "INSTALLATION COMPLETE" with Web UI URL
5. Remove USB and reboot
6. Access Web UI at `http://<IP>`

### 4. Deployment Targets

- **Development Server**: `192.168.1.228` (always up to date)
- **Test Devices**: 
  - Dell OptiPlex (current test device)
  - Start9 Server Pure (Intel i7, NVMe)
  - HP ProDesk 400 G4 DM
- **Production**: Any x86_64 device with NVMe/SSD

## Architecture

### Frontend (Web UI)
- **Framework**: Vue.js 3 + Vite
- **Build Output**: `web/dist/neode-ui/` (NOT `neode-ui/dist/`)
- **Deployment**: Copied to `/opt/archipelago/web-ui` on dev server
- **Served By**: Nginx on port 80
- **API Proxy**: Nginx proxies `/rpc/`, `/ws/`, `/health` to `localhost:5678`

### Backend (API Server)
- **Language**: Rust
- **Binary Location**: `/usr/local/bin/archipelago`
- **Bind Address**: `0.0.0.0:5678`
- **Systemd Service**: `archipelago.service`
- **Managed By**: systemd (auto-start on boot)

### System Integration
- **OS**: Debian 12 (Bookworm)
- **Web Server**: Nginx (port 80)
- **Container Runtime**: Podman (rootless)
- **Apps**: Bitcoin Core, LND, BTCPay, Nostr relays, etc.

## Build Scripts

### `build-auto-installer-iso.sh` (CORRECT SCRIPT)
Creates a bootable auto-installer ISO (like the working build from this morning).

**Features**:
- Pre-built rootfs (no network needed during install)
- Auto-detects internal disk
- One-button installation
- Boots directly to web UI after install
- Pre-bundles container images (Bitcoin, LND, etc.)

**Usage**:
```bash
cd image-recipe
sudo bash build-auto-installer-iso.sh
```

**IMPORTANT**: Must capture LIVE SERVER state, not build from source.

### `build-debian-iso.sh` (DEPRECATED)
Creates a live system ISO (boots into a live environment, doesn't install).
**DO NOT USE** - This was causing the boot-to-prompt issue.

## Deployment to Dev Server

### Dev server access

- **Host:** `archipelago@192.168.1.228`
- **Password:** `archipelago` — use this for deployment. For non-interactive sync/deploy from scripts or the agent, use: `sshpass -p "archipelago"` (e.g. `sshpass -p "archipelago" rsync ...` or prepend it to ssh/rsync when running `./scripts/deploy-to-target.sh` or equivalent).
- **Build approach:** We build **directly on the server** by SSHing in and running `cargo build --release` there. Do not build the backend on macOS and copy the binary.

### ⚠️ CRITICAL: Backend Compilation Architecture

**NEVER compile the Rust backend on macOS and deploy to Linux!**

The dev server (`192.168.1.228`) is **x86_64 Linux (Debian 12)**. Binaries compiled on macOS (even with cross-compilation) can cause "Exec format error" due to:
- Different architecture (macOS ARM64/Intel vs Linux x86_64)
- Different libc (macOS vs glibc)
- Different system call interfaces

**ALWAYS build the backend directly on the Linux dev server.**

### Deployment Procedures

1. **Backend** (MUST build on Linux — use rsync then build on server):
   ```bash
   # From project root. Sync source to server (exclude local target/.git).
   sshpass -p "archipelago" rsync -avz --exclude target --exclude .git -e "ssh -o StrictHostKeyChecking=no" \
     core/ archipelago@192.168.1.228:~/archy/core/

   # Build on server and deploy binary
   sshpass -p "archipelago" ssh -o StrictHostKeyChecking=no archipelago@192.168.1.228 \
     'source ~/.cargo/env && cd ~/archy/core/archipelago && cargo build --release && \
      sudo systemctl stop archipelago && \
      sudo cp ~/archy/core/target/release/archipelago /usr/local/bin/ && \
      sudo systemctl start archipelago'
   ```
   **Do not** build the binary on macOS and copy it; always rsync source and build on the server.

2. **Frontend** (can build locally):
   ```bash
   # Build locally (macOS is fine for frontend)
   cd neode-ui
   npm run build
   
   # Deploy to server
   rsync -avz ../web/dist/neode-ui/ archipelago@192.168.1.228:/tmp/neode-ui-build/
   ssh archipelago@192.168.1.228 'sudo rm -rf /opt/archipelago/web-ui/* && sudo cp -r /tmp/neode-ui-build/* /opt/archipelago/web-ui/ && sudo chown -R www-data:www-data /opt/archipelago/web-ui'
   ```

3. **Container Images** (Docker/Podman):
   ```bash
   # Build locally and push to server
   cd docker/<app-name>
   podman build -t localhost/<app-name>:latest .
   podman save localhost/<app-name>:latest | ssh archipelago@192.168.1.228 'podman load'
   ```

## CRITICAL RULES

### 🚨 NEVER VIOLATE THESE

1. **ALWAYS deploy to the live development server (192.168.1.228)** for testing
2. **After every change: sync and build on the live server.** When you finish implementing a feature or fix, run the deploy script so the live server has the latest code. Command: `./scripts/deploy-to-target.sh --live` (from project root). If SSH is not available in the current environment, tell the user to run it locally. Do not skip this step. **App UIs** (e.g. `docker/lnd-ui/`, `docker/bitcoin-ui/`) are served by their own containers; the deploy script rebuilds the LND UI image and restarts its container so changes to the LND UI are visible after deploy.
3. **🔴 NEVER EVER compile the Rust backend on macOS and deploy to Linux** 
   - Dev server is `x86_64 Linux (Debian 12)`
   - Always build backend **ON the Linux server** using `source ~/.cargo/env && cargo build --release`
   - macOS binaries will cause "Exec format error" and break the system
   - Frontend (Vue.js) CAN be built on macOS - it's just HTML/CSS/JS
4. **The ISO must capture the CURRENT STATE of the dev server**, not build from source
5. **Frontend build output is in `web/dist/neode-ui/`**, NOT `neode-ui/dist/`
6. **Nginx serves on port 80** and proxies backend on `localhost:5678`
7. **App icons are in `neode-ui/public/assets/img/app-icons/`**
8. **The auto-installer ISO is the ONLY way to deploy** - no live systems

## Testing Checklist

Before creating ISO:
- [ ] Backend running on dev server (`curl http://192.168.1.228:5678/health`)
- [ ] Frontend accessible (`curl http://192.168.1.228/`)
- [ ] Web UI shows correct apps and icons
- [ ] API calls working (check browser console)
- [ ] All systemd services enabled and running

After flashing ISO:
- [ ] ISO boots to installer menu
- [ ] Auto-installer detects internal disk
- [ ] Installation completes without errors
- [ ] System reboots and shows Web UI URL
- [ ] Web UI accessible at `http://<IP>`
- [ ] Backend API responding
- [ ] Apps visible in marketplace

## Common Issues

**Issue**: ISO boots to prompt instead of auto-starting
- **Cause**: Using `build-debian-iso.sh` (live system) instead of `build-auto-installer-iso.sh`
- **Fix**: Use correct auto-installer script

**Issue**: macOS backend binary on Linux server ("Exec format error")
- **Cause**: Compiling Rust backend on macOS and copying to Linux server
- **Symptom**: `systemd` service fails with "status=203/EXEC" and "Failed to execute: Exec format error"
- **Why it happens**: Different architectures and system ABIs between macOS and Linux
- **Fix**: **ALWAYS build the backend ON the Linux server**:
  ```bash
  ssh archipelago@192.168.1.228
  cd ~/archy/core/archipelago
  source ~/.cargo/env
  cargo build --release
  sudo systemctl stop archipelago
  sudo cp ../target/release/archipelago /usr/local/bin/
  sudo systemctl start archipelago
  ```
- **Prevention**: Never use local `cargo build` for deployment - always build on target system

**Issue**: Frontend not updating on server
- **Cause**: Building to wrong output directory or not deploying to correct Nginx root
- **Fix**: Build to `web/dist/neode-ui/`, deploy to `/opt/archipelago/web-ui`

**Issue**: ISO doesn't have latest changes
- **Cause**: Building from source instead of capturing live server state
- **Fix**: Modify build script to snapshot dev server, not compile from scratch

## Next Steps

- [ ] Fix `build-auto-installer-iso.sh` to capture live server state
- [ ] Create snapshot script for dev server
- [ ] Document container image bundling process
- [ ] Create automated testing framework
- [ ] Set up CI/CD for ISO builds