diff --git a/.gitignore b/.gitignore index 894b91d..2f341f5 100644 --- a/.gitignore +++ b/.gitignore @@ -34,4 +34,8 @@ server/docker-compose.yml .idea/ *.swp .DS_Store -.claude/ +.claude/* +# ...except Claude Code skills (e.g. agent-run instructions for this +# app) -- those are project tooling worth sharing, not personal/local +# state like settings.local.json or worktrees/. +!.claude/skills/ diff --git a/server/.claude/skills/run-server/.gitignore b/server/.claude/skills/run-server/.gitignore new file mode 100644 index 0000000..63e2157 --- /dev/null +++ b/server/.claude/skills/run-server/.gitignore @@ -0,0 +1,2 @@ +# Generated by setup.sh -- bakes in this host's /tmp paths, not portable. +env.sh diff --git a/server/.claude/skills/run-server/SKILL.md b/server/.claude/skills/run-server/SKILL.md new file mode 100644 index 0000000..e4edf27 --- /dev/null +++ b/server/.claude/skills/run-server/SKILL.md @@ -0,0 +1,205 @@ +--- +name: run-server +description: Build, run, and drive the espresso_frame FastAPI server (server/) -- start it against a scratch DB, browser-test its UI, run its pytest suite. Use when asked to start the server, take a screenshot of a frame page, click through the web UI, or verify a server/UI change actually works. +--- + +The espresso_frame server is a FastAPI app (`app.main:app`) with a +server-rendered Jinja UI. For agent/automated use it's driven by a +Playwright REPL at `.claude/skills/run-server/driver.py`, run under +tmux -- `chromium-cli` isn't available in this container, so this +driver replaces it (same command vocabulary: nav/wait-for/click/fill/ +screenshot/eval/console-errors). + +This container ships with **no Python, Node, Docker, or browser, and +no sudo**. `setup.sh` bootstraps everything non-root; it's the bulk of +what makes this skill non-obvious. Run it once per fresh container. + +All paths below are relative to `server/`. + +## Prerequisites + +None to install manually -- `setup.sh` does it all without root, using +only `curl`/`apt-get download`/`dpkg-deb -x` (never `apt-get install`, +which needs root). It downloads ~450MB total (Python, Chromium, tmux, +shared libs) on first run. + +```bash +bash .claude/skills/run-server/setup.sh +``` + +Re-running is safe and fast -- every step checks whether it already +happened (venv exists? chromium downloaded? libs extracted? tmux +present?) before doing any work. + +This creates: +- `.venv/` -- Python 3.12 + `requirements.txt` + `playwright`, via `uv` + (a static Rust binary that fetches its own Python -- no compiler + needed, install via `curl -LsSf https://astral.sh/uv/install.sh | sh`) +- `~/.cache/ms-playwright/` -- Chromium (full `chrome` + headless-shell) +- `/tmp/run-server-chromium-deps/` -- Chromium's + tmux's shared libs + and fonts, extracted (not installed) from `.deb` files +- `.claude/skills/run-server/env.sh` -- the `PATH`/`LD_LIBRARY_PATH`/ + `FONTCONFIG_PATH`/`RUN_SERVER_CHROME_BIN` exports the driver needs + (gitignored -- host-specific `/tmp` paths, regenerated by setup.sh) + +## Run (agent path) + +**1. Start the server** against a scratch DB (never the real +deployment's data -- see `CLAUDE.md`): + +```bash +bash .claude/skills/run-server/start-server.sh +# -> server PID up on http://127.0.0.1:8420 (log: /tmp/run-server-scratch/server.log) +``` + +Optional args: `start-server.sh [scratch-dir] [port]` (defaults +`/tmp/run-server-scratch` / `8420`). + +**2. Drive it**, wrapped in tmux so you can send one command at a time +and read the response before sending the next. `tmux` itself was +extracted the same non-root way as Chromium's libs (see Prerequisites) +and needs `LD_LIBRARY_PATH` set in *this* shell too, not just inside +the pane -- source `env.sh` before the first `tmux` call: + +```bash +source .claude/skills/run-server/env.sh +tmux new-session -d -s runserver -x 200 -y 50 +tmux send-keys -t runserver \ + 'source .claude/skills/run-server/env.sh && .venv/bin/python .claude/skills/run-server/driver.py' Enter +timeout 20 bash -c 'until tmux capture-pane -t runserver -p | grep -q "driver>"; do sleep 0.3; done' + +# Every fresh scratch DB starts with no users -- bootstrap-admin +# completes first-run /setup and logs in (see Commands table): +tmux send-keys -t runserver 'bootstrap-admin' Enter +timeout 15 bash -c 'until tmux capture-pane -t runserver -p | grep -q "bootstrapped admin"; do sleep 0.3; done' + +tmux send-keys -t runserver 'nav /frames/1/config' Enter +tmux send-keys -t runserver 'wait-for #frame-preview-thumb' Enter +tmux send-keys -t runserver 'screenshot before' Enter +tmux capture-pane -t runserver -p +``` + +Poll for a specific marker between `send-keys` and `capture-pane` +(`driver>`, `bootstrapped admin`, `screenshot:`, ...) rather than a +fixed `sleep` -- it's faster and fails loudly instead of capturing a +half-rendered screen. Give each poll its own `timeout` (~15-20s); don't +chain many polling loops inside one shell invocation -- see Gotchas. + +Screenshots land in `/tmp/run-server-shots/` (override: +`SCREENSHOT_DIR`). **Actually look at them** -- a blank or error-page +screenshot is a failure to launch, not success. + +### Driver commands + +| command | what it does | +|---|---| +| `nav ` | navigate (relative paths resolve against `http://127.0.0.1:8420`, override with `RUN_SERVER_BASE_URL`) | +| `wait-for ` | wait up to 10s for a selector (plain CSS -- see Gotchas for attribute selectors) | +| `click ` | click an element | +| `fill ` | fill an input | +| `press ` | keyboard press (e.g. `Enter`) | +| `screenshot [name]` | → `/tmp/run-server-shots/.png` | +| `eval ` | evaluate JS in the page, prints JSON | +| `console-errors` | prints all captured console/page errors as a JSON array | +| `is-open ` | prints `true`/`false` for a `` element's `.open` -- use this instead of `wait-for sel[open]` | +| `bootstrap-admin [user] [pass]` | completes first-run `/setup` (defaults `admin`/`testpassword123`); links frame #1 and logs in | +| `quit` | closes the browser, exits the driver | + +**3. Stop the server** when done: + +```bash +bash .claude/skills/run-server/stop-server.sh +tmux kill-session -t runserver +``` + +## Run (human path) + +```bash +cd server +DATABASE_URL="sqlite:////tmp/dev.db" CONFIG_PATH="/tmp/dev-config.json" \ + .venv/bin/uvicorn app.main:app --reload --port 8420 +``` + +Open `http://localhost:8420` in a real browser. `start.sh` (the Docker +entrypoint) is not this -- it also launches the Node whiteboard +render-service sidecar, which needs Node (not installed here) and +isn't needed for most UI testing. + +## Test + +```bash +.venv/bin/pytest +``` + +Uses its own tempfile SQLite per run (`tests/conftest.py`) -- no setup +needed beyond the venv. + +--- + +## Gotchas + +- **`chrome-headless-shell` (Playwright's default headless target) + crashes on basic calls in this container**, e.g. `page.set_content()` + returns `TargetClosedError`, even after every `ldd`-reported missing + library is resolved. The full `chrome` binary (`chromium-*/chrome-linux64/chrome`) + + `--no-sandbox` is stable; `driver.py` and `setup.sh` both use it, + not the headless-shell default. + +- **Missing fonts silently break `fill()`, not just rendering.** Before + `fontconfig`/`libfontconfig1` were extracted and `fonts.conf` pointed + at the extracted font dir, `page.fill()` ran with no error but left + inputs empty (`input_value()` returned `""`), and all text rendered + invisible in screenshots. It looks like a scripting bug, not a + missing-lib problem -- if `fill` silently no-ops, suspect fonts + before suspecting the selector or a race. + +- **No `apt-get install` / `playwright install-deps` (no root) and no + `apt-get update` into the real `/var/lib/apt/lists` (root-owned).** + Worked around by redirecting apt's state dirs to a scratch, + user-writable path (`-o Dir::State::Lists=... -o Dir::Cache=...`), + which makes plain `update` and `install --download-only --print-uris` + work as a non-root user; then `dpkg-deb -x ` (extract, + not install) needs no root either. `setup.sh` does this for + Chromium's deps *and* for `tmux` itself, which also isn't + preinstalled. + +- **`wait-for` with an attribute selector like `#frame-preview-dialog[open]` + is unreliable through `tmux send-keys`** -- shell/tmux escaping of + `[`/`]` easily mangles it (seen: a real 10s Playwright timeout from a + garbled selector, not a fast failure). Use the app-specific `is-open + ` command instead of `wait-for sel[open]` to check a + ``'s open state. + +- **Don't chain many `tmux send-keys` + polling-`timeout` loops inside + one shell invocation.** Each poll can legitimately take up to its own + timeout (e.g. 15s) if a selector is wrong; five or six chained in one + command can add up past this tool's own command timeout even though + each individual step is fine. Send one or two commands per shell + call and check the pane before continuing. + +- **A crashed `chrome-headless-shell` process can leave a large core + dump file** (`server/core`, ~170MB, from the crash described above) + if core dumps are enabled. It's not part of the app -- delete it, and + use full `chrome` (as `driver.py` does) to avoid triggering it again. + +## Troubleshooting + +- **`error while loading shared libraries: libglib-2.0.so.0` (or similar) + when launching Chromium directly**: `LD_LIBRARY_PATH` isn't set -- + source `.claude/skills/run-server/env.sh` first, or run through + `driver.py`, which reads `RUN_SERVER_CHROME_BIN` from it. +- **`fc-list` prints nothing after extracting fonts**: `fonts.conf`'s + `` entries still point at the real (unpopulated) `/usr/share/fonts`. + `setup.sh` patches this with `sed`; if you extracted packages by hand, + do the same. +- **`E: Could not open lock file ... Permission denied` from `apt-get`**: + you're missing the `-o Dir::State::Lists=... -o Dir::Cache=...` + overrides -- plain `apt-get update`/`install` always needs root here. +- **`tmux: command not found`**: not preinstalled and no sudo; run + `setup.sh`, which fetches it the same non-root way as Chromium's libs. +- **`tmux: error while loading shared libraries: libutempter.so.0`**: + you sourced `env.sh` inside the driver's tmux pane but not in the + shell that *invokes* `tmux` itself -- `tmux` was extracted from the + same non-root `.deb` set as Chromium and needs `LD_LIBRARY_PATH` too. + `source .claude/skills/run-server/env.sh` before the first `tmux` + command, not just inside `send-keys`. diff --git a/server/.claude/skills/run-server/driver.py b/server/.claude/skills/run-server/driver.py new file mode 100644 index 0000000..988dcf3 --- /dev/null +++ b/server/.claude/skills/run-server/driver.py @@ -0,0 +1,156 @@ +#!/usr/bin/env python3 +"""REPL driver for the espresso_frame server's web UI. + +Playwright-based since chromium-cli isn't available in this container. +Reads one command per line from stdin, prints a result line -- built +for tmux send-keys/capture-pane use by an agent. Vocabulary mirrors +chromium-cli where it overlaps (nav/wait-for/click/fill/screenshot/ +eval/console-errors). + +Requires setup.sh to have run first (Python venv + Playwright Chromium ++ the non-root shared-lib/font extraction). Run via: + + server/.claude/skills/run-server/env.sh sourced, then + .venv/bin/python server/.claude/skills/run-server/driver.py + +See SKILL.md for the full agent-path invocation (tmux wrapping etc). +""" +import glob +import json +import os +import sys + +from playwright.sync_api import sync_playwright + +SHOT_DIR = os.environ.get("SCREENSHOT_DIR", "/tmp/run-server-shots") +os.makedirs(SHOT_DIR, exist_ok=True) +BASE = os.environ.get("RUN_SERVER_BASE_URL", "http://127.0.0.1:8420") + + +def find_chrome() -> str: + override = os.environ.get("RUN_SERVER_CHROME_BIN") + if override and os.path.exists(override): + return override + matches = glob.glob(os.path.expanduser("~/.cache/ms-playwright/chromium-*/chrome-linux64/chrome")) + if not matches: + sys.exit("chrome binary not found -- run setup.sh first") + return matches[0] + + +pw = sync_playwright().start() +# The FULL `chrome` binary, not chrome-headless-shell (Playwright's +# default headless target): chrome-headless-shell crashed on basic +# calls like set_content() in this container even once every +# ldd-reported missing lib was resolved. Full chrome + --no-sandbox is +# stable here. +browser = pw.chromium.launch(executable_path=find_chrome(), args=["--no-sandbox"]) +page = browser.new_page(viewport={"width": 1280, "height": 900}) +console_errors: list[str] = [] +page.on("console", lambda msg: console_errors.append(msg.text) if msg.type == "error" else None) +page.on("pageerror", lambda exc: console_errors.append(str(exc))) + + +def cmd_nav(arg): + url = arg if arg.startswith("http") else BASE + arg + page.goto(url) + print("nav ->", page.url) + + +def cmd_wait_for(arg): + page.wait_for_selector(arg, timeout=10000) + print("found:", arg) + + +def cmd_click(arg): + page.click(arg) + print("clicked:", arg) + + +def cmd_fill(arg): + sel, _, value = arg.partition(" ") + page.fill(sel, value) + print("filled:", sel, "=", value) + + +def cmd_press(arg): + page.keyboard.press(arg) + print("pressed:", arg) + + +def cmd_screenshot(arg): + name = arg or f"ss-{len(os.listdir(SHOT_DIR))}" + path = os.path.join(SHOT_DIR, name + ".png") + page.screenshot(path=path) + print("screenshot:", path) + + +def cmd_eval(arg): + try: + print(json.dumps(page.evaluate(arg))) + except Exception as e: + print("ERROR:", e) + + +def cmd_console_errors(_arg): + print(json.dumps(console_errors)) + + +def cmd_is_open(arg): + """App-specific: print whether a element is open (true/false).""" + print(json.dumps(page.eval_on_selector(arg, "el => el.open"))) + + +def cmd_bootstrap_admin(arg): + """App-specific: complete first-run /setup (username/password args, + default admin/testpassword123). Every scratch DB starts with no + users, and /setup is the only way in -- it also auto-links the + pre-existing frame #1 (created by migrations) to the new admin, so + /frames/1/... is reachable right after this.""" + parts = arg.split() + username = parts[0] if len(parts) > 0 else "admin" + password = parts[1] if len(parts) > 1 else "testpassword123" + page.goto(BASE + "/setup") + page.fill("input[name=username]", username) + page.fill("input[name=password]", password) + page.click("button[type=submit]") + page.wait_for_load_state("networkidle") + print("bootstrapped admin, now at:", page.url) + + +def cmd_quit(_arg): + browser.close() + pw.stop() + sys.exit(0) + + +COMMANDS = { + "nav": cmd_nav, + "wait-for": cmd_wait_for, + "click": cmd_click, + "fill": cmd_fill, + "press": cmd_press, + "screenshot": cmd_screenshot, + "eval": cmd_eval, + "console-errors": cmd_console_errors, + "is-open": cmd_is_open, + "bootstrap-admin": cmd_bootstrap_admin, + "quit": cmd_quit, +} + +print("run-server driver -- commands:", ", ".join(COMMANDS), flush=True) +print("driver>", end=" ", flush=True) +for line in sys.stdin: + line = line.strip() + if not line: + print("driver>", end=" ", flush=True) + continue + cmd, _, rest = line.partition(" ") + fn = COMMANDS.get(cmd) + if fn is None: + print("unknown command:", cmd, "-- try one of:", ", ".join(COMMANDS)) + else: + try: + fn(rest) + except Exception as e: + print("ERROR:", e) + print("driver>", end=" ", flush=True) diff --git a/server/.claude/skills/run-server/setup.sh b/server/.claude/skills/run-server/setup.sh new file mode 100755 index 0000000..52b78bb --- /dev/null +++ b/server/.claude/skills/run-server/setup.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# One-time (idempotent) environment bootstrap for running the +# espresso_frame FastAPI server and browser-driving its UI, in a +# container that ships with NO Python/Node/Docker/browser and NO sudo. +# Re-run any time; every step checks whether it already happened. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")"/../../.. # -> server/ + +UV_BIN="$HOME/.local/bin/uv" +DEPS_ROOT="/tmp/run-server-chromium-deps" +APT_WORK="/tmp/apt-work-run-server" +SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="$SKILL_DIR/env.sh" + +# 1. uv: a static Rust binary that can fetch its own Python build with +# no C compiler needed (this container has none). +if [ ! -x "$UV_BIN" ]; then + echo "installing uv..." + curl -LsSf https://astral.sh/uv/install.sh | sh +fi + +# 2. Python 3.12 + venv + server deps +if [ ! -x .venv/bin/uvicorn ]; then + echo "creating venv + installing server deps..." + "$UV_BIN" python install 3.12 + "$UV_BIN" venv --python 3.12 .venv + "$UV_BIN" pip install -r requirements.txt +fi + +# 3. Playwright (Python) + its Chromium download (~280MB: full chrome + +# chrome-headless-shell + ffmpeg) +if ! .venv/bin/python -c "import playwright" 2>/dev/null; then + echo "installing playwright..." + "$UV_BIN" pip install playwright +fi +if ! ls "$HOME"/.cache/ms-playwright/chromium-*/chrome-linux64/chrome >/dev/null 2>&1; then + echo "downloading chromium..." + .venv/bin/playwright install chromium +fi + +# 4. Chromium's shared libs + fonts. `playwright install-deps` and +# `apt-get install` both need root; neither is available. Instead: +# download the .deb files directly (apt-get download works read-only +# without root once given a user-writable state dir) and extract +# (not install) them with dpkg-deb -x, which needs no root either. +if [ ! -f "$DEPS_ROOT/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0" ]; then + echo "fetching chromium's shared libs + fonts (non-root)..." + mkdir -p "$APT_WORK/lists" "$APT_WORK/cache/archives/partial" "$APT_WORK/debs" "$DEPS_ROOT" + + apt-get -o Dir::State::Lists="$APT_WORK/lists" -o Dir::Cache="$APT_WORK/cache" \ + -o Dir::Etc::SourceParts=/dev/null update + + PKGS="libglib2.0-0t64 libnspr4 libnss3 libatk1.0-0t64 libatk-bridge2.0-0t64 + libdbus-1-3 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 + libxkbcommon0 libasound2t64 libatspi2.0-0t64 libcups2t64 libcairo2 + libpango-1.0-0 libpangocairo-1.0-0 libx11-6 libxcb1 libxext6 + fonts-liberation fontconfig libfontconfig1" + # ^ first row: what chrome-headless-shell's ldd reported missing. + # second row: what the FULL chrome binary additionally needed (we use + # full chrome, not headless-shell -- see Gotchas in SKILL.md). + + apt-get -o Dir::State::Lists="$APT_WORK/lists" -o Dir::Cache="$APT_WORK/cache" \ + -o Dir::Etc::SourceParts=/dev/null install --download-only --reinstall -y \ + --print-uris $PKGS | grep -oP "^'[^']+'" | tr -d "'" > "$APT_WORK/urls.txt" + + (cd "$APT_WORK/debs" && xargs -n1 -P8 curl -sS -O --max-time 30) < "$APT_WORK/urls.txt" + for f in "$APT_WORK"/debs/*.deb; do dpkg-deb -x "$f" "$DEPS_ROOT"; done + + # fonts.conf as shipped points at the real /usr/share/fonts, which is + # root-owned and has nothing extracted into it. Point it at our + # extracted copy instead, and give it a writable cache dir. + mkdir -p /tmp/run-server-fontcache + sed -i "s#/usr/share/fonts#$DEPS_ROOT/usr/share/fonts#" \ + "$DEPS_ROOT/etc/fonts/fonts.conf" + sed -i "s#.*#/tmp/run-server-fontcache#" \ + "$DEPS_ROOT/etc/fonts/fonts.conf" + + PATH="$DEPS_ROOT/usr/bin:$PATH" \ + LD_LIBRARY_PATH="$DEPS_ROOT/usr/lib/x86_64-linux-gnu:$DEPS_ROOT/lib/x86_64-linux-gnu" \ + FONTCONFIG_PATH="$DEPS_ROOT/etc/fonts" \ + "$DEPS_ROOT/usr/bin/fc-cache" -f +fi + +# 5. tmux -- also missing, also no apt/sudo. Same non-root download + +# dpkg-deb -x trick, into the same extracted root (so its `usr/bin` is +# already on PATH via env.sh). +if [ ! -f "$DEPS_ROOT/usr/bin/tmux" ]; then + echo "fetching tmux (non-root)..." + mkdir -p "$APT_WORK/lists" "$APT_WORK/cache/archives/partial" "$APT_WORK/debs" "$DEPS_ROOT" + apt-get -o Dir::State::Lists="$APT_WORK/lists" -o Dir::Cache="$APT_WORK/cache" \ + -o Dir::Etc::SourceParts=/dev/null install --download-only --reinstall -y \ + --print-uris tmux | grep -oP "^'[^']+'" | tr -d "'" > "$APT_WORK/tmux_urls.txt" + (cd "$APT_WORK/debs" && xargs -n1 -P3 curl -sS -O --max-time 30) < "$APT_WORK/tmux_urls.txt" + for f in $(sed -E 's#.*/##' "$APT_WORK/tmux_urls.txt"); do dpkg-deb -x "$APT_WORK/debs/$f" "$DEPS_ROOT"; done +fi + +CHROME_BIN="$(ls "$HOME"/.cache/ms-playwright/chromium-*/chrome-linux64/chrome | head -1)" + +cat > "$ENV_FILE" < $ENV_FILE" diff --git a/server/.claude/skills/run-server/start-server.sh b/server/.claude/skills/run-server/start-server.sh new file mode 100755 index 0000000..7e65326 --- /dev/null +++ b/server/.claude/skills/run-server/start-server.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Background-launch the server against a scratch DB/config -- never the +# real deployment's data (see CLAUDE.md). Waits for readiness, prints +# the PID and log path. Usage: ./start-server.sh [scratch-dir] [port] +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")"/../../.. # -> server/ + +SCRATCH="${1:-/tmp/run-server-scratch}" +PORT="${2:-8420}" +mkdir -p "$SCRATCH" + +DATABASE_URL="sqlite:///$SCRATCH/test.db" \ +CONFIG_PATH="$SCRATCH/config.json" \ +.venv/bin/uvicorn app.main:app --host 127.0.0.1 --port "$PORT" \ + > "$SCRATCH/server.log" 2>&1 & +PID=$! +echo "$PID" > "$SCRATCH/server.pid" + +for _ in $(seq 1 30); do + curl -sf -o /dev/null "http://127.0.0.1:$PORT/" && break + sleep 0.5 +done + +if ! curl -sf -o /dev/null "http://127.0.0.1:$PORT/"; then + echo "server did not become ready -- check $SCRATCH/server.log" >&2 + exit 1 +fi + +echo "server PID $PID up on http://127.0.0.1:$PORT (log: $SCRATCH/server.log, db: $SCRATCH/test.db)" diff --git a/server/.claude/skills/run-server/stop-server.sh b/server/.claude/skills/run-server/stop-server.sh new file mode 100755 index 0000000..921aa51 --- /dev/null +++ b/server/.claude/skills/run-server/stop-server.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Usage: ./stop-server.sh [scratch-dir] +SCRATCH="${1:-/tmp/run-server-scratch}" +if [ -f "$SCRATCH/server.pid" ]; then + kill "$(cat "$SCRATCH/server.pid")" 2>/dev/null || true + rm -f "$SCRATCH/server.pid" + echo "stopped" +else + echo "no $SCRATCH/server.pid -- nothing to stop" +fi