From 914eaed71c4dc3e56cc6d3ea266cdfc7da3403ef Mon Sep 17 00:00:00 2001 From: Thomas Faour Date: Fri, 24 Jul 2026 20:04:08 -0400 Subject: [PATCH] Add CLAUDE.md and docs/widgets.md; fix stale single-mode architecture docs Repo-tracked context so a fresh Claude Code session (this machine or a remote/cloud one) gets accurate project context without relying on this session's local, machine-specific memory: repo conventions (no co-author trailers, flag copyleft deps explicitly, scope security gates broadly -- each backed by a real past incident), testing/deploy workflow, and pointers into the existing docs. docs/architecture.md's sequence diagram and boot-flow text still described the pre-widget-system single-photo-queue model (e.g. "force- advance to next queued photo") even though that was fully replaced by the widget system across this branch's recent history -- fixed, and added docs/widgets.md distilling the widget system's actual design (data model, grid placement, compositor, button-action dispatch) as current-state documentation, including the still-open Phase 6 cleanup (legacy Frame columns not yet dropped) as a known gap. --- CLAUDE.md | 71 +++++++++++++++++++++ README.md | 5 +- docs/architecture.md | 21 ++++-- docs/widgets.md | 149 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 239 insertions(+), 7 deletions(-) create mode 100644 CLAUDE.md create mode 100644 docs/widgets.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..650d664 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,71 @@ +# espresso_frame + +A DIY e-ink photo frame: an ESP32-C6 (`firmware/`, ESP-IDF) driving a +Waveshare 7.3" E Ink Spectra 6 panel (800x480, 6-color, SPI), paired with a +self-hosted FastAPI server (`server/`) that pulls from Immich, does all +image processing (crop/dither/quantize/pack), and serves a placeable +photos/calendar/whiteboard widget system to the device. + +Start here, don't re-derive from scratch: +- [`docs/architecture.md`](docs/architecture.md) -- how firmware and + server talk (sequence diagram, boot flow). +- [`docs/widgets.md`](docs/widgets.md) -- the server-side widget system + (data model, grid placement, compositor, button-action dispatch). Notes + a known gap at the bottom (legacy `Frame` columns not yet dropped). +- [`docs/hardware.md`](docs/hardware.md) -- wiring. +- [`server/README.md`](server/README.md), [`firmware/README.md`](firmware/README.md) + -- per-component setup, config, and a lot of accumulated gotchas + (Immich API shape, TLS trust-anchor details, button GPIO wakeup + quirks, etc.) -- check these before assuming something is a new bug. + +## Conventions specific to this repo + +- **No `Co-Authored-By: Claude` trailers in commits.** Attribution lives + in the root [`README.md`](README.md) instead (see its last line) -- + the maintainer's explicit preference, not the default. +- **Copyleft dependencies need an explicit flag, not a silent decision.** + Before adding anything LGPL/GPL/AGPL (or unclear), verify the actual + license via `pip show`/package metadata -- including transitive deps, + not just the top-level package -- and present the finding and tradeoff + in plain text rather than picking an approach unilaterally (hand-rolling + an alternative, swapping packages, silently accepting it). This project + has knowingly accepted AGPL-3.0-or-later exposure once already + (`icalendar-searcher`, a transitive dep of `caldav`) as a deliberate, + explicit call -- not a precedent for skipping the check next time. +- **Scope new auth/access-control broadly, not just to the literal + endpoint named.** When a request changes the trust model (e.g. adding + public-internet exposure), apply the new gate to every endpoint serving + real data or performing a real action, and call out anything you're + tempted to exclude and why. This repo shipped a token gate once that + covered `/api/*` but left `/frame/image` -- the actual photo bytes -- + open; caught immediately in production. + +## Working in this repo + +- **Server tests**: `cd server && pytest` (SQLite, fixtures wipe/reseed + between tests -- see `tests/conftest.py`). Migration changes need a + matching test in `tests/test_migrations.py`; anything touching + `require_frame_view`/`require_frame_control` boundaries needs a + same-shape permission test (see `tests/test_permission_boundaries.py` + and `tests/test_button_actions.py` for the pattern: owner, linked user, + unrelated user, logged out). +- **UI changes**: verify in a real browser (Playwright), not just by + reading the JS -- this project has hit multiple bugs that only showed up + live (mobile viewport CSS collapse, a dialog's status message landing + behind its own backdrop, a JSON/form-urlencoded body mismatch). Spin up + `uvicorn app.main:app` against a scratch `DATABASE_URL`/`CONFIG_PATH` + sqlite file, don't touch the real deployment's data. +- **Deploy**: Gitea Actions at `git.thumeit.com/tfaour/espresso_frame` + (`.gitea/workflows/server-docker-build.yml`: `test` -> `build-and-push` + -> `deploy` on any push to `main` touching `server/**`; `deploy` SSHes + into the host as `espressoframe_deployer` and runs `docker compose pull + && docker compose up -d`). A separate workflow + (`firmware-release-build.yml`) builds+publishes firmware binaries as + Gitea release assets when `firmware/version.txt` changes. Poll CI status + with `curl https://git.thumeit.com/api/v1/repos/tfaour/espresso_frame/actions/tasks` + rather than asking the user to check. +- **Device-facing paths are frozen.** `/frame/image`, `/frame/advance`, + `/frame/back`, `/frame/config`, `/frame/battery`, `/frame/firmware` and + their exact JSON key names (`refresh_interval_s`, `firmware_version`, + etc.) are baked into deployed firmware -- never rename or restructure + these without a firmware-side migration story to match. diff --git a/README.md b/README.md index 64bd937..d628a14 100644 --- a/README.md +++ b/README.md @@ -24,9 +24,10 @@ time in deep sleep. - ESP32-C6 dev board (8MB flash) - [Waveshare 7.3" E Ink Spectra 6 (E6)](https://www.waveshare.com/7.3inch-e-paper-hat-e.htm) panel -- 800x480, 6-color, SPI -See [`docs/hardware.md`](docs/hardware.md) for wiring and +See [`docs/hardware.md`](docs/hardware.md) for wiring, [`docs/architecture.md`](docs/architecture.md) for how the two halves talk -to each other. +to each other, and [`docs/widgets.md`](docs/widgets.md) for the server's +placeable photos/calendar/whiteboard widget system. ## Getting started diff --git a/docs/architecture.md b/docs/architecture.md index e661365..a95c5c6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -22,17 +22,17 @@ sequenceDiagram Frame->>Frame: Connect to home WiFi alt next-photo button pressed Frame->>Server: POST /frame/advance - Server->>Server: Force-advance to next queued photo, reset interval clock + Server->>Server: Run every action assigned to NEXT, in order
(may span several widgets -- see docs/widgets.md) else back-photo button pressed Frame->>Server: POST /frame/back - Server->>Server: Return to previously-current photo (bounded history),
reset interval clock + Server->>Server: Run every action assigned to BACK, in order else normal wake Frame->>Server: GET /frame/image - Server->>Server: Advance only if refresh_interval_s has elapsed
since the current photo was set -- otherwise a no-op + Server->>Server: Render every widget on the panel into its own region
(each independently idempotent -- a photo widget only
actually advances once its own refresh_interval_s has elapsed) end - Server->>Immich: List album assets / download preview / faces + Server->>Immich: List album assets / download preview / faces
(once per photo widget on the panel) Immich-->>Server: JPEG + face bounding boxes - Server->>Server: Crop (face-aware) + quantize (dither) + pack 4bpp + Server->>Server: Composite every widget's region onto one canvas,
then enhance/overlay/quantize (dither)/pack 4bpp once Server-->>Frame: 192,000 raw bytes, streamed Frame->>Frame: Write to panel SPI buffer, compute CRC32 alt CRC unchanged since last physical refresh @@ -45,6 +45,17 @@ sequenceDiagram Frame->>Frame: Deep sleep (server-configured interval, or a short
retry interval on any failure) ``` +The device-facing endpoints above (`/frame/image`, `/frame/advance`, +`/frame/back`, `/frame/config`) are frozen -- baked into deployed firmware +-- and unchanged by any of this. What *does* change server-side: a frame's +panel isn't a single fixed "mode" anymore, it holds an arbitrary +arrangement of independently placed/sized widgets (photos/calendar/ +whiteboard, including several of the same type), each rendered into its +own region and composited together, with NEXT/BACK each mapped to their +own ordered list of per-widget actions rather than one fixed meaning. See +[`docs/widgets.md`](widgets.md) for the widget system's data model, +placement grid, and button-action dispatch. + ## Firmware boot flow 1. **No stored config** (first boot, or NVS erased): bring up the display, diff --git a/docs/widgets.md b/docs/widgets.md new file mode 100644 index 0000000..5fc4347 --- /dev/null +++ b/docs/widgets.md @@ -0,0 +1,149 @@ +# Widget system + +A frame's panel isn't one fixed "mode" anymore -- it holds N independently +placed/sized widgets (photos/calendar/whiteboard), like arranging icons on +an Android home screen. A frame can hold several widgets of the same type +(e.g. two photo widgets pointed at different Immich albums side by side). +This replaced an earlier design where `Frame.mode` picked exactly one +full-panel renderer; that column (and the other now-dead per-mode `Frame` +columns it left behind -- `album_id`, `calendar_*`, `whiteboard_*`, etc.) +is still physically present but unused, pending a final cleanup migration +(see "Known gaps" below). + +The device-facing contract is unchanged by any of this: `GET /frame/image`, +`POST /frame/advance`, `POST /frame/back` are the same frozen paths +firmware has always called (see `docs/architecture.md`) -- what changed is +entirely server-side, in how those endpoints decide what to render and what +a button press does. + +## Data model + +- `Widget` (`server/app/models.py`): `id`, `frame_id`, `widget_type` + (`"photos"` | `"calendar"` | `"whiteboard"`), `x`/`y`/`w`/`h` (grid + cells), `sort_order`. Widgets never overlap (enforced server-side in + `routers/api_widgets.py`, re-validated regardless of what the client + already checked) -- that's what keeps compositing simple: no z-order, + no blending, just N independent regions pasted onto one shared canvas. +- Per-type 1:1 extension tables -- `PhotoWidgetConfig`, + `CalendarWidgetConfig`, `WhiteboardWidgetConfig`, each keyed by + `widget_id` with `ondelete="CASCADE"` -- rather than one wide table with + every type's mostly-irrelevant columns. `PhotoWidgetConfig` mirrors + `app/photo_queue.py`'s attribute names exactly, so that module's + advance/back/queue logic ports across widget instances unchanged. +- `FrameCalendar` is keyed by `widget_id` (not `frame_id`) since a frame + can now have more than one independent calendar widget, each with its + own set of included calendars. +- `FrameButtonAction` (`id`, `frame_id`, `button` [`"next"`|`"back"`], + `widget_id`, `action`, `sort_order`) -- see "Button actions" below. + +## Placement: a grid, not freeform pixels + +`app/grid.py` is pure grid math, no I/O. The grid is `GRID_LONG=8` x +`GRID_SHORT=5` cells, defined relative to the panel's long/short axis +(not "landscape" specifically) so it stays valid across +`image_pipeline.logical_render_size(orientation)`'s genuine width/height +swap for portrait -- landscape orientations are 8 cols x 5 rows, portrait +are 5 cols x 8 rows, same cell size either way. **Changing a frame's +orientation invalidates its existing layout** (an 8x5 arrangement isn't +valid on a 5x8 grid) -- the server resets to one full-panel widget on an +orientation change rather than trying to remap coordinates. + +Each widget type has a minimum grid footprint (`grid.MIN_FOOTPRINT`): +photos 1x1, calendar 3x2 (a crammed calendar is illegible regardless of +size-tier scaling), whiteboard 2x2. Enforced both client-side (UX, in the +Layout tab's drag/resize canvas -- `static/frame_layout.js`) and +server-side (`routers/api_widgets.py`) -- the client is never trusted +alone. + +## Rendering: one shared compositor + +`app/widgets/` is the render/action registry -- one module per +`widget_type` (`photos.py`, `calendar.py`, `whiteboard.py`), each +exposing: + +- `render(db, frame, widget, target_w, target_h, is_normal_wake) -> Image`: + an unquantized RGB image exactly `target_w x target_h`, the widget's + content composed into its own region. Never raises for a foreseeable + failure (an Immich hiccup, an unconfigured widget) -- falls back to a + small placeholder within its own region instead, so one widget having a + bad moment doesn't blank the whole panel. +- `ACTIONS: dict[str, Callable]` -- named button actions this type + supports (`"advance"`/`"back"` for photos and calendar, `"check_now"` + for whiteboard). +- `ACTION_LABELS: dict[str, str]` -- human labels for the button- + assignment UI. + +`routers/device.py`'s `_render_widgets` loads every `Widget` row for the +frame, maps each one's grid rect to pixels (`grid.cell_to_pixels`), calls +its module's `render()`, and hands the whole list of `(rect, image)` +regions to `image_pipeline.render_panel` -- which pastes every region onto +one shared canvas, then runs enhance/manage-overlay/quantize/dither/pack +**once** over the composited result. Quantizing the whole canvas together +(not each region separately before pasting) is what keeps the 6-color +e-ink dithering pattern consistent across a widget boundary instead of a +visible seam at the edge. + +Calendar widgets pick from discrete size tiers (`calendar_render.py`'s +`_SIZE_TIERS`) for font size/margins/row heights based on their actual +grid footprint, rather than continuously scaling constants tuned for a +full ~800x480 canvas -- falls back to agenda view if a widget is too small +for month view to stay legible. + +## Button actions + +Each physical button (NEXT/BACK) maps to an **ordered list** of +`(widget, action)` bindings, not a fixed meaning -- e.g. NEXT can be +"photo widget A: advance" *and* "calendar widget B: advance" together, or +even a mismatched combination on purpose. On a press, +`routers/device.py`'s `_run_button_actions` runs every assigned action for +that button in order (each in its own `widget_locked` span -- never nested, +since the underlying per-frame lock isn't reentrant), catching and +logging any single action's failure without blocking the rest, then +re-renders and returns the whole composed panel once at the end regardless +of which actions succeeded. + +The web UI for this is the "Button assignments" card on a frame's +Configuration tab (`static/frame_config.js`, `GET`/`PUT +/api/frames/{id}/buttons`) -- add/remove/reorder, autosaved. Two widgets of +the same type would otherwise both just say "Photos" in the assignment +dropdowns; the UI disambiguates using each widget's grid position (e.g. +"Photos 1 (left)" / "Photos 2 (right)"), the same way you'd tell them +apart by eye on the Layout canvas. + +A newly-created widget (including the one auto-migrated from a frame's old +`mode` on upgrade) gets a sensible default binding reproducing its old +button behavior -- see `migration.py`'s `_default_button_actions`. + +## Per-widget config UI + +Each widget has a gear-icon button on the Layout canvas that opens a +`` with that widget's own settings (album, calendar inclusion, +whiteboard source, etc.) -- not a per-frame tab, since a frame can now +have several widgets of the same type with independent settings. The +dialog HTML is injected server-rendered (`routers/frame_pages.py`'s +`widget_dialog`, dispatching on `widget.widget_type`); its JS is a +top-level, always-loaded file (`static/widget_dialog_*.js`) exposing +`initDialog()`/`closeDialog()`, since dynamically-injected +HTML can't carry executable `