espresso_frame/docs/widgets.md

# Widget system

A frame's panel isn't one fixed "mode" anymore -- it holds N independently
placed/sized widgets (photos/calendar/whiteboard/tasks/static image/text/
weather/battery), like arranging icons on an Android home screen. A frame
can hold several widgets of the same type (e.g. two photo widgets pointed
at different Immich albums side by side).
This replaced an earlier design where `Frame.mode` picked exactly one
full-panel renderer; that column (and the other now-dead per-mode `Frame`
columns it left behind -- `album_id`, `calendar_*`, `whiteboard_*`, etc.)
is still physically present but unused, pending a final cleanup migration
(see "Known gaps" below).

The device-facing contract is unchanged by any of this: `GET /frame/image`,
`POST /frame/advance`, `POST /frame/back` are the same frozen paths
firmware has always called (see `docs/architecture.md`) -- what changed is
entirely server-side, in how those endpoints decide what to render and what
a button press does.

## Data model

- `Widget` (`server/app/models.py`): `id`, `frame_id`, `widget_type`
  (`"photos"` | `"calendar"` | `"whiteboard"` | `"tasks"` | `"static"` |
  `"text"` | `"weather"` | `"battery"`), `x`/`y`/`w`/`h`
  (grid cells), `sort_order`. Widgets never overlap (enforced server-side in
  `routers/api_widgets.py`, re-validated regardless of what the client
  already checked) -- that's what keeps compositing simple: no z-order,
  no blending, just N independent regions pasted onto one shared canvas.
  Also carries an optional per-widget border (`border_style` -- `"none"`
  | `"solid"` | `"dashed"` | `"dotted"` | `"fancy"`, `border_thickness`,
  `border_color_index`, an index into the frame's palette so a border
  always renders as one of the panel's exact 6 ink colors) directly on
  `Widget` itself rather than a per-type config table, since every
  widget type can have one regardless of `widget_type`. Drawn by
  `image_pipeline.draw_widget_border` onto each widget's own region in
  `routers/device.py`'s `_render_widgets`, before that region is pasted
  onto the shared canvas -- one central integration point instead of
  every `app/widgets/*.py` module needing to know about it. Set via the
  gear-icon dialog's shared "Border" card (`_widget_border_fields.html`,
  included by every `_widget_dialog_*.html` template) and
  `POST .../widgets/{id}/border`, its own endpoint (not folded into
  `api_widget_config_save`) since that endpoint's per-type dispatch is
  keyed on a config row via `widget_locked`, and border fields live on
  `Widget` itself, not any per-type config table.
- Per-type 1:1 extension tables -- `PhotoWidgetConfig`,
  `CalendarWidgetConfig`, `WhiteboardWidgetConfig`, `TaskWidgetConfig`,
  `StaticWidgetConfig`, `TextWidgetConfig`, `WeatherWidgetConfig`,
  `BatteryWidgetConfig`, each keyed by `widget_id` with
  `ondelete="CASCADE"` -- rather than one wide table with every type's
  mostly-irrelevant columns. `TextWidgetConfig.content` is parsed rich
  text (paragraphs of styled runs), never raw HTML -- see
  `server/app/text_content.py`'s module docstring for why that parse
  step is the widget's actual stored-XSS sanitization boundary.
  `PhotoWidgetConfig`
  mirrors `app/photo_queue.py`'s attribute names exactly, so that module's
  advance/back/queue logic ports across widget instances unchanged.
  `PhotoWidgetConfig.locked` (migration 27) freezes `current_asset_id`
  against both the timer-elapsed auto-advance
  (`photo_queue.get_current`) and the advance/back button actions
  (`app/widgets/photos.py`'s `ACTIONS`) until unlocked -- toggled via a
  "Lock this photo" button in the widget's own dialog
  (`POST .../widgets/{id}/lock`), shown as a lock badge on the widget's
  box on the Layout tab canvas.
  `TaskWidgetConfig` used to be a handful of `tasks_*` columns bolted onto
  `CalendarWidgetConfig` (a week-view-only, single-list task list); split
  into its own widget type (migration 17) so a task list can be placed
  and sized independent of any calendar's view/footprint, then (migration
  18) given the same multi-source shape a calendar widget already has.
  `WeatherWidgetConfig` similarly lifts `CalendarWidgetConfig`'s embedded
  weather strip (still present and unchanged, `weather_*` columns) out
  into its own placeable widget type (migration 24) -- see "Weather
  widget" below. `BatteryWidgetConfig` (migration 25) is the odd one out
  -- its actual content (`Frame.battery_percent`/`battery_as_of`) isn't
  in this table at all, already existing frame-level state set by
  `routers/device.py`'s `frame_battery` regardless of whether a battery
  widget is even placed; the config row only holds a display-mode
  setting (`"compact"` | `"detailed"`).
- `FrameCalendar`/`FrameTaskList` are keyed by `widget_id` (not
  `frame_id`) since a frame can now have more than one independent
  calendar/tasks widget, each with its own included set. Identical
  shape and permission model (owner-added, anyone-linked-can-mute, see
  "Per-widget config UI" below) -- `FrameTaskList` just has no `"ics"`
  calendar_key variant, since a plain ICS subscription has no VTODO
  (task) collection.
- `FrameButtonAction` (`id`, `frame_id`, `button` [`"next"`|`"back"`],
  `widget_id`, `action`, `sort_order`) -- see "Button actions" below.

## Placement: a grid, not freeform pixels

`app/grid.py` is pure grid math, no I/O. The grid is `GRID_LONG=8` x
`GRID_SHORT=5` cells, defined relative to the panel's long/short axis
(not "landscape" specifically) so it stays valid across
`image_pipeline.logical_render_size(orientation)`'s genuine width/height
swap for portrait -- landscape orientations are 8 cols x 5 rows, portrait
are 5 cols x 8 rows, same cell size either way. **Changing a frame's
orientation invalidates its existing layout** (an 8x5 arrangement isn't
valid on a 5x8 grid) -- the server resets to one full-panel widget on an
orientation change rather than trying to remap coordinates.

Each widget type has a minimum grid footprint (`grid.MIN_FOOTPRINT`):
photos 1x1, calendar 3x2 (a crammed calendar is illegible regardless of
size-tier scaling), whiteboard 2x2, tasks 2x2, static image 1x1, text 2x1,
weather 2x2 (its hourly/daily strips need the room; current/multi_city
modes would tolerate smaller, but every mode shares one footprint value),
battery 1x1 (just an icon + a percent, legible even at a single cell,
like photos/static -- though see `MIN_FOOTPRINT`'s own comment in
`grid.py` on a mobile-width gear-icon click-target gap at that size,
already pre-existing for photos/static too). Enforced both client-side
(UX, in the Layout tab's drag/resize canvas -- `static/frame_layout.js`)
and server-side (`routers/api_widgets.py`) -- the client is never trusted
alone.

## Rendering: one shared compositor

`app/widgets/` is the render/action registry -- one module per
`widget_type` (`photos.py`, `calendar.py`, `whiteboard.py`, `tasks.py`,
`static_image.py`, `text.py`, `weather.py`, `battery.py`), each exposing:

- `render(db, frame, widget, target_w, target_h, is_normal_wake) -> Image`:
  an unquantized RGB image exactly `target_w x target_h`, the widget's
  content composed into its own region. Never raises for a foreseeable
  failure (an Immich hiccup, an unconfigured widget) -- falls back to a
  small placeholder within its own region instead, so one widget having a
  bad moment doesn't blank the whole panel.
- `ACTIONS: dict[str, Callable]` -- named button actions this type
  supports (`"advance"`/`"back"` for photos and calendar, `"check_now"`
  for whiteboard and weather -- both throttled external fetches with a
  forced-refetch action). Empty for tasks, static image, text, and
  battery -- nothing to advance/back/force for a passive checklist, a
  fixed uploaded image, a fixed block of authored text, or a number the
  device itself pushes on every wake.
- `ACTION_LABELS: dict[str, str]` -- human labels for the button-
  assignment UI.

`routers/device.py`'s `_render_widgets` loads every `Widget` row for the
frame, maps each one's grid rect to pixels (`grid.cell_to_pixels`), calls
its module's `render()`, and hands the whole list of `(rect, image)`
regions to `image_pipeline.render_panel` -- which pastes every region onto
one shared canvas, then runs enhance/manage-overlay/quantize/dither/pack
**once** over the composited result. Quantizing the whole canvas together
(not each region separately before pasting) is what keeps the 6-color
e-ink dithering pattern consistent across a widget boundary instead of a
visible seam at the edge.

Calendar widgets pick from discrete size tiers (`calendar_render.py`'s
`_SIZE_TIERS`) for font size/margins/row heights based on their actual
grid footprint, rather than continuously scaling constants tuned for a
full ~800x480 canvas -- falls back to agenda view if a widget is too small
for month view to stay legible.

## Button actions

Each physical button (NEXT/BACK) runs the `(widget, action)` binding of
every widget on the frame that has one -- **at most one binding per
widget per button** (a widget can't be bound to two different actions on
the same button). On a press, `routers/device.py`'s `_run_button_actions`
runs every widget's assigned action for that button (each in its own
`widget_locked` span -- never nested, since the underlying per-frame lock
isn't reentrant), catching and logging any single action's failure
without blocking the rest, then re-renders and returns the whole composed
panel once at the end regardless of which actions succeeded. Which
widget's action runs first never matters -- each only touches its own
state, and the shared re-render happens once, after all of them finish.

The UI for this lives in each widget's own gear-icon config dialog (the
"Button actions" card, `templates/_widget_button_fields.html` +
`static/widget_dialog_button_actions.js`, `POST
/api/frames/{id}/widgets/{widget_id}/button-actions`) -- not a frame-level
tab, since assigning a widget's next/back behavior is naturally part of
configuring that widget. The card only renders for widget types with a
non-empty `ACTIONS` (photos, calendar, whiteboard, weather); tasks/
static/text/battery have nothing to bind so the card is omitted for
them. An empty selection ("(none)") clears that button's binding for the
widget.

A newly-created widget (including the one auto-migrated from a frame's
old `mode` on upgrade) gets a sensible default binding reproducing its
old button behavior -- see `widgets.default_button_actions` (called from
both `migration.py`'s backfill and `api_widgets.py`'s
`api_widget_create`), so a widget is never left with nothing bound until
someone deliberately reassigns it.

### Hold-for-global-action

Holding NEXT or BACK past a configurable duration (`Frame.hold_duration_ms`,
minimum 3000ms, set on the Configuration tab) triggers a **global**
action instead of the per-widget one -- not scoped to any widget, e.g.
cycling through the user's saved layouts. See `app/global_actions.py`'s
`GLOBAL_ACTIONS`/`GLOBAL_ACTION_LABELS` registry and
`routers/device.py`'s `/frame/global-next`/`/frame/global-back` (the
device calls these instead of `/frame/advance`/`/frame/back` once it
detects a long press -- see `firmware/main/next_button.c`/`back_button.c`).
`Frame.next_hold_action`/`back_hold_action` pick which registry entry (if
any) each button's hold triggers; unset is a silent no-op, same
convention as an unbound short-press button.

## Per-widget config UI

Each widget has a gear-icon button on the Layout canvas that opens a
`<dialog>` with that widget's own settings (album, calendar/task-list
inclusion, whiteboard source, etc.) -- not a per-frame tab, since a
frame can now have several widgets of the same type with independent
settings. The
dialog HTML is injected server-rendered (`routers/frame_pages.py`'s
`widget_dialog`, dispatching on `widget.widget_type`); its JS is a
top-level, always-loaded file (`static/widget_dialog_*.js`) exposing
`init<Type>Dialog()`/`close<Type>Dialog()`, since dynamically-injected
HTML can't carry executable `<script>` tags. While a dialog is open,
`window.FRAME_API` is repointed at that widget's own API base
(`/api/frames/{id}/widgets/{widget_id}`) and restored on close;
`window.FRAME_BASE_API` stays pointed at the frame-level base throughout
for the always-present header/status-bar JS.

## Saved layouts

A user can snapshot a frame's whole widget arrangement -- every widget's
type/placement/settings, calendar/task sources, and button-action
bindings -- under a name (`SavedLayout` + `SavedLayoutWidget` +
`SavedLayoutSource` + `SavedLayoutButtonAction`, `server/app/models.py`),
then switch back to it later, or apply it to a *different* frame. Saved
layouts are owned by the **user**, not any one frame -- the same set
shows up (with a per-frame `compatible` flag) on every frame that user
controls whose grid matches (`grid.grid_dims(orientation)`'s cols/rows,
landscape-class 8x5 vs. portrait-class 5x8), not just the frame it was
captured from.

Saving only captures an authored *setting*, never runtime/cache state --
a photo widget's current queue position, a calendar's fetch cache, a
whiteboard's rendered-image cache, etc. are deliberately left out (see
`routers/api_layouts.py`'s `LAYOUT_CONFIG_FIELDS` allowlist per
`widget_type`), so applying a layout feels like a fresh widget of that
type with its settings pre-filled, not a resurrection of stale state
from whenever it was saved. A static-image widget's uploaded bytes are
the one exception carried through verbatim (`SavedLayoutWidget.image`).
Saving again under a name the user already has overwrites that layout's
snapshot in place rather than erroring or creating a duplicate --
`SavedLayout`'s own docstring.

Applying a layout to a frame (`api_layout_apply`, `require_frame_control`)
deletes every widget currently on that frame and recreates the saved
arrangement from scratch, remapping calendar/task sources and button
bindings onto the newly-created widget ids -- same "act unconditionally
on the server, confirm on the client" posture as the Layout tab's own
"Clear all". A source whose owning user account no longer exists is
silently dropped rather than left dangling (config is JSON, not
FK-checked, so nothing else would catch that).

The web UI lives in the Layout tab's "Saved layouts" card
(`static/saved_layouts.js`, `GET`/`POST /api/frames/{id}/layouts`,
`PATCH`/`DELETE /api/layouts/{id}`, `POST
/api/frames/{id}/layouts/{id}/apply`) -- name + Save, then a list of
saved layouts each with Apply/rename/delete, incompatible ones shown
greyed-out with a "different orientation" badge rather than hidden.

## Weather widget

A standalone widget type (`models.WeatherWidgetConfig`, `app/widgets/
weather.py`) -- distinct from, and unrelated in code to,
`CalendarWidgetConfig`'s own embedded weather strip (still present,
still Open-Meteo-only, still working exactly as before). Four display
modes (`WeatherWidgetConfig.mode`, switchable in the widget's dialog like
`calendar_view`):

- `current` -- one city's current temp + a condition icon.
- `hourly` -- one city, a row of ticks across the day at a configurable
  interval (`hourly_interval_hours`: 3/4/6/12).
- `daily` -- one city, a multi-day strip (`daily_days`, 1-14).
- `multi_city` -- several cities' current-day high/low/icon side by
  side -- the calendar widget's embedded strip, as a standalone
  widget's whole content instead of a strip above an agenda day.

**Render style** (`WeatherWidgetConfig.render_style`, `"classic"` default
| `"modern"`, experimental): `current`/`daily` only -- `hourly`/
`multi_city` always render classic regardless of this setting. `modern`
draws the widget as an HTML/CSS card (Jinja2 templates under
`app/templates/widget_html/`) through a persistent headless Chromium
browser (`app/html_render.py`, Playwright) instead of `app/weather_render.
py`'s hand-drawn PIL primitives -- gradients/shadows/soft icon shading
PIL can't easily do. Its own `ordered_dither` (Bayer/ordered, not Floyd-
Steinberg) commits the rendered widget to exact palette colors *before*
compositing, so it's safe to mix with photo/other classic-rendered
widgets on the same frame without a Floyd-Steinberg seam at the boundary
(see that module's docstring for why ordered dithering doesn't have this
problem and Floyd-Steinberg does) -- no `Frame`-level dithering setting
was needed. Playwright/Chromium is a real, heavyweight runtime dependency
imported lazily only when a weather widget actually uses this style, and
its Docker packaging has a known likely-blocking image-size problem not
yet resolved (see `server/Dockerfile`'s own comment) -- treat this style
as unshipped/local-only until that's sorted out.

`current`/`hourly`/`daily` share one configured location
(`city_label`/`city_latitude`/`city_longitude`, set via `POST .../
weather-location`, geocoded through `weather.geocode_city`); `multi_city`
has its own list (`cities`, add/remove via `POST .../weather-widget-
cities/add`|`remove` -- named to avoid colliding with the calendar
widget's own, differently-scoped `weather-cities/add`|`remove` routes,
which share the same `{widget_id}`-parameterized path shape).

**Providers** (`app/weather/`, a dispatch registry over pluggable
implementations mirroring `app/widgets/` itself): `WeatherWidgetConfig.
provider` selects which of `app/weather.PROVIDERS` actually fetches --
`"open_meteo"` (worldwide, no API key), `"nws"` (api.weather.gov, US
only, no API key, approximates "current" with the first hourly forecast
period rather than a real station observation), or `"ec"` (Environment
Canada, api.weather.gc.ca's MSC GeoMet OGC API, Canada only, no API key).
Every provider function returns already-normalized `{"category": ...}`
entries (one of `clear`/`partly_cloudy`/`cloudy`/`fog`/`rain`/`snow`/
`thunderstorm`) so `app/weather_render.py`'s drawing code never needs to
know which provider supplied an entry. `geocode_city` (name -> lat/lon)
always goes through Open-Meteo's free geocoder regardless of which
provider is chosen to fetch with the result.

EC's `citypageweather-realtime` collection is only queryable by bounding
box (OGC API - Features), not a direct by-coordinate endpoint -- unlike
Open-Meteo/NWS's simple lat/lon REST, `app/weather/ec.py`'s
`_nearest_site` widens the box progressively and picks the closest site
by straight-line distance, rejecting anything beyond 300 km (calibrated
against a real bug caught in development: an unconditional "nearest
site, however far" matched a Miami, FL query to a site in Ontario,
1824 km away, once the box widened enough to cover the whole country).

`app/weather_render.py` holds every weather-related drawing primitive:
`draw_weather_icon`/`draw_weather_row` (extracted out of
`calendar_render.py`, which still imports `draw_weather_row` for its own
embedded strip, unchanged) plus this widget's own `build_current`/
`build_hourly`/`build_daily`/`build_multi_city`, dispatched by `build()`
-- the weather analogue of `calendar_render.py`'s own `_build_tasks`/
`render_tasks_preview_png` relationship. Icons are hand-drawn (no custom
font/icon asset), styled after Environment Canada's own icon set
(pointed sun rays, a puffy cloud, teardrop rain, dendrite snowflakes, a
zigzag bolt) but filled with the panel's *exact* ink RGB values rather
than an arbitrary bitmap's anti-aliased colors -- a flat fill that's
already a palette color quantizes with zero dithering error to diffuse,
where a fetched/vendored icon's colors (almost never an exact match)
dither into a visible speckle at these small on-panel sizes (confirmed
by actually running one through the real quantize pass during
development). Used for every provider's rendering, not just when EC is
selected as the provider.

## Battery widget

The simplest widget type (`models.BatteryWidgetConfig`, `app/widgets/
battery.py`): shows this frame's own last-reported battery level. Unlike
every other widget type, there's no live upstream to poll and nothing to
cache -- the content is `Frame.battery_percent`/`battery_as_of`, set by
`routers/device.py`'s `frame_battery` on every device wake-on-battery
report, which already existed for the Device panel's own history chart
regardless of whether a battery widget is placed anywhere. The widget's
own config is just a display mode: `"compact"` (icon + percent) or
`"detailed"` (default, adds `routers/common.py`'s existing
`battery_estimate_s` time-remaining estimate and the last report's age).
`render()` falls back to a "No reports yet" placeholder for a frame that
has never reported (never run on battery, or not yet claimed by a
device) rather than showing a stale or fabricated number. The battery
icon fill color (red/yellow/green by percent) uses the same exact-panel-
ink-RGB approach as the weather icons above and `manage_overlay.py`'s own
battery glyph on the "scan to manage" overlay -- a separate, unrelated
piece of code with its own fixed small size, not shared with this
widget, but drawing from the same thresholds/colors so a battery glyph
reads the same wherever one shows up on a panel.

## Known gaps (Phase 6, not yet done)

The original 8-phase rollout plan's last phase is still open:

- Legacy per-mode `Frame` columns (`mode`, `album_id`,
  `current_asset_id`, all `calendar_*`, all `whiteboard_*`, etc.) are
  still physically present in the schema but no longer read or written
  anywhere -- they need a dedicated final migration to drop them. Left in
  place deliberately through the widget-system rollout (a much larger
  blast radius cutover than this project's usual same-migration-drop
  convention) but there's no reason to keep carrying them now that every
  phase has shipped.
- `server/README.md` still describes photos/calendar/whiteboard as
  per-frame "modes" in several places rather than widgets -- needs a pass
  once the column drop above is safely deployed.
- Whiteboard rendering is tagged **(alpha)** in the UI -- not fully
  reliable yet, treat it as experimental if extending it.