The weather widget's icons/layout are hand-drawn PIL primitives -- clean
under quantization but flat, no gradients/shadows. Adds an opt-in
render_style="modern" (current/daily modes only) that instead renders a
Jinja2 template through a persistent headless-Chromium browser
(app/html_render.py), following the approach of Tesserae, an open-source
e-ink dashboard targeting this same panel family.
Key design points:
- The Chromium dependency (Playwright) is lazily imported only when a
weather widget actually uses "modern" style, and the background browser
itself only launches on first use -- every other widget type, and this
one's own classic/hourly/multi_city paths, never pay for it.
- No Frame-level dithering setting needed: html_render dithers its own
rendered widget to exact palette colors (Bayer/ordered, not
Floyd-Steinberg) before compositing, so the shared whole-canvas
Floyd-Steinberg pass sees zero quantization error there and leaves it
untouched -- same trick draw_text/hand-drawn icons already use. Floyd-
Steinberg keeps working unchanged for photos and every other widget.
- A "Load calibrated Spectra 6 preset" button in Advanced configuration
offers a community-measured palette (data ported from
paperlesspaper/epdoptimize, Apache 2.0) as an alternative starting
point to the existing idealized DEFAULT_PALETTE_RGB -- fills the
existing palette table, doesn't save by itself.
Known open risk, not resolved here: a headless Chromium binary is far
larger than the ~100MB single-layer limit that already forced this
project's pip/npm installs into split layers, and (unlike those) is a
single ~180MB file that can't be split across layers by ordinary
Dockerfile restructuring. Flagged prominently in server/Dockerfile and
docs/widgets.md -- treat this render style as experimental/local-only
until that's resolved.
Introduces app/panel_style.py, a shared style module every render
module now draws through instead of independently duplicating margins/
colors/fonts: Inter Bold/Regular (already vendored, previously only
used by widgets/text.py) replace PIL's single-weight bundled default
font everywhere else; a per-widget-kind accent color (calendar=blue,
tasks=green, weather=black header) replaces plain black-on-white chrome
and is centralized in one THEME mapping so a future global theme only
needs to touch panel_style.py; a small per-widget gutter separates
adjacent widgets without touching grid.py's cell math; header bars,
color chips, and the battery icon get rounded corners.
Also drops the MUTED gray text color used throughout calendar_render.py
and weather_render.py -- a non-palette color that has no close match in
the panel's 6-ink palette and dithers into visible speckle once the
composited canvas is quantized. Secondary text now reads through size/
weight alone, always exact black.
widgets/battery.py and manage_overlay.py's previously-duplicated
battery-glyph-drawing code now share one implementation (panel_style.
draw_battery_icon). widgets/_shared.py's placeholder image is fixed to
use exact palette colors and route through image_pipeline.draw_text,
same as everything else -- it was quietly violating both rules already.
image_pipeline.draw_widget_border gains an opt-in radius param (default
0, unused by any call site) for a possible future rounded-border
setting -- doesn't touch the exact-corner-pixel behavior test_widget_
border.py already pins.
Deliberately out of scope: DEFAULT_PALETTE_RGB and the Floyd-Steinberg
quantization pipeline are untouched, per the prior reverted measured-
palette/OKLab attempt (05b417a/dfe9d701).
DEFAULT_PALETTE_RGB was a guessed approximation of the panel's ink
colors (pure sRGB primaries); swap in epdoptimize's measured spectra6
palette instead, which is far more muted/darker, matching how these
inks actually look.
_quantize now matches against the palette in OKLab space (perceptual
distance) instead of PIL's raw-RGB quantize(), with lightness weighted
down relative to hue/chroma when selecting the nearest color -- this
palette's inks are lit so differently from their sRGB namesakes
(muted dark red, bright yellow) that unweighted distance let lightness
dominate and mismatch hue (pure red nearest "yellow").
Dithering switched from Floyd-Steinberg error diffusion to a Bayer
ordered dither: true error diffusion is an inherently serial per-pixel
loop, and doing that in pure Python for a full 800x480 panel took
~1s, blowing past the render-latency budget the "render widgets
concurrently" fix (previous commit) exists to protect. The ordered
dither finds each pixel's true nearest and second-nearest palette
color and mixes between them (via projection onto that segment, not
distance ratio) using a tiled Bayer threshold -- fully vectorized, no
Python-level pixel loop.
The server now records exactly what was last sent to the device on
every device-facing render (/frame/image, /frame/advance, /frame/back,
and the global hold actions), persisted as Frame.last_displayed_image/
_at and served back via GET /api/frames/{id}/now-displaying. The
header thumbnail is split into that frozen "now displaying" snapshot
and the existing live "up next" re-render, with an arrow between them
-- so editing a layout shows the change immediately on the right while
the left stays exactly what's actually on the panel until the device's
next real wake.
A Widget-level property (border_style/border_thickness/border_color_index),
not a per-type config field, since every widget type can have one -- drawn
once centrally in device.py's _render_widgets before compositing, using
an exact panel palette color so it never dithers. Styles: solid, dashed,
dotted, and a fancy double-line picture-frame-mat look. Configurable from
a shared "Border" card in every widget's gear-icon dialog.
A new widget type showing a single user-uploaded image with no live
upstream to poll -- decoded once at upload time (PDF's first page via
pypdfium2, BSD-3-Clause/Apache-2.0, no copyleft exposure) into plain RGB
PNG bytes, then composed per a crop/stretch/letterbox display mode like
the photos widget.
Small header thumbnail showing exactly what the frame is currently
displaying -- same widget compositor /frame/image uses, handed back as
a plain PNG (image_pipeline.render_panel/render_placeholder gain an
as_png option) instead of packed native-panel bytes. New session-authed
GET /api/frames/{id}/preview exposes it; click-to-refresh plus a slow
60s poll on the frame header so it doesn't hammer Immich/calendar
sources just for a header thumbnail.
calendar_render.py's _build_* functions now take a real target box
and pick font sizes/margins from three discrete size tiers (nearest
pixel-area fit) instead of always laying out at full panel size and
resizing after the fact -- a calendar widget placed smaller than the
full panel gets an actually-legible layout instead of shrunk text.
Month view falls back to agenda below the smallest tier, where 7
columns can no longer stay readable.
The old photo-inlay split (inlay_region/_content_region/_paste_inlay)
is deleted along with it -- arbitrary widget placement already
subsumes what a fixed half-panel split did, and every call site has
passed photo_inlay=None since the Phase 2 cutover.
Also adds HTTP-level test coverage for GET .../preview/calendar,
which had none before this -- it's what caught a stale photo_inlay
kwarg left over from the _build signature change that would have
TypeError'd on every request.
New app/widgets/ package (photos.py, calendar.py, whiteboard.py, plus
the WIDGET_TYPES registry) -- the widget-system analogue of
routers/device.py's old RENDERERS/ADVANCE_RENDERERS/BACK_RENDERERS,
generalized from "one mode owns the whole panel" to "each widget renders
into its own region and responds to named button actions." Each module
exposes render(db, frame, widget, target_w, target_h) -> Image.Image
(never raises -- a widget's own fetch hiccup falls back to a small
placeholder rather than taking the whole panel's render down) and an
ACTIONS registry for NEXT/BACK button assignment.
Supporting changes needed to give the widget modules something to call,
all mechanical/behavior-preserving for every existing caller:
- image_pipeline.py: render_panel(regions, ...) generalizes render_frame's
tail (paste, enhance once, overlay once, quantize once, pack once) from
one photo to N regions -- not a restructuring, since calendar mode's
photo-inlay feature already pastes a second composed image onto the
canvas before that single shared pipeline runs.
- photo_queue.py: advance_forced/back_forced/remove_from_rotation/
get_current take an explicit `frame` param now that `cfg` won't always
be the Frame itself once photo-queue state moves to PhotoWidgetConfig
-- caught a real latent bug while doing this: get_current was reading
refresh_interval_s off `cfg`, but that's a frame-level wake-cadence
setting, not something that becomes per-widget, so it now reads that
off `frame` explicitly instead.
- routers/common.py: list_assets/fetch_source_and_faces take album_id/
display_mode directly instead of a whole Frame (both only ever read
that one attribute off it); new get_or_refresh_*_for_widget siblings
of the existing calendar/weather/tasks/whiteboard cache helpers, read/
writing the new per-widget config tables -- the Frame-scoped originals
are untouched and still what routers/device.py's actual dispatch calls
until the Phase 2 cutover.
26 new tests (95 total): render_panel size/placement/orientation
coverage, and per-widget-type render/action tests (unconfigured ->
placeholder, a fetch failure -> placeholder not a crash, actions mutate
the right state). Full suite passes; diff-reviewed to confirm
device.py's actual RENDERERS dispatch and the old Frame-scoped
get_or_refresh_* bodies are unchanged, so this is safe to deploy on its
own despite being step 1 of a two-step cutover (see the project plan on
why the *next* step, not this one, has to ship atomically).
Responds to post-launch feedback on calendar mode: configurable
week-start day for week/month views, crisper non-antialiased text
(threshold-masked instead of drawn straight, so Floyd-Steinberg
dithering doesn't speckle glyph edges), a color-coded/proportionally
filled battery icon on the manage overlay, word-wrapped placeholder
text so "Calendar isn't set up yet" no longer clips in portrait, photo
inlay support extended from agenda-only to every view, and a fix so
manage-overlay face labels reposition correctly when a photo inlay is
active (they previously assumed the photo filled the whole canvas).
Also adds a fourth calendar view, "Today & Tomorrow" -- a two-day
agenda that reuses the same per-day row-layout helper the single-day
agenda view already has.
calendar_feed.py/calendar_render.py: fetch/merge per-user ICS feeds,
render agenda/week/month views. manage_overlay.py: composites the
manage-button overlay server-side (QR, battery, location/date,
share-QR, face labels), reused by every render mode. device.py/common.py
wire both together: mode dispatch for /frame/image+advance+back, and
the &manage=1 flag. Plus UI (frame_config.html Calendar card, settings
calendar URL field) and the icalendar/recurring-ical-events deps.
/api/frames/{id}/firmware/check could silently stage new firmware as a
side effect (the auto-apply path, when firmware_auto_update is on and
a newer release exists) but was gated by require_frame_view instead of
require_frame_control like its sibling firmware routes, and being a
GET, was exempt from the app's CSRF check (which only applies to
non-GET/HEAD/OPTIONS). A linked viewer without control -- or a
cross-site page riding a control-holding victim's session via a plain
GET -- could trigger an unreviewed firmware install. Now POST +
require_frame_control, matching /firmware/apply-latest; the frontend's
two callers (passive poll on page load, "Check now" button) both
already handle a 409 from a non-controller gracefully via the existing
apiError()/control-banner pattern, so this doesn't change UX for a
frame's actual controller.
Separately: Immich has been observed to return a face detection entry
with a null bounding-box field (a still-pending or otherwise
incomplete detection). Both places that do arithmetic on those fields
-- image_pipeline._face_aware_crop_box (crop_faces display mode) and
face_labels.compute_face_labels (manage-menu name labels) -- crashed
with an unhandled TypeError on such an entry, taking down that frame's
whole photo instead of the intended graceful fallback. Both now skip
any face missing a bounding-box field via a shared _has_bounding_box()
check; a face list with zero valid entries already degrades cleanly to
the plain center crop (the existing inf/-inf sentinel math already
handled "no faces" correctly, it just couldn't tell "none passed
Immich" apart from "one broken entry" before).
Advanced configuration gains three sliders (PIL ImageEnhance factors
for color/contrast, 0-2, 1=unchanged; a 0-1 dithering strength) applied
to every photo this frame renders. Confirmed the parameter conventions
against a similar project (jwchen119/EPF: ImageEnhance.Color/Contrast,
1.0 baseline) before implementing; dithering strength isn't natively
exposed by PIL's quantize(), so it's implemented by blending the source
toward its own flat/undithered quantization before running Floyd-
Steinberg on the blend -- at 0 there's no quantization error left to
diffuse (exactly the flat result), at 1 it's the original unmodified
behavior, with a smooth continuum between rather than dithering being
an on/off toggle.
image_pipeline.py split into composition (_compose), enhancement
(_enhance), quantization (_quantize), and transpose+pack stages so
render_frame (device bytes) and the new render_preview_png (a normal
viewable PNG, upright logical orientation) share the same pipeline
instead of duplicating it. Named-face overlay label math (face_labels.py)
was already routed through the shared _placement_transform, so it
needed no changes for the new params.
Also added the requested before/after comparison: the Configuration
tab's new Preview card shows the current photo's untouched Immich
preview next to that same photo run through the frame's actual saved
rendering pipeline (two new GET endpoints, /preview/original and
/preview/rendered) -- immediate visual feedback for tuning the palette
and these new sliders. "Refresh preview" re-fetches after saving.
Schema migration v6 adds color_boost/contrast_boost/dither_strength,
defaulting to 1.0/1.0/1.0 -- reproduces the exact previous rendering
until a frame's Configuration tab changes one.
Verified against the live-shaped test database: the migration, sliders
persisting and clamping out-of-range input, both preview endpoints
(real JPEG passthrough / real PNG at correct logical size+orientation),
confirmed dither_strength=0 actually changes the rendered bytes vs.
default, and the standing legacy-device curl suite.
Replaces the smart_crop_faces boolean with a 4-way display_mode select
on each frame's Configuration tab (image_pipeline.DISPLAY_MODES):
- Crop to fill / Crop to faces: the previous False/True behavior,
unchanged (center-crop trimming excess, optionally shifted to keep
faces on screen).
- Stretch to fill (new): fills the panel exactly, aspect ratio not
preserved -- a plain resize, no crop.
- Shrink to fit (new): the whole photo visible, letterboxed with white
where it doesn't fill the panel.
Named-face overlay label positioning (face_labels.py, the manage menu's
"who's in this photo") now goes through a shared _placement_transform()
in image_pipeline.py instead of duplicating crop-box math, so label
placement stays correct (and in-bounds) under all four modes, not just
the two crop ones -- letterbox/stretch never crop a face out, so labels
just use straight scale+offset math there.
Schema migration v5 adds display_mode, backfills it from the old
boolean (True/False -> crop_faces/crop_fill), and drops the boolean.
Verified against the live-shaped test database: the migration
(existing frames correctly preserved as crop_faces), the config page's
new 4-option select, actual renders under letterbox (confirmed real
white letterbox padding in the packed panel-code bytes) and
stretch_fill, invalid-input fallback, and the standing legacy-device
curl suite.
Advanced configuration (Configuration tab, collapsed <details> section):
a color picker per ink color (black/white/yellow/red/blue/green),
overriding image_pipeline.DEFAULT_PALETTE_RGB for that frame's actual
panel -- different units can vary enough from the documented
approximations to be worth calibrating once you can compare a rendered
photo against the real hardware. Stored as Frame.palette_rgb (NULL =
default, schema migration v4), threaded through render_frame/
render_placeholder/_quantize_and_pack (which now builds the PIL palette
image per call instead of once at import) so both photos and the
unclaimed/unconfigured placeholder screen respect it. "Reset to
defaults" clears back to NULL. Config-save validates exactly 6 #rrggbb
values, rejecting anything else with a 400.
Also: each frame's sidebar entry now shows its last-reported battery
percent (🔋NN%) next to the name, using the frame_dot's existing
recently-seen indicator conventions -- silent when never reported
(mains-only frames, or before the first report), matching how battery
is hidden everywhere else it's not applicable.
Verified against the same live-shaped database as the SMTP work: the
v3->v4 migration, save/reload/reset round trip through the real HTTP
route, an actual rendered image using a custom palette (confirmed via
its packed panel-code bytes), input validation, and the sidebar badge
against real battery data -- plus the standing legacy-device curl suite.
The frame-claiming pipeline, end to end. Firmware: every request now
carries ?id=<12-hex STA MAC> via build_url (mirrored in build_ota_url),
and the captive portal's success page became a redirect that hands the
user's browser to <server>/claim?device_id=... after ~7s -- enough time
for the phone to drop the provisioning AP while the device reboots.
The server pushes a per-frame device token through /frame/config during
a one-time handshake; the firmware persists it to NVS (a dedicated
single-key write that deliberately doesn't reset the connected-once
flag or WiFi cache) and prefers it over the provisioned shared token
from the next request on. Config response buffer grows 256->512. Both
board variants compile clean; new firmware also works against an old
server (which ignores ?id=) and old firmware against this server (the
phase A legacy mapping), so either deploy order survives.
Server: /claim lands the captive-portal redirect -- claim-gated signup
(a valid unclaimed/unregistered device id IS the enrollment invitation),
pending claims for the user-beats-the-frame race (auto-attached at
self-registration, 24h expiry), and a waiting page that refreshes until
the frame checks in. Unclaimed/unconfigured frames get a rendered
instruction placeholder with a QR from /frame/image (200, never an
error loop) -- new qrcode dep, placeholder shares the exact
quantize/pack path photos use.
The on-frame manage QR now resolves to a limited no-login page: scans
of / carrying device credentials (new ?id&token or the legacy shared
token) 303 to /m/<manage_token>, which allows exactly view queue,
show-next, advance, back, and scoped thumbnails -- no settings, no
removal, no other frames. Full control means logging in.
One real protocol hole found by simulating full wake cycles: after
self-registration the device could never authenticate again (the wake
cycle fetches the image BEFORE /frame/config delivers its token).
require_device now treats the id itself as the credential until the
first authenticated request flips device_token_ack -- the same trust
level as open registration, closing permanently once the handshake
completes.
Battery (firmware + server, disabled by default): new battery.c reads
a 2x200k voltage divider via ADC oneshot with curve-fitting calibration
(the ESP32-C6's scheme), maps through a piecewise LiPo discharge curve,
and restores the pin to button duty after each read -- the settled
XIAO ESP32-C6 design shares the back button's GPIO0/A0, time-shared per
wake. Skipped entirely when on mains (a 2x100k VBUS divider into a
spare digital pin -- the 5V pin is dead on battery power, so presence =
mains, where the charging voltage would read misleadingly full) or when
the reading is implausible. The manage overlay gains a battery region
(static outline glyph + "NN%", below the manage QR, all menu levels),
and the device POSTs to the new /frame/battery endpoint after a
successful fetch; the server stores percent + as-of timestamp, exposed
via /api/queue and shown in the web UI. FRAME_BATTERY_ADC_GPIO /
FRAME_VBUS_SENSE_GPIO default to -1 (fully inert on the dev board);
compile-verified both disabled and enabled, hardware bring-up deferred
until the ordered XIAO + batteries arrive.
Orientation (server-side only): new config setting + web UI dropdown
(landscape / portrait / landscape_flipped / portrait_flipped). Photos
are composed/cropped at the logical hanging shape (portrait crops at
480x800, so face-aware crops match how the frame actually hangs), then
rotated losslessly into the panel's native 800x480 byte layout after
dithering -- the device never knows. Face-label anchors are transformed
through the same rotation (logical_to_native()) so they stay attached
to faces on rotated frames. Known documented limitation: the on-device
manage overlay still renders in native orientation, so it appears
sideways on a portrait-hung frame (QRs scan at any rotation; text reads
sideways).
Two rounds of follow-up work on the manage-button overlay:
1. Location formatting: US/Canada now show abbreviated state/province
("CA", "ON") instead of the full name, other countries show the full
country name, and each is its own line (was one line, now wraps to
two) so longer international place names have more room without
threatening to overlap the top-right QR box. The bottom-left share QR
also gets a "SCAN TO DOWNLOAD" caption.
2. Escalating menu: pressing the manage button again while its overlay
is already up adds a second level -- each Immich-identified person's
name labeled next to their face in the photo (using Immich's own
face recognition/People data, no detection/recognition added to this
project). A third press exits immediately instead of waiting out the
30s auto-revert timer. No new Immich API needed -- GET /api/faces
already embeds a nullable person.name per face; new
server/app/face_labels.py maps a named face's box into the final
800x480 frame's pixel space (reusing crop-box math extracted from
image_pipeline.py's face-aware cropping). Capped at 4 named faces,
sized to a real firmware RAM budget: each label is its own malloc'd
overlay region on the device, alongside the 4 fixed corner regions
already in use. New GET /frame/face-labels returns a flattened
fixed-slot JSON shape (not a real array) so firmware's existing
flat-scalar parser can read it without needing an actual array
parser. No persistent state needed for the escalation itself -- it's
all local control flow within one continuous awake session
(frame_client.c's run_management_menu()).
_face_aware_crop_box() previously always centered the crop on the union
of all detected faces' centroid, even when the plain center-crop already
kept every face fully on screen -- unnecessarily moving a composition
that didn't need fixing. Now starts from the plain center-crop and only
shifts it the minimum amount needed to bring an otherwise-cropped-out
face back into frame; already-fine framing is left untouched (falls back
to centering on the faces' midpoint only if they're spread too wide for
any single shift to contain them all, which is unchanged from before).
Verified: a face safely inside the plain center-crop now produces byte-
identical output to the no-shift case (previously it still would have
been re-centered); an edge face gets a 100px shift instead of the 1050px
a full re-center would have applied. Re-ran against the real 4-face test
photo from earlier -- all four were already fully visible, so the refined
box now exactly matches the plain center-crop instead of shifting
unnecessarily.
Two features, both toggleable/settable from the web config UI:
Refresh interval: new GET /frame/config returns
{"refresh_interval_s": ...} as plain JSON. Reuses the endpoint the frame
already needs to hit for a reachability check each wake cycle (previously
/health) rather than adding a third round trip, and always returns 200
with current settings regardless of Immich-configured state so it stays
valid as a pure reachability signal. Clamped to [60, 86400] seconds in
POST /api/config.
Face-aware cropping: GET /api/faces?id={assetId} on Immich already
returns real per-photo face bounding boxes from its own People-feature
ML -- confirmed against a live instance, boxes scaled to the asset's
native resolution. No face detection built or bundled here at all, just
an API call plus rectangle math. image_pipeline.render_frame() gains an
optional `faces` param: when present, computes the largest crop window
matching the panel's aspect ratio that fits in the source image, centered
on the union of all face boxes' centroid (scaled into the downloaded
preview's actual resolution) instead of the image's geometric center,
clamped to stay within bounds. No faces (or the smart_crop_faces config
toggle off) falls straight back to the existing ImageOps.fit() center-crop
-- zero behavior change in that case. A faces-lookup failure logs and
degrades to center-crop rather than failing the whole request.
Verified: unit tests for the crop-box math (horizontal shift toward an
off-center face, edge clamping), a full mock-Immich end-to-end pass
(extended to serve /faces) confirming the toggle changes output and the
response is still exactly 192,000 bytes, and a live comparison against a
real 4-face photo on the user's Immich instance (crop top shifted from
528px to 246px toward the detected faces).
Implements the server side of the architecture decided on: the ESP32-C6
has no PSRAM and a tight RAM budget, so all the heavy lifting (JPEG
decode, resize, Floyd-Steinberg dithering, 6-color quantization, 4bpp
packing) happens here instead of on-device. The frame just does a single
GET and streams the response straight to SPI.
- GET /frame/image: looks up the current cursor's asset in the configured
Immich album, downloads its preview thumbnail, and returns it packed
into the panel's exact 800x480/4bpp/2px-per-byte format
(application/octet-stream, always exactly 192,000 bytes).
- GET / + POST /api/config + GET /api/albums: a small web UI for entering
the Immich URL/API key and picking an album, rather than cramming that
into the ESP32's captive portal form.
- Config (Immich creds, selected album, cursor) persists to a JSON file
via a docker-compose volume mount.
Verified locally with a venv (Docker isn't available in this environment):
unit-tested image_pipeline against a synthetic image (exact byte count,
valid panel color codes only), and ran a full end-to-end pass against a
mock Immich HTTP server exercising the real /frame/image path.
Pinned dependency versions in requirements.txt after hitting a real bug
with unpinned floors: the latest starlette (1.3.1) resolved by `pip
install fastapi` breaks Jinja2Templates outright.
Not yet wired to the ESP32 side (task 6) or authenticated -- /frame/image
is unauthenticated for now, fine on a trusted LAN but worth revisiting
once the firmware sends a shared device token.
tfaour
Thomas Faour