A new widget type showing a single user-uploaded image with no live
upstream to poll -- decoded once at upload time (PDF's first page via
pypdfium2, BSD-3-Clause/Apache-2.0, no copyleft exposure) into plain RGB
PNG bytes, then composed per a crop/stretch/letterbox display mode like
the photos widget.
Small header thumbnail showing exactly what the frame is currently
displaying -- same widget compositor /frame/image uses, handed back as
a plain PNG (image_pipeline.render_panel/render_placeholder gain an
as_png option) instead of packed native-panel bytes. New session-authed
GET /api/frames/{id}/preview exposes it; click-to-refresh plus a slow
60s poll on the frame header so it doesn't hammer Immich/calendar
sources just for a header thumbnail.
calendar_render.py's _build_* functions now take a real target box
and pick font sizes/margins from three discrete size tiers (nearest
pixel-area fit) instead of always laying out at full panel size and
resizing after the fact -- a calendar widget placed smaller than the
full panel gets an actually-legible layout instead of shrunk text.
Month view falls back to agenda below the smallest tier, where 7
columns can no longer stay readable.
The old photo-inlay split (inlay_region/_content_region/_paste_inlay)
is deleted along with it -- arbitrary widget placement already
subsumes what a fixed half-panel split did, and every call site has
passed photo_inlay=None since the Phase 2 cutover.
Also adds HTTP-level test coverage for GET .../preview/calendar,
which had none before this -- it's what caught a stale photo_inlay
kwarg left over from the _build signature change that would have
TypeError'd on every request.
New app/widgets/ package (photos.py, calendar.py, whiteboard.py, plus
the WIDGET_TYPES registry) -- the widget-system analogue of
routers/device.py's old RENDERERS/ADVANCE_RENDERERS/BACK_RENDERERS,
generalized from "one mode owns the whole panel" to "each widget renders
into its own region and responds to named button actions." Each module
exposes render(db, frame, widget, target_w, target_h) -> Image.Image
(never raises -- a widget's own fetch hiccup falls back to a small
placeholder rather than taking the whole panel's render down) and an
ACTIONS registry for NEXT/BACK button assignment.
Supporting changes needed to give the widget modules something to call,
all mechanical/behavior-preserving for every existing caller:
- image_pipeline.py: render_panel(regions, ...) generalizes render_frame's
tail (paste, enhance once, overlay once, quantize once, pack once) from
one photo to N regions -- not a restructuring, since calendar mode's
photo-inlay feature already pastes a second composed image onto the
canvas before that single shared pipeline runs.
- photo_queue.py: advance_forced/back_forced/remove_from_rotation/
get_current take an explicit `frame` param now that `cfg` won't always
be the Frame itself once photo-queue state moves to PhotoWidgetConfig
-- caught a real latent bug while doing this: get_current was reading
refresh_interval_s off `cfg`, but that's a frame-level wake-cadence
setting, not something that becomes per-widget, so it now reads that
off `frame` explicitly instead.
- routers/common.py: list_assets/fetch_source_and_faces take album_id/
display_mode directly instead of a whole Frame (both only ever read
that one attribute off it); new get_or_refresh_*_for_widget siblings
of the existing calendar/weather/tasks/whiteboard cache helpers, read/
writing the new per-widget config tables -- the Frame-scoped originals
are untouched and still what routers/device.py's actual dispatch calls
until the Phase 2 cutover.
26 new tests (95 total): render_panel size/placement/orientation
coverage, and per-widget-type render/action tests (unconfigured ->
placeholder, a fetch failure -> placeholder not a crash, actions mutate
the right state). Full suite passes; diff-reviewed to confirm
device.py's actual RENDERERS dispatch and the old Frame-scoped
get_or_refresh_* bodies are unchanged, so this is safe to deploy on its
own despite being step 1 of a two-step cutover (see the project plan on
why the *next* step, not this one, has to ship atomically).
Responds to post-launch feedback on calendar mode: configurable
week-start day for week/month views, crisper non-antialiased text
(threshold-masked instead of drawn straight, so Floyd-Steinberg
dithering doesn't speckle glyph edges), a color-coded/proportionally
filled battery icon on the manage overlay, word-wrapped placeholder
text so "Calendar isn't set up yet" no longer clips in portrait, photo
inlay support extended from agenda-only to every view, and a fix so
manage-overlay face labels reposition correctly when a photo inlay is
active (they previously assumed the photo filled the whole canvas).
Also adds a fourth calendar view, "Today & Tomorrow" -- a two-day
agenda that reuses the same per-day row-layout helper the single-day
agenda view already has.
calendar_feed.py/calendar_render.py: fetch/merge per-user ICS feeds,
render agenda/week/month views. manage_overlay.py: composites the
manage-button overlay server-side (QR, battery, location/date,
share-QR, face labels), reused by every render mode. device.py/common.py
wire both together: mode dispatch for /frame/image+advance+back, and
the &manage=1 flag. Plus UI (frame_config.html Calendar card, settings
calendar URL field) and the icalendar/recurring-ical-events deps.
/api/frames/{id}/firmware/check could silently stage new firmware as a
side effect (the auto-apply path, when firmware_auto_update is on and
a newer release exists) but was gated by require_frame_view instead of
require_frame_control like its sibling firmware routes, and being a
GET, was exempt from the app's CSRF check (which only applies to
non-GET/HEAD/OPTIONS). A linked viewer without control -- or a
cross-site page riding a control-holding victim's session via a plain
GET -- could trigger an unreviewed firmware install. Now POST +
require_frame_control, matching /firmware/apply-latest; the frontend's
two callers (passive poll on page load, "Check now" button) both
already handle a 409 from a non-controller gracefully via the existing
apiError()/control-banner pattern, so this doesn't change UX for a
frame's actual controller.
Separately: Immich has been observed to return a face detection entry
with a null bounding-box field (a still-pending or otherwise
incomplete detection). Both places that do arithmetic on those fields
-- image_pipeline._face_aware_crop_box (crop_faces display mode) and
face_labels.compute_face_labels (manage-menu name labels) -- crashed
with an unhandled TypeError on such an entry, taking down that frame's
whole photo instead of the intended graceful fallback. Both now skip
any face missing a bounding-box field via a shared _has_bounding_box()
check; a face list with zero valid entries already degrades cleanly to
the plain center crop (the existing inf/-inf sentinel math already
handled "no faces" correctly, it just couldn't tell "none passed
Immich" apart from "one broken entry" before).
Advanced configuration gains three sliders (PIL ImageEnhance factors
for color/contrast, 0-2, 1=unchanged; a 0-1 dithering strength) applied
to every photo this frame renders. Confirmed the parameter conventions
against a similar project (jwchen119/EPF: ImageEnhance.Color/Contrast,
1.0 baseline) before implementing; dithering strength isn't natively
exposed by PIL's quantize(), so it's implemented by blending the source
toward its own flat/undithered quantization before running Floyd-
Steinberg on the blend -- at 0 there's no quantization error left to
diffuse (exactly the flat result), at 1 it's the original unmodified
behavior, with a smooth continuum between rather than dithering being
an on/off toggle.
image_pipeline.py split into composition (_compose), enhancement
(_enhance), quantization (_quantize), and transpose+pack stages so
render_frame (device bytes) and the new render_preview_png (a normal
viewable PNG, upright logical orientation) share the same pipeline
instead of duplicating it. Named-face overlay label math (face_labels.py)
was already routed through the shared _placement_transform, so it
needed no changes for the new params.
Also added the requested before/after comparison: the Configuration
tab's new Preview card shows the current photo's untouched Immich
preview next to that same photo run through the frame's actual saved
rendering pipeline (two new GET endpoints, /preview/original and
/preview/rendered) -- immediate visual feedback for tuning the palette
and these new sliders. "Refresh preview" re-fetches after saving.
Schema migration v6 adds color_boost/contrast_boost/dither_strength,
defaulting to 1.0/1.0/1.0 -- reproduces the exact previous rendering
until a frame's Configuration tab changes one.
Verified against the live-shaped test database: the migration, sliders
persisting and clamping out-of-range input, both preview endpoints
(real JPEG passthrough / real PNG at correct logical size+orientation),
confirmed dither_strength=0 actually changes the rendered bytes vs.
default, and the standing legacy-device curl suite.
Replaces the smart_crop_faces boolean with a 4-way display_mode select
on each frame's Configuration tab (image_pipeline.DISPLAY_MODES):
- Crop to fill / Crop to faces: the previous False/True behavior,
unchanged (center-crop trimming excess, optionally shifted to keep
faces on screen).
- Stretch to fill (new): fills the panel exactly, aspect ratio not
preserved -- a plain resize, no crop.
- Shrink to fit (new): the whole photo visible, letterboxed with white
where it doesn't fill the panel.
Named-face overlay label positioning (face_labels.py, the manage menu's
"who's in this photo") now goes through a shared _placement_transform()
in image_pipeline.py instead of duplicating crop-box math, so label
placement stays correct (and in-bounds) under all four modes, not just
the two crop ones -- letterbox/stretch never crop a face out, so labels
just use straight scale+offset math there.
Schema migration v5 adds display_mode, backfills it from the old
boolean (True/False -> crop_faces/crop_fill), and drops the boolean.
Verified against the live-shaped test database: the migration
(existing frames correctly preserved as crop_faces), the config page's
new 4-option select, actual renders under letterbox (confirmed real
white letterbox padding in the packed panel-code bytes) and
stretch_fill, invalid-input fallback, and the standing legacy-device
curl suite.
Advanced configuration (Configuration tab, collapsed <details> section):
a color picker per ink color (black/white/yellow/red/blue/green),
overriding image_pipeline.DEFAULT_PALETTE_RGB for that frame's actual
panel -- different units can vary enough from the documented
approximations to be worth calibrating once you can compare a rendered
photo against the real hardware. Stored as Frame.palette_rgb (NULL =
default, schema migration v4), threaded through render_frame/
render_placeholder/_quantize_and_pack (which now builds the PIL palette
image per call instead of once at import) so both photos and the
unclaimed/unconfigured placeholder screen respect it. "Reset to
defaults" clears back to NULL. Config-save validates exactly 6 #rrggbb
values, rejecting anything else with a 400.
Also: each frame's sidebar entry now shows its last-reported battery
percent (🔋NN%) next to the name, using the frame_dot's existing
recently-seen indicator conventions -- silent when never reported
(mains-only frames, or before the first report), matching how battery
is hidden everywhere else it's not applicable.
Verified against the same live-shaped database as the SMTP work: the
v3->v4 migration, save/reload/reset round trip through the real HTTP
route, an actual rendered image using a custom palette (confirmed via
its packed panel-code bytes), input validation, and the sidebar badge
against real battery data -- plus the standing legacy-device curl suite.
The frame-claiming pipeline, end to end. Firmware: every request now
carries ?id=<12-hex STA MAC> via build_url (mirrored in build_ota_url),
and the captive portal's success page became a redirect that hands the
user's browser to <server>/claim?device_id=... after ~7s -- enough time
for the phone to drop the provisioning AP while the device reboots.
The server pushes a per-frame device token through /frame/config during
a one-time handshake; the firmware persists it to NVS (a dedicated
single-key write that deliberately doesn't reset the connected-once
flag or WiFi cache) and prefers it over the provisioned shared token
from the next request on. Config response buffer grows 256->512. Both
board variants compile clean; new firmware also works against an old
server (which ignores ?id=) and old firmware against this server (the
phase A legacy mapping), so either deploy order survives.
Server: /claim lands the captive-portal redirect -- claim-gated signup
(a valid unclaimed/unregistered device id IS the enrollment invitation),
pending claims for the user-beats-the-frame race (auto-attached at
self-registration, 24h expiry), and a waiting page that refreshes until
the frame checks in. Unclaimed/unconfigured frames get a rendered
instruction placeholder with a QR from /frame/image (200, never an
error loop) -- new qrcode dep, placeholder shares the exact
quantize/pack path photos use.
The on-frame manage QR now resolves to a limited no-login page: scans
of / carrying device credentials (new ?id&token or the legacy shared
token) 303 to /m/<manage_token>, which allows exactly view queue,
show-next, advance, back, and scoped thumbnails -- no settings, no
removal, no other frames. Full control means logging in.
One real protocol hole found by simulating full wake cycles: after
self-registration the device could never authenticate again (the wake
cycle fetches the image BEFORE /frame/config delivers its token).
require_device now treats the id itself as the credential until the
first authenticated request flips device_token_ack -- the same trust
level as open registration, closing permanently once the handshake
completes.
Battery (firmware + server, disabled by default): new battery.c reads
a 2x200k voltage divider via ADC oneshot with curve-fitting calibration
(the ESP32-C6's scheme), maps through a piecewise LiPo discharge curve,
and restores the pin to button duty after each read -- the settled
XIAO ESP32-C6 design shares the back button's GPIO0/A0, time-shared per
wake. Skipped entirely when on mains (a 2x100k VBUS divider into a
spare digital pin -- the 5V pin is dead on battery power, so presence =
mains, where the charging voltage would read misleadingly full) or when
the reading is implausible. The manage overlay gains a battery region
(static outline glyph + "NN%", below the manage QR, all menu levels),
and the device POSTs to the new /frame/battery endpoint after a
successful fetch; the server stores percent + as-of timestamp, exposed
via /api/queue and shown in the web UI. FRAME_BATTERY_ADC_GPIO /
FRAME_VBUS_SENSE_GPIO default to -1 (fully inert on the dev board);
compile-verified both disabled and enabled, hardware bring-up deferred
until the ordered XIAO + batteries arrive.
Orientation (server-side only): new config setting + web UI dropdown
(landscape / portrait / landscape_flipped / portrait_flipped). Photos
are composed/cropped at the logical hanging shape (portrait crops at
480x800, so face-aware crops match how the frame actually hangs), then
rotated losslessly into the panel's native 800x480 byte layout after
dithering -- the device never knows. Face-label anchors are transformed
through the same rotation (logical_to_native()) so they stay attached
to faces on rotated frames. Known documented limitation: the on-device
manage overlay still renders in native orientation, so it appears
sideways on a portrait-hung frame (QRs scan at any rotation; text reads
sideways).
Two rounds of follow-up work on the manage-button overlay:
1. Location formatting: US/Canada now show abbreviated state/province
("CA", "ON") instead of the full name, other countries show the full
country name, and each is its own line (was one line, now wraps to
two) so longer international place names have more room without
threatening to overlap the top-right QR box. The bottom-left share QR
also gets a "SCAN TO DOWNLOAD" caption.
2. Escalating menu: pressing the manage button again while its overlay
is already up adds a second level -- each Immich-identified person's
name labeled next to their face in the photo (using Immich's own
face recognition/People data, no detection/recognition added to this
project). A third press exits immediately instead of waiting out the
30s auto-revert timer. No new Immich API needed -- GET /api/faces
already embeds a nullable person.name per face; new
server/app/face_labels.py maps a named face's box into the final
800x480 frame's pixel space (reusing crop-box math extracted from
image_pipeline.py's face-aware cropping). Capped at 4 named faces,
sized to a real firmware RAM budget: each label is its own malloc'd
overlay region on the device, alongside the 4 fixed corner regions
already in use. New GET /frame/face-labels returns a flattened
fixed-slot JSON shape (not a real array) so firmware's existing
flat-scalar parser can read it without needing an actual array
parser. No persistent state needed for the escalation itself -- it's
all local control flow within one continuous awake session
(frame_client.c's run_management_menu()).
_face_aware_crop_box() previously always centered the crop on the union
of all detected faces' centroid, even when the plain center-crop already
kept every face fully on screen -- unnecessarily moving a composition
that didn't need fixing. Now starts from the plain center-crop and only
shifts it the minimum amount needed to bring an otherwise-cropped-out
face back into frame; already-fine framing is left untouched (falls back
to centering on the faces' midpoint only if they're spread too wide for
any single shift to contain them all, which is unchanged from before).
Verified: a face safely inside the plain center-crop now produces byte-
identical output to the no-shift case (previously it still would have
been re-centered); an edge face gets a 100px shift instead of the 1050px
a full re-center would have applied. Re-ran against the real 4-face test
photo from earlier -- all four were already fully visible, so the refined
box now exactly matches the plain center-crop instead of shifting
unnecessarily.
Two features, both toggleable/settable from the web config UI:
Refresh interval: new GET /frame/config returns
{"refresh_interval_s": ...} as plain JSON. Reuses the endpoint the frame
already needs to hit for a reachability check each wake cycle (previously
/health) rather than adding a third round trip, and always returns 200
with current settings regardless of Immich-configured state so it stays
valid as a pure reachability signal. Clamped to [60, 86400] seconds in
POST /api/config.
Face-aware cropping: GET /api/faces?id={assetId} on Immich already
returns real per-photo face bounding boxes from its own People-feature
ML -- confirmed against a live instance, boxes scaled to the asset's
native resolution. No face detection built or bundled here at all, just
an API call plus rectangle math. image_pipeline.render_frame() gains an
optional `faces` param: when present, computes the largest crop window
matching the panel's aspect ratio that fits in the source image, centered
on the union of all face boxes' centroid (scaled into the downloaded
preview's actual resolution) instead of the image's geometric center,
clamped to stay within bounds. No faces (or the smart_crop_faces config
toggle off) falls straight back to the existing ImageOps.fit() center-crop
-- zero behavior change in that case. A faces-lookup failure logs and
degrades to center-crop rather than failing the whole request.
Verified: unit tests for the crop-box math (horizontal shift toward an
off-center face, edge clamping), a full mock-Immich end-to-end pass
(extended to serve /faces) confirming the toggle changes output and the
response is still exactly 192,000 bytes, and a live comparison against a
real 4-face photo on the user's Immich instance (crop top shifted from
528px to 246px toward the detected faces).
Implements the server side of the architecture decided on: the ESP32-C6
has no PSRAM and a tight RAM budget, so all the heavy lifting (JPEG
decode, resize, Floyd-Steinberg dithering, 6-color quantization, 4bpp
packing) happens here instead of on-device. The frame just does a single
GET and streams the response straight to SPI.
- GET /frame/image: looks up the current cursor's asset in the configured
Immich album, downloads its preview thumbnail, and returns it packed
into the panel's exact 800x480/4bpp/2px-per-byte format
(application/octet-stream, always exactly 192,000 bytes).
- GET / + POST /api/config + GET /api/albums: a small web UI for entering
the Immich URL/API key and picking an album, rather than cramming that
into the ESP32's captive portal form.
- Config (Immich creds, selected album, cursor) persists to a JSON file
via a docker-compose volume mount.
Verified locally with a venv (Docker isn't available in this environment):
unit-tested image_pipeline against a synthetic image (exact byte count,
valid panel color codes only), and ran a full end-to-end pass against a
mock Immich HTTP server exercising the real /frame/image path.
Pinned dependency versions in requirements.txt after hitting a real bug
with unpinned floors: the latest starlette (1.3.1) resolved by `pip
install fastapi` breaks Jinja2Templates outright.
Not yet wired to the ESP32 side (task 6) or authenticated -- /frame/image
is unauthenticated for now, fine on a trusted LAN but worth revisiting
once the firmware sends a shared device token.
Thomas Faour
tfaour