#R04 — Offline-First Architecture and the Order Domain Model

Scope: the counter. Order capture, kitchen routing, cash, multi-station consistency, and the sync substrate underneath. This is the part of a POS that cannot be bought, cannot be faked, and is where most POS startups quietly die.

Bottom line up front: the offline core is buildable by a small team, but only if you (a) treat the order as an event-sourced aggregate with a deterministic fold, not a mutable row; (b) refuse to let any managed sync product own conflict semantics; (c) accept that LAN multi-station sync is ~15 of the ~34 engineer-months and decide deliberately whether v1 needs it. Getting this wrong is not a bug backlog — it is a rewrite of the counter app plus a per-venue data migration.

Fact labels used throughout: [verified] = checked this session with a URL, [known] = confident from training data, [estimate] = reasoned guess.

#1. The hard requirement, stated precisely

"Works offline" is marketing. Engineers need a degradation matrix. Define four network states and what must still work in each.

#1.1 Capability degradation matrix

Design consequence: N3 (single device fully isolated, printer unreachable) cannot be made to print. Do not promise it. What you promise is: "the check is never lost, cash is always takeable, and the kitchen can be served with a handwritten chit if the printer is dead — and the POS tells the server that clearly, on the button they just pressed." Honest failure UX is a feature. Silent queueing that the server assumes printed is how you lose an account.

#2. Topologies

#2.1 Candidates

#2.2 Recommendation: T3 primary, T2 fallback, T1 substrate

State replication is T1 (every device holds the full event log for open checks — leaderless, converges by fold). On top of that, a single coordinator role exists for the three things that genuinely need one writer:

1. Printer dispatch (exactly-once effects — §4.4)
2. Sequence/number leasing (ticket numbers, check numbers)
3. Payment terminal routing (which station owns which terminal right now)

The coordinator is the hub when the hub is alive; otherwise devices elect one. Correctness never depends on the coordinator being the hub — the hub is a convenience and a reliability upgrade, not a correctness requirement. If you make the hub required for correctness, every hub SD-card failure is a venue outage and a truck roll.

#2.3 Hub hardware

Strong recommendation: Pi 5 + PoE, no SD card, ext4 on NVMe, read-mostly rootfs, all mutable state on a separate partition. PoE matters more than it sounds: the #1 cause of "the server died" in small venues is a human unplugging a power strip [estimate, from general field-support patterns]. PoE puts the hub's power on the same switch as the APs, and you can UPS the switch (~EUR 60–120 for a 600VA line-interactive unit) instead of UPS-ing a device in the ceiling.

#2.4 Discovery: mDNS, and why it will fail on you

Plan: advertise _rbpos._tcp.local with TXT records venue=<uuid> role=hub|peer epoch=<n> proto=3 devid=<uuid>.

Real-world failure modes you will hit:

Rule: discovery is an optimization. Every peer relationship must be expressible as an explicit, persisted (device_id, ip, port, cert_fingerprint) tuple that survives discovery being broken forever.

#2.5 Transport and trust on the LAN

• Per-venue CA generated at venue provisioning (cloud-side), private key never leaves the cloud; each device gets a leaf cert with device_id in SAN, 90-day rotation while online, 1-year hard expiry.
• Peer links: TLS 1.3 over TCP with mutual auth and pinned venue CA. WebSocket framing (easiest across RN/iOS/Android/desktop) or raw length-prefixed CBOR. QUIC is attractive (connection migration when a tablet roams APs) but adds platform pain on RN — [estimate] +1.5 eng-months for QUIC vs WS; defer.
• Never trust "it's the LAN so it's safe." Restaurant Wi-Fi is shared with the guest network more often than anyone admits.

#2.6 Split-brain: what actually breaks

Partition the venue into groups A (2 tablets) and B (3 tablets + hub) for 20 minutes.

Fencing: every coordinator election increments a monotonic epoch stored durably on each participating device (not just the leader). Any message carrying epoch < max_seen_epoch is rejected. Printers and number leases are stamped with epoch. This is the standard fencing-token pattern and it is the difference between "two coordinators is confusing" and "two coordinators double-fires 40 tickets."

#2.7 Election and the "hub's battery dies mid-service" spec

Election: not Raft. A 2-device venue cannot form a quorum, and a POS must work with one device. Use a deterministic priority-bully scheme:

priority = (is_dedicated_hub ? 3 : 0)
         + (is_mains_powered  ? 1 : 0)
         + (battery_pct > 40  ? 1 : 0)
tiebreak = lowest device_uuid

Every peer heartbeats every 1s. Coordinator loss declared after 5 missed heartbeats (5s). Election window 2s. Total takeover budget: < 8 seconds, and during those 8 seconds order entry and cash still work (they're local); only printing and new number leases stall.

Timeline when the hub dies at 19:42 on a Friday:

That last row matters: automatic failback causes a second failover storm. Failback is manual or scheduled.

The "battery dies" case specifically: if the coordinator is a tablet on battery (T2 fallback in a venue with no hub), add a pre-emptive handoff at battery < 20%: the coordinator voluntarily elects a successor and hands over the print outbox and number leases while it is still alive. This is a clean handoff, not a failover — no verify-then-resend, no duplicate tickets. Costs ~0.3 eng-months, removes the ugliest failure mode.

#3. Concurrency model for the order aggregate

#3.1 The aggregate

Check (id, venue_id, business_day, table_id?, tab_name?, party_size, opened_by, status)
 ├─ Seat        (seat_no, guest_id?)                        # seat-level splits
 ├─ Line        (line_id, sku, menu_version_id, qty,
 │               unit_price_minor, tax_code, tax_rate_bp,   # PRICE AT TIME OF ORDER
 │               seat_no?, course_no, state: ACTIVE|VOID|COMPED,
 │               fired_at?, print_job_id?)
 │    └─ Modifier (mod_id, sku, price_delta_minor)
 ├─ Discount    (discount_id, code, scope: CHECK|LINE, kind: PCT|AMT, value, authorised_by)
 ├─ Payment     (payment_id, tender, amount_minor, tip_minor, psp_ref?, state)
 └─ FiscalDoc   (fiscal_seq, signature, issued_at)          # allocated by fiscal device, never by us

Non-negotiable invariant: unit_price_minor, tax_rate_bp and menu_version_id are snapshotted onto the line at order time and never recomputed. A price change at 20:00 must not reprice a check opened at 19:30. This single rule kills an entire class of "the total changed while I was looking at it" bugs and is also a legal requirement in most consumer-protection regimes [known].

#3.2 Why naive last-writer-wins loses money — with numbers

Assume LWW at the document level (the classic Firestore/Realm-shaped mistake): each device writes the whole check object, last write wins.

At a venue doing EUR 4,000/day with 6 stations, [estimate] 0.5–2% of check value silently lost to document-LWW during busy periods. The restaurateur does not see it that day — they see it in the weekly P&L as an unexplained gap, and they will attribute it to staff theft before they attribute it to your software. That is how you lose accounts and get a reputation you can't argue with.

Even field-level LWW (Firestore-ish) doesn't save you: it converges collections badly (arrays), and it cannot express "void is absorbing" or "payments are additive, never overwritten."

#3.3 Three models compared

#3.4 Recommendation: C — event sourcing with per-device append-only streams and a deterministic fold

Use A only as a UX affordance ("Anna is editing table 12" advisory badge, expires in 30s) and never as a correctness mechanism. Use B (Automerge or Yjs) for exactly one thing: free-text collaborative fields — order notes, kitchen notes, floor-plan editing in back office. Those are genuinely text-CRDT-shaped and worthless to hand-roll. Money is not.

The core of the design:

Event {
  event_id      UUIDv7          # client-generated; primary idempotency key
  device_id     UUID
  device_seq    u64             # strictly monotonic per device, gapless — detects loss
  aggregate_id  UUID            # check_id
  hlc           (u48 phys_ms, u16 counter, device_id)   # hybrid logical clock
  kind          enum
  payload       CBOR (unknown fields preserved)
  actor_id      UUID            # staff member; needed for audit + comps authority
  schema_v      u16
}

• Per-device stream is totally ordered and gapless (device_seq). A receiver seeing device_seq jump from 41 to 43 knows event 42 exists and must request it before folding — this is how you detect silent loss without a full-log hash.
• Cross-device order = sort by (hlc.phys, hlc.counter, device_id). Deterministic everywhere.
• fold(events) → CheckState is a pure function. No clocks, no RNG, no I/O. This is what makes the whole thing testable (§6).

Command set (intent-based, all carrying a client-generated id):

OpenCheck, AddLine(line_id,…), ChangeQty(line_id, delta), VoidLine(line_id, reason, actor), CompLine(line_id, reason, actor), AddModifier(mod_id, line_id,…), SetSeat(line_id, seat_no), FireCourse(course_no), ApplyDiscount(discount_id, code,…), RecordPayment(payment_id, tender, amount_minor, tip_minor), MoveLine(line_id, from_check, to_check), SplitCheck(new_check_id, line_ids[]), MergeChecks(src, dst), TransferCheck(check_id, to_staff/to_table), CloseCheck, ReopenCheck(actor, reason).

Conflict semantics encoded in the fold (this table is the spec):

86'd items: treat as a soft allocation, not a counter. Coordinator distributes remaining=10 as leases (e.g. 4/3/3 across three stations). A station sells against its lease without coordination. When a station exhausts its lease it requests more; if the coordinator is unreachable it allows the sale and flags it. Refusing to sell food you might have is worse than a kitchen phone call. Reconciliation on heal produces an Oversold(sku, by=2) event and a kitchen alert.

Seat-level splits: seat_no on the line is the primitive. "Split by seat" is a view, not a mutation, until the guest actually pays — then it becomes SplitCheck with explicit line_ids. Keeping split as a view for as long as possible removes 80% of the split/merge conflict surface [estimate].

Snapshotting: fold cost is O(events). A 3-hour table has [estimate] 40–120 events; trivial. Compact on CloseCheck + fiscal seal: write a frozen CheckSnapshot, keep events for 90 days on device / 7–10 years in cloud (tax retention: DE 10 years, PL 5 years, UA 3–7 years [known] — verify per market).

#4. Sync mechanics

#4.1 Outbox and idempotency

CREATE TABLE outbox (
  event_id     BLOB PRIMARY KEY,
  aggregate_id BLOB NOT NULL,
  device_seq   INTEGER NOT NULL,
  payload      BLOB NOT NULL,
  target       TEXT NOT NULL,        -- 'cloud' | 'peer:<device_id>'
  state        TEXT NOT NULL,        -- PENDING | INFLIGHT | ACKED
  attempts     INTEGER NOT NULL DEFAULT 0,
  next_attempt INTEGER NOT NULL      -- monotonic ms
);
CREATE INDEX outbox_due ON outbox(state, next_attempt);

• Rows are written in the same SQLite transaction as the event append. No exceptions. An event that exists locally but not in the outbox is an event that never syncs.
• Server ingest: INSERT … ON CONFLICT (event_id) DO NOTHING RETURNING server_seq. Idempotent by construction.
• HTTP command endpoints (payments, PSP calls) additionally carry Idempotency-Key: <event_id> — every serious PSP supports this [known].
• Backoff: 0.5s, 1s, 2s, 5s, 15s, 60s, then 60s forever with jitter. Never give up. A device offline 3 weeks must still drain its outbox.
• Outbox rows are deleted only after cloud ack and peer ack (or peer marked permanently gone).

#4.2 Clock skew

• Ordering never uses wall clock. HLC only. hlc.phys is nudged forward when a peer's HLC is ahead, clamped to +2 minutes/step to survive a device with a wildly wrong year.
• Durations and timeouts use monotonic clocks: SystemClock.elapsedRealtime() (Android), mach_continuous_time() / CLOCK_MONOTONIC_RAW (iOS/Linux) [known]. A wall-clock-based retry timer breaks when NTP jumps.
• Store both: occurred_at_wall (for humans, receipts, reports — may be wrong) and hlc (for ordering — always right). Reconcile occurred_at_wall server-side against ingest time and flag deltas > 5 min.
• Business day is venue-configured (e.g. 04:00–04:00) and stamped by the device at event creation. A device with a wrong date will stamp the wrong business day; provide ReassignBusinessDay as an audited admin event rather than pretending it can't happen. A Z-report that silently swallows a mis-dated check is a tax problem.

#4.3 Numbering: three counters, never one

This is where teams conflate things and create a compliance incident.

The single most important structural decision in this document: never allocate a fiscal sequence number in application code. Delegate it to the certified fiscal component, which owns its own gapless sequence and has legally-defined offline behaviour:

UA's PRRO model is the pattern to copy internally: the authority hands you a lease of numbers to use offline. Design your check-number leasing identically — leases with epochs and expiry, gaps tolerated, duplicates impossible.

#4.4 Exactly-once effects on printers

Printers are the only place in the system with irreversible side effects. A duplicate database row is fixable; a duplicate ribeye is EUR 24 of food and a chef who stops trusting the tickets.

Cost of getting it wrong: [estimate] EUR 3–15 of food per duplicate ticket; a systemic reconnect-storm bug across 200 venues on one Friday = EUR 10–40k of comped food plus churn.

Architecture:

1. One print coordinator per printer. Printing is the one place where single-writer is worth the complexity.
2. Intent log before send, confirm after:

   fsync(job_id → SENDING, epoch, payload_hash)
   send to printer
   read status
   fsync(job_id → PRINTED | FAILED)

   On coordinator restart, all SENDING jobs go to a verify queue, not a resend queue.
3. Prefer pull-based printers. Star CloudPRNT (mC-Print3, TSP143IV) has the printer poll an HTTP endpoint and POST back a delivery confirmation for a specific job id [known]. That confirmation is what makes exactly-once actually achievable. Epson ePOS-Print XML over LAN (TM-m30III, TM-T88VII) is push-based: you get an HTTP response with a status, but a lost response is genuinely ambiguous [known]. Epson's ePOS-Device / status APIs let you read printer state, which narrows but does not eliminate the ambiguity window.
4. Never resend blind. For push printers, on ambiguity: mark the job AMBIGUOUS, print a new ticket header *** POSSIBLE REPRINT — job 7F3A — VERIFY WITH PASS ***. Humans are a valid part of an exactly-once protocol when the alternative is silence.
5. Every ticket carries a short job id (4 hex chars) and a monotonic per-printer ticket number. Cooks learn to spot 7F3A twice.
6. Kitchen Display Systems are strictly better than printers here — a KDS is a networked peer with state, so delivery is idempotent and confirmable. Where you can sell KDS instead of a printer, do; it removes the hardest correctness problem in the product. [estimate] a 15" Android KDS panel is EUR 250–450 vs EUR 280 for a TM-m30III — a genuinely easy upsell.

Transactional outbox for prints: the FireCourse event and the print job row are written in one SQLite transaction on the coordinator. If the coordinator changes mid-flight (§2.7), the incoming coordinator inherits the job table via the event log and resumes at the verify step.

#4.5 Replay, backfill, tombstones

#4.6 Schema migration on a device that was offline 3 weeks

Four separate migration surfaces — teams typically remember one:

Hard operational rule: a device more than K app-versions behind (or whose fiscal_ruleset_version is stale) may open checks and take orders, but may not tender payments or close checks. VAT rates change, service-charge rules change, fiscal formats change. Blocking money on a stale device is the cheap version of that problem; the expensive version is a tax reassessment.

#5. Storage / sync technology evaluation

#5.1 Verdict

Build the counter on SQLite + a bespoke event-sync protocol. Own the conflict semantics.

Layer it:

#5.2 "Is it sane to trust a managed sync product with money-critical data?"

Trust it as a transport. Never as a semantics engine. Three concrete reasons:

1. They define conflict resolution; you cannot afford theirs. Every managed product ships LWW or server-rebase defaults. Your §3.3 table is your product's actual value. Outsourcing it is outsourcing correctness.
2. Vendor mortality. Atlas Device Sync deprecation [known] is a live example of a category leader exiting. Your POS must outlive any sync vendor by 10 years, because tax retention says so.
3. The offline requirement they don't serve. Every managed product is device↔cloud. None does venue-LAN peer sync with the cloud unreachable. You are writing that regardless — at which point the managed product only saves you the easy half.

Mitigation if you do adopt one: keep the event log as the wire format so the sync vendor is carrying opaque blobs. Migration to a different transport then becomes a 2–3 week job instead of a rewrite.

#6. Test strategy

Standard test pyramids do not catch distributed money bugs. Budget for a simulator, not just a test suite.

#6.1 Deterministic simulation (the core investment)

Model the entire node — domain, sync, print dispatch, election — as a pure state machine over (input, injected time, injected network, seeded RNG). Nothing in the core touches Date.now(), Math.random(), sockets, or disk directly; all are injected. FoundationDB/TigerBeetle style.

Then: run 10⁵–10⁶ randomized scenarios per CI run. On failure, print the seed. The seed reproduces the bug exactly, forever, including on a laptop three years later.

[estimate] This costs ~3.5 eng-months and is the highest-ROI line item in the entire document. Teams that skip it debug distributed bugs by adding logs to production.

#6.2 Property-based invariants ("money in = money out")

Asserted after every event in every simulated scenario:

All money in integer minor units. Never a float. Ever. [known] — this is the most common single bug in POS codebases.

#6.3 Network partition injection

• Docker-compose lab with tc netem: latency 20/150/400 ms, loss 0/2/10%, reorder, duplication.
• Partition shapes: clean 2-way split; asymmetric (A can send to B, B cannot send to A — the nastiest and the most realistic on flaky Wi-Fi); flapping (5s up / 5s down for 10 min); partial (3 of 5 nodes isolated); healing while a print job is in flight.
• Durations: 30s, 5 min, 3 h, 3 weeks (simulated clock).
• Kill/restart during partitions: SIGKILL mid-transaction, SIGKILL between the intent-log fsync and the printer send.

#6.4 Physical device lab

Simulation catches logic; only hardware catches "the iPad suspended the WebSocket in the background and Bonjour never came back."

Scripted hardware chaos (the smart PDU earns its keep): cut printer power mid-job; cut hub power at a random point in a 4-hour soak; reboot the AP; pull a tablet off Wi-Fi onto LTE and back.

#6.5 Soak: the simulated Friday rush

Nightly, on the physical rig:

#6.6 Production replay

Every venue's event log is uploadable on demand (pseudonymize guest PII in the replay pipeline — GDPR, and it makes the fixtures shareable internally). Support can replay a customer's actual Friday in the simulator and reproduce their complaint deterministically. This is a support superpower and it converts "the totals were wrong last Friday" from a 2-day archaeology exercise into a 20-minute investigation.

#6.7 Divergence detector (production)

Each device periodically publishes hash(CheckState) for every open check with its fold_version. The coordinator compares. Any mismatch fires an alert before the check closes. [estimate] 0.5 eng-months, and it is how you find out about a fold bug from telemetry instead of from an angry restaurateur.

#7. What it looks like when a team gets this wrong

Concrete, in the order it usually happens:

1. Month 3 — "it works." Built on Firestore or Realm. Demos beautifully. One device, office Wi-Fi.
2. Month 7 — first 3-station venue. Items intermittently disappear. Team adds "refresh" buttons and blames the network.
3. Month 9 — the theft accusation. Owner's weekly P&L is short ~1.2%. He fires a server. Six weeks later he works out it's the POS. You will not get that account back, and he tells the other owners in his WhatsApp group.
4. Month 10 — the double-fire Friday. A reconnect storm re-sends the print queue. Kitchen gets 40 duplicate tickets across 8 venues. Chefs start ignoring the printer and calling out orders verbally — your product is now decorative.
5. Month 12 — the fiscal incident. App-allocated "fiscal" numbers gapped during an outage in PL or RO. Now it's an accountant's problem, then a tax authority's problem, and possibly a re-certification cycle ([estimate] 2–5 months and EUR 5–20k per regime).
6. Month 13 — the diagnosis. The state model cannot express what the business needs. Fixing it means changing the write path, the merge path, the reports, and every deployed device's local schema simultaneously.
7. Rebuild cost: [estimate] 9–18 engineer-months, plus a per-venue migration with a 4-hour maintenance window that no restaurant will grant you on any day they're open. In practice most teams ship "v2" as a parallel app and migrate venues one at a time over 6–9 months, while maintaining both.
8. Meanwhile: a managed sync vendor deprecates (Atlas Device Sync [known]) and you port under time pressure with zero option value.

The asymmetry that should drive the decision: doing this right costs ~34 eng-months up front. Doing it wrong costs ~34 eng-months anyway, 12 months later, plus your first 50 customers, plus the founder's credibility in a market that runs on word of mouth.

#Build effort

Engineer-months for the offline/sync/domain core only. Excludes: UI/UX of the POS, menu management, back office, CRM/loyalty, payments PSP integration, per-country fiscal drivers, hardware certification matrix (all covered in sibling docs).

Reduced scopes worth costing:

Calendar time:

S1 strategic read: a bootstrapped team should ship T0-only (≈13 eng-months) and pick a beachhead segment where single-station is genuinely sufficient, then earn the right to build T3. Attempting full multi-station offline sync as a first deliverable with 3 engineers is the most likely single cause of the project running out of runway before first revenue.

#Open questions / what would change this answer

1. Does v1 need LAN multi-station at all? This is a ~15 eng-month swing (13 vs 34) and it is a market question, not an engineering one. What fraction of the CEE beachhead's addressable venues run 2+ POS stations? If it is under ~40%, ship T0 first. Would be settled by: 30 discovery calls with target venues counting actual station counts, plus competitor station-count data from Poster/SmartTouch/Syrve deployments.
2. Is a cloud-TSE (fiskaly) legally acceptable for German offline operation, or does DE require a local hardware TSE for our offline claim? [verified] that TSE failure is legally accommodated with receipt marking and tamper-proof failure logging — but a cloud-TSE means every WAN outage is a TSE outage, which turns an exceptional path into the normal path. If DE is in scope, this determines whether we must ship Swissbit hardware TSEs with every venue (~EUR 60–150/unit [estimate]). Would be settled by: reading fiskaly's and Swissbit's compliance documentation plus one German Steuerberater consultation.
3. Do we sell a hub SKU? EUR 150–190 BOM + a support/RMA burden + firmware OTA infrastructure, against materially better reliability and a simpler correctness story. Would be settled by: pricing the support cost of hub failures at ~2% annual hardware failure rate across N venues.
4. Does PowerSync's Open Edition source-available license permit redistribution inside a commercial on-prem product? Cloud pricing is [verified] ($0 / $49 / $599 / custom), but source-available ≠ open source and on-prem redistribution is exactly the case such licenses restrict. Would be settled by: reading the actual LICENSE file in the powersync-service repo, 20 minutes of work.
5. Will our target PSPs (Adyen, Nexi, SumUp, Viva Wallet, Ukrainian acquirers) permit store-and-forward card payments with a floor limit, and who bears the chargeback? This determines whether "take payments with the internet down" means "cash only" or "cash and cards." That is a large difference in the sales pitch. Would be settled by: written answers from 2 PSPs' integration teams.
6. Confirm the Atlas Device Sync EOL date and check whether any managed sync vendor offers a contractual data-format escape hatch. [known] currently. If no vendor will contractually commit to an exportable, documented on-device format, that hardens the "own your sync protocol" recommendation from "strongly advised" to "mandatory." Would be settled by: MongoDB's deprecation notice + asking PowerSync/Zero for their format-stability commitments in writing.