Local durable work

Scope

Migration 021 initially added work.operation. Migration 022 preserves those records as work.outbox and adds a separate work.job_queue transport table. Both live in the normal application database, in the work schema:

Application changes and outbox creation share ITransaction and commit or roll back together. Only committed intent is eligible for dispatch. Queue publication and acknowledgement deliberately run outside business transactions, matching the boundaries of a future external broker rather than hiding them in one PG commit.

API charges and CLI payouts always enqueue through PaymentQueue and IInvoiceWork. There is no synchronous bypass or local-only business-flow flag. The standalone Worker host claims the work, executes payment handlers, and records ledger/invoice updates atomically. Bootstrap reads the Services definitions from shared appsettings.json, selected by ENVIRONMENT:

Environment Stripe ModT
Development FileStripe FileModT
Test / Tests FileStripe FileModT
Prod / Production RealStripe RealModT

These names select implementations of the same IGateway contract. Payment workflows never choose a mode or inspect the environment. File implementations simulate pending payments without moving money or making payment-provider HTTP requests. Missing or invalid definitions fail startup rather than falling back to another provider. There is no file-payment command-line switch.

Ledger v1 and future processing

Ledger v1 will use our existing standalone Worker, PG outbox and job queue, dispatcher, processor, and payment-status polling. Adopting Hangfire or an Azure queue is not a v1 prerequisite. Keep the current handler contracts and payment safety guarantees; do not add a second processing mode for this future option.

Hangfire is the preferred candidate to evaluate after v1 for durable job execution and its management dashboard. The proposed flow is our transactional outbox -> Hangfire execution -> our business handlers. Replace overlapping delivery, scheduling, and retry machinery rather than stacking two competing processors. This requires an integration review, not merely swapping an IJobQueue adapter.

Any later adoption must preserve these boundaries:

SigNoz remains the planned complementary observability UI for structured logs and correlated traces across outbox dispatch, handlers, provider calls, polling, and ledger recording. Telemetry is not the authoritative queue or financial record and must not block processing when unavailable. Until a management UI is adopted, operational controls can be exposed through explicit CLI commands as needed.

References: Hangfire retry safety, dashboard and access control, PostgreSQL provider.

Poll checkpoints

Migration 024 adds work.polling, keyed by the registered poll job name. The shared WorkerEngine.PollAsync orchestration supplies a PollContext with LastSuccessfulRun (null before the first success) and StartedAt. After the handler finishes discovery and durably queues all work, the engine saves that start time. Empty successful scans advance it; failed or canceled scans do not. Overlapping successful runs cannot move the persisted timestamp backwards. Both the local Worker and Azure timer hosts use this path and share the checkpoint in their application database. Diagnostic PollPaymentsAsync sweeps, which can target one work item, do not advance the whole job's checkpoint.

Handlers own their query windows and first-run policy; orchestration does not apply a lookback. A date-based return discovery job can use the previous successful scan minus one day through StartedAt. The existing bank-deposit query still uses its ledger-date anchor below, while pending payment checks have no date filter. All providers within a job must succeed before its checkpoint advances. Handlers must propagate incomplete scans rather than merely log failures. Already queued observations are deduplicated when a failed scan is retried. A future change of provider/account scope requires a new job identity or an explicit checkpoint reset.

Incoming Stripe payouts to Tendr's bank account

Customer collection completion at Stripe does not mean those funds have reached Tendr's business bank account. BankDepositPoller discovers posted bank credits at ModT independently of PaymentStatusPoller. Both implement IPollHandler, with separate schedules: bank discovery every four hours, recent payment statuses every five seconds, and older payment statuses every five minutes. Both use six-field cron schedules in UTC. Console polling waits for the next scheduled occurrence, just like the Azure timers with RunOnStartup = false; --poll-once runs an explicit immediate scan. Runs do not overlap themselves; the next occurrence is calculated after completion rather than accumulating missed executions.

The ModT adapter queries transactions with posted=true, direction=credit, the configured originating account, and as_of_date_start/end. It follows X-After-Cursor even across empty pages. Only posted USD credits enter this flow; pending statement rows have different provider IDs and are not ledger deposits. The API must supply vendor_description for classification; missing descriptions are not guessed from metadata or amounts and are skipped. Arrivals backdated beyond the one-day overlap require an operator rescan/backfill.

Local simulation uses deposit rows in ~/.tendrme/dev/payments/FileModT.csv, with account_xid=tendr, a unique bank transaction ID in operation_id and payment_xid, an empty idempotency_key, and processed_at carrying the bank date. Put the bank description with an Originator field in statement_descriptor; the file provider uses the same parser as real ModT, including unknown originator names. Other payment fields may be blank. Set status to Completed to simulate a posted deposit; Pending is ignored. The ledger still needs an initial backfilled deposit.

Example CSV row (using the existing header):

deposit,bank-txn-001,tendr,,4321,,,Type:ACH;Role:receiving;Originator:STRIPE;Recipient:TENDRME;Message:TRANSFER,Completed,bank-txn-001,2026-10-22T00:00:00.0000000+00:00,

--poll-once runs all registered discovery handlers without waiting four hours. BankDepositObserved is handled only by BankDepositHandler, recording a positive, Completed ledger entry with the generated outbox ID as its idempotency key. It does not emit PaymentSettled, invoice/RT payouts, or payment emails.

Bank-balance accounting rule, not a separate feature or v1 release blocker: sum confirmed incoming Stripe bulk deposits at ModT and subtract confirmed outgoing vendor/RT payouts. Do not also count the individual Stripe customer collections as bank credits. Assume sufficient operating float; do not hold vendor payouts until Stripe's bulk transfer arrives. No separate profit transfer, second bank account, or processor-fee ledger entries are required for this accounting model.

The optional tendr ledger balance --env Development command now reports actual balance, pending outflows, and available balance (float) from the ledger. It makes no bank API calls, excludes unsubmitted payout work, and adds no operating reserve rules. See CLI bank balance.

Deferred: payment failure recovery and alerts

Post-v1 UI follow-up: surface ACH invoice payments still awaiting completion and their not-yet-issued vendor payouts. Keep this separate from payouts already submitted but still Pending. No new ACH-lag UI or RFP checkout is needed for v1.

This is post-v1 work, not a prerequisite for ledger/worker v1. Keep asynchronous payment-failure recovery manual for v1; do not add automatic rail-recovery handlers or queued email delivery yet. V1 records PaymentFailed and PaymentCanceled observations through reconciliation handlers, without attempting automatic recovery. Failed/canceled charges and payouts (including RT) and returned payouts also have a PaymentOperatorAlert subscriber. After financial recording commits, it emits an Error log routed through the existing ErrorAlerts email sink. The alert includes invoice/vendor, ledger/payment-work and provider references, amount in cents, and return code where applicable. Repeated observations/redelivery do not rerun a completed event's alert; rollback sends no financial-outcome alert. Delivery remains best-effort: SMTP failure or shutdown after commit can lose the email. No automatic payment retry or customer email is triggered by this subscriber. Orchestration error logs pass the exception object to the email sink so alerts include the full exception chain and stack. Explicit retryable failures remain Warning while attempts remain, but exhaustion emits Error; missing prerequisites remain deferred warnings. The outbox error field stores only operational classifications; work.log includes full exception text in its JSON fields. Operators can use tendr vendor rail and tendr invoice pay for a definitively failed/canceled invoice payout, as described below. This is not RT retry or returned- payout recovery; those cases still require separate review.

Near-term follow-up: failure-status handlers should send customer/vendor failure-notification emails after the status update commits. This notification work is independent of automatic recovery and does not need to wait for rail fallback or first-class email outbox delivery. Keep it out of v1 for now; the operator handles failed payments manually. Notification failure must not resubmit payments, and repeated status observations must not repeatedly notify recipients.

Deferred customer UI change: a failed customer payment should display DUE, as a new unpaid invoice does, rather than FAILED. Preserve the underlying PaymentStatus.Failed and its history; this is presentation only, not an automatic payment reset or resubmission. Vendor payout failures are a separate concern.

The intended future flow is:

  1. Status polling atomically records a definitive provider payment failure and creates a PaymentFailed outbox event. Future webhooks use the same path. The original submission job remains completed; acceptance and settlement are separate outcomes.
  2. A failure-recovery handler classifies the provider reason. An explicitly recognized unsupported-rail failure may authorize a new payout on the next lower rail through the existing generic payout machinery. Generic failure, NSF, ambiguous errors, and unknown outcomes do not justify rail downgrade.
  3. A replacement is a new outbox item with new provider idempotency and external IDs, linked to the original attempt. Preserve the original frozen request and history. Stop at ACH, never cycle rails, and only update account preference if the bank account and attempted preference still match.
  4. Fatal or unrecognized failures create a separate SendEmail outbox item for an operator alert. Ambiguous outcomes require review, not another payment; they can also generate an alert. An email handler retries delivery independently without replaying payment submission or blocking recovery completion.

Failure-event publication, replacement scheduling, and alert scheduling must be duplicate-safe. Tie downstream deduplication to the original failure event and action, and commit child intent with handler completion. Replayed observations or events must not create duplicate replacement payments or alert jobs. This is an explicit future recovery policy, not permission for ordinary job retries to mint new payment identities. Submitted failures remain in the ledger with Failed status, but do not contribute to settled balances.

Deferred: automatic invoice cancellation

A separate scheduled sweep should identify eligible old, unpaid invoices and publish CancelInvoice outbox items with the invoice ID and reason Expired. Use the existing invoice.canceled_at; do not add expired_at or an Expired invoice status. This is not part of ledger/worker v1.

Deduplicate cancellation requests across repeated or concurrent sweeps. The handler must recheck eligibility under the same invoice locking used by payment initiation, excluding already-paid invoices and payments in flight. Apply canceled_at and complete the work in one transaction. If payment began after the sweep, skip cancellation and record that outcome. The outbox preserves the reason, retries, and outcome; already-canceled invoices are an idempotent no-op.

Choose the age/due-date policy and sweep cadence before implementation. Three months was an example, not an agreed expiration threshold.

Deferred: notifications and priority queues

After ledger/worker v1, move email and SMS delivery to outbox events such as SendEmail and SendSms. Commit notification intent with the business change, then deliver through independent handlers and configured providers. Deduplicate intent against the originating event/action. Recheck SMS consent at delivery; login and verification messages also need expiry and latency safeguards. Provider acceptance followed by a failed completion write still requires provider idempotency or reconciliation to avoid duplicate delivery; an outbox alone does not guarantee exactly-once messaging.

Keep one work.outbox as the durable intent/history store, with a system-defined priority discriminator such as WorkPriority.RealTime and Normal. Do not split it into high/low outbox tables. Dispatch into two independent delivery queues:

These names are provisional. Both local queue tables use the same schema and share one implementation; the lanes can map to separate Azure queues later. Dispatch and consumption must both honor the priority lane so slower standard work does not delay realtime dispatch. Handlers remain unaware of scheduling.

Priority controls pickup cadence, not retry eligibility: both lanes still honor available_at, leases, and duplicate-safe processing. A missing-bank payout remains deferred for an hour even if marked realtime. RealTime is a low-latency class, not a guaranteed one-second execution or delivery SLA. This design is recorded only; v1 retains its existing single job queue and notification paths.

Shared bootstrap

Bootstrap/Bootstrap.csproj builds Tendr.Bootstrap, referenced by the API, CLI, and Worker without referencing any executable or Azure Functions package itself. It packages the shared appsettings.json into each host's output/publish directory. API, CLI, and Worker use AddTendrConfiguration to load that required base file, then optional appsettings.{Environment}.json, then environment variables and host configuration command-line arguments. Worker also supports its explicit --database-config file before environment-variable overrides. JSON files are read at startup without reload; settings are validated by the requested services.

// Each host/command declares the services it uses.
services.AddTendr(configuration, environment)
    .Require<IInvoiceStore>()
    .Require<IVendorStore>();
// After building the host, before serving requests or accepting commands:
host.Services.ValidateTendrServices();

Bootstrap owns an explicit recipe for each supported service: dependencies, lifetimes, configuration, provider selection, and startup validation. Require<T>() recursively registers those dependencies once. Repeated requirements (including across AddTendr calls with the same context) reuse registrations. There is no assembly scanning, inferred "all stores" collection, or storage/payment profile. Unknown services/keys and conflicting bootstrap contexts fail clearly. Host-owned services still use normal IServiceCollection registration.

// Bank setup: only the ACH provider, not Stripe, pricing, or workers.
services.AddTendr(configuration, environment)
    .Require<IVendorStore>()
    .Require<IGateway>(PaymentMethodType.Ach);

// Payment producer: scheduler and its stores/pricing, no payment gateways.
services.AddTendr(configuration, environment).Require<PaymentQueue>();

// Worker: dependencies include both gateways, handlers, and settlement stores.
services.AddTendr(configuration, environment)
    .Require<WorkProcessor>()
    .Require<OutboxDispatcher>()
    .Require<PaymentStatusPoller>();

Gateway and browser-link requirements need a PaymentMethodType key. Explicit IEnumerable<IGateway> / IEnumerable<IServiceLink> requirements register both configured methods; IEnumerable<IWorkHandler> registers the two built-in payment handlers, RT's settlement scheduler, and the separate RT planner while preserving host-added handlers. Unkeyed single gateways are ambiguous and rejected. Request individual stores by their specific interfaces.

IWorkHandler.Event, WorkItem.Event, and WorkCreateOptions.Event use the system-owned WorkEvent enum (ChargeInvoice, PayVendor, PaymentSettled, PaymentFailed, PaymentCanceled, BankDepositObserved, PaymentReturned, RtPayout, Probe). The outbox's existing type column stores enum names, not integers; JSON also writes names. The processor invokes all handlers registered for an event, sequentially in DI registration order. Each has a stable string Name, unique within that event. Duplicate names are rejected; multiple distinct subscribers are supported. Ledger Kind remains a caller-defined string.

All ExecuteAsync calls finish before recording begins. A failure stops the current phase, so later handlers in that phase do not run. All RecordAsync calls and outbox completion commit together; recording failure rolls back that entire transaction. The outbox result is a JSON object keyed by handler name. Payment polling reads only its submission handler's result, not another subscriber's result.

Retry replays the full chain: there are no per-handler completion checkpoints or independent deliveries. The worker gates the whole event only after completion. Handlers must replay their own external operations safely with stable, distinct operation keys and return the original matching result, rather than skip recording. An external success does not prove the local transaction committed. Conflicts with different facts are errors, not successful duplicates. Uncertain failures remain held for review under the existing policy; only retryable failures retry automatically.

IOutboxStore and IOutboxDispatchStore resolve the same scoped outbox store. IJobQueue resolves a separate PG transport implementation. Outbox result writes share the transaction used by ledger and invoice stores; transport operations do not. Startup checks only requested dependencies: relevant work/ledger tables, provider configuration, and the file-provider destination guard when file gateways are selected. Explicit host IConfigValidator registrations run too, once per instance per validation. Hosts still load their environment/config files and add their own services; no payment business-flow switches are involved.

Only the API requires IEnumerable<IServiceLink> for browser setup. Its existing debug-only embedded PgLite option supplies a database factory to the shared registration; the actual connection is still checked during startup. Normal shared database bootstrap never starts/stops a server or applies migrations.

The shared bootstrap never starts processing loops inside API or CLI. Worker host registration and lifecycle are provided by Worker.Engine as described below.

Worker hosting

Worker.Engine/Worker.Engine.csproj is the reusable Tendr.Worker.Engine library. It references Bootstrap/Core and standard .NET hosting/options abstractions, not the console executable or Azure Functions. Core retains the business processor, dispatcher, handlers, settlement logic, and stores. The library owns their host registration, scope lifetimes, validated worker settings, and optional polling loops.

Both hosts discover concrete, closed IPollHandler types in the Core assembly. Bootstrap registers their constructor dependencies through the existing service recipes and exposes each handler as a scoped concrete service plus an IPollHandler alias. There is no per-poller registration list. New handlers need one public constructor; a genuinely new dependency still needs its usual bootstrap recipe. Discovery is assembly-bounded, not an AppDomain-wide scan of arbitrary loaded code.

Hosts compose configuration with the normal .NET Generic Host APIs, then call builder.AddTendrWorker(). This uses IHostEnvironment.EnvironmentName; the library does not load files, inspect environment variables, construct a second container, or select a local/cloud business mode. Supply appsettings.json and the host's database/pricing/provider settings before registration. The console supports --env (alias -e), then DOTNET_ENVIRONMENT, then the existing ENVIRONMENT variable, in that precedence order. An environment must be explicitly supplied; there is no default. The environment selects service definitions; --database-config still supplies the database/settings file explicitly.

dotnet run --project Worker -- --env Development --database-config Bill/appsettings.Development.json

Worker timing options are bound and validated at startup via the options system and ValidateOnStart, with no reload subscription. The same immutable snapshot is injected downstream. Missing/invalid timings fail startup; there are no timing defaults. PaymentQueueOptions remains the single shared source of MaxAttempts. A hosted lifecycle startup check validates storage, providers, and handler registration before hosted services start. Hosts should not separately call ValidateTendrServices when using AddTendrWorker.

For continuous console hosting:

AddTendrWorkerPolling discovers concrete background services in Worker.Engine and registers each once as a hosted service. Azure does not start those console loops: it uses the same discovered poll handlers with Azure's generated timer metadata and startup cron validation instead.

// Configuration and EnvironmentName already supplied by the host.
builder.AddTendrWorker().AddTendrWorkerPolling();
using var host = builder.Build();
await host.RunAsync();

The existing API Function App now hosts five timer functions in Bill/WorkerFunctions.cs:

Function UTC schedule Work
ProcessWork */5 * * * * * (every five seconds) Dispatch then process bounded batches
PollEveryFiveSeconds */5 * * * * * (every five seconds) Recent Pending payment status checks
PollEveryFiveMinutes 0 */5 * * * * (every five minutes) Older Pending payment status checks
PollEveryFourHours 0 0 */4 * * * (every four hours) All pollers declaring this exact cron
PollDaily 0 0 0 * * * (daily at midnight) Payout return discovery and any other pollers declaring this exact cron

API registration calls services.AddTendrWorker(startupConfig, environment, "api"), reusing its existing Bootstrap context, email provider, and template renderer. It does not register the standalone polling services. RegisterWorkerEngine also registers AzurePollRegistry as a singleton hosted lifecycle service. During StartingAsync, before any hosted service starts, it asks Azure's IFunctionMetadataManager for the SDK-generated function metadata (including registered metadata transforms). It reads the timer bindings for entry points in PollFunctions; WorkerFunctions.ProcessWork is deliberately outside that class. There is no custom marker attribute, assembly reflection scan, or per-function registration list. Add poll timers to PollFunctions with normal Azure Function/TimerTrigger attributes. Missing or invalid metadata fails startup. Every IPollHandler.Schedule.Cron must match exactly one timer by ordinal string equality; missing or ambiguous matches fail startup with the handler name and required cron. Semantically equivalent but differently written expressions do not match. Six-field syntax is validated using NCrontab, with seconds enabled; there is no hand-written cron-to-interval conversion. Calendar expressions and uneven steps such as */7 minutes are supported. Configuration placeholders are not accepted as literal schedules. Execution is UTC; leave Azure timezone overrides unset so both hosts evaluate the same schedule. Several handlers can share one timer. Each retains its own DI scope, checkpoint, and failure handling; one failing handler does not prevent siblings from running, but the timer reports failure. The queue dispatch/consume timer is separate and is not a discovery route.

All functions use the same host-neutral IWorkerEngine; discovery timers route via AzurePollRegistry to PollScheduleAsync(cron), not hardcoded handler names. Each dispatch/process batch has a 100-item limit and a two-second budget for starting new claims; an in-flight operation may exceed that budget. Host cancellation flows to the engine. No timer runs on startup; the five-minute/four-hour/daily timers use Azure schedule monitoring, while the five-second timers do not. Schedule monitoring requires AzureWebJobsStorage.

Local Functions uses AzureWebJobsStorage=UseDevelopmentStorage=true. Timer host leases also need Blob Storage when UseMonitor=false; this is separate from Tendr's PostgreSQL outbox and polling checkpoints. The VS Code API debug task runs node scripts/ensure-azurite.mjs before starting the host. For manual func start, run that command first. It reuses an authenticated, healthy local Azurite Blob endpoint or starts azurite-blob from PATH/the installed Azurite VS Code extension, bound only to 127.0.0.1:10000. Install that extension or npm install -g azurite if neither is present. Emulator data and logs live in .tmp/azurite and persist across debugging sessions. The helper never kills existing listeners or deletes data; an occupied but unhealthy endpoint fails startup instead. No queue/table emulator is needed by these timer functions. Do not disable schedule monitoring to work around a missing emulator.

For local Functions debugging, start Azurite with the existing development storage settings. Starting the API now also starts these timer listeners, so a separate console Worker is no longer needed. To intentionally run console Worker instead, disable the timers using Azure's normal function settings: AzureWebJobs.ProcessWork.Disabled=true, AzureWebJobs.PollEveryFiveSeconds.Disabled=true, AzureWebJobs.PollEveryFiveMinutes.Disabled=true, AzureWebJobs.PollEveryFourHours.Disabled=true, and AzureWebJobs.PollDaily.Disabled=true. The frequency-named timers replace the previous PollPaymentStatus/PollBankDeposits function names; update any per-function settings. Do not run both hosts against the same dev records unless deliberately testing concurrency. Nothing is deployed by adding these functions. scripts/ship.sh discovers new timer names from published metadata and disables them alongside HTTP functions before migrations.

IWorkerEngine.PollSchedules exposes registered IPollHandler names and cron strings. The generic DiscoveryPollingWorker runs each in an independent loop/scope, so a slow or failed bank scan does not stall payment status polling. Additional poll handlers register through DI without adding a new host loop. Names and six-field cron syntax are validated at startup. Console scheduling uses NCrontab to calculate the next UTC occurrence from Schedule.Cron, honoring calendar boundaries rather than approximating them as fixed intervals. Long waits are cancelable and split into bounded delays to support schedules beyond Task.Delay's maximum duration. Checkpoints still advance only after successful scans and remain independent of the scheduler.

Azure function entry points are implemented; production deployment remains pending. Each batch stops when empty, at its item limit, or before claiming another item after the budget expires. An already-running operation may exceed the batch budget; host cancellation is propagated independently. Payment sweeps snapshot pending operations once and use the validated lease duration as each provider request's timeout. Choose invocation limits appropriate to the hosting plan. A sweep returns checked/failed counts so the host can report partial failures without losing them.

DispatchOnceAsync, ProcessOnceAsync, and PollPaymentsAsync also accept an operation ID for targeted debugging. Every call creates/disposes its own async DI scope; batches never share a database connection across jobs. Calls do not swallow infrastructure failures or host cancellation. Interrupted leases remain recoverable.

The optional console polling services run dispatch/consume loops with the validated idle interval (configured as five seconds) and payment sweeps on their five-second/five-minute schedules. One-shot console commands start the host for validation but do not register these loops, so they cannot accidentally process unrelated work. Probe and administrative commands remain console-owned.

Contract

Handlers implement ExecuteAsync and RecordAsync. Execution must be retry-safe using the separately persisted provider idempotency key and must resolve ambiguous outcomes by that same identity. Recording must use the same scoped connection/transaction as the processor and must not call external services. Post-commit callbacks are best-effort only, not a durable delivery mechanism. The processor never guesses whether a payment settled.

A process crash can occur after a successful provider call and before its result is stored. Every charge/payout request carries both InternalId and IdempotencyKey. UniqueId validates their 10-character lowercase alphanumeric format; payment validation rejects equal values. Keys are generated once and saved in the outbox payload. Stripe receives the supplied key as RequestOptions.IdempotencyKey; ModT receives it verbatim as Idempotency-Key, with InternalId as external_id. For newly scheduled charges and payouts, InternalId is the reserved ledger ID, not the work-item ID or invoice ID. LedgerId is generated and persisted in the immutable outbox payload before any provider call. The ledger row is inserted with that exact ID after acceptance, once the immutable provider source_reference is known; pending settlement still records a Pending ledger row. Replays reuse the reserved ID, including recovery after a provider success followed by DB rollback. Status polling uses the same provider reference, and payout emails use that ledger ID after commit. Charge and payout each reserve distinct ledger identities. Legacy work without LedgerId keeps its original provider reference and ledger creation behavior; existing provider records are not rewritten. The ledger's idempotency_key still records the work-item ID for deduplication and audit linkage. Discovery keys for each rail are also generated once and persisted. Separate rail-candidate keys prevent a cached ACH success being mistaken for an RTP success during replay of discovery. Neither worker nor collision handling increments an external-ID version. New attempts require a future explicit human-authorized resubmission workflow; that management UI is not implemented here.

File providers persist idempotency_key in the CSV and reject conflicting payloads. Legacy receipts without a key remain readable for status polling, but cannot be resubmitted by guessing a key. AuthNet is deprecated: its source and legacy test commands are retained but excluded from compilation, and its SDK is no longer a build dependency.

Stripe and ModT have finite idempotency retention (24 hours). Replays of operations at least 23 hours old fail closed to NeedsReview before calling a provider, leaving a safety margin. Age is measured conservatively from durable creation, not the client clock. Automatic provider lookup/reconciliation of such ambiguous submissions remains future work. ModT conflicts likewise stop rather than inventing a new identity. Leases protect database writes, not external side effects. Tests use mocked HTTP providers and actual file receipts, not real money movement.

Invoice and payment flow

IInvoiceWork schedules work, rather than executing it. The PG implementation locks the invoice while checking/enqueueing; a unique invoice/type index also prevents duplicate charge/payout jobs. Repeating a submission returns the same operation. Conflicting charge references are rejected. Scheduling participates in an existing transaction, or opens one if there is none. Charge scheduling atomically marks collection Pending; payout scheduling requires Completed collection. Bank setup is checked before submission, not before creating payout intent. A missing bank defers only that payout. No ledger entry is written by the producer.

Scheduling persists the selected gateway's source in the immutable work payload. Handlers resolve by that exact source and method, never a replacement provider chosen from changed configuration. A missing provider fails closed. Both record the provider reference in the outbox result and use PaymentReconciler for status effects. InvoicePaymentProjection additionally updates invoice fields for invoice payments only. Provider submission creates a Pending InvoicePayment or negative VendorPayout. Confirmed collection/disbursement updates that entry's status and its audit history. Unknown rails use the existing precise unsupported-rail discovery policy. Failed/canceled submitted movements stay in ledger, excluded from balances. A pending charge does not set TotalPaid; a failed payout leaves the collected invoice Paid.

IGateway.Source declares the provider identity. Submission persists that named source in the outbox result; polling resolves the gateway by the saved source and payment method, and ledger posting uses the same source. File gateways retain the identity they simulate. Missing providers, missing source in older results, and observations from a different source fail rather than guessing from the payment method. Source is an opaque, case-sensitive, nonblank string supplied by the provider, not a Core enum or registry. Migration 020 creates the ledger without a source allowlist; adding a gateway does not require a new source value in Core or a schema change. This does not change initial provider selection: choosing between multiple card providers for new submissions remains separate work.

Payment status reconciliation

Migration 023 separates invoice cancellation from the two string-backed financial states:

Collection retains payment_xid and payment_method; payout has payout_xid. Enqueue marks only the corresponding financial status Pending. Provider submission adds its reference, and settlement updates only that financial status. Neither changes canceled_at. A failed payout leaves collection Completed. Canceling an invoice does not cancel or reverse provider payments; those retain their own states. The existing immutable outbox payload links each operation to its invoice; there is no extra payment-tracking table, duplicate operation-ID column, or per-invoice polling timestamp. 023 replaces the previous integer status: Created and Unknown become active with no collection; Pending/Failed/Refunded keep their corresponding collection status; Paid becomes Completed collection; Disbursed becomes both collection and payout Completed. Legacy Canceled invoices get canceled_at = created_at and payment_status = 'Unknown': invoice cancellation does not establish a provider payment cancellation. Financial Canceled statuses require a provider-confirmed cancellation of an initiated payment. Creation time is a historical fallback, not the known time of cancellation; other legacy states keep a null cancellation timestamp. Existing provider references are left unchanged; no historical payout ID is invented. No outbox history is imported. Down restores the old combined integer projection; the old schema cannot retain independent payout failures/cancellations, cancellation timestamps, or distinguish invoice cancellation from collection cancellation. The removed Unknown sentinel normalizes to Created on rollback.

An active invoice with collection Unknown displays Created. The bill API's status is an enum name (for example "Paid" or "Pending"), derived through GetDisplayStatus, not a state used to authorize financial work. The separate InvoiceDisplayStatus enum keeps these presentation labels. Completed collection displays Paid until payout completes, then displays Disbursed. The API also returns paymentStatus and payoutStatus as normalized lifecycle names. The frontend API client binds that name to its numeric BillStatus enum once, so local comparisons and range checks remain numeric. Unknown names are rejected. The CLI status filter uses the same derived presentation. Payout Pending/Failed/Canceled still display Paid after collection, as before. Document cancellation displays Canceled; collection Refunded takes precedence over a prior payout. Ledger entries represent submitted movements with current status. Migration 023 adds status and trigger-written ledger_history to the unreleased ledger schema. Balances sum Completed entries; pending amounts are returned separately. Runtime permissions allow only status updates; financial facts remain immutable.

The shared PaymentStatusPoller implementation snapshots unresolved submitted payments from work.outbox across both providers and checks each once per sweep. Two code-defined PaymentStatusPollProfile configurations select the schedule and submission-age window: PaymentStatusFast runs every five seconds (*/5 * * * * *) for submissions up to one minute old; PaymentStatus runs every five minutes (0 */5 * * * *) for older ones. The discovered FastPaymentStatusPoller delegates to the same implementation. Each has its own checkpoint. Windows use the run's start time and work.outbox.completed_at (submission committed), not creation time or the last checkpoint. The exact cutoff belongs to the fast profile; the normal profile selects strictly older submissions. Pending rows are not excluded by a checkpoint, so missed runs remain recoverable. No schema change is needed. Console scheduling and Azure's matching timers use these same crons; neither requests an immediate startup run. Use --poll-once for that. Manual PollPaymentsAsync / RunSweepAsync calls without a window still check all Pending payments regardless of age. Pending observations wait for that next sweep, not another immediate turn. Each check has its own timeout; errors are reported without stopping the remaining checks. Unchanged status, network errors, and process crashes remain eligible for a later sweep; they do not create new jobs or consume submission attempts. Multiple Worker instances may read the same provider status. PaymentSettledPublisher locks the original payment row to validate its state, then delegates publication to IOutboxStore.PublishOnceAsync, the same operation used by bank discovery. It uses existing request scope/key/fingerprint columns: scope identifies terminal status, key is the original operation ID, and fingerprint includes stable payment facts, provider, status, and provider reference, never poll time. Repeated observations reuse the first event and its original payload, whether Pending, Processing, or Completed. The unique request index backs this identity; no new columns are needed.

Polling (and future authenticated webhooks) calls ObserveAsync, not reconciliation. It queues PaymentSettled, PaymentFailed, or PaymentCanceled without changing payment/invoice state or writing the ledger. Pending observations create no event. An immediate terminal submission result uses this same queued path. The payment remains Pending until its observation is processed; further polls safely deduplicate.

PaymentSettlementHandler runs reconciliation in the worker's existing recording transaction. For successful settlements it is registered before InvoicePayoutHandler and RtPayoutSchedulingHandler: ledger status/history, payment/invoice updates, invoice payout intent, one RtPayout work item, and event completion commit together. Settlement does not calculate RT, inspect referrals, or decide eligibility; every completed invoice collection (including ACH) queues RT processing. A publication failure rolls settlement back, so completed payments cannot lose their RT follow-up. Failed/canceled events update existing ledger status/history, without scheduling payouts. Conflicting terminal states require review, never a silent rewrite. Late Pending observations cannot undo settlement. Recovery/alert policies remain post-v1 work.

Shared payouts and local Rising Tide

IPayoutService owns payout preparation and submission for both invoice payouts and RT bonuses. PayVendorHandler is the one execution handler; no second RT remittance or ledger implementation exists. Requests carry a caller-defined string kind and recipient/context. Kind has no enum, registry, or database allowlist; the shared payout path preserves it without needing to recognize it. Each immutable outbox payload contains the amount, selected provider, and replay keys. A write-once preparation snapshot pins the destination before a provider call is possible.

work.outbox.payment_status tracks settlement independently of job completion. Null means this job has no recorded payment submission (including nonpayment work). Submitted payments start Pending. Invoice fields are projections, not the polling source: RT payments may reference the originating invoice but never overwrite its payout_xid or payout status. There is no extra payment table or per-payment polling timestamp.

RisingTideHandler subscribes to the separate RtPayout event and owns eligibility checks and the shared RisingTidePayouts planner. Ineligible jobs finish successfully without recipient payouts. The job is published once with scope rt-payout, keyed by the collection ledger ID (original charge work ID for legacy payloads without a reserved ledger ID). Observation timestamps are excluded from its fingerprint. Catch-up respects existing RT jobs in every state, including completed zero-recipient jobs and failed jobs awaiting repair; it must not bypass their retry state. For historical payments without such a job, catch-up calls the same planner without fabricating settlement events or adding a catch-up event type. The planner calls IRisingTideCalculator and IPayoutService.ScheduleAsync for each recipient inside the event-completion transaction (or a transaction owned by catch-up). No separate calculation plan is stored on disk or in the database: the PayVendor outbox items record the recipient, amount, and ReferralBonus reason. A failure rolls all inserts and completion back; retry can recalculate. Completed redelivery is gated by WorkProcessor before handlers run. RT recording failures are logged through the worker error path and recover through the RT job's own lease/retry budget, eventually requiring review if exhausted. They cannot revert a committed payment to Pending or block the independent invoice payout. After fixing the cause, retry the RT job, never the customer charge. Recipient jobs are independent: one failed payout does not resubmit successful siblings. Each payout gets a generated request key when inserted; vendor IDs are references, not request keys, and may have older lengths.

RT catch-up

After historical ledger backfill, run:

dotnet run --project Cli -- rt catchup 1 --env Development

Preview the same selection without creating outbox items:

dotnet run --project Cli -- rt catchup list 1 --env Development
dotnet run --project Cli -- rt catchup all --dry-run --env Development

list shows only invoice payments eligible for outstanding RT payouts, without recipient details. It excludes payments whose calculated chain has no payouts. --dry-run shows the full payout calculation instead. Both require the limit and reuse queueing eligibility checks. Already-handled and ineligible payments are omitted; an empty selection prints No RT catch-up work. The limit counts invoices needing RT payouts, not skipped ones or individual recipients. Both read-only modes use the same RT calculator as Worker and the original payment date. Dry-run shows every recipient payout in the RT email's concise style: amount and recipient, then tier, year and reward factors. Worker calculation logs use the same format. Basis-dollar arithmetic is omitted; the invoice's total RT payout remains visible. Actual queueing displays the same calculation and each recipient's PayVendor work ID. Preview creates no outbox items and sends no payments or emails. Referral or pricing changes between preview and queueing can change the result; queued payout amounts are immutable. Dry-runs explicitly show zero-recipient results; list omits them.

The positional limit is required: a positive integer or explicit all (for example, rt catchup all --env Development). Missing or invalid values fail before bootstrap; there is no default. The number limits invoices newly queued for RT payouts, not the number of referral recipients. Selection is oldest first by payment ledger occurred_at, then ID. Already-requested and ineligible payments do not use the limit. The command prints invoice, payment ledger entry, and work IDs. Run it again after observing the previous request; it does not wait for payout settlement. Use --env Production only after backfill and deployment of the updated handlers. Do not run old and new RT handler versions simultaneously.

The command calculates RT bonuses and creates only normal PayVendor outbox items with kind ReferralBonus. It does not fabricate PaymentSettled, queue invoice-vendor payouts, or call a payment provider. The normal dispatcher/Worker submits these independent recipient jobs and records their ledger entries. Calculation uses the original payment ledger timestamp for the age factor, with the current calculator policy, pricing and referral relationships; no historical chain snapshots exist.

Both producers lock the original payment ledger row before checking existing ReferralBonus payout work/ledger history and inserting recipient jobs. This serializes concurrent catch-up/live producers without a new marker event or table. All existing work states, including Failed and NeedsReview, are preserved; catch-up never resets a job or replaces its keys. Retain payout work records. Recipient inserts commit together, so failed recording rolls back the entire group. Partial historical sets require manual review, not automatic repair. Completed live settlement results also exclude previously handled zero-recipient runs. A zero-recipient catch-up creates no marker and may be calculated again. Backfilled invoices must agree with the ledger's amount, provider reference, vendor and customer. Ambiguous mismatches stop the run for review; earlier requests, if any, remain safely queued. Previously processed legacy RT settlements and existing referral ledger history are skipped rather than paid again. Historical referral payouts must retain their originating invoice link in the backfill for that check to work. No schema migration or extra plan store is needed for catch-up.

Local RT providers

All environments use RisingTideCalculator, which loads referral data from the same database and delegates to the pure PayoutCalculator. Existing files under the retired ~/.tendrme/rising-tide directory are unused. Remittances use ~/.tendrme/dev/payments/FileModT.csv, initially Pending. Change those rows to Completed to confirm transfers, or Failed/Canceled to simulate failure.

An eligible settled card collection triggers RT automatically when Worker processes its settlement event. The basis is the actual charged amount minus the invoice subtotal and configured Stripe percentage/flat costs, using the shared IPaymentPricing service. Stripe's percentage applies to the full charged amount, including the surcharge. Zero or negative net revenue produces no RT award. For example, a $3,000 invoice charged at $3,110.13 with Stripe costs of 2.9% + $0.30 has a $19.64 basis; Tier 1 at 20% in Year 1 pays $3.93. Existing queued payout amounts are not recalculated by this change. ACH collections and outgoing payouts produce no RT awards, preventing recursive bonuses. The shared bootstrap always registers RisingTideCalculator in every environment; there is no environment-specific calculator selection setting.

PayoutCalculator.Calculate(RisingTideInput) accepts only POCO payment facts, processor cost cents, an explicit transaction timestamp, and an ordered referral chain with recipient overrides. It returns payout values without I/O, clock reads, logging, generated IDs, or input mutation. Repeating identical inputs produces identical values; scheduling idempotency remains the responsibility of the existing RisingTidePayouts/outbox machinery, not the pure calculation. Shared payment pricing computes processor costs before calling the calculator. Empty or truncated referral chains produce only the awards supported by the supplied nodes.

The standard program has three tiers at 20%, 10%, and 5% of net revenue, with each link's referred-vendor age measured in 365.25-day years at payment time. Displayed years 1-3 have factors 100%, 50%, and 25%; standard eligibility expires at the start of Year 4. LiveForever remains available with its existing behavior pending separate review: Year 4 pays at a 12.5% factor, then later years remain at 12.5% with an Ongoing label. Custom recipient rates are retained. The pure service rejects cyclic/misordered chains, negative monetary inputs, out-of-range rates, and payments predating a referred vendor instead of generating invalid awards. Database-free unit tests cover these rules; separate integration tests cover data loading, shared pricing, and the normal scheduling path.

RT simplification implemented 2026-10-29: three referral tiers and three full 365.25-day years of standard eligibility. Already queued or recorded awards are not rewritten; new calculations, including catch-up previews, use these limits. LiveForever policy changes remain pending discussion.

Migrations 020-023 remain unreleased and are edited in place. Existing dev databases need manual synchronization for the renamed ledger, outbox payment status/request columns, and the invoice-only uniqueness predicate before restarting Worker.

Status readers validate payment ID, amount, currency, and ModT direction. ModT completed confirms movement; sent/approved remain pending per the payment-order contract. Stripe succeeded with the expected amount received records collection, per the PaymentIntent contract. This is captured collection, not a claim that Stripe funds have reached the bank or become available for payout. Post-settlement refunds/returns require new money-movement entries and separate handling; terminal rows are not continuously polled.

The file facades initially return Pending. Their CSV stores live at <directory>/FileStripe.csv and <directory>/FileModT.csv (both charges and payouts). Each row contains the operation ID, requested references/amount, stable payment ID, named status, and processing timestamp. Calls serialize through an in-process gate and OS-level exclusive handles on the sidecar .lock and CSV itself for separate Worker processes. Lock acquisition waits up to two minutes total (or earlier caller cancellation), then throws a timeout. The CSV handle stays locked through reading, updating, and flushing. Writes use that same handle rather than replacing the locked file, so this local simulator is not crash-atomic during a write. Concurrent callers and later processes reuse the same row; mismatched inputs require review. FileStripe has exact charged-amount scenarios (after fees): $0.01 stays Pending, $0.02 becomes Completed, and $0.03 becomes Failed on the first status poll. Submission still returns Pending, so completion/failure exercises the real status poller and settlement work. Other amounts stay Pending; FileModT does not use these scenarios. FileStripeGateway.SetPaymentStatusAsync(paymentXid, status, cancellationToken) also allows explicit status changes using the same CSV locks, without modifying application database state. Scenario polling never overrides a terminal status set explicitly. E2E hosts use isolated per-run payment CSVs. Only the dedicated card-payment-outcome fixture uses zero card fees, so its 1/2/3-cent invoice scenarios reach FileStripe unchanged. Other E2E tests and production retain normal pricing. To simulate settlement, edit only the row's status column: use Completed for a successful charge or payout, or Failed / Canceled for either. On macOS these OS locks are advisory: an editor that ignores locks or replaces the file can still race a write. Avoid saving manual edits while submitting new payments; pausing Worker is optional protection. Leave all other columns unchanged. The status poller rereads the provider CSV and looks up the requested operation on each sweep. Invalid/duplicate rows fail rather than being overwritten. An already-terminal invoice payment is no longer polled, so receipt edits cannot reverse a previously recorded settlement. Keep this directory across worker restarts: deleting receipts destroys simulated provider idempotency history. Never put actual bank/card credentials in these commands.

The file provider rejects non-loopback PG hosts and explicit non-local environments. Do not point a loopback tunnel at Production. Local invoice and ledger tables really are mutated, so use Development/test data only. API/CLI use the same queue regardless of environment; the file-provider guard does not select a different business flow. Separate bootstrap definitions select browser setup services: Development uses StripeLink: Stripe and PlaidLink: Plaid with the existing sandbox credentials; Test/Tests use Stub for both. GATEWAY_STUB no longer selects these services. File gateways return synthetic account references and safe masks without storing card numbers, bank numbers, or setup tokens in receipt files.

V1 still needs reviewed production service configuration, a final backfill refresh, and production migration/deployment verification. Consolidated 020 passed a local rehearsal against a real production snapshot on October 28. See the remaining v1 checklist. Bank balance is an accounting rule, not an outstanding feature. Incoming bank discovery/outbox recording is implemented. Keep the existing DIY worker and PG queue. Automatic ambiguous-submission recovery, webhooks, durable queued notifications, and automated failed-operation repair remain deferred; uncertain results stop for manual review. Payment-received emails run after the provider accepts the charge and submission is committed, not after settlement. The ledger balance here is cash movement, not an assertion of profit or vendor amount owed.

PaymentReceivedEmailHandler subscribes to ChargeInvoice, alongside the charge handler. After a Pending or Completed provider submission and its ledger entry commit, it uses the existing customer and vendor payment-received templates via SendAndForget. The customer receipt shows the charged amount (including processing fees); the vendor notice shows the invoice subtotal. Invoice customer snapshots take precedence over current contact data. Each delivery is best-effort and independent: a missing template or send failure does not block the other recipient or retry monetary work. Both templates describe submission/processing, not finalized collection or payout. No settlement polling is required, and later status events send no additional receipts. Completed-work redelivery skips these callbacks. Host failure after commit may still lose these non-durable notifications; first-class email outbox items remain future work. Worker bootstrap requires BASE_URL and validates it once for invoice links; the Development Bill configuration already supplies it when passed to Worker. Use a path-based site root (http://127.0.0.1:3030/ locally). The local server supports direct routes; BASE_URL must not contain a query or fragment.

Failure and recovery

If publishing fails, the dispatch lease expires and a dispatcher retries the same ID. If publication succeeds but marking dispatched fails, the same ID can be sent again. PG publication deduplicates a still-queued ID; consumers also tolerate duplicates that arrive after acknowledgement. No shared transaction spans sending and marking dispatched. A lost acknowledgement after the business commit leaves a recoverable delivery, not an unrecorded payment.

Pending work becomes Processing while leased. Successful execution and recording become Completed. Explicit retryable failures return to Pending after a configured delay; permanent failures or exhausted explicit retries become Failed. Uncertain/unclassified errors and unknown handler types become NeedsReview, not an automatic repeat of a potentially successful payment. Retryable outcomes set the outbox retry time and release the same delivery's lease. The queue has no separate retry timestamp. Terminal outcomes are acknowledged only after their outbox state is persisted.

Missing payout bank setup reports WorkFailureKind.PrerequisiteMissing, leaves work in Pending, logs a warning, and releases the queue lease without consuming a failure attempt. The outbox's available_at defers another attempt for one hour. No handler sleeps; the worker continues with other deliveries and retries the blocked payout on a later eligible polling pass.

Expired processing leases can be reclaimed with the same operation ID, bounded by max_attempts. An expired final attempt becomes NeedsReview. Stale lease tokens cannot complete work or record ledger effects. Cancellation on host shutdown leaves the lease for recovery. Error fields contain safe classification codes, never raw provider response bodies or tokens.

The local worker runs one delivery at a time; run multiple processes to exercise concurrent claiming. Handlers receive a cancellation token bounded by the lease; there is no heartbeat renewal in this first implementation. For long debugging pauses, increase Worker__LeaseSeconds (up to one day) and avoid a second worker. An expired lease cannot commit a late result. Handlers must honor cancellation; ignoring it does not extend ownership.

Run locally

Start the existing local PG service, and explicitly apply Development migrations using the normal migrator. The worker never starts/stops databases or migrates them automatically.

Always start the local database through scripts/ensure-pglite.sh, including agent-assisted startup. Keep required recovery behavior in that script rather than using direct runner commands or manual PID cleanup. It serializes startup attempts, matches port listeners to the cluster's open-file owners, and runs a short SELECT 1 health check before declaring the database ready. It requires psql (PATH, Homebrew libpq, or PGLITE_PSQL). Stale PgLite postmaster.pid cleanup requires no port listener, no SQL response, and no open cluster files. A bound but unhealthy server, mismatched listener, or failed process inspection blocks cleanup; the script never kills processes or removes database contents.

./scripts/ensure-pglite.sh
ENVIRONMENT=Development dotnet run --project Migrations -- --task up
ENVIRONMENT=Development dotnet run --project Worker -- --database-config Bill/appsettings.Development.json --enqueue-probe
ENVIRONMENT=Development dotnet run --project Worker -- --database-config Bill/appsettings.Development.json --dispatch-once
ENVIRONMENT=Development dotnet run --project Worker -- --database-config Bill/appsettings.Development.json --once
ENVIRONMENT=Development dotnet run --project Worker -- --database-config Bill/appsettings.Development.json --list

Omit the final command switch to run dispatch and consumption continuously. VS Code has a Worker: Development launch profile with stores at $HOME/.tendrme/dev/payments/FileStripe.csv and $HOME/.tendrme/dev/payments/FileModT.csv. Run it independently or alongside the API. The host validates storage and prints the selected providers before accepting work. --list shows the latest 50 records without exposing payloads/results. Information-level logs show dispatched operation IDs, execution attempts, committed work completion, and payment-status sweep results. Work completion means the submission was recorded, not that money settled. Logs omit payment payloads, account details, and raw provider responses. Empty dispatch/consumer polls stay quiet; the less-frequent status sweep reports its counts. Use --dispatch-once --work-item-id <id> to publish one operation, then --once --work-item-id <id> to consume it without claiming other pending work. --once never bypasses dispatch by scanning the outbox directly. --reconcile-once checks all unresolved payments once and queues terminal observations, without updating financial state or submitting payments again. Dispatch and consume the observation events to apply them; --work-item-id <id> restricts the sweep to one original payment operation. It reports checked/failed counts and exits nonzero if any check failed. Continuous mode runs this third loop alongside dispatch/consume.

The explicit database config file supplies the existing PG_CONNECTION contract. Environment variables override file settings. Shared appsettings.json sets service definitions, the file receipt directory, polling, lease, retry delay, and the attempt budget for API, CLI, and worker. Override the receipt directory with Payments__FileDirectory; override timing and attempts through Worker__PollIntervalSeconds, Worker__LeaseSeconds, Worker__RetryDelaySeconds, and Worker__MaxAttempts. Payment-status polling uses five-second and five-minute profiles, independent of the job queue polling interval. Startup rejects invalid/missing configuration and a missing work schema. No production connection or gateway credentials are bundled.

App and CLI

Run the API against the same migrated database as the worker. Submit payment normally; the API returns HTTP 202 with { "status": "Pending" }. Queue state and work-item identifiers are internal implementation details, not public payment fields. The UI keeps its progress indicator running and submission disabled while polling the invoice for up to 36 seconds. Animated progress notices appear immediately and every 6 seconds through 30 seconds, each lasting 5 seconds. Confirmed success/failure ends the wait immediately; otherwise the progress bar stops and the page transitions to its normal Pending view with: "Your payment is submitted and being processed. Check your email for confirmation." The invoice's paymentSubmitted flag indicates a recorded provider reference, not merely acceptance of the customer's request.

The existing payout command now schedules rather than pays inline:

ENVIRONMENT=Development dotnet run --project Cli -- invoice pay INVOICE_ID

invoice list requires IInvoiceStore and IVendorStore: it needs only the normal PG config, not payment credentials, card pricing, or workers. The invoice schema must include consolidated migration 020's cancellation and financial-status columns even for read-only operational commands. invoice pay requires PaymentQueue and does not bypass the queue. It queues an initial payout or, for a failed/canceled payout, prompts for a retry. Pending or completed payouts are left unchanged. invoice list STATUS keeps its required status argument and shows the payout status/reference separately: a paid invoice with a failed or pending payout still appears under Paid.

Manual invoice payout retry

tendr vendor rail VENDOR_ID --env Development
tendr invoice pay INVOICE_ID --env Development

vendor rail updates the latest verified payout account after confirmation. It suggests the next lower rail; 0/auto restores RTP -> same-day ACH -> ACH discovery. Changing a rail does not modify already-prepared payment requests or submit money.

invoice pay includes retry handling instead of a separate retry/reset command. It shows the failed ledger/provider reference, invoice amount, and preferred rail, and requires confirmation. It uses PaymentQueue/IInvoiceWork to schedule ordinary PayVendor work; it does not call the provider or reset the customer's completed charge. Scheduling locks the invoice and requires a matching current Failed/Canceled VendorPayout ledger entry, with no unresolved work or Pending/Completed invoice payout. A Completed payout subsequently returned is not eligible for this command.

The failed ledger ID is the stable retry authorization key: repeated/concurrent requests for that same attempt reuse its one replacement. Each new attempt gets new work, ledger, and provider idempotency IDs. RetryOfLedgerId links the work to the previous ledger entry, and the replacement ledger memo preserves that link. The original attempt and history remain unchanged. Only the invoice payout projection becomes Pending and its current provider reference is cleared until submission records the replacement reference. The worker reads the recipient's current account/rail when it prepares the new request. Repeated failure requires another explicit authorization for that replacement's failed ledger entry.

Unreleased migration 020's outbox uniqueness key now includes the retry predecessor, while still permitting only one original payout per invoice. Existing local copies already at 020 require that index definition refreshed; do not rerun the production seed.

The CLI payout command needs normal PG config, Worker:MaxAttempts, and the four pricing settings listed below because it resolves the shared invoice scheduler. Supply those in CLI config or environment, using the same values as the API/worker. It does not configure or validate gateways; Worker owns payment execution. add payout requires the vendor store, transaction, and keyed ACH gateway only. It needs no Stripe/Plaid definitions, pricing, or work/ledger tables. There is no environment flag to restore synchronous charging/payouts.

Exercise payments

Use an existing local unpaid invoice. Rates are read from the supplied config's CARD_PERCENT_RATE, CARD_FLAT_RATE, STRIPE_COST_PERCENT_RATE, and STRIPE_COST_FLAT_RATE; missing/malformed rates fail startup. The payment references can be synthetic because the file facade does not contact a provider.

ENVIRONMENT=Development dotnet run --project Worker -- \
  --database-config Bill/appsettings.Development.json \
  --charge --invoice INVOICE_ID --method card --account test-account --payment-method test-method

ENVIRONMENT=Development dotnet run --project Worker -- \
  --database-config Bill/appsettings.Development.json

The second command polls continuously. Edit the charge row's status to Completed while no new payments are being submitted, then wait for the next status sweep or run --reconcile-once with the same database config. Continuous dispatch/processing applies the observation and automatically schedules the invoice payout plus any eligible RT payouts. Use --method ach for a ModT charge. Each payout also starts Pending; set its CSV status to Completed, then poll and consume its observation to settle the existing Pending ledger entry. Queued payouts awaiting bank setup have no entry yet. For one-shot debugging, run --dispatch-once --work-item-id ID, followed by --once --work-item-id ID, in place of continuous polling. Every payment consumer must use the same provider definitions and receipt directory.

OpenTelemetry

API platform registration, AddTendrWorker(), and CLI bootstrap all register logging/tracing through Bootstrap's AddTendrTelemetry() extension. Business code uses standard ILogger<T> and ActivitySource, not SigNoz APIs. The CLI initializes its registered tracer during service validation because it has no Generic Host.

Telemetry is disabled in appsettings.json. When the SigNoz OTLP collector is available, enable export and restart the relevant processes. For Worker:

Telemetry__Enabled=true ENVIRONMENT=Development dotnet run --project Worker -- \
  --database-config Bill/appsettings.Development.json

For VS Code API debugging, set Telemetry:Enabled to true in appsettings.json and rebuild/restart, or supply Telemetry__Enabled=true to the API process. The same environment override enables CLI exports. Each host uses the same config.

The checked-in endpoint is http://localhost:4318 (OTLP/HTTP, not the SigNoz UI). Override with Telemetry__Endpoint for another collector origin. Logs and traces use /v1/logs and /v1/traces; service.name is tendr.me, with tendr.component (api, worker, or cli) and deployment.environment.name for filtering. Telemetry__Headers accepts comma-separated name=value ingestion headers if the collector requires authentication. Do not use SigNoz's JWT signing secret.

Settings are read/validated once during bootstrap and retained as an immutable snapshot. When enabled, all export settings must be supplied; the checked-in config explicitly defines queue/batch sizes, sampling, delay, and timeout. There is no startup health check or synchronous network export on the work path. Background exporters use bounded memory queues and a one-second request timeout by default in config. Overflow drops telemetry rather than delaying work; an unavailable collector is not a processing failure. Shutdown may spend time flushing queued batches. Telemetry is best-effort, not the financial audit record.

Spans cover API function requests, CLI workflows, payment enqueueing, dispatch, processing, individual handler execution/recording, and payment-status checks. API middleware wraps authentication/exception translation and records function name, invocation ID, duration, and HTTP outcome. It uses the active trace or Functions' W3C trace parent, never request URLs, headers, bodies, or baggage. Logs within spans carry trace/span IDs; enqueue/replay logs link API/CLI requests to operation IDs.

Application Tendr.* log categories are exported at Information and above. Only approved operational fields (IDs, statuses, counts, error types, timings) keep their values. Other fields, including email, names, paths, tokens, summaries, and provider response messages, are redacted. Raw exceptions and arbitrary scopes are excluded. Always use static message templates with structured fields, not interpolated strings containing customer data. Existing console/Application Insights logging is unchanged; this export policy applies to OTLP only. Framework/SDK categories are not exported. There is no automatic HTTP-client/SQL instrumentation, browser instrumentation, or metrics export yet. Process-startup Console.WriteLine output remains local rather than being captured as application logs.

Outbox creation captures the immediate producer's parent_id, W3C trace_parent, optional trace_state, and correlation_id. The first outbox ID is the saga's internal correlation ID; it is not exposed in the public payment response. The initial request and enqueue span exist before that ID is allocated, so their trace links to the saved operation without delaying request logging. Idempotent replays keep the original saved context.

parent_id references work.outbox.id and is captured automatically from the currently executing work context by both outbox creation methods. Root jobs have no parent. It is indexed and immutable, including across retries and duplicate publications; it is independent of tracing/sampling. For example: ChargeInvoice -> PaymentSettled -> RtPayout -> PayVendor, with the normal vendor PayVendor branching directly from PaymentSettled. The foreign key prevents deleting a parent while its children still reference it. Old rows are not assigned guessed parents. Daily return discovery restores the original payout work as its parent when available; returns of historical backfilled payouts without work history start a new root and retain their original ledger reference in the payload.

To investigate from any known work item, retrieve the whole correlated flow:

with flow as (
    select coalesce(correlation_id, id) as root_id
    from work.outbox where id = :work_item_id
)
select id, parent_id, type, status, attempts, last_error, created_at, completed_at
from work.outbox, flow
where correlation_id = flow.root_id or id = flow.root_id
order by created_at, id;

Use parent_id to reconstruct the branches rather than assuming timestamp order is a causal chain.

Dispatch, processing, and payment polling restore that durable parent. Every retry gets a fresh span, not a reused span ID. Dispatch and processing are siblings under the saved producer span; the queue still carries only an outbox ID. Follow-up events automatically inherit saga identity and the active span through the shared outbox store, including settlement and RT payouts. Direct reconciliation also restores context. A separately requested invoice payout starts its own saga.

Spans carry work.item.id and work.correlation.id; structured logs carry WorkItemId and CorrelationId. Ambient saga identity works even with telemetry disabled or sampling off. Missing/malformed trace context never rejects work; old items fall back to their own operation ID for correlation. No baggage, credentials, or customer data is propagated. These fields are immutable metadata, not part of the request fingerprint or provider idempotency key. Migration 020 defines the columns; an already-migrated local database needs its schema synchronized before running this version. Browser tracing is still separate future work; no client outbox ID or custom header is required to correlate the first server request.

WorkItemId identifies work.outbox.id, not an invoice or provider payment. The operator CLI filter is --work-item-id; public payment responses omit this ID. Existing settlement JSON and request fingerprints retain their OperationId field name for replay compatibility. The file-provider CSV's legacy operation_id header represents the provider request's InternalId.

TelemetryTests checks disabled/invalid configuration, immutable settings, structured OTLP logs and correlated traces from API/CLI/Worker, API request outcomes, trace parenting, privacy filtering, and collector outages. WorkTracingTests covers durable context, replay, retries, follow-up events, disabled telemetry, and scope isolation. RisingTideWorkTests checks charge-to-settlement-to-RT propagation and polling. WorkerEngineTests also processes real test outbox/queue work with telemetry enabled and its collector unavailable.

Payout recipient emails

PayVendorHandler starts a non-critical IEmail.SendAndForget notification after an accepted provider submission has been recorded and its ledger/outbox transaction has committed. The payment reference is the committed ledger entry's ID, not the work-item or provider ID. Worker runs best-effort AfterCommitAsync callbacks outside the recording transaction; callback failures do not retry monetary work. Invoice payouts and Rising Tide rewards select separate Markdown templates shipped with Core. There is no generic fallback: an unmapped payout kind or missing template logs a warning and skips the email without failing the payout. Recipient copy announces the payout and amount without exposing provider status values. Later Pending-to-Completed observations do not send a second recipient email. Rejected submissions and missing bank prerequisites do not send success emails.

RT snapshots recipient-visible scalar EmailFields into each payout intent: tier, tier factor, one-based year (or "Ongoing" for the lifetime override), age multiplier, and tier amount. Each recipient/tier has its own payout; recipients are not grouped or aggregated for email. The template uses {{tier}}, {{tier_factor}}, {{year}}, {{age_factor}}, {{total_basis_factor}} (tier factor times age factor), and the payout's {{amount}}. There are no email loops, payout arrays, or preformatted Markdown. The database already rejects referral cycles. The eligible basis and originating customer/vendor identities are not included. No separate RT file store or extra lookup/recalculation is involved in email rendering.

These emails are independent of WorkTraces and use the configured email provider (Development writes to ~/.tendrme/dev/email). Rendering or send startup errors are logged without undoing payment recording. Recording/commit failures send no email; completed-work redelivery does not resend it. Delivery is not durable: shutdown after commit but before notification, or during delivery, can lose messages. First-class email outbox work is deferred.

Temporary diagnostic email alerts

ErrorAlerts adds a shared ILoggerProvider to the API and Worker. Every Error or Critical log (including framework categories in those processes) schedules a best-effort email through the host's existing IEmail. It is independent of OpenTelemetry export being enabled. This captures logs, not arbitrary console output, process crashes, or configuration failures before logging is initialized.

appsettings.json explicitly enables ErrorAlerts:Enabled and sets ErrorAlerts:Recipients to abe.gillespie+tm.error@gmail.com. Subjects start with [Tendr][Production][ERROR] or [Tendr][Production][CRITICAL] in Production, followed by host component and logger category. Development instead shows [Development] and writes through its file email provider to ~/.tendrme/dev/email. Use the dedicated recipient for mailbox filtering; no Gmail settings are changed by the application. Recipients and the enable flag are validated and captured once at bootstrap. Override with ErrorAlerts__Recipients or ErrorAlerts__Enabled.

Bodies include the static message template, permitted operational fields, trace/span IDs, event ID, and full Exception.ToString() output: exception types, messages, inner/aggregate exceptions, and stack traces with available source paths and line numbers. Exception text is not redacted and may contain sensitive data; restrict access to the alert mailbox and Development email files accordingly. Unrelated interpolated log payloads/customer data and arbitrary scopes remain excluded. This does not change OpenTelemetry's redaction policy. Log call sites must use static templates rather than interpolating secrets into the template itself. Email-triggered logs are suppressed in the error sink's async execution context to prevent recursive alerts. Sending failures go to stderr rather than back through logging. There is no rate limiting, durable outbox, or delivery guarantee in this stop-gap implementation.

Worker uses Services:<environment>:Email to select FileEmail or SmtpEmail for Error/Critical alerts and business notifications. Development selects FileEmail with Services:Development:EmailDirectory set to ~/.tendrme/dev/email. Worker does not inherit Functions' local.settings.json. SMTP requires its corresponding SMTP settings; no Production worker email provider is selected yet. SMTP_FROM_PRIV, when provided, selects the private sender. Configuration is read and validated once at bootstrap. The unused Azure Communication email implementation and API registration are excluded from compilation; their source is retained for reference only, and the Azure Communication Email SDK dependency is removed.

Buffered work logs

Routine work traces no longer send email. WorkTraces (a boolean, default true) registers a standard ILoggerProvider through the shared worker bootstrap, for both console Worker and Azure Functions. It uses NLog.Database with Npgsql and an AsyncTargetWrapper, not a replacement for application ILogger<T> calls. The old WorkAlerts switch is removed; use WorkTraces=false to disable this sink without affecting console logging, OpenTelemetry, or Error/Critical emails.

SigNoz cutover reminder

Migration 020 creates work.log. The sink uses the existing PG_CONNECTION / ConnectionStrings:PG_CONNECTION configuration and credentials, with its own connection pool (one connection, 3-second connection/command timeouts, no ambient transaction enlistment). It does not provision tables or roles at runtime. Existing development/rehearsal databases already at 020 need the added table before enabling this sink; production will receive it when running 020.

Dispatch, processing, handler start/completion, retries, missing handlers, delivery resolution, and worker/poller warnings/errors are recorded. Every scheduled poll also emits PollStarted followed by PollCompleted, PollFailed, or PollCanceled from the shared WorkerEngine.PollAsync, including empty successful scans. These events carry PollHandler, a unique per-run CorrelationId, PollStatus, and elapsed milliseconds on the outcome. Completion is emitted only after the checkpoint is saved. Host-requested cancellation is informational, not an error email; a token already canceled before entry produces no run. The outer scheduling loops do not log the same failure again. All application/framework Error and Critical logs, including PaymentRequiresAttention and logged unhandled exceptions, are also captured. Queue-empty checks and unrelated lower-severity application logs remain excluded. Rows contain the original UTC timestamp, environment, host component, logger category, level/event, work/correlation/trace IDs, and JSONB operational fields. Context is captured before queuing. Error/Critical events include MessageTemplate; exceptions include full Exception.ToString() in fields.Exception, including inner/aggregate exceptions and stack traces. This text is not redacted and may contain sensitive data. Raw message rendering and arbitrary scopes/payload fields are excluded. Errors can appear in both the table and an error email, but there is no second lifecycle email. Exceptions must reach ILogger to be captured; this sink does not intercept process crashes or exceptions never logged by the host.

Database writes happen on NLog's background writer. The in-memory queue is bounded at 2,048 events with batches of up to 100. Overflow drops events instead of waiting; WorkLogProvider.DroppedCount counts queue overflow and malformed diagnostic state, not database write failures. Database errors are swallowed by NLog, not routed back through ILogger/email or retried as payment work. Shutdown attempts a short flush; crashes, saturation, or database outages can lose logs. This is best-effort diagnostics, not financial audit history or a durable delivery queue. There is no automatic table retention yet; rows can be deleted independently of ledger/outbox data.

For example:

select occurred_at, component, level, event_name, work_item_id, fields
from work.log
order by occurred_at desc, id desc
limit 100;

Azure later

Keep the outbox, dispatcher, and handlers. Replace the PG IJobQueue transport with an Azure adapter that carries the same outbox IDs, and adapt worker hosting for Azure delivery/trigger semantics. Publication, execution completion, and transport acknowledgement already have separate states and failure boundaries. An Azure adapter still needs visibility/acknowledgement mapping, integration tests, and recovery/poison-message handling. Do not mark an operation completed on publication.

Payout Returns

PaymentReturnPoller declares 0 0 0 * * * (daily at midnight UTC). Both hosts discover it normally; Azure routes it through PollDaily. Both payment status sweeps only check pending submissions, not completed payout history.

The daily poller queries ModT's Events API with resource=payment_order, event_name=returned, and event_time_start / event_time_end. The UTC date window starts at the previous successful scan minus one day, through today's date. With no checkpoint it starts two days ago; historical returns before that initial window must be covered by backfill. After successful initialization, downtime is covered by the persisted checkpoint, not a fixed two-day window. The API key needs read access to developer resources/events.

All pages are read, including continuations after empty pages. Event time is the filter, not payout creation time, so a six-week-old payout returning today is found. Events for other originating accounts and customer-collection returns are excluded. For each payout, the gateway fetches the current order and posted bank-credit evidence; it does not rely on potentially incomplete event snapshots. The original ledger row is matched by ModT payment order ID, including backfilled payouts without an outbox item. No metadata external-ID fallback is used.

The orchestrator advances work.polling.last_successful_run only after successful discovery and durable publication. Incomplete bank evidence, ambiguous/missing ledger matches, permission errors, and pagination failures fail the scan and preserve its checkpoint for retry. Successfully published earlier pages may replay safely through normal outbox deduplication. Ordinary completed payouts are never scanned again.

A returned payout must supply a completed current_return linked to the original order and a matching posted bank credit (account, environment, currency and full amount). The daily poller (or a pending-payment status check that sees an early return) publishes PaymentReturned through normal outbox/work processing, deduplicated by the return bank transaction ID. The handler retains the original Completed debit and writes a Completed Reversal credit with reversal_of_id pointing to it. The credit uses the bank date, not discovery time. A return seen before the initial settlement poll first completes the original Pending debit; it does not erase the outflow or rerun settlement fan-out.

The invoice payout becomes Failed only if it still points to that returned order; a newer replacement payout is not overwritten. Original payment work remains Completed because its transfer occurred; the return has its own work item. No replacement payment, RT recalculation, or notification is automatically triggered. Ordinary deposit discovery excludes receiving-return transactions to prevent counting the reversal credit twice. Returned customer collections, partial returns, and returns without matching posted bank evidence still require separate review.

For local FileModT simulation, append a separate return row to FileModT.csv: operation_id is the return ID; account_xid is tendr; payment_method_xid is the original payout's provider payment ID; amount_cents is the returned amount; statement_descriptor is the return code; status is Completed; payment_xid is the unique bank return-credit ID; processed_at is the return date/time. Leave idempotency_key empty. Keep the original payout row Completed with its original date. File simulation uses processed_at for both discovery time and bank date; real ModT discovery uses event time separately from the bank credit's as_of_date.

Verification

OutboxStoreTests runs real migrations in disposable PgLite databases. It covers transaction visibility/rollback, immutable inputs, enqueue deduplication, concurrent claims, skipped locks, lease recovery/fencing, delayed bounded retries, uncertain failures, shutdown, atomic ledger recording, completed redelivery, and recovery after a simulated provider-success/recording-failure boundary. OutboxQueueTests covers the separate publication/consumption boundary, dispatch lease fencing, send-before-mark recovery, delivery lease recovery, delayed retry, and acknowledgement failure after commit without repeated payment or ledger effects. Migration tests preserve preexisting work through the 021/022 up/down/up transition. PaymentReconciliationTests covers submitted ledger entries and audited transitions, pending/network retry without new jobs, failed payouts, concurrent observations, stale Pending observations, one-time follow-up scheduling, and transactional rollback. PaymentStatusGatewayTests covers ModT/Stripe HTTP reads and file receipt reads. PaymentWorkTests verifies transactional scheduling, concurrent submissions, charge/payout projections and balances, and crash recovery using file receipts. FilePaymentGatewayTests covers cross-instance replay, concurrent execution, input conflicts, and invalid payments. PaymentProvidersTests checks the checked-in definitions, configuration overrides, and rejection of invalid/missing providers. BootstrapTests verifies shared service lifetimes, startup validation, and the absence of worker consumers in producer hosts. LocalWorkerTests starts separate worker and CLI processes, using configuration alone, to exercise both probes and the complete charge/payout path.