Deterministic engines and bounded storage¶
The default engine is Python. It uses cached codec metadata, a linear registry
index and an expiry heap rather than repeatedly scanning terminal receipts.
NavigationConfig(..., engine="python") preserves the earlier positional
configuration parameters; the new engine option is keyword only. Memory and Redis
constructors also accept engine= explicitly.
engine="rust" requires the separately built, exactly matching
aiogram-nav-native==0.3.0 companion. There is no native dependency or installation
extra in the base package. The companion is not published until the measured
performance gate passes. Presence of Rust source or a successful wheel build is
not evidence of a speedup. Local development can build it with Maturin from
native/; see its README and benchmark findings.
Engine acceptance checks the exact version, ABI, required functions and registry
methods before a backend can initialize its namespace. Missing binaries, loader
errors and incompatible capabilities raise ConfigurationError. No mutation or
failed effect falls back to another engine. Both engines keep Telegram requests,
application callbacks, locks, sessions, ownership and Redis I/O in Python.
The Rust core implements strict JSON parsing, registry validation/indexing, history/state transitions, token comparisons and hashing of the original Python canonical bytes. It does not reserialize floats to redefine existing fingerprints. All valid Unicode scalars, including non-BMP characters, remain supported. Lone or paired surrogate code units supplied as Python strings are rejected early, rather than normalized or silently accepted in memory and rejected later by Redis. Valid JSON surrogate-pair escapes decode to their corresponding non-BMP scalar.
Fresh format-three Redis namespaces¶
from aiogram_nav import NavigationConfig
from aiogram_nav.storage.redis import RedisNavigationRepository
repository = RedisNavigationRepository(
redis=application_redis,
prefix="application-navigation",
storage_format=3,
config=NavigationConfig(engine="python"),
)
await repository.initialize()
Fresh format-three initialization atomically establishes the profile, history,
format manifest and live-index witness. Explicit storage_format=3 refuses an
existing format-one/two namespace: use the drained migration below. Omitting the
option preserves compatibility: fresh namespaces use format one and existing
namespaces are detected without an implicit upgrade. Memory storage also accepts
storage_format=3; it remains single-process and non-durable.
Format three separates the current projection from canonical operation rows:
- The hot record contains current navigation, active controls, external binding, revision and pending/current-owner references. It contains no operation history.
- Each canonical operation has its own STRING key, scoped by conversation and a SHA256 of its operation ID. The row verifies the complete conversation and ID.
- Terminal rows expire physically through Redis
PXAT, anchored toTIMEin the terminal commit. Rewriting a terminal integration payload preserves its original deadline. Prepared/unknown rows have no automatic TTL. - Metadata contains row-key/status/control-index information, not another snapshot
or delivery receipt. Bounded expiry cleanup removes at most a page per update;
sweep()reclaims inactive conversations and remaining index metadata.
Reads fetch only requested operations, current/pending owners, indexed controls and a bounded expiry page. They do not download all retained receipts. A missing row before its promised expiry, a missing protected row or inconsistent ownership metadata fails closed. An expired row may be absent only after the server's expiry cutoff. Active controls are independent of terminal replay retention, so a still current keyboard does not become invalid just because its render receipt expired.
The replay promise is finite: recovery_seconds defaults to 3600 seconds from the
terminal commit. In Nav this setting controls terminal receipt retention only;
prepared and unknown operations have no automatic expiry. Unexpired receipts and unresolved evidence are never evicted to
admit another operation. Capacity exhaustion raises Overloaded. Expired terminal
receipts can be reclaimed, allowing paced traffic to exceed lifetime capacity.
ActionStatus("not_found") means not retained, not never executed. After the
advertised window, producers must revalidate their source and session; arbitrary
string IDs cannot provide permanent global deduplication. Unknown external stages
remain protected and are never automatically replayed.
Atomic adoption and optional component hosts¶
track() adopts a screen already rendered by the application. Default memory
storage, and Redis after format-three activation, can prepare and complete that
adoption in one guarded mutation through adopt_render(...). Authority is checked
again after application guard/locale callbacks. Events and generation-specific
cleanup run after commit. Invalid inputs or failed pure reductions do not expose a
partially accepted activity. A crash before calling track() still belongs to the
application's rendering workflow; use prepared rendering for recoverable sends.
AtomicAdoptionRepository is an optional capability. Legacy/custom repositories
choose the earlier sequence before mutation. An atomic failure is never retried
through that sequence. Custom subclasses conservatively retain their public
method hooks unless they explicitly opt into the atomic capability.
Optional hosts such as aiogram-idle use the public LedgerRead, LedgerDelta and
NavigationLedgerReducer. Its repository executes pure bounded operations;
record/replace_record() support a validated owned normalization view.
integration_payload() returns a detached first accepted activity result from the
canonical operation, never the current screen. A host commits the delta, activity,
deadline and authority binding in one fenced transaction. It must not keep another
historical navigation snapshot or acknowledgement ledger. Navigation stays usable
without Idle, and Idle does not become a base dependency.
Migration and verification¶
Drain all old workers before explicit activation. The profile/history rotation fences old readers and writers; changing a flag without actually draining in-flight external effects is insufficient.
# Read-only assessment; supplies no approval or activation implicitly.
uv run python tools/migrate_storage.py --target 3 --expected 1
# Explicit upgrade after all old workers have drained.
uv run python tools/migrate_storage.py --target 3 --expected 1 --apply \
--operation-id navigation-format3 --old-workers-drained
# Optional bounded eager conversion after activation (otherwise conversion is lazy).
uv run python tools/migrate_storage.py --target 3 --migrate-all \
--batch-size 100 --max-pages 100
Use --expected 2 for a format-two namespace. These commands use the existing
Redis environment variable and namespace prefix options. Record conversion is one
atomic CAS after pure off-thread conversion; interruption can be resumed. Original
absolute replay deadlines are retained, and unknown/protected operations survive.
An old-format envelope is distinguishable from partial/corrupt format-three state;
corruption does not trigger legacy fallback. The page cursor contains only offsets.
With concurrent admissions, repeat assessment/passes as with any SSCAN traversal.
benchmarks/run.py runs against the installed distribution, without inserting
src into the import path. It records fixed-payload track workloads aged 50, 500 and
5000 receipts, CPU medians, individual latency median/p95, versions and hardware.
--redis-url adds whole protected Dispatcher.feed_update measurements, exact
consumed RESP bytes, library round trips, separate Lua command counts and cold
bootstrap diagnostics. It removes only its newly generated benchmark prefixes.
Every driver result sets qualification=false; the independent release gate owns
qualification. --quick is only a smoke test. Normal CPU results use at least ten
independent batches, and individual-update p95 uses nearest-rank sampling.
Run full comparisons serially on the same machine. The native publication gate
requires at least 20% representative CPU benefit and no greater than 5% realistic
p95 regression against optimized Python. Published baseline comparisons, optimized
Python and Rust results must remain separate; unsupported baseline capacity is
reported as unavailable rather than replaced with invented timing values. Pyperf
kernel probes are provided in benchmarks/pyperf_codec.py, and Criterion lives in
native/benches.
Capacity and authoritative host generations¶
Size max_receipts for the promised window: peak accepted operations per second
multiplied by recovery_seconds, plus headroom for protected unfinished or unknown
effects. The defaults (256 receipts and a 3,600-second replay window) are finite.
Atomic adoption retains one operation instead of four stage snapshots; it does not
remove admission limits. For example, 2 operations/second and a 60-second window
need 120 terminal slots; 256 slots leave 136 for protected work and bursts. Never
shorten the business recovery promise to disguise insufficient capacity. The
5,000-operation slope benchmark explicitly configures 10,000 slots; separate tests
exercise default-capacity rejection and more than ten capacity windows of paced
traffic.
Canonical component hosts call the pure
NavigationLedgerReducer.bind_session_generation(session_id, expires_at) before
preparing an action or selecting its scoped view index. This hook requires an
already verified host session; it performs no authorization or network access.
The host fences the final atomic commit against that observation. Fresh Nav
state uses the host's absolute deadline from the start. A live legacy generation
with an earlier Nav deadline keeps that deadline. An expired Nav generation
cannot reopen under the same session id, even if its host still has time left;
replace the host generation explicitly. Pending effects retain their original
session and deadline. Ordinary standalone Nav generation ownership is unchanged.
Format 3 stores view ownership under a session-scoped hash. A separate integer index-count witness detects missing reverse-index fields before reads, garbage collection or writes. Selected pointers are validated against their canonical owner row. Coordinated external modification of both the index and its witness is outside the storage corruption model; Redis keys remain library-owned.
0.3.0 native qualification result¶
The independent repeated serial installed-artifact gate did not qualify the
native companion for publication. Rust consumed 21.7% more process CPU than the
optimized Python engine on the preselected memory-track/50 workload; its
individual-update p95 was 22.4% higher. Whole-handler Redis latency remained near
parity, which does not satisfy the required 20% CPU benefit. The pure Python
package can release independently. Native source and platform CI remain, but no
native PyPI package, release tag or [native] extra is published for this result.
Selecting Rust requires an explicitly built, exact-version local companion;
ordinary installations continue to use Python.
Development native decoding boundary¶
The unpublished development companion materializes strict JSON directly into
Python objects in one native pass. It avoids an intermediate Rust JSON tree and
a second conversion traversal for every Redis projection. The existing parser
remains authoritative for validation-only/internal Rust operations. Duplicate
decoded keys, insertion order, finite floats, raw depth/byte bounds and arbitrary
integers retain their existing semantics. Python's runtime integer digit limit
is still applied; -0 is integer zero and -0.0 retains its floating sign bit.
No stored format or fingerprint encoding changes.
Each memory reducer owns a deferred native codec. Read-only Redis projections
that do no JSON freezing/fingerprinting allocate no native derived-data cache.
The first pure-data operation creates that backend's own fork; no cache, mutable
record or authorization result is shared across backends. Capability checks still
happen eagerly before mutation, and unused/reused codecs keep the same bounded
cache and task-ownership rules. Cold dispatch is cycle-free: dropping an unused
projection releases its codec immediately, even with cyclic GC disabled. After
first use, calls bind directly to native methods without retaining the wrapper.
Revoking a codec context clears its codec and task references even in long-lived
context copies. Actual lock ownership is fenced independently until release.
Generic mapping canonicalization unpacks exactly
two iterable values from each items() pair, matching Python's reference path.
The typed companion API includes Canonicalizer and InvalidJson, matching the
existing extension capabilities. These changes still require the unchanged full
performance qualification and platform gates. All Rust companions remain
unpublished; a faster decoder alone is not a release qualification.
Fresh-window benchmark observations¶
benchmarks/run.py --latency-windows N repeats Redis observations using N
fresh, independently owned namespace/session fixtures; its default is 1,
preserving historical sampling. --samples remains the update count per
fixture, and receipt capacity is still age + samples + 10. For example,
--ages 50 --samples 100 --latency-windows 10 records ten separate track
windows progressing from age 50 to 150, never one conversation progressing
to 1050. No-op observations stay at age 50 before the track phase. All captured
latencies are pooled in original window order, without trimming or retries.
A failed window marks the run unavailable and causes a nonzero CLI exit;
completed windows/scenarios and captured partial observations remain diagnostic
evidence rather than qualifying a successful subset.
Rows expose latency_windows, exact [start, stop)
latency_window_boundaries, unique latency_window_fixtures, and
total_calls. calls and operations_per_cpu_sample remain the operation
count for each entry in process_cpu_seconds. Aggregate Redis command/byte
counters cover counter_operation_count == total_calls, not just calls.
Per-update traffic arrays contain every observation. Cold bootstrap counters
are separately itemized per fixture and summed outside measured update counts.
Memory CPU repeats already construct fresh repositories: --cpu-samples 30
with --samples 100 still performs 30 separate 100-operation fixtures, with
matching boundaries, not one larger state-growth trace. Gate thresholds and
workload names are unchanged; the requested sampling plan is explicit metadata.