Loading vLEI.wiki
Fetching knowledge base...
Fetching knowledge base...
This comprehensive explanation has been generated from 52 GitHub source documents. All source documents are searchable here.
Last updated: October 7, 2025
This content is meant to be consumed by AI agents via MCP. Click here to get the MCP configuration.
Note: In rare cases it may contain LLM hallucinations.
For authoritative documentation, please consult the official GLEIF vLEI trainings and the ToIP Glossary.
The streamer is a utility class implemented across multiple KERI codebases (primarily Python in keripy and Rust in keride) that provides infrastructure for parsing CESR streams. As noted by Kent Bull during a KERI technical meeting (August 6, 2024), the streamer occupies a unique conceptual position: it is "akin to a cryptographic primitive yet is not a cryptographic primitive itself." This distinction emphasizes that the streamer is infrastructure for operating on primitives rather than being a primitive data type.
The streamer's primary purpose is to handle the complex parsing requirements of CESR's composable encoding scheme, which supports both textual and binary streaming applications of cryptographic material. Unlike simple parsers that process flat data structures, the streamer must handle hierarchical compositions where streams contain other streams, encrypted segments require decryption during parsing, and multiple primitive types are interleaved.
CESR defines a dual text-binary encoding protocol with a critical property called text-binary concatenation composability. This property ensures that any set of self-framing concatenated primitives expressed in either the Text domain or Binary domain may be converted as a group to the other domain and back again without loss. The streamer class provides the complementary parsing capability needed to decompose CESR streams back into their constituent primitives, enabling the round-trip processing necessary for KERI's event streaming architecture.
The CESR specification defines streams as "any set of concatenated Primitives, concatenated groups of Primitives, or hierarchically composed groups of Primitives." The streamer implements the parsing logic to handle all these cases, making it essential infrastructure for KERI implementations.
In keripy version 2.0.0-dev1, the Streamer class was moved from keri.core.streaming to keri.core.signing to resolve circular import issues. The streaming.py module was renamed to annotating.py. Backwards compatibility is maintained through import aliases, so from keri.core import Streamer continues to work.
The streamer is explicitly not a subclass of Bexter (a CESR primitive type), reinforcing that it is infrastructure code rather than a cryptographic primitive. This design decision prevents confusion about the streamer's role and ensures it's not accidentally serialized into CESR streams.
Implementations should use generator/iterator patterns rather than loading entire streams into memory. This is critical for:
High-performance implementations should use lookup tables for framing code interpretation rather than sequential conditionals. The CESR specification defines multiple code tables, and efficient table switching based on version codes is essential for parsing performance.
When implementing nested stream support, maintain separate parsing contexts for each nesting level. This includes:
The streamer should coordinate with the sniffer component for format detection in mixed streams. When the sniffer identifies non-CESR content (JSON, CBOR, MGPK), the streamer should:
Implement cold start recovery by:
The streamer's core capability is processing CESR streams that are fundamentally concatenations of primitives. A primitive in CESR is the serialization of a value associated with a cryptographic operation, including digests, salts, seeds, private keys, public keys, and signatures. The streamer can handle:
One of the streamer's most sophisticated capabilities is handling nested streams - streams that contain other streams as components. This hierarchical structure is common in KERI's security architecture where:
The streamer maintains parsing context across nesting levels, enabling it to correctly extract primitives from deeply nested structures without losing track of boundaries or mixing data from different nesting levels.
The streamer provides specialized support for tunneled streams - CESR data encapsulated within other protocols or data structures. This capability is particularly important for:
The sniffer component (part of Parside) works in conjunction with the streamer to detect format boundaries and enable proper parsing of tunneled content. When the sniffer identifies non-CESR formats (JSON, CBOR, MGPK), it uses regex to locate version strings and extract length information, allowing the streamer to skip over non-CESR sections and resume parsing at the next CESR boundary.
The streamer handles encrypted stream segments as part of its parsing flow. This capability is essential for KERI's security model where:
The streamer coordinates with decryption operations, maintaining stream integrity while processing encrypted segments. After decryption, the streamer can recursively parse the revealed CESR content, handling nested encrypted streams where multiple layers of encryption may be present.
The streamer leverages CESR's self-framing property, where each primitive includes sufficient information (via framing codes) to determine its own boundaries. This enables the streamer to:
The streamer works in conjunction with other parsing components in the KERI ecosystem:
Parside Integration: Parside is responsible for parsing group codes and orchestrating higher-level stream structure, while the streamer handles the extraction of individual primitives and nested stream segments. The strip parameter determines which portions of the stream are handled by which parsing code.
Cesride Coordination: Cesride provides the foundational primitive types (Diger, Verfer, Signer, Siger, Cigar, Salter) that the streamer instantiates during parsing. The streamer reads framing codes to determine which primitive type to create and how many bytes/characters to extract.
Version Management: The streamer respects version codes that determine which CESR code tables to use for parsing. When encountering a version count code, the streamer switches to the specified version and loads the appropriate parsing tables.
The streamer is explicitly designed as a convenience class rather than a core protocol element. This design choice reflects several architectural principles:
The explicit statement that the streamer is "not a cryptographic primitive itself" is architecturally significant. Unlike Diger, Verfer, or Signer, which represent actual cryptographic material, the streamer is:
This distinction clarifies the streamer's role in the KERI architecture: it's a tool for working with primitives, not a primitive itself.
The streamer implements a generator/iterator pattern for efficient stream processing. Rather than loading entire streams into memory, the streamer can:
This design aligns with CESR's emphasis on pipelining for high-bandwidth applications, where network pipelines operate at 40 GHz while individual processor cores max out at 5 GHz.
The streamer implements cold start stream parsing capabilities, addressing the fundamental challenge of resynchronizing after errors or reboots. When a stream processor encounters malformed data or loses synchronization:
This capability is essential for robust KERI implementations that must handle network interruptions, corrupted data, and mixed serialization formats.
While specific API details vary by implementation, the general pattern for using a streamer involves:
For nested streams, the typical pattern is:
When encountering encrypted segments:
The streamer concept is implemented across multiple KERI codebases:
Python (keripy): The primary Python implementation includes a Streamer class in the keri.core.signing module (moved from keri.core.streaming in version 2.0.0-dev1 to resolve circular import issues). The Python implementation provides the most mature and feature-complete streamer.
Rust (keride, keri-ox): Rust implementations provide high-performance streaming parsing with memory safety guarantees. The Rust streamer implementations are optimized for throughput and can be used via FFI from other languages.
JavaScript/TypeScript: JavaScript implementations provide streamer functionality for browser and Node.js environments, enabling KERI applications in web contexts.
All implementations must maintain compatibility with the CESR specification's composability requirements, ensuring that streams parsed by one implementation can be generated by another without loss of information.
The streamer's design directly impacts KERI system performance:
Pipelining Support: By enabling extraction of discrete data chunks from streams, the streamer supports demultiplexing across multiple processor cores, essential for matching network pipeline speeds (40 GHz) with processor core speeds (5 GHz).
Memory Efficiency: Generator-based iteration allows processing of large event logs without loading entire streams into memory, critical for resource-constrained IoT devices and mobile applications.
Parsing Overhead: The streamer's efficiency in extracting primitives directly affects overall system throughput. Optimized implementations use lookup tables for framing codes and minimize string operations.
Cold Start Recovery: Efficient resynchronization after errors reduces downtime and data loss, improving system reliability in production environments.
The streamer plays a critical role in KERI's event processing pipeline:
KEL Parsing: Key Event Logs are CESR streams containing inception, rotation, and interaction events. The streamer extracts these events for validation and key state computation.
Receipt Processing: Witness receipts are CESR-encoded signatures attached to events. The streamer extracts receipt signatures for verification.
ACDC Handling: Authentic Chained Data Containers may include CESR-encoded attachments and signatures. The streamer processes these cryptographic commitments.
OOBI Resolution: Out-Of-Band Introductions may include CESR-encoded endpoint information and cryptographic material that the streamer must parse.
In all these contexts, the streamer provides the foundational parsing capability that enables KERI's cryptographic verification and event processing logic.
For high-throughput applications:
Streamer instances should generally not be shared across threads. For parallel processing:
Comprehensive streamer testing should cover: