Key Features & Capabilities

Stream Composition Handling

The streamer's core capability is processing CESR streams that are fundamentally concatenations of primitives. A primitive in CESR is the serialization of a value associated with a cryptographic operation, including digests, salts, seeds, private keys, public keys, and signatures. The streamer can handle:

1. Simple concatenated primitives: Sequential primitives of potentially different types
2. Grouped primitives: Primitives organized using count codes or group framing codes
3. Hierarchically composed groups: Nested structures where groups contain other groups
4. Mixed primitive types: Streams containing combinations of different cryptographic material types

Nested Stream Processing

One of the streamer's most sophisticated capabilities is handling nested streams - streams that contain other streams as components. This hierarchical structure is common in KERI's security architecture where:

• Key events may be encapsulated within multiple layers of signatures
• Encrypted segments contain inner CESR streams that must be parsed after decryption
• Receipts and attachments form nested structures

The streamer maintains parsing context across nesting levels, enabling it to correctly extract primitives from deeply nested structures without losing track of boundaries or mixing data from different nesting levels.

Tunneled Stream Support

The streamer provides specialized support for tunneled streams - CESR data encapsulated within other protocols or data structures. This capability is particularly important for:

• Protocol interoperability: CESR streams embedded in HTTP, WebSocket, or other transport protocols
• Format mixing: CESR primitives within JSON, CBOR, or MessagePack serializations
• Envelope structures: CESR content wrapped in application-specific containers

The sniffer component (part of Parside) works in conjunction with the streamer to detect format boundaries and enable proper parsing of tunneled content. When the sniffer identifies non-CESR formats (JSON, CBOR, MGPK), it uses regex to locate version strings and extract length information, allowing the streamer to skip over non-CESR sections and resume parsing at the next CESR boundary.

Encrypted Stream Processing

The streamer handles encrypted stream segments as part of its parsing flow. This capability is essential for KERI's security model where:

• Confidential data may be encrypted within event streams
• Delegation chains may include encrypted commitments
• Private ACDCs use encryption to protect attribute disclosure

The streamer coordinates with decryption operations, maintaining stream integrity while processing encrypted segments. After decryption, the streamer can recursively parse the revealed CESR content, handling nested encrypted streams where multiple layers of encryption may be present.

Self-Framing Exploitation

The streamer leverages CESR's self-framing property, where each primitive includes sufficient information (via framing codes) to determine its own boundaries. This enables the streamer to:

• Extract primitives atomically: Read exactly the right number of characters/bytes without examining entire content
• Handle variable-length primitives: Process primitives of different sizes without external schema
• Support cold start parsing: Resynchronize after errors by finding the next framing code
• Enable pipelining: Process streams in parallel across multiple cores

Stream Parsing Architecture

The streamer works in conjunction with other parsing components in the KERI ecosystem:

Parside Integration: Parside is responsible for parsing group codes and orchestrating higher-level stream structure, while the streamer handles the extraction of individual primitives and nested stream segments. The strip parameter determines which portions of the stream are handled by which parsing code.

Cesride Coordination: Cesride provides the foundational primitive types (Diger, Verfer, Signer, Siger, Cigar, Salter) that the streamer instantiates during parsing. The streamer reads framing codes to determine which primitive type to create and how many bytes/characters to extract.

Version Management: The streamer respects version codes that determine which CESR code tables to use for parsing. When encountering a version count code, the streamer switches to the specified version and loads the appropriate parsing tables.

Architecture Highlights

Convenience Class Design

The streamer is explicitly designed as a convenience class rather than a core protocol element. This design choice reflects several architectural principles:

1. Separation of concerns: The streamer provides parsing infrastructure without being part of the encoded data itself
2. Implementation flexibility: Different KERI implementations can provide streamers with varying performance characteristics
3. Optional usage: Applications can implement custom parsing if the standard streamer doesn't meet their needs
4. Composability preservation: The streamer operates on CESR's composable primitives without affecting their encoding

Not a Cryptographic Primitive

The explicit statement that the streamer is "not a cryptographic primitive itself" is architecturally significant. Unlike Diger, Verfer, or Signer, which represent actual cryptographic material, the streamer is:

• Infrastructure code: Provides parsing services rather than cryptographic operations
• Not serialized: Never appears in CESR streams as encoded data
• Implementation-specific: Different implementations may have different streamer designs
• Not versioned: Not subject to CESR primitive versioning schemes

This distinction clarifies the streamer's role in the KERI architecture: it's a tool for working with primitives, not a primitive itself.

Stream Processing Model

The streamer implements a generator/iterator pattern for efficient stream processing. Rather than loading entire streams into memory, the streamer can:

• Yield primitives one at a time as they're parsed
• Support streaming processing of arbitrarily large event logs
• Enable pipeline architectures where parsing and processing occur in parallel
• Minimize memory footprint for resource-constrained environments

This design aligns with CESR's emphasis on pipelining for high-bandwidth applications, where network pipelines operate at 40 GHz while individual processor cores max out at 5 GHz.

Cold Start Recovery

The streamer implements cold start stream parsing capabilities, addressing the fundamental challenge of resynchronizing after errors or reboots. When a stream processor encounters malformed data or loses synchronization:

1. The streamer can skip forward to the next well-defined boundary
2. Special count codes serve as synchronization points
3. In-transit buffer contents are preserved rather than flushed
4. Parsing resumes at the next valid framing code

This capability is essential for robust KERI implementations that must handle network interruptions, corrupted data, and mixed serialization formats.

Usage Patterns

Basic Stream Parsing

While specific API details vary by implementation, the general pattern for using a streamer involves:

1. Initialization: Create a streamer instance with a CESR stream (bytes or text)
2. Iteration: Iterate over the streamer to extract primitives
3. Type handling: Process each primitive according to its type (digest, signature, key, etc.)
4. Nesting: Recursively create new streamers for nested stream segments

Nested Stream Handling

For nested streams, the typical pattern is:

1. Parse outer stream until encountering a nested stream marker
2. Extract the nested stream segment
3. Create a new streamer instance for the nested content
4. Parse the nested stream recursively
5. Resume parsing the outer stream after nested content is processed

Encrypted Stream Processing

When encountering encrypted segments:

1. Parse up to the encryption boundary
2. Extract encrypted content using framing codes
3. Decrypt the content
4. Create a new streamer for the decrypted CESR stream
5. Parse the decrypted content
6. Continue parsing the outer stream

Related Implementations

The streamer concept is implemented across multiple KERI codebases:

Python (keripy): The primary Python implementation includes a Streamer class in the keri.core.signing module (moved from keri.core.streaming in version 2.0.0-dev1 to resolve circular import issues). The Python implementation provides the most mature and feature-complete streamer.

Rust (keride, keri-ox): Rust implementations provide high-performance streaming parsing with memory safety guarantees. The Rust streamer implementations are optimized for throughput and can be used via FFI from other languages.

JavaScript/TypeScript: JavaScript implementations provide streamer functionality for browser and Node.js environments, enabling KERI applications in web contexts.

All implementations must maintain compatibility with the CESR specification's composability requirements, ensuring that streams parsed by one implementation can be generated by another without loss of information.

Performance Considerations

The streamer's design directly impacts KERI system performance:

Pipelining Support: By enabling extraction of discrete data chunks from streams, the streamer supports demultiplexing across multiple processor cores, essential for matching network pipeline speeds (40 GHz) with processor core speeds (5 GHz).

Memory Efficiency: Generator-based iteration allows processing of large event logs without loading entire streams into memory, critical for resource-constrained IoT devices and mobile applications.

Parsing Overhead: The streamer's efficiency in extracting primitives directly affects overall system throughput. Optimized implementations use lookup tables for framing codes and minimize string operations.

Cold Start Recovery: Efficient resynchronization after errors reduces downtime and data loss, improving system reliability in production environments.

Integration with KERI Event Processing

The streamer plays a critical role in KERI's event processing pipeline:

KEL Parsing: Key Event Logs are CESR streams containing inception, rotation, and interaction events. The streamer extracts these events for validation and key state computation.

Receipt Processing: Witness receipts are CESR-encoded signatures attached to events. The streamer extracts receipt signatures for verification.

ACDC Handling: Authentic Chained Data Containers may include CESR-encoded attachments and signatures. The streamer processes these cryptographic commitments.

OOBI Resolution: Out-Of-Band Introductions may include CESR-encoded endpoint information and cryptographic material that the streamer must parse.

In all these contexts, the streamer provides the foundational parsing capability that enables KERI's cryptographic verification and event processing logic.

Performance Optimization

For high-throughput applications:

• Minimize string operations and allocations
• Use byte-level operations where possible
• Implement zero-copy parsing for large primitives
• Consider SIMD operations for Base64 decoding
• Profile framing code lookup performance

Thread Safety Considerations

Streamer instances should generally not be shared across threads. For parallel processing:

• Create separate streamer instances per thread
• Use the streamer to demultiplex the stream into chunks
• Process chunks in parallel with thread-local streamers
• Recombine results after parallel processing

Testing Requirements

Comprehensive streamer testing should cover:

• Simple concatenated primitives of various types
• Nested streams with multiple levels of nesting
• Encrypted segments requiring decryption
• Tunneled streams with mixed serialization formats
• Malformed data and cold start recovery
• Large streams for memory efficiency validation
• Performance benchmarks for pipelining scenarios