Key Features & Capabilities

Multi-Format Detection

The sniffer performs comprehensive format detection across five distinct serialization types. Each format has unique identifying characteristics:

CESR Formats: Both CESR binary and CESR text formats contain group codes or object codes at the stream's beginning. These codes provide unique three-bit combinations that enable immediate recognition without ambiguity.

Structured Formats: JSON, CBOR, and MessagePack each have distinct structural markers that the sniffer can recognize through pattern matching.

Detection Process Architecture

The sniffer implements different processing strategies depending on the detected format:

Non-CESR Format Handling

When the sniffer detects JSON, CBOR, or MGPK formats, it initiates a specialized extraction process:

1. Regex Matching - The parser applies regular expressions to locate the version string embedded within the serialized data. The version string is a critical field in KERI field maps that appears first in any top-level structure.

2. Length Extraction - From the version string, the parser extracts the character count (for text formats) or byte count (for binary formats) that defines the total length of the serialized structure. This length information is embedded within the version string itself.

3. Boundary Detection - Using the extracted length, the parser determines exactly where the current data structure ends. This enables precise segmentation of the stream without parsing the entire structure.

4. Resume Sniffing - After processing the bounded structure, the sniffer resumes operation on the remaining stream. This allows for heterogeneous streams containing multiple serialization formats in sequence.

This approach is necessary because JSON, CBOR, and MessagePack are not self-framing in the same way CESR primitives are. They require explicit length information to determine boundaries.

CESR Format Handling

When the sniff result indicates CESR format at the top level, the processing differs significantly:

• The parser searches for the CESR version count code which identifies the specific CESR encoding tables in use. This version code determines which code table the parser should load for interpreting subsequent primitives.

• It also looks for other count codes that define the number and types of primitives in groups. Count codes are special framing codes that enable self-framing at the group level.

• These codes enable the parser to properly decode the self-framing primitives in the stream without requiring external length information.

The fundamental difference is that CESR formats are self-framing - each primitive contains sufficient information to determine its own boundaries. This property, described as "relatively new" in the design discussions, enables modular parser construction where parsing responsibilities can be dispatched based on the strip parameter.

Sniffability Criteria

A stream achieves sniffability when it begins with specific recognizable markers:

• Group codes: Structural markers indicating grouped primitives
• Field maps: Object-level markers indicating structured data
• Object codes: Format-specific identifiers

Each supported datablock type has either an Object code or Group code (available in both binary and text variants) that provides a unique three-bit signature for immediate recognition. This design enables:

• Immediate type detection without buffering or look-ahead
• Parser synchronization from the first bytes
• Multiple format support through discriminating code prefixes
• Error-free demultiplexing of interleaved stream types

Streams lacking these markers are classified as non-sniffable and require alternative parsing strategies.

Version Management Integration

The sniffer works in conjunction with CESR's version management system:

• The Parside parser starts with a default version for CESR
• When a version count code is encountered, the parser changes the active version
• Version count codes must appear at the top level of the stream
• The version count code determines which CESR table to load when parsing the stream

This version management is critical because count code interpretation depends on the active version context. The parser must maintain a default version to handle scenarios where:

• The stream does not begin with an explicit version code
• Processing resumes after a cold start
• Count code interpretation must proceed without explicit version context

Architecture and Integration

Position in Parsing Pipeline

The sniffer operates as the first stage in CESR stream processing:

1. Stream Reception - Raw byte/character stream arrives
2. Sniffer Detection - Format identification occurs
3. Parser Dispatch - Appropriate parsing handler is invoked
4. Primitive Extraction - Individual elements are parsed
5. Validation - Cryptographic verification proceeds

This pipeline architecture ensures that format detection happens before any parsing attempts, preventing errors from misinterpreted data.

Relationship to Parside

The sniffer is explicitly described as "part of Parside" in the source documentation. Parside is the component responsible for parsing group codes and orchestrating higher-level stream structure, while Cesride handles individual CESR primitives.

This division of responsibilities reflects CESR's modular design:

• Cesride: Concerned with parsing individual CESR primitives
• Parside: Handles parsing of group codes and stream orchestration
• Sniffer: Performs initial format detection for Parside

The strip parameter determines which portion of the CESR stream will be parsed by which code component, enabling dynamic dispatch based on detected format.

Hierarchical Parsing Support

When the sniffer identifies CESR format, it enables hierarchical parsing:

• Parser looks for CESR version count codes or other count codes at the top level
• Upon encountering a group count code, the parser may descend into nested group parsing
• This recursive descent can nest arbitrarily deep, following the hierarchical structure
• The sniffer's initial detection enables this entire hierarchical processing chain

Technical Significance

Composability Enablement

The sniffer is fundamental to CESR's composability property - the ability to convert any set of self-framing concatenated primitives between text and binary domains without loss. By enabling automatic format detection, the sniffer allows:

• Heterogeneous streams - Multiple serialization formats coexisting in a single stream
• Domain transitions - Seamless conversion between text and binary representations
• Primitive separability - Individual elements remain independently verifiable

Cold Start Problem Solution

The sniffer directly addresses the cold start problem identified in KERI documentation. Without sniffability:

• Parsers would require external configuration specifying format
• Stream processing would need buffering to detect format
• Format transitions would require explicit delimiters
• Error recovery would be complex and unreliable

With the sniffer's detection capabilities:

• Parsing begins immediately upon stream reception
• No external configuration is required
• Format transitions are automatically detected
• Error recovery can restart from any sniffable boundary

Interoperability Support

By supporting multiple serialization formats (JSON, CBOR, MessagePack) alongside native CESR formats, the sniffer enables KERI to integrate with existing data interchange standards while maintaining CESR's composability properties. This is critical for:

• Legacy system integration - Existing systems using JSON/CBOR can participate
• Protocol bridging - KERI can interface with non-CESR protocols
• Gradual adoption - Systems can transition incrementally to CESR

Implementation Considerations

Performance Characteristics

The sniffer's design prioritizes efficiency:

• Minimal buffering - Detection occurs from initial bytes
• Single-pass processing - No backtracking required
• Deterministic recognition - Unique three-bit combinations prevent ambiguity
• Constant-time detection - Format identification is O(1) operation

These characteristics enable high-throughput stream processing essential for KERI's scalability goals.

Error Handling

While not extensively documented in the source materials, the sniffer's role in error handling is implicit:

• Malformed streams - Unrecognizable initial markers indicate invalid streams
• Format violations - Detected format must match actual content
• Version mismatches - Version codes must correspond to supported tables

Proper error handling at the sniffer stage prevents downstream parsing failures.

Design Evolution

The source documentation indicates the sniffer concept emerged from design discussions in February 2023 (Cesride Slack thread). The design reflects CESR's "relatively new" self-framing property, suggesting the sniffer architecture evolved alongside CESR's maturation.

Key design principles established in these discussions:

1. Separation of concerns - Sniffer detects, Parside orchestrates, Cesride parses
2. Count code focus - Parside processes count codes at stream beginning
3. Dynamic table loading - Version codes enable runtime table selection
4. Generator patterns - Consideration for iterator/generator return values

These principles reflect a mature understanding of stream processing requirements in cryptographic protocols.

Relationship to KERI Protocol

The sniffer enables several critical KERI capabilities:

Event Stream Processing

KERI key event logs (KELs) and transaction event logs (TELs) are transmitted as CESR streams. The sniffer enables:

• Event multiplexing - Multiple event types in single stream
• Witness receipt integration - Receipts interleaved with events
• Attachment handling - Signatures and seals properly segmented

OOBI Protocol Support

Out-of-Band Introductions (OOBIs) may contain various serialization formats. The sniffer enables OOBI endpoints to accept:

• JSON OOBIs - Human-readable introductions
• CESR OOBIs - Compact binary introductions
• Mixed format responses - Flexible endpoint behavior

ACDC Credential Exchange

Authentic Chained Data Containers (ACDCs) may be serialized in multiple formats. The sniffer enables:

• Format negotiation - Issuers and verifiers can use preferred formats
• Compact disclosure - Binary formats for bandwidth efficiency
• Full disclosure - JSON formats for human readability

The sniffer's multi-format support is thus essential for KERI's practical deployment across diverse use cases and integration scenarios.

Integration with Parside

The sniffer operates as the first stage in Parside's parsing pipeline:

1. Sniffer performs format detection
2. Parside dispatches to appropriate parser based on detection result
3. For CESR: Parside handles group codes, Cesride handles primitives
4. For non-CESR: Parside extracts bounded section, resumes sniffing

The strip parameter mechanism determines which code handles which stream portion based on sniffer results.

Hierarchical Parsing Support

When CESR format is detected:

• Enable hierarchical descent into nested group structures
• Support recursive parsing of group count codes
• Maintain version context throughout nested parsing
• Allow arbitrary nesting depth following CESR's hierarchical design

Testing Considerations

Implementations should be tested with:

• Pure CESR streams: Binary and text variants
• Pure non-CESR streams: JSON, CBOR, MessagePack
• Mixed format streams: Multiple format transitions
• Malformed streams: Invalid markers, missing version codes
• Edge cases: Empty streams, single-primitive streams, deeply nested structures