Key Participants

The cold-start process involves several components:

• Stream Parser (Parside): The primary parsing engine that processes CESR streams
• Sniffer Component: Detects stream format types (CESR binary, CESR text, JSON, CBOR, MGPK)
• Framing Codes: Count codes and group codes that mark boundaries
• Version Codes: Specify which CESR code tables to use for parsing
• Buffer Management: In-transit data storage that must be preserved during recovery

Process Flow

Initial Cold Start Sequence

When a parser performs a cold start from system initialization:

Step 1: Default Version Loading

The parser initializes with a default CESR version that specifies which code tables to load. This default ensures the parser can begin processing even if the stream doesn't start with an explicit version code.

Step 2: Stream Format Detection (Sniffing)

The sniffer component examines the first bytes/characters of the stream to detect format:

• CESR Binary: Begins with binary framing codes
• CESR Text: Begins with Base64 URL-safe character framing codes
• JSON: Begins with { or [ characters
• CBOR: Begins with CBOR major type indicators
• MessagePack: Begins with MGPK format markers

Each format has unique three-bit combinations in its object codes or group codes that enable unambiguous detection. This property makes CESR streams sniffable.

Step 3: Framing Code Extraction

Once format is detected, the parser extracts the initial framing code:

• For CESR streams: Reads the count code or group code that specifies how many primitives or bytes follow
• For JSON/CBOR/MGPK: Uses regex or format-specific parsing to locate the version string field and determine the serialized section length

Step 4: Boundary-Aligned Parsing

CESR's 24-bit alignment constraint ensures all primitives align on boundaries that are:

• Integer multiples of 4 Base64 characters (24 bits) in text domain
• Integer multiples of 3 bytes (24 bits) in binary domain

This alignment guarantees that framing codes always begin at predictable positions, enabling atomic extraction of elements.

Step 5: Hierarchical Descent

When the parser encounters a group code:

• It may descend into nested group parsing
• Each group has its own count code specifying member count
• Parsing can nest arbitrarily deep following the hierarchical structure
• The parser maintains a stack of group contexts for proper unwinding

Error Recovery and Re-synchronization

When the parser encounters malformed data or ambiguous framing:

Step 1: Error Detection

The parser identifies parsing failures through:

• Invalid framing code values
• Mismatched primitive lengths
• Failed cryptographic verification of signatures or digests
• Unexpected end-of-stream conditions

Step 2: Forward Scanning

Instead of flushing buffers, the parser skips forward to locate the next well-defined boundary:

• Scans for recognizable count codes or group codes
• Identifies format transition markers (CESR to JSON boundaries)
• Locates version codes that reset parsing context

Step 3: Boundary Validation

When a potential boundary is found, the parser validates it:

• Checks that the framing code is valid for the current version
• Verifies that subsequent data matches the declared length
• Confirms that primitives are properly self-framing

Step 4: Recovery Cold Start

Once a valid boundary is confirmed:

• The parser executes a cold start from that position
• Discards only the malformed segment between the error and the boundary
• Preserves all buffered data after the boundary
• Resumes normal parsing with the recovered context

State Changes During Cold Start

The parser transitions through several states:

1. Uninitialized: No parsing context, awaiting first data
2. Sniffing: Detecting stream format from initial bytes
3. Synchronized: Successfully parsing with valid framing context
4. Confused: Encountered ambiguous or malformed data
5. Scanning: Searching for next valid boundary
6. Recovering: Executing cold start from found boundary
7. Re-synchronized: Resumed normal parsing after recovery

Decision Points

Key decision points in the cold-start process:

Version Code Handling

When encountering a CESR version count code:

• Decision: Load new code tables or continue with current version?
• Criteria: Version codes must appear at top level (not nested in groups)
• Action: If valid, switch to specified version and reload tables

Format Transition Handling

When the sniffer detects a format change:

• Decision: Continue CESR parsing or switch to JSON/CBOR/MGPK parser?
• Criteria: Presence of format-specific markers at expected boundaries
• Action: Delegate to appropriate parser for the bounded section

Error Recovery Strategy

When parsing fails:

• Decision: Attempt re-synchronization or abort stream processing?
• Criteria: Availability of valid boundaries within buffer, criticality of data
• Action: If boundaries found, recover; otherwise, signal fatal error

Technical Requirements

Cryptographic Requirements

While cold-start parsing itself doesn't perform cryptographic operations, it must preserve cryptographic integrity:

Primitive Integrity

• Framing codes must correctly identify cryptographic primitives (keys, signatures, digests)
• Parsing errors must not corrupt cryptographic material
• Recovery must not split primitives across boundaries

Signature Verification Context

• Parser must maintain sufficient context for signature verification
• Indexed signatures require correct association with signing keys
• Receipt attachments must be properly associated with events

Digest Chain Integrity

• KEL parsing must preserve hash chain continuity
• SAID references must be correctly extracted
• Seal attachments must maintain cryptographic commitments

Timing Considerations

Latency Requirements

Cold-start parsing must minimize latency:

• Initial synchronization: Should complete within milliseconds for typical streams
• Error recovery: Should not introduce multi-second delays
• Format detection: Sniffing should require examining only first few bytes

Throughput Impact

The cold-start mechanism must not significantly degrade throughput:

• Overhead: Framing code processing adds minimal per-primitive cost
• Recovery cost: Scanning for boundaries is linear in malformed segment size
• Buffering: Must balance memory usage against recovery capability

Real-time Constraints

For witness and watcher systems:

• Must keep pace with event generation rates
• Cannot accumulate unbounded parsing backlogs
• Must handle bursty traffic patterns

Error Handling

Recoverable Errors

Errors that cold-start parsing can recover from:

• Malformed primitives: Invalid length or type codes
• Truncated data: Incomplete primitives at buffer boundaries
• Format confusion: Ambiguous serialization format markers
• Version mismatches: Unexpected CESR version codes

Fatal Errors

Errors requiring stream abort:

• No valid boundaries: Cannot locate synchronization points within buffer
• Persistent corruption: Repeated recovery failures
• Resource exhaustion: Buffer overflow during scanning
• Protocol violations: Fundamental CESR specification violations

Error Reporting

Parsers must provide diagnostic information:

• Error location: Byte/character offset where parsing failed
• Error type: Classification of parsing failure
• Recovery status: Whether re-synchronization succeeded
• Data loss: Extent of discarded malformed data

Usage Patterns

Typical Scenarios

Scenario 1: Witness Initialization

When a witness starts and connects to a controller's event stream:

1. Parser performs cold start with default CESR version
2. Sniffer detects CESR text format from Base64 characters
3. Parser extracts initial count code indicating event count
4. Processes inception event and subsequent establishment events
5. Maintains synchronized state for ongoing event stream

Scenario 2: Network Interruption Recovery

When a watcher reconnects after network failure:

1. Buffer contains partial event from before disconnection
2. New data arrives but parser is mid-primitive
3. Parser detects framing inconsistency
4. Scans forward to next event boundary (marked by count code)
5. Executes recovery cold start from that boundary
6. Resumes processing with minimal data loss

Scenario 3: Mixed Format Stream

When processing a KERI event stream with embedded JSON:

1. Parser processes CESR-encoded events normally
2. Encounters count code indicating JSON section
3. Sniffer confirms JSON format from { character
4. Parser extracts JSON length from count code
5. Delegates JSON parsing to format-specific handler
6. Resumes CESR parsing after JSON section boundary
7. No cold start needed due to explicit boundary markers

Scenario 4: Malicious Data Injection

When an attacker attempts to confuse parser with malformed data:

1. Parser receives crafted data with invalid framing codes
2. Detects parsing failure when primitive length doesn't match
3. Enters scanning mode to find next valid boundary
4. Locates legitimate count code from trusted source
5. Recovers and continues processing
6. Malformed segment is isolated and can be logged for analysis

Best Practices

Parser Implementation

• Maintain default version: Always have a fallback CESR version for cold starts
• Limit scan distance: Set maximum bytes to scan before declaring fatal error
• Preserve buffers: Never flush buffers unless absolutely necessary
• Log recovery events: Track all cold starts for debugging and security analysis

Stream Production

• Explicit boundaries: Always include count codes at major boundaries
• Version codes: Include version codes at stream start and after format changes
• Avoid ambiguity: Don't create patterns that could be mistaken for framing codes
• Test malformed data: Verify parser recovery with intentionally corrupted streams

System Design

• Buffer sizing: Allocate buffers large enough to span typical event groups
• Timeout handling: Implement timeouts for incomplete primitives
• Monitoring: Track cold-start frequency as a health metric
• Graceful degradation: Design systems to tolerate occasional parsing failures

Integration Considerations

KERI Protocol Integration

Cold-start parsing is essential for KERI components:

• KEL Processing: Witnesses and watchers must reliably parse event logs
• KERL Verification: Validators need robust parsing for receipt logs
• OOBI Resolution: Discovery mechanisms depend on parsing endpoint data
• TEL Management: Credential registries require parsing transaction logs

Multi-Agent Systems

In KERIA deployments:

• Multiple agents share parsing infrastructure
• Cold starts must not interfere with other agents' streams
• Recovery must be isolated to affected stream
• Shared code tables must be thread-safe

High-Availability Systems

For production deployments:

• Cold-start capability enables zero-downtime updates
• Parser can recover from transient failures without service interruption
• Monitoring cold-start frequency helps detect infrastructure issues
• Graceful degradation maintains partial service during problems

Cross-Platform Compatibility

Implementations across languages (Python, Rust, TypeScript):

• Must implement identical cold-start behavior
• Should produce same recovery decisions for same input
• Need consistent error reporting formats
• Must maintain interoperability despite implementation differences

Conclusion

Cold-start stream parsing is a critical capability that enables CESR's robustness and composability. By providing well-defined synchronization points through count codes and supporting recovery without buffer flushing, CESR parsers can maintain high reliability even in the face of network issues, malformed data, or system restarts. This capability is essential for the production deployment of KERI-based identity systems where continuous operation and data integrity are paramount.

Default Version Selection: Choose default CESR version based on:

• Most recent stable specification
• Compatibility with deployed systems
• Support for required primitive types
• Current recommendation: CESR 1.0

Version Code Handling: When encountering version codes:

• Verify version code appears at top level (not nested)
• Load appropriate code tables atomically
• Maintain backward compatibility with older versions
• Log version transitions for audit trails

Performance Optimization

Fast Path for Normal Operation: Optimize for the common case:

• Cache frequently used code table entries
• Use lookup tables instead of conditional logic
• Minimize allocations during normal parsing
• Reserve cold-start overhead for actual error cases

Lazy Boundary Validation: During scanning:

• Perform quick checks first (valid code range)
• Only do expensive validation (length verification) for promising candidates
• Use early exit conditions to avoid unnecessary work

Thread Safety

Shared Code Tables: In multi-threaded environments:

• Code tables should be immutable after loading
• Use copy-on-write for version transitions
• Protect parser state with appropriate synchronization
• Consider per-thread parser instances for high concurrency

Testing Requirements

Malformed Data Testing: Implementations must be tested with:

• Truncated primitives at various positions
• Invalid count codes with plausible values
• Format confusion (CESR text that looks like JSON)
• Repeated recovery scenarios
• Boundary cases (recovery at buffer edges)

Interoperability Testing: Verify that:

• Different implementations recover at same boundaries
• Error reporting is consistent across implementations
• Version transitions work identically
• Mixed format streams parse identically

Security Considerations

Denial of Service Prevention: Protect against:

• Malicious data designed to trigger excessive scanning
• Crafted streams that cause repeated cold starts
• Resource exhaustion through buffer overflow
• Timing attacks based on recovery behavior

Isolation: Ensure that:

• Recovery in one stream doesn't affect others
• Malformed data can't corrupt parser state
• Error conditions are properly contained
• Cryptographic material is never exposed during recovery