API reference / @evolu/common / Evolu/Protocol
Evolu/Protocol
Evolu Protocol
Evolu Protocol is a local-first, end-to-end encrypted binary synchronization protocol optimized for minimal size and maximum speed. It enables data sync between a client and a relay, clients in a peer-to-peer (P2P) setup, or relays with each other.
Evolu Protocol is designed for SQLite but can be extended to any database. It implements Range-Based Set Reconciliation. To learn how RBSR works, check Negentropy. Evolu Protocol is similar to Negentropy but uses different encoding and also provides data transfer and ownership.
Message Structure
| Field | Notes |
|---|---|
| Header | |
| - protocolVersion | |
| - OwnerId | Owner |
| Initiator | |
| - WriteKeyMode | |
| - WriteKey | If WriteKeyMode >= 1 |
| - WriteKey | If WriteKeyMode = 2 (new) |
| Non-initiator | |
| - ProtocolErrorCode | |
| Messages | |
| - NonNegativeInt | A number of messages. |
| - EncryptedCrdtMessage | |
| Ranges | |
| - NonNegativeInt | Number of ranges. |
| - Range |
WriteKey Validation
The initiator sends WriteKeyMode and optionally one or two WriteKeys. One key for write operations and two for key rotation (current and new). Note that it's ok to not send any key if initiator is going to be synced with readonly owner. The non-initiator validates them immediately after parsing the initiator header, before processing any messages or ranges.
WriteKey Rotation
When initiator's WriteKeyMode is Rotation, two WriteKeys are
present:
- Current WriteKey (for validation)
- New WriteKey (to be stored)
Synchronization
- Messages: Sends EncryptedCrdtMessages in either direction.
- Ranges: Determines messages to sync. Usage varies by transport—e.g., sent only on WebSocket connection open or with every fetch request.
Synchronization involves an initiator and a non-initiator. The initiator
is typically a client, and the non-initiator is typically a relay. Each
side processes the received message and responds with a new ProtocolMessage
if further sync is needed or possible, continuing until both sides are
synchronized.
Both Messages and Ranges are optional, allowing each side to send, sync, or only subscribe data as needed.
When the initiator sends data, the WriteKey is required in Messages as
a secure token proving the initiator can write changes. The non-initiator
responds without a WriteKey, since the initiator’s request already
signals it wants data. If the non-initiator detects an issue, it sends an
error code via the Error field in the header back to the initiator. In
relay-to-relay or P2P sync, both sides may require the WriteKey
depending on who is the initiator.
Protocol Errors
The protocol uses error codes in the header to signal issues:
- ProtocolWriteKeyError: The provided WriteKey is invalid or missing.
- ProtocolWriteError: A write operation failed (e.g., due to storage limits or billing).
- ProtocolSyncError: A generic or unexpected synchronization failure occurred.
- ProtocolUnsupportedVersionError: Protocol version mismatch.
- ProtocolInvalidDataError: The message is malformed or corrupted.
All protocol errors except ProtocolInvalidDataError include the ownerId
to allow clients to associate errors with the correct owner.
Message Size Limit
The protocol enforces a strict maximum size for all messages, defined by maxProtocolMessageSize. This ensures every ProtocolMessage is less than or equal to this limit, eliminating the need for applications to fragment and reconstruct messages during transmission.
Why Binary?
The protocol avoids JSON because:
- Encrypted data doesn’t compress well, unlike plain JSON.
- Message size must be controlled during creation.
- Sequential byte reading is faster than parsing and can avoid conversions.
It uses structure-aware encoding, significantly outperforming generic binary serialization formats with the following optimizations:
- NonNegativeInt: Up to 33% smaller than MessagePack.
- Base64Url Strings: Up to 25% size reduction.
- DateIso: Up to 75% smaller.
- Timestamp Encoding: Delta encoding for milliseconds and run-length encoding (RLE) for counters and NodeIds.
- Small Integers (0 to 19): Reduces size by 1 byte per integer.
To avoid reinventing serialization where it’s unnecessary—like for JSON and certain numbers—the Evolu Protocol relies on MessagePack.
Versioning
Evolu Protocol uses explicit versioning to ensure compatibility between
clients and relays (or peers). Each protocol message begins with a version
number and an ownerId in its header.
How version negotiation works:
-
The initiator (usually a client) sends a
ProtocolMessagethat includes its protocol version and theownerId. -
The non-initiator (usually a relay or peer) checks the version.
- If the versions match, synchronization proceeds as normal.
- If the versions do not match, the non-initiator responds with a message
containing its own protocol version and the same
ownerId.
-
The initiator can then detect the version mismatch for that specific owner and handle it appropriately (e.g., prompt for an update or halt sync).
Version negotiation is per-owner, allowing Evolu Protocol to evolve safely over time and provide clear feedback about version mismatches.
Interfaces
| Interface | Description |
|---|---|
| ApplyProtocolMessageAsClientOptions | - |
| ApplyProtocolMessageAsRelayOptions | - |
| CrdtMessage | A CRDT message that combines a unique Timestamp with a DbChange. |
| EncryptedCrdtMessage | An encrypted CrdtMessage. |
| FingerprintRange | - |
| ProtocolErrorBase | Base interface for all protocol errors. |
| ProtocolInvalidDataError | Error for invalid or corrupted protocol message data. |
| ProtocolMessageBuffer | Mutable builder for constructing ProtocolMessage respecting size limits. |
| ProtocolSyncError | Error indicating a synchronization failure during the protocol exchange. Used for unexpected or generic sync errors not covered by other error types. |
| ProtocolUnsupportedVersionError | Represents a version mismatch in the Evolu Protocol. Occurs when the initiator and non-initiator are using incompatible protocol versions. |
| ProtocolWriteError | Error when a write fails due to storage limits or billing requirements. Indicates the need to expand capacity or resolve payment issues. |
| ProtocolWriteKeyError | Error when a WriteKey is invalid, missing, or fails validation. |
| SkipRange | - |
| Storage | Evolu Protocol Storage |
| StorageDep | - |
| TimestampsBuffer | - |
| TimestampsRange | - |
| TimestampsRangeWithTimestampsBuffer | - |
Type Aliases
| Type Alias | Description |
|---|---|
| Base64Url256 | - |
| Base64Url256Variant | Union type for all variants of Base64Url strings with limited length. All these types use Base64Url alphabet and are < 256 characters. |
| BinaryId | Binary representation of Id. |
| BinaryOwnerId | Binary representation of OwnerId. |
| DbChange | - |
| EncryptedDbChange | Encrypted DbChange |
| Fingerprint | A cryptographic hash used for efficiently comparing collections of BinaryTimestamps. |
| InfiniteUpperBound | - |
| ProtocolError | - |
| ProtocolMessage | Evolu Protocol Message. |
| Range | - |
| RangeType | - |
| RangeUpperBound | Union type for Range's upperBound: either a BinaryTimestamp or InfiniteUpperBound. |
Variables
| Variable | Description |
|---|---|
| Base64Url256 | Base64Url string with maximum length of 256 characters. Encoding strings as Base64UrlString saves up to 25% in size compared to regular strings. |
| binaryIdLength | - |
| DbChange | A DbChange is a change to a table row. Together with a unique Timestamp, it forms a CrdtMessage. |
| decodeLength | - |
| fingerprintSize | - |
| InfiniteUpperBound | - |
| maxProtocolMessageRangesSize | Maximum size of the ranges in bytes. |
| maxProtocolMessageSize | Maximum size of the entire protocol message in bytes. |
| ProtocolErrorCode | - |
| ProtocolValueType | - |
| protocolVersion | Evolu Protocol version. |
| RangeType | - |
| WriteKeyMode | - |
| zeroFingerprint | A fingerprint of an empty range. |
Functions
| Function | Description |
|---|---|
| applyProtocolMessageAsClient | - |
| applyProtocolMessageAsRelay | - |
| base64Url256ToBytes | Converts a Base64Url string to a Uint8Array for binary storage. This encoding is more space-efficient than UTF-8 for Base64Url strings. |
| binaryIdToId | - |
| binaryOwnerIdToOwnerId | - |
| binaryTimestampToFingerprint | - |
| createProtocolMessageBuffer | - |
| createProtocolMessageForSync | Creates a ProtocolMessage for sync. |
| createProtocolMessageForWriteKeyRotation | Creates a ProtocolMessage for WriteKey rotation. |
| createProtocolMessageFromCrdtMessages | Creates a ProtocolMessage from CRDT messages. |
| createTimestampsBuffer | - |
| decodeBase64Url256 | - |
| decodeBase64Url256WithLength | - |
| decodeNodeId | - |
| decodeNonNegativeInt | Decodes a non-negative integer from a variable-length integer format. |
| decodeNumber | - |
| decodeProtocolMessageToJson | Decodes a ProtocolMessage into a readable JSON object for debugging. |
| decodeSqliteValue | - |
| decodeString | - |
| decryptAndDecodeDbChange | Decrypts and decodes an EncryptedDbChange using the provided owner's encryption key. |
| encodeAndEncryptDbChange | Encodes and encrypts a DbChange using the provided owner's encryption key. Returns an encrypted binary representation as EncryptedDbChange. |
| encodeBase64Url256 | - |
| encodeLength | - |
| encodeNodeId | - |
| encodeNonNegativeInt | Encodes a non-negative integer into a variable-length integer format. It's more efficient than encoding via encodeNumber. |
| encodeNumber | Evolu uses MessagePack to handle all number variants except for NonNegativeInt. For NonNegativeInt, Evolu provides more efficient encoding. |
| encodeSqliteValue | - |
| encodeString | - |
| idToBinaryId | - |
| ownerIdToBinaryOwnerId | - |