Websockets Protocol

We use standard websockets handshake with Sec-WebSocket-Protocol: ciruela.v1 and no extensions.

Serialization

Payload is serialized using CBOR. There are three kinds of messages:

1. Request
2. Response
3. Notification

All three types of messages can be sent at any time into any direction. Each request includes a numeric identifier that is used in corresponding response. Each side of the connection can create request identifiers independently. Each request has exactly one response. If more than one response is provided it’s built by some higher level construct.

Every message is contiguous, messages can’t interleaved. Protocol has no flow control besides what TCP provides. If more concurrency desired than multiple connections might be used.

We will use CDDL for describing message format. Here is the basic structure of a message:

message = $message .within message-structure

message-structure = [message-kind, message-type, *any] .and typed-message
message-kind = &( notification: 0, request: 1, response: 2 )
message-type = $notification-type / $request-type

typed-message = notification / request / response
notification = [0, $notification-type, *any]
request = [1, $request-type, request-id, *any]
response = [2, $request-type, request-id, *any]
request-id = uint

Signing Uploads

Signature of the upload consists of the following fields packed as the CBOR length-prefixed array in this specific order:

signature-data = [
    path: text,      ; destination path
    image: bytes,    ; binary hashsum of the image (bottom line of the
                     ; index file but in binary form)
    timestamp: uint, ; milliseconds since unix epoch when image was signed
]

Ciruela currently only supports ed25519 algorithm for signatures, but more alorithms (RSA in particular) can be used in future.

The signature itself is an array of at least two arguments with type as the first element and rest depends on the signature algorithm:

signature = ["ssh-ed25519", bytes .size 64]

Note: the ed25519 signature includes public key as a part of the signature as per standard. Other signatures might require different structure.

Commands

AppendDir

Schedule a an adding the new directory. This sends only a signed hash of the directory index and marks this directory as incoming.

Note

If different images have been scheduled for upload by different peers in the cluster cluster may end up with different images on different nodes

If upload for this path and image already exists at node another signature is added.

If there is no such index on the peer it asks this peer or any other available connection for the index data itself and subsequently asks for missing chunks (some chunks may be reused from different image).

Content of the message is a dictionary (CBOR object):

$message /= [1, "AppendDir", request-id, append-dir-params]
$message /= [2, "AppendDir", request-id, append-dir-response]
append-dir-params = {
    path: text,                 ; path to put image to
    image: bytes,               ; binary hashsum of the image (bottom line
                                ; of the index file but in binary form
    timestamp: uint,            ; milliseconds since the epoch
    signatures: [+ signature],  ; one or more signatures
}
append-dir-response = {
    accepted: bool,             ; whether directory accepted or not
    ? reject_reason: text,      ; a machine-parseable reason for rejection
    ? hosts: {* bytes => text}, ; hosts that will probably accept the
                                ; directory
}

Note: accepted response here doesn’t mean that this is new directory (i.e. same directory might already be in place or might still be downloaded). Also it doesn’t mean that download is already complete. Most probably it isn’t, and you should wait for a completion notification.

The hosts field may or may be not sent both in case of accepted is true or not. In the latter case, it might be useful to reconnect to one of these hosts. In the former case, we can track ReceiveImage messages from all these hosts. Note: we transmit machine ids (key in mapping) and host names. Client should track notifications by machine_id, but may use name for human-readable output. Note2: while in most cases hosts will be exhaustive list for all clusters it may be not so, if not is just restarted and has not picked up all the data in gossip subsystem.

ReplaceDir

Schedule a replacing the directory with the new image. This sends only a signed hash of the directory index and marks this directory as incoming.

Note

If different images have been scheduled for upload by different peers in the cluster the one with latest accross the cluster timestamp in the signature will win

If there is no such index on the peer it asks this peer or any other available connection for the index data itself and subsequently asks for missing chunks (some chunks may be reused from different image).

$message /= [1, "ReplaceDir", request-id, replace-dir-params]
$message /= [2, "ReplaceDir", request-id, replace-dir-response]
replace-dir-params = {
    path: text,                 ; path to put image to
    image: bytes,               ; binary hashsum of the image (bottom line
                                ; of the index file but in binary form)
    ? old_image: bytes,         ; hash olf the previous image
    timestamp: uint,            ; milliseconds since the epoch
    signatures: [+ signature],  ; one or more signatures
}
replace-dir-response = {
    accepted: bool,             ; whether directory accepted or not
    ? reject_reason: text,      ; a machine-parseable reason for rejection
    ? hosts: {* bytes => text}, ; hosts that will probably accept the
                                ; directory
}

Note: if no old_image is specified the destination directory is not checked. Use AppendDir to atomically update first image.

See AppendDir for the explanation of hosts usage.

PublishImage

Notifies peer that this host has data for the specified index. This is usually executed before AppendDir, so that when receiving latter command server is already aware where to fetch data from.

$message /= [0, "PublishImage", publish-index-params]
publish-image-params = {
    id: bytes,               ; binary hashsum of the image (bottom line
                             ; of the index file but in binary form)
}

This notification basically means that peer can issue GetIndex in backwards direction.

ReceivedImage

Notifies peer that some host (maybe this one, or other peer) received and commited this image. The notification is usually sent after PublishImage for the specified id.

The notification can be used by cicuela command-line client to determine that at least one host (or at least N hosts) received the image and it’s safe to disconnect from the network and also to display progress.

$message /= [0, "ReceivedImage", received-image-params]
received-image-params = {
    id: bytes,               ; binary hashsum of the image (bottom line
                             ; of the index file but in binary form)
    path: text,              ; path where image was stored
    machine_id: bytes,       ; machine-id of the receiver
    hostname: text,          ; hostname of the receiver
    forwarded: bool,         ; whether message originated from this host
                             ; or forwarded
}

The forwarded field might be used to skip check on hostname field.

AbortedImage

Notifies peer that some host (maybe this one, or other peer) have aborted receiving this image. The notification is usually sent after PublishImage for the specified id.

The notification can be used by cicuela command-line client to notify that image can’t be written for some reason, or to determine when it’s find to retry upload in case of already_uploading_different_version (-x flag of CLI).

$message /= [0, "AbortedImage", aborted-image-params]
aborted-image-params = {
    id: bytes,               ; binary hashsum of the image (bottom line
                             ; of the index file but in binary form)
    path: text,              ; path where image was stored
    machine_id: bytes,       ; machine-id of the receiver
    hostname: text,          ; hostname of the receiver
    forwarded: bool,         ; whether message originated from this host
                             ; or forwarded
    reason: text,            ; reason of why image was aborted
}

The forwarded field might be used to skip check on hostname field.

GetIndex

Fetch an index data by it’s hash. This method is usually called by server after AppendDir and ReplaceDir has been received. And it is sent to the original client (in backwards direction). But the call only takes place if no index already exists on this host or on one of the peers.

$message /= [1, "GetIndex", request-id, get-index-params]
$message /= [2, "GetIndex", request-id, get-index-response]
get-index-params = {
    id: bytes,               ; binary hashsum of the image (bottom line
                             ; of the index file but in binary form)
    ? hint: text             ; virtual_path where index can be found
}
get-index-response = {
    ? data: bytes,           ; full original index file
}

Note: index file can potentially be in different formats, but in any case:

• Consistency of index file is verified by original id which is also a checksum
• Kind of index can be detected by inspecting data itself (i.e. first bytes of index file should contain a signature of some kind)

Note 2: server implementation can ignore or can use hint value, client implementation can supply or can skip hint. Current state is: ciruela upload does not use hint, while ciruela-server always sends but never uses a hint value (still, the virtual path where index resides is used internally, so it may become useful in future if we will ever forward the GetIndex requests)

GetIndexAt

Fetch an index data by it’s path. It’s usually used to download image by a client (perhaps to execute modify and update cycle).

Note: image id is a part of index data so is not provided separately.

$message /= [1, "GetIndexAt", request-id, get-index-at-params]
$message /= [2, "GetIndexAt", request-id, get-index-at-response]
get-index-at-params = {
    path: text                  ; virtual_path to check image at
}
get-index-at-response = {
    ? data: bytes,              ; full original index file
    ? hosts: {* bytes => text}, ; hosts that contain a directory
}

The index file returned is a similar way to GetIndex. If there is no such config response may include a list of hosts to search for a directory at. Similarly to how it’s done in AppendDir and ReplaceDir.

GetBlock

Fetch a block with specified hash.

$message /= [1, "GetBlock", request-id, get-block-params]
$message /= [2, "GetBlock", request-id, get-block-response]
get-block-params = {
    hash: bytes,                ; binary hashsum of the block
    ? hint: [text, text, uint], ; virtual_path, path, and position where
                                ; the blocks can be found found
}
get-block-response = {
    ? data: bytes,           ; full original index file
}

Note: server implementation can ignore or can use hint value, client implementation can supply or can skip hint. Current state is: ciruela upload does not use hint, while ciruela-server always sends and uses a hint value.