Part of Stem. This page defines the Reconcile method of the sync RPC specification. A sync run repeats it until every range is agreed, and comes away knowing exactly which readable blobs it lacks.

The formal schema is attached as the schemaDefinition of this page: a struct {key, input, output} with key pinned to Reconcile.

One round of range-based set reconciliation over a scope. The server derives the scope set from its index, removes every blob the caller may not read, and answers the caller's ranges. Fingerprints are folded over the filtered set, so an unauthorized caller learns nothing about hidden blobs. The server is stateless between rounds.

Input

field	type	required	meaning
`scope`	data/scope	yes	The scope whose blob set is being compared.
`ranges`	list of rpc/type/range	yes	The caller's ranges for its part of the set.

Output

field	type	required	meaning
`ranges`	list of rpc/type/range	yes	The server's answer: skips, fingerprints to split further, or lists.

Rules

The set. Both sides compute the scope set for scope from their own index with the same rule, then the server removes every blob whose readers do not include an account bound to the connection (or everyone).

Items. An item is (ts, cid): the blob's signed timestamp in milliseconds (zero for unsigned file blobs) and the full CID bytes. Items are ordered by timestamp, then by CID bytes. An item's hash is the SHA-256 of its CID bytes.

Fingerprint. For a range, sum the item hashes as 256-bit integers with wraparound, append the item count as a little-endian 64-bit integer, take SHA-256, keep the first 16 bytes. When the server removed any hidden item from a range, the fingerprint is folded over the filtered items only, so it equals what a peer holding exactly the readable items would compute.

Rounds. The initiator splits its whole set into sixteen ranges and sends one fingerprint each, or a single list range when it holds fewer than thirty-two items. The server answers each range: skip when fingerprints match; otherwise it splits the range into sixteen sub-ranges with fingerprints, or lists it when fewer than thirty-two items remain. A list from the caller is answered with the server's list for the same bounds, and the caller computes what it lacks. A message is at most 1 MiB; ranges that do not fit are collapsed into one trailing fingerprint and resumed next round.

Limits. A round is bounded at fifteen seconds and a session at one thousand rounds. The server is stateless between rounds and limits concurrent inbound reconciliations; past the limit it fails with resource-exhausted after a short wait.

Direction. The result is one-directional: the caller learns what the server has that it lacks. What the caller has that the server lacks is offered with Offer if the caller's policy says so.

A Reconcile creates no durable state on the server for scopes it does not itself materialise.

Today (HM24)

HM24's Syncing.ReconcileBlobs takes filters[]{resource, recursive, depth_one, types[]} and ranges[]{mode SKIP|FINGERPRINT|LIST, bound_timestamp, bound_value, values, fingerprint}. The algorithm, items, bucket count, list threshold, message budget and round limits are kept unchanged, and the folded fingerprint for hidden items is kept. The filter becomes a scope, and the hidden set is defined by readers rather than by the per-space visibility table. HM24's accumulator never incremented the item count; Stem's fingerprint includes it, which is one reason for the new protocol id.

Example

{"key": "Reconcile", "input": {"scope": {"space": {"/": {"bytes": "7QEA…"}}, "node": "bafyrein6a6wfhym6l3vfz5zfkkibj5j6wjibagi3mnbqnspuq2idw52ijb", "depth": "subtree"}, "ranges": [
  {"mode": "fingerprint", "boundTs": 1759000000000, "boundCid": {"/": "bafy…01"}, "fingerprint": {"/": {"bytes": "a1…"}}},
  {"mode": "fingerprint", "boundTs": 1759910400000, "fingerprint": {"/": {"bytes": "b2…"}}}
]}}

{"ranges": [
  {"mode": "skip", "boundTs": 1759000000000, "boundCid": {"/": "bafy…01"}},
  {"mode": "list", "boundTs": 1759910400000, "cids": [{"/": "bafy…11"}, {"/": "bafy…12"}]}
]}

Input

Output

Rules

Today (HM24)

Example

See also