Part of Stem. This page defines the scope set, the thing two peers compare when they sync, and gives the derivation that every implementation must reproduce bit for bit.

The scope set of a scope, for a requesting peer, is the set of blobs the answering peer holds that belong to the scope by the derivation below and that the requesting peer may read, where "may read" is membership of the requester's authenticated accounts in the readers of a node the blob maps to.

Range-based set reconciliation compares two sets without listing them, which only works if both peers build their set by the same rule from the same inputs. The rule is therefore part of the protocol and is versioned by the protocol id: two peers on different rules will not sync, which is the behaviour HM24 already has for its scope derivation. The rule is a function of the scope and of the peer's own index; it never asks another peer.

The derivation

Given a scope (space, node, depth, facets), with node defaulting to the space root, depth to exact and facets to state, files, comments and authority:

In-scope nodes. The node itself; plus, for children, the nodes whose head placement names it as parent; plus, for subtree, every node reachable by following parents up to it. Only nodes whose creating Node blob is indexed count. Tombstoned and redirected nodes count: their Node blobs are part of the set so that deletions and moves propagate.

Facet state. Every Node blob for every in-scope node (all of them, not only heads, so that prev chains reconcile). For every head target: for heads, every Change reachable from the heads through deps, to the genesis; for snapshot, the Snapshot and every blob reachable through prev. A Change or Snapshot reached through deps or prev that is itself a Snapshot continues through its own prev.

Facet files. Every blob reached from the state blobs of step 2 through file links, and from those through their own UnixFS links, to the leaves.

Facet comments. Every node of kind comment, in any space the answering peer holds, whose state target names an in-scope node; then steps 2 and 3 for those comment nodes; then the same again for comments whose target is one of those comments, to a fixpoint. Comments live in their authors' spaces, so this is the one place a scope reaches outside its space.

Facet authority. Every Grant whose subject covers an in-scope node (the node itself or an ancestor with exact false), every Grant whose subject is a group that any such Grant's audience names, every Revocation that names any of those Grants, every Group blob any of them names, and every Grant that any blob already in the set names as proof, together with its chain; repeated to a fixpoint. This facet is what lets a receiving peer evaluate authority for everything else in the set.

Facet profile. The Node blobs of the space root and the state blobs behind its heads, as in step 2, and nothing else. A scope with only this facet is what a client uses to render a name and an avatar.

Union the facets selected. Drop any blob the answering peer holds only as a placeholder (known by hash, bytes not present) or that is stashed, since those cannot be served.

Filter by readers. For each remaining blob, keep it if some node it maps to has, among its readers, one of the requester's authenticated accounts, or everyone. The mapping from blobs to nodes is the one defined in Readers. An unauthenticated requester keeps only blobs of public nodes.

The result is sorted by the blob's signed timestamp and then by CID bytes, which is the item order range-based set reconciliation uses. Blobs without a signed timestamp (files) sort at timestamp zero.

Properties

The answering peer computes the filtered set before it computes any fingerprint, so a requester never learns anything about blobs outside its readers, not even that a range is non-empty. This is the uniform denial rule applied to sync.

A requester reconciling its own copy of the set against a peer applies the same filter with the peer's authenticated accounts, so it never advertises blobs the other side may not read.

Because step 5 brings grants along with content, a receiving peer can authorize what it receives without a second round trip. Because Fetch serves the authority facet first, it usually can do so without stashing.

The set is a function of the index, so the daemon can maintain it incrementally: when a blob is indexed, the handler knows which nodes it maps to and which scopes those nodes fall in, and patches the materialised sets. HM24's maintained reconciliation index with its oracle and shadow verifier is the same idea; Stem's node-to-blob mapping makes the oracle exact instead of over-approximate.

Where it is used

Reconcile compares two scope sets.

Watch streams additions to a scope set.

Offer claims that blobs belong to a scope; the receiver checks them against this derivation.

Database structure describes the materialised per-scope item tables.

Today (HM24)

The Network page gives the current derivation: Refs, capabilities, comments, profiles and contacts anchored on the IRIs in scope, changes through heads and deps, every linked media blob, inbound contacts for a whole space, and AGENT capabilities to a fixpoint, then filtered by blob_visibility for the authenticated peer. Stem's derivation is the same shape with nodes in place of IRIs, facets in place of type allowlists, and readers in place of the two visibility values.

The derivation

Properties

Where it is used

Today (HM24)

See also