Hosted onseed.hyper.mediavia theHypermedia Protocol

The “compute haves” work is mainly here:

Main path: discovery local store

File:

backend/hmnet/syncing/discovery.go

Function:

func (s *Service) DiscoverObjectWithProgress(...)

Around the middle of that function, it builds the local RBSR store once:

store := newAuthorizedStore()
err := s.db.WithSave(ctx, func(conn *sqlite.Conn) error {
    return loadRBSRStore(conn, dkeys, store)
})
store.Seal()

That is the local “what do we have?” dataset for this discovery.

Then the same store is passed to all peer syncs:

res := s.syncWithManyPeers(ctxLocalPeers, subsMap, store, prog, auth, blobTypes)

The expensive DB part

Same file:

func loadRBSRStore(conn *sqlite.Conn, dkeys map[DiscoveryKey]struct{}, store *authorizedStore) error

This calls:

collectBlobs(conn, dkeys, false)

Then runs a big select over rbsr_blobs, blobs, structural_blobs, blob_visibility, ordered by timestamp/hash, and inserts each item into the in-memory RBSR store:

store.Insert(ts, cidBytes)

So the DB-heavy “compute local haves” part is:

DiscoverObjectWithProgress
  -> loadRBSRStore
    -> collectBlobs
      -> fillTables
    -> SELECT ... FROM rbsr_blobs ...
    -> store.Insert(...)
  -> store.Seal()

Per-peer reconciliation haves

File:

backend/hmnet/syncing/syncing.go

Functions:

syncWithManyPeers(...)
syncWithPeer(...)
syncResources(...)

Each peer gets the same store, filtered by authorization:

filteredStore := store.WithFilter(authorizedSpaces)

Then syncResources creates an RBSR session:

ne, err := rbsr.NewSession(store, rbsrMsgSizeBytes)

And each reconcile round computes haves/wants here:

msg, err = ne.ReconcileWithIDs(msg, &haves, &wants)

RBSR algorithm code

File:

backend/hmnet/syncing/rbsr/rbsr.go

Function:

func (n *Session) reconcile(...)

In ListMode, it compares our store range with the peer’s range:

n.store.ForEach(lower, upper, func(_ int, item Item) bool {
    have := item.Value
    if _, exists := theirElems[string(have)]; !exists {
        *haveIDs = append(*haveIDs, have)
    }
    ...
})

That is literally where “haves” are appended.

Important nuance

Current code already builds loadRBSRStore once per DiscoverObjectWithProgress call, then reuses it across peers in syncWithManyPeers.

We compute haves once per peer:

  1. the RBSR range comparison work in ReconcileWithIDs, which does happen per peer; or

  2. multiple frontend-triggered discoveries for the same document, each rebuilding its own store; or

  3. server-side ReconcileBlobs, where each inbound peer request builds a store separately.

Recap

The main haves code is:

  • DB/local dataset build: backend/hmnet/syncing/discovery.goloadRBSRStore

  • per-peer haves/wants computation: backend/hmnet/syncing/syncing.gosyncResourcesReconcileWithIDs

  • actual haves append: backend/hmnet/syncing/rbsr/rbsr.goSession.reconcile

Current discovery code appears to build the expensive DB-backed local store once per discovery, not once per peer.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime