The Underlay Protocol

Overview

Underlay is a protocol for publishing, versioning, and collaborating on structured data. Every piece of content (records, schemas, and files) is identified by its SHA-256 hash. Versions are manifests that reference these hashes. This means storage is deduplicated globally, transfers only move data the other side doesn't have, and provenance is built in: any record can be traced back to every collection and version that includes it.

A single-file implementation of the Underlay protocol can be found at sus.knowledgefutures.org/

Data model

The protocol has four primitives:

• Record: A JSON object with an id, a type, and a data payload. Records are the rows of your dataset. Each record is content-addressed by the SHA-256 hash of its canonical JSON representation.
• Schema: A JSON Schema document that describes the structure of a record type. Schemas are also content-addressed. They define validation rules, mark private fields, and annotate cross-record references.
• Version: An immutable snapshot: a manifest of record hashes, schema hashes, file hashes, and a metadata bag. Versions are identified by semver (e.g. v1.2.0).
• File: A binary blob (PDF, image, etc.) stored by SHA-256 hash. Records reference files with the {"$file": "sha256:..."} convention.

Record identity

A record's identity is the SHA-256 hash of its canonical JSON. The canonical form is:

canonical = JSON.stringify({ id: "pub-001", type: "Publication", data: { ... } })
hash = SHA256(canonical)  // hex-encoded

The private flag is not part of the hash. Two records with identical id, type, and data but different privacy flags produce the same hash. This is intentional. The record's content identity doesn't change when you change who can see it.

That flag is declared on the manifest entry at push time, never on the record body, and it belongs to the reference rather than the content: it is stored on the version→record edge, never on the content-addressed record object. Three properties follow, and an implementation must honour all three.

• Per-version. The same bytes are legitimately public in one collection and hidden in another. An implementation that stores privacy on the deduplicated record object cannot represent this, and makes one collection’s public: hash depend on another collection’s push history.
• Omitted means public. There is no inheritance from the previous version. Privacy is re-declared, in full, on every push — a record private in v1 whose v2 manifest entry omits the flag is public in v2. The manifest read endpoint echoes the flag back so a round-trip is lossless.
• Forward-only. Versions are immutable, so marking a record private in v2 hides it in v2 and does not reach back into v1 — the record stays fetchable at its old version, and any file it referenced stays downloadable. “Redacted” means absent from the latest version’s public projection, not erased. Retroactive purging is a separate operation and is not part of the protocol.

A record whose type declares private fields has a second address: its public record hash, the SHA-256 of the same canonical form with the private fields stripped. Public manifests list records by their public hash, and the record endpoints resolve either address, so a public reader can always verify that hashing the document they received reproduces the address they requested. When a type has no private fields the two addresses coincide.

Wire format is JSONL: one record per line, independently hashable and streamable:

{"id":"pub-001","type":"Publication","data":{"title":"The Structure of Scientific Revolutions","doi":"10.1234/example"}}

Version identity

A version's hash is the SHA-256 of a canonical JSON object containing sorted hashes:

canonical = JSON.stringify({
  schemas: { "Publication": "abc123...", "Author": "def456..." },  // sorted by slug
  records: ["0a1b2c...", "3d4e5f...", ...],                        // sorted, hex SHA-256
  files: ["7a8b9c...", ...],                                       // sorted, hex SHA-256
  metadata: { "license": "CC-BY-4.0", "readme": "# My Collection\n..." }  // canonicalized JSON
})
hash = "private:" + SHA256(canonical)

Two versions with the same content produce the same hash, regardless of when or where they were created. A version is identified by both digests, though: the server rejects a push as a duplicate only when the private: hash and the public: hash both match an existing version. That matters for redaction — re-pushing identical content with a record newly marked private yields the same private: hash and a different public: one, and is a legitimate new version rather than a “no changes” conflict.

A separate public: hash covers the public projection of the version: private records and private types are omitted entirely, and private fields are stripped from the records that remain before re-hashing. This lets external verifiers confirm the public content without access to private data — and because it is computed from the pushed version alone, never from any global state, the same authored version yields the same public: hash on every server.

Semver semantics

Versions are identified by semver strings (e.g. v1.2.0). The server auto-derives the next version based on what changed:

• Major bump: a schema changed (e.g. v1.2.0 -> v2.0.0)
• Minor bump: records or files changed (e.g. v1.2.0 -> v1.3.0)
• Patch bump: neither the schema set nor the record set changed — a metadata edit such as readme or license, or a push that changes only which records are private (e.g. v1.2.0 -> v1.2.1)

Push

All pushes use the negotiate protocol, a three-step flow similar to git's pack negotiation. The client sends a manifest of record hashes, the server says which it needs, the client sends those records (in one or more batches), then commits.

# 1. Client sends manifest of record hashes
POST /api/collections/:owner/:slug/versions/negotiate
{
  "base_version": "v1.1.0",
  "schemas": { "Publication": { ... } },
  "manifest": [
    { "id": "pub-001", "type": "Publication", "hash": "abc123..." },
    { "id": "pub-002", "type": "Publication", "hash": "def456...", "private": true }
  ],
  "files": ["7a8b9c..."],
  "message": "Add new publication"
}

# 2. Server responds with what it needs
{
  "session_id": "...",
  "needed_records": ["def456..."],
  "needed_files": [],
  "total_records": 2,
  "already_have_records": 1
}

# 3. Client sends only the missing records as JSONL (repeatable for large batches)
POST /api/collections/:owner/:slug/versions/negotiate/:sessionId/records
Content-Type: application/x-ndjson

{"id":"pub-002","type":"Publication","data":{"title":"...","doi":"..."}}
# -> { "received": 1, "remaining": 0, "total_needed": 1 }

# 4. Client commits — server validates schemas, creates version
POST /api/collections/:owner/:slug/versions/negotiate/:sessionId/commit
# -> { "semver": "v1.2.0", "hash": "...", "recordCount": 2, "fileCount": 1 }

The negotiate step checks every record and file hash against the server's global store. If 100,000 records already exist and only 5 are new, only those 5 are transferred.

For large pushes, the /records endpoint can be called multiple times (up to 10,000 records per batch). The server tracks which records have been received. Once all needed records are submitted, commit to finalize the version. Sessions expire after 10 minutes of inactivity — every manifest chunk and record batch pushes the expiry back, so a push that runs for an hour will not expire underneath you.

Pushes larger than one request

Two steps of the flow assume the collection fits comfortably in one request: the manifest arrives as a single JSON body, and commit holds the connection open while it validates and hashes everything. At a few million records neither holds — the manifest would be hundreds of megabytes and the commit would run for minutes. Both have a chunked form, and they change the shape of the exchange rather than its meaning: the version hash a chunked, asynchronous push produces is identical to the one the simple flow produces from the same content.

# Manifests above 500k entries upload in chunks instead of one body.
# Declare the count; the server opens the session without asking for anything yet.
POST /api/collections/:owner/:slug/versions/negotiate
{ "base_version": null, "schemas": {...}, "manifest_expected": 3110000 }
# -> { "session_id": "...", "manifest_expected": 3110000, "manifest_received": 0 }

# Send the manifest as JSONL, <= 50,000 entries per request. Each response says
# which records from THAT chunk are needed, so bodies can start flowing early.
POST /api/collections/:owner/:slug/versions/negotiate/:sessionId/manifest
Content-Type: application/x-ndjson

{"id":"pub-001","type":"Publication","hash":"abc123..."}
# -> { "received": 50000, "needed_records": [...], "manifest_received": 150000 }

# Commit in the background rather than holding a request open for minutes.
POST /api/collections/:owner/:slug/versions/negotiate/:sessionId/commit?async=true
# -> 202 { "session_id": "...", "status": "committing" }

# Poll until the version lands. The finalize does not depend on your connection.
GET /api/collections/:owner/:slug/versions/negotiate/:sessionId
# -> { "status": "committed", "result": { "semver": "v1.2.0", "hash": "...", ... } }

A chunked manifest has no natural end-of-stream, so manifest_expected is part of the contract rather than a hint: commit compares it against what actually arrived and refuses to build a version if they differ. Chunks are keyed by hash and therefore idempotent — re-sending one after a timeout is safe. An asynchronous commit publishes nothing until it finishes, so there is no window in which a partially built version can be read.

Pull

Clients can fetch a full manifest or a delta between two versions. Combined with the batch records endpoint, this enables efficient pull synchronization.

# Full manifest
GET /api/collections/:owner/:slug/versions/v2.0.0/manifest

# Delta since a previous version
GET /api/collections/:owner/:slug/versions/v2.0.0/manifest?since=v1.1.0
{
  "version": "v2.0.0",
  "since": "v1.1.0",
  "delta": {
    "added":   [{ "id": "pub-004", "type": "Publication", "hash": "..." }],
    "updated": [{ "id": "pub-001", "type": "Publication", "hash": "...", "previousHash": "..." }],
    "removed": [{ "id": "pub-003", "type": "Publication", "hash": "..." }]
  }
}

# Fetch only the records you need
POST /api/records/batch
{ "hashes": ["abc123...", "def456..."] }
# Returns JSONL stream

# Or read the whole version in one streamed response
GET /api/collections/:owner/:slug/versions/v2.0.0/records.ndjson
Content-Type: application/x-ndjson
X-Underlay-Record-Count: 3113504

{"id":"pub-001","type":"Publication","data":{...},"hash":"..."}
{"id":"pub-002","type":"Publication","data":{...},"hash":"..."}
# ... one object per line, ordered by id; ?after=<id> resumes

Both are keyset-paginated: pass pagination.nextCursor back as ?cursor= until hasMore is false. A delta of any size can be walked to completion, and the three lists drain independently, so a page late in the walk may hold only updated entries. The cursor is opaque — pass back what you were given rather than constructing one.

Reading a whole version

Paging is the wrong shape for "give me everything": each request pays a round trip to re-establish a cursor the server just had, so a three-million-record collection costs over fifteen hundred of them. records.ndjson streams the entire version in one response, read through a database cursor and written as it goes, so neither side holds more than a chunk.

Four properties make it usable as a protocol rather than a convenience, and an implementation is expected to honour all four:

• Ordered by record id, ascending. This is what gives ?after= meaning, and it is the difference between a stream you can resume and one you must restart.
• One JSON object per line, of the form {id, type, data, hash}. hash is the same content-address the paged endpoint serves — the full record hash for owners, the public hash for everyone else.
• Privacy filtering is identical to the paged endpoint. Private types and private records are absent; private fields are stripped. A reader must not be able to learn more by choosing a different transport.
• Completeness is the reader's to verify. A stream that fails partway cannot say so — its 200 and headers left before the failure did.X-Underlay-Record-Count states how many lines to expect — the count for that request, privacy-filtered for the reader, so the check is exact at every access level; count them, and resume from the last complete line with ?after=.

That last point is a deliberate trade rather than an oversight. Any single-response bulk format has it — the alternative is paging, which buys per-page error reporting at the cost of a round trip per page. Making the expected count explicit lets a client get the safety without the round trips.

Schema semantics

Schemas are JSON Schema documents with a few protocol-level extensions:

{
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "doi": { "type": "string" },
    "authors": {
      "type": "array",
      "items": { "type": "string", "x-ref-type": "Author" }
    },
    "pdf": { "type": "object" },
    "internalNotes": { "type": "string", "private": true }
  }
}

• "private": true on a property: the field is stripped from public views and excluded from the public hash.
• "private": true on the schema root: the entire type is hidden from public views.
• "x-ref-type": "Author": marks a field as a reference to another record type (advisory, not enforced).

Schemas are content-addressed by their SHA-256 hash. Two collections that use an identical Author schema share the same underlying schema object, with zero duplication. Schema changes trigger a major semver bump.

Unknown field handling

When records contain fields not defined in the schema, the server rejects the push with a 422 response listing the extra fields per record. This protects against accidentally storing data outside the schema contract.

To accept stripping, set "strip_unknown_fields": true in the negotiate request. The server strips the extra fields before hashing and storing, so the stored records match the schema exactly. Hashes are recomputed after stripping.

Files

Files are binary blobs stored by SHA-256 hash. Upload a file, then reference it from a record:

# Upload (content-addressed by SHA-256)
PUT /api/collections/:owner/:slug/files/sha256:a1b2c3...
Content-Type: application/pdf
<binary data>

# Reference in a record
{ "pdf": { "$file": "sha256:a1b2c3..." } }

Files are verified on upload (the server recomputes the hash and rejects mismatches). Like records and schemas, files are globally deduplicated. The same PDF in ten collections is stored once.

Provenance

Because records are content-addressed, every record hash can be traced back to every version and collection that includes it. The provenance endpoint returns this lineage:

GET /api/records/:hash/provenance
{
  "hash": "abc123...",
  "recordId": "pub-001",
  "type": "Publication",
  "firstSeen": "2026-01-15T...",
  "references": [
    { "owner": "alice", "collection": "papers", "version": "v1.2.0" },
    { "owner": "bob", "collection": "reading-list", "version": "v1.0.0" }
  ]
}

firstSeen is the earliest version creation date across all references, the record's birthday on this server. This enables citation-like provenance: "this record first appeared in alice/papers v1.2.0 on 2026-01-15."

Collaboration

Underlay supports collaboration through a small set of primitives:

• Versioning. Every push creates a new immutable version. The full history is always available. Versions are identified by semver strings and use optimistic locking: base_version (a semver string, or null for the first push) must match the current latest, or the push is rejected with a 409 conflict.
• Diffing. Any two versions of a collection can be diffed (GET .../versions/v2.0.0/diff?from=v1.1.0), returning added, updated, and removed records with hash-level comparison.
• Cross-collection references. Records reference each other by ID. Because record hashes are global, the same record appearing in two collections can be identified as identical content.
• Mirroring. Any Underlay instance can pull from another, using hash negotiation to transfer only new data. Mirrors maintain verified, independent copies.
• Forking. POST .../fork creates a new collection under your org with the source's latest version. Because records, schemas, and files are content-addressed, forking copies only the manifest; zero additional storage. The fork tracks its origin via forkedFrom. A fork references the full record bodies and gives the forker owner-level access to them, so a caller who is not a member of the source org is refused with 403 when the source holds any private content — private records, private types, or records with private fields. Members of the source org can always fork.

Errors

All error responses return JSON with an error field and an HTTP status code:

• 400 - Bad request (missing fields, invalid JSONL, hash mismatch)
• 401 - Missing or invalid credentials
• 403 - Authenticated, but not permitted: an API key used outside the collections it is scoped to, or a fork of a collection whose private content the caller cannot see
• 404 - Collection, version, or record not found
• 409 - Version conflict (base_version doesn't match), or duplicate content — both the private: and public: digests match an existing version
• 413 - File upload exceeds the instance's size limit
• 422 - Schema validation failed, missing schemas/files, or records contain fields not defined in the schema (set strip_unknown_fields to accept stripping)
• 429 - Rate limited (includes Retry-After header)
• 503 - Query timed out under load (includes Retry-After); page large result sets with keyset pagination

Content the caller may not see returns 404, not 403 — private collections and inaccessible files alike — so that a response cannot confirm their existence. 403 is reserved for cases where the caller’s identity is already established as insufficient for a resource they can see.