1. Shared conventions
These rules apply to every format.
1.1 Canonical JSON#
Canonical JSON is the JSON Canonicalization Scheme of RFC 8785.
In brief: no whitespace between tokens; object members sorted by their names compared as sequences of UTF-16 code units; strings escaped only where JSON requires, with \b, \t, \n, \f, \r, \" and \\ for those characters and \u00xx with lowercase hex for the other characters below U+0020; numbers written as ECMAScript writes a double.
Strings are not normalized: canonical JSON keeps every code point as given.
A document containing a lone surrogate, NaN or an infinity has no canonical form and is invalid.
Every hash of a JSON value in these specifications is a hash of its canonical JSON, encoded as UTF-8.
The vectors in canonical-json.jsonl give JSON text to parse as IEEE 754 doubles and the canonical output.
A store may keep its own canonical form for its stored hashes; that form is outside this text.
1.2 Numbers#
A decimal that carries meaning is a string, so that 310.0 and 1.5260 keep their printed digits.
The schemas call this a decimal: an optional minus sign, an integer part without leading zeros, and an optional fraction.
JSON numbers are used only for counts, indexes, offsets, sizes and geometry.
Geometry is in PDF points. A producer SHOULD round coordinates to hundredths of a point, which keeps their canonical form short and identical across languages.
1.3 Hashes#
A hash is SHA-256, written as 64 lowercase hexadecimal digits.
Six digits are for display and for pins in addresses; no record stores a shortened hash.
document.json names the function in hashing.algorithm, sha-256 in this version, and every hash in the package, its records and its proofs uses the one function the document names; a consumer refuses an algorithm it does not implement (doc.hash.algorithm).
Property names such as sha256 and page_fp belong to that function, so a later algorithm arrives as a new major version of the formats whose values it changes, with the name in the data letting two versions share a store.
Inside a team's own store, hashes are plain SHA-256; any content value that leaves the team is keyed, as 5.8 defines.
1.4 Text normalization for hashing#
To normalize a string for hashing:
- Apply Unicode Normalization Form C.
- Replace every maximal run of whitespace characters with one U+0020.
- Remove a leading and a trailing U+0020.
Whitespace means the 25 characters with the Unicode property White_Space: U+0009 to U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000 to U+200A, U+2028, U+2029, U+202F, U+205F and U+3000.
U+200B ZERO WIDTH SPACE is not whitespace.
Stored text is kept exactly as witnessed; normalization applies only when a value is computed.
The vectors in normalize.jsonl cover the cases, including the angstrom sign, which NFC turns into Å.
Normalization forms, General Categories and the White_Space property are those of Unicode 16.0.
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference tools refuse to run on another, and moving the formats to a later version is a change to every format that hashes text (0.3).
1.5 Character offsets#
Offsets count Unicode code points, never UTF-16 code units or bytes.
An offset pair [start, end] names the code points from start up to but not including end.
A browser implementation converts to UTF-16 at its edge; offsets.jsonl gives both forms for text with characters outside the Basic Multilingual Plane.
1.6 Times#
A time is UTC, written YYYY-MM-DDTHH:MM:SSZ, with no fractions.
A date is YYYY, YYYY-MM or YYYY-MM-DD.
1.7 Determinism#
Every value these specifications define is a pure function of its inputs. Readings from machine-learning backends are not deterministic, so a producer caches them by the hash of their inputs and records them in the package's provenance, and never recomputes them silently. From the cached readings onward, the same inputs and tool versions MUST give byte-identical files. No file carries the time it was written, unless the time is part of the record, as a receipt's is.
1.8 IDs and keys#
| Kind | Form | Pattern |
|---|---|---|
| Node ID | Seven characters of lowercase Crockford base32, the first a letter. | ^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6}$ |
| Record ID | 7 to 26 such characters, the first a letter. Derived records use 13. | ^[a-hjkmnp-tv-z][0-9a-hjkmnp-tv-z]{6,25}$ |
| Work key | A Scholar-style part, a hyphen and 4 to 51 characters of Crockford base32 (2.4). At most 128 characters. | ^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}$ |
| Document key | A work key, with a dotted suffix for a version other than the version of record. At most 128 characters. | ^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\.[a-z0-9][a-z0-9-]*)*$ |
| Key | A store, project or source key, or a legacy document key kept as an alias. At most 128 characters. | ^[a-z0-9][a-z0-9._-]*$ |
| Alias | A legacy ID; it always holds a ., _ or -, so it never looks like a node ID. | ^[a-z0-9][a-z0-9._-]*[._-][a-z0-9._-]*$ |
| Dataset ID | ds- and a key. | ^ds-[a-z0-9][a-z0-9._-]*$ |
The Crockford alphabet is 0123456789abcdefghjkmnpqrstvwxyz; it has no i, l, o or u.
Its 22 letters are abcdefghjkmnpqrstvwxyz.
Because the first character is a letter, every ID is a valid HTML id, CSS selector and URL fragment without escaping.
A key qualified by its store, smithlab/cornell1995second, names a document anywhere.
1.9 Files#
JSON Lines files hold one record per line, in canonical JSON, each line ending with one U+000A. JSON files hold one value in canonical JSON followed by one U+000A. So a corrected sentence is a one-line change, and every file has exactly one valid byte sequence. Each JSON Lines file has a defined order, given where the file is defined.
1.10 Prose#
Prose in these formats, and in this repository, is US English, one sentence per line.