Formata design notes

Copy Markdown View Source

Scope

One Mix library and one Hex package (formata, namespace Formata) with no runtime dependencies. It composes backends that are maintained forks of existing packages (see "Forked backends"); it adds no processes, caches, umbrella, schema AST, schema inference, or schema conversion.

ModuleResponsibility
FormataPublic entry points: decode/3, encode/3, convert/4, validate/3, cast/3
Formata.CodecCodec behaviour, format lookup, shared value-domain checks
Formata.Codec.JSON / .JSON5 / .TOML / .YAMLFormat adapters
Formata.Codec.JSON.StreamIncremental single-document JSON parser (events)
Formata.Codec.JSON5 parser (internal JSON5.Parser)JSON5 1.0.0 parser
Formata.Codec.TOML.EncoderFocused TOML 1.0.0 encoder
Formata.Schemacompile/3, check/3, compiled-schema struct
Formata.Schema.NativeFormata.NativeSchema adapter, pure-validation preflight, casting
Formata.Schema.JSONSchemaFormata.JSONSchema adapter
Formata.JSONApplication/Phoenix/Plug JSON (strict decode, exact float callbacks, ordered objects, JSON.Encoder, pretty output)
Formata.TransformExplicit data transformations
Formata.ErrorStructured errors

Contracts

  • Argument order is data first: decode(input, format, opts), encode(data, format, opts), convert(input, from, to, opts), validate(data, schema, opts), cast(data, schema, opts), Schema.compile(kind, definition, opts).
  • Producing operations return {:ok, result}; validate/3 and Schema.check/3 return :ok.
  • Expected failures return {:error, [%Formata.Error{}, ...]} (non-empty). Error kinds: :parse, :encode, :schema, :validation, :transform, :unsupported. Programming errors (wrong argument types, unknown compile options) raise.
  • location is set only from backend data (Formata.JSON.Backend byte offsets, Formata.TOML line/column in messages, yamerl line/column).
  • validate/3 requires a compiled schema; compiling is never repeated per call.

Forked backends

The backends were published packages; they are now forked into this repository, renamed into the Formata namespace (so a consumer can still use the originals side by side), and maintained here. Upstream test suites are ported alongside the corresponding module tests under test/formata/, with helpers under test/support/formata/. Fixtures are grouped by backend under test/fixtures/. Provenance, licenses, and the common fork-time modifications are in NOTICE.md.

UpstreamVersionForkUse
jason1.4.5Formata.JSON.BackendJSON decode (objects: :ordered_objects for duplicate detection) and encode
gsmlg_toml1.0.0Formata.TOMLTOML decode
yaml_elixir2.12.2Formata.YAML.ReaderYAML decode (read_all_from_string/2)
yamerl0.10.0formata_yaml* (Erlang)Detailed node tree for duplicate keys, merge keys, null roots
ymlr5.1.6Formata.YAML.WriterYAML encode
backplane_schema1.10.76Formata.NativeSchemaNative schemas (public vocabulary)
backplane_json_schema1.10.76Formata.JSONSchemaJSON Schema compile/validate
ex_json_pointer0.9.0Formata.JSONPointerJSON Pointer for the JSON Schema backend

Fork rules: no compile-time optional integrations (their presence would depend on the consumer's dependency tree), no application environment, no processes. The minimum Elixir is 1.18 (built-in JSON, used by the JSON Schema fork); CI tests Elixir 1.18 and 1.20.

Audit findings and decisions

Formata.NativeSchema.validate/3 transforms

It applies defaults, coercions and transforms, and in strict mode drops undeclared fields and re-keys output by schema keys; permissive mode leaves the input keys and writes updated values under the data's own key (fork fix for gsmlg-opt/backplane#171, which duplicated them under atom keys). There is no pure mode and no reject-unknown mode.

Decision: pure validation uses a plan, a rewritten definition that only adds checks:

  • A preflight walks the definition and every reachable {:ref, {Mod, name}} target and rejects default, transform, encode, coerce, and {:dependent, callback} with :unsupported errors and schema paths.
  • {:custom, v} validators are wrapped so a result other than :ok/error is a :non_pure_custom error. Without this, the native engine accepts {:enum, choices, type: {:custom, fn -> {:ok, changed} end}} by checking the changed value.
  • References become lazily expanded {:dependent, fun} nodes so the wrapping also applies inside referenced (including recursive) definitions.
  • unknown: :reject wraps each map schema as {:schema, map, {:additional_keys, {:custom, reject}}}, keeping the check in the native engine; Formata expands the resulting error to one :unknown_field error per undeclared key.
  • The plan runs in permissive mode; its output is discarded.

cast/3 runs the original definition (strict mode for :strip, the reject plan in strict mode for :reject, permissive mode for :allow, keeping the input keys; a field added by a default uses the schema key). Coercion to :atom! is refused because it calls String.to_atom/1 on input.

JSON Schema

Ordinary JSON Schema validation goes directly to the JSON Schema backend, without conversion. Explicit interchange is available through Formata.Schema.Native.to_json_schema/2 and from_json_schema/2: wire export uses anyOf for native unions and requires explicit opt-in to omit custom validators; the conservative importer rejects oneOf and constraints it cannot represent. Native absent/nil and unknown-field semantics still apply to imported definitions. The old fork conversion entry points remain private.

  • Dialects: Draft 7 and 2020-12 via $schema or dialect:; conflicts are backend :draft_conflict errors (:dialect_conflict). An unrecognized $schema without dialect: or :loader is rejected, because its vocabularies cannot be known.
  • format: annotation by default, including without $schema (fork fix for gsmlg-opt/backplane#170: the default vocabularies are the standard 2020-12 set, without format-assertion). Values that need the IDNA2008 tables (A-labels, non-ASCII labels in hostname/email/idn-*) stop validation with :indeterminate_format, reported as :unsupported / :unsupported_format; it cannot be inverted by not/anyOf/oneOf/if.
  • multipleOf uses exact decimal arithmetic on integers (no Decimal, no float quotient); it never raises on data.
  • References: local only unless a :loader is passed. Unresolvable refs are backend validation errors that Formata reports as :schema / :unresolved_ref.
  • Compilation validates keyword shapes against the bundled meta-schema. Formata.Schema.JSONSchema.check_document/2 exposes document checking; preflight/2 additionally audits every reference in schema locations, including unused branches/definitions. It reuses the backend URI, pointer and resource-context helpers, retains recursive references without expansion, rejects ambiguous IDs/anchors, bounds input/loaded resources, and captures explicit loaded documents in an immutable loader closure. Annotation values are not scanned as schemas unless selected by a pointer.
  • validate/3 accepts no options, so nothing can override additionalProperties / unevaluatedProperties.
  • Data must be in the JSON data model; other terms are :non_json_value.
  • Error messages: format_error/1 is called without the path because Formata.Error renders it (the backend's path rendering, #172, is fixed in the fork).
  • Opaque base URIs (urn:...) resolve relative references per RFC 3986 on every supported Elixir version (Elixir < 1.20's URI.merge/2 requires a host).

Codecs

  • JSON: Formata.JSON.Backend silently keeps the last duplicate key; Formata decodes ordered objects and rejects duplicates.
  • TOML: gsmlg_toml had no encoder, so Formata ships its own. The forked decoder had four bugs found by toml-test (gsmlg-dev/gsmlg_umbrella#13 offset sign, #14 empty array-of-tables elements, #15 true/false keys, #16 local date-time in arrays) plus many more exposed by the invalid suite (lexer escapes, table redefinition rules, inline tables, crashes on invalid UTF-8); all are fixed. The decoder passes the full toml-test 1.0.0 suite (208 valid, 509 invalid); encoder output is also checked with Python's tomllib. The correctness review records subsequent assignment/comment, numeric underscore, date-time separator, and UTF-8 encoding-key fixes, with deterministic boundary coverage. Historical upstream issue state is independent of fixes maintained in this fork.
  • YAML: the facade uses one detailed parse preserving duplicate entries and source scalar metadata. Constructor/parser limits bound pending nesting and expanded aliases before a fused validation/conversion pass creates maps. Plain implicit merge keys and complex keys are explicitly rejected; quoted/string-tagged << is preserved. Scalar key types remain distinct. The emitter quotes implicit numeric lookalikes and uses YAML escaping when block scalar normalization would alter a string. Every output is still decoded and compared, with the caller's limits. See the YAML review for accounting, schema decisions, independent checks and the pinned official fixture selection.
  • JSON decode options (comments, trailing_commas): see "JSON extensions" below.
  • Special floats use the atoms produced by Formata.TOML (:infinity, :negative_infinity, :nan); YAML's :"+inf"/:"-inf" are normalized to them and converted back for Formata.YAML.Writer.

JSON extensions (comments, trailing commas)

  • The extensions live in the Formata.JSON.Backend decoder, not in a pre-pass. A / clause was added to every bytecase position that accepts whitespace (value, array, object, key, terminate); it consumes a // or /* */ comment when comments is set and is an error otherwise. Comments therefore cannot appear inside strings, numbers, or literals, and byte offsets in errors stay those of the original input. Comment text must be valid UTF-8; block comments do not nest.
  • Trailing commas reuse the decoder's stacks: after a comma the array/object accumulator is non-empty, so ]/} in value/key position closes the container only when the accumulator is non-empty and trailing_commas is set. Empty slots never reach that state and stay errors.
  • With both options false (the default) the decoder takes exactly the previous paths; the new clauses only replace the generic error clause for /.
  • The :json codec's decode/2 used to ignore its options. It now validates them (:unsupported / :invalid_option), so callers that passed unknown options (also through convert/4's :decode) get an error.
  • There is no :jsonc codec: JSONC is a parsing mode of :json, and encoding is always strict JSON.

Incremental JSON parser

Formata.Codec.JSON.Stream is a separate state machine (the internal Stream.Machine module). The one-shot Formata.JSON.Backend decoder was not reused: it is a recursive parse over one binary with no resumable state, and making it resumable would slow the common path. Agreement is enforced by differential tests instead (every two-way split, byte-by-byte and random chunkings, strict and extended inputs).

  • State: an immutable struct holding the options, the absolute offset/line/column of the next byte, the carried bytes, and a small record with the container stack ({:array, index, path}, {:object, seen_keys, key_path, path}), the grammar mode (expect value/key/colon/after-value/done), the depth, and the token in progress (string, number, or comment, with its pieces).
  • Each feed/2 scans carry <> chunk once. Indivisible units that are cut by the chunk end are not consumed but carried: a partial UTF-8 character (≤ 3 bytes), a \uXXXX escape or surrogate pair (≤ 11 bytes), a literal prefix (≤ 4 bytes), /, or the * of */. Strings and numbers keep their pieces in the token; at the end of the feed, pieces that reference the chunk are copied (:binary.copy/1) so the state never keeps a chunk alive. On OTP 28+, slices of up to 64 bytes (including the carry) are already standalone binaries and are not copied again.
  • Valid prefixes wait, invalid prefixes fail immediately: tru, 1e+, "\u00, or a UTF-8 lead byte wait, while trx, {"a" 1, or an overlong UTF-8 lead (E0 80) fail at once. Numbers are emitted only when a byte that cannot continue them arrives, or at finish/1.
  • Events are paths plus values, never partial values; string_deltas adds preview fragments for string values that contain only complete characters and escapes.
  • Duplicate keys are detected per object from the set of keys seen so far, so memory grows with the number of keys of open objects as well as the depth.
  • Limits (max_depth 1024, max_document_bytes 64 MiB, max_token_bytes 8 MiB) return :parse / :limit_exceeded errors. The 1024-digit integer limit of Formata.JSON.Backend also applies.
  • Error codes: :syntax, :incomplete (input ended early; reported by finish/1), :duplicate_key, :limit_exceeded. Acceptance always matches Formata.decode/3; codes and some offsets can differ because the stream reports the first problem in input order (for example a duplicate key before a later syntax error).
  • decode_stream/2 is a Stream.resource/3 that pulls one chunk at a time through a suspended Enumerable.reduce/3, so the upstream is read only on demand and is halted after an error or when the consumer stops.

Performance

Profiled with :tprof (call time, call memory), process reductions, and GC statistics; reproduced with MIX_ENV=test mix run bench/json_stream.exs. The cost model is: one scan per byte, a fixed cost per feed/2 call, and the events themselves (each event carries its own path list).

Bottlenecks found in 0.2.0 and what changed in 0.2.1:

  • Position tracking (largest cost). Every feed/2 re-traversed the consumed bytes with a binary comprehension to count code points: 32% of the time and 45% of all allocated words at 4 KiB chunks. Positions are now advanced by a loop over 7-byte words (and 4-byte words for short chunks) using SWAR bit tricks on BEAM small integers: one mask for "all ASCII", the exact zero-byte test for "no newline", and a mask plus shift-and-add popcount for UTF-8 continuation bytes. Only words containing a newline are handled byte by byte, once. Columns still count code points.
  • A quadratic copy per number. On each number terminator the scanner rebuilt <<b, rest::bits>>, which copies the rest of the chunk, so cost grew with chunk size times the number of numbers (1.19 µs per number in a 396 KiB chunk versus 0.10 µs in 256-byte chunks). It now resumes through a sub-binary of the chunk. This is why large chunks were slower than 256-byte chunks.
  • Sub-binary allocation per token. dispatch passed the remaining input under two names to a function that first matched on the grammar mode, so the compiler could not reuse the binary match context (ERL_COMPILER_OPTIONS=bin_opt_info reported "BINARY CREATED"). Each mode now has its own function whose first argument is the input.
  • State copies. Each grammar transition copied the 15-field parser struct. Per-token state (mode, stack, depth, token) now lives in a 4-field record inside the struct, which is updated once per feed; string pieces and the decoded size are loop arguments, so escapes no longer rebuild the token.
  • Paths. Event paths were rebuilt by walking the stack. Each frame now caches its container path, and a member's key path is shared by its :key and :value events.
  • Strings. Unescaped ASCII is scanned eight bytes per step; a string made of a single run is detached without iolist_to_binary/1.
  • Not bottlenecks: settle (≤ 3%) and the decode_stream/2 adapter (≤ 10% over a manual feed/2 loop at 1-byte chunks, about 0 for larger chunks).

Before (0.2.0) and after (0.2.1) on Elixir 1.20.4 / OTP 28, Apple M1 Max. Median of 5–15 runs, each in a fresh process. reductions counts BEAM reductions of the parsing process; alloc (kw) is the thousands of heap words reclaimed by GC during the run (a proxy for allocation).

workloadchunkingmodechunksbefore msafter msspeedupbefore reductionsafter reductionsbefore alloc (kw)after alloc (kw)
structured (396 KiB)whole documentevents141.3723.751.74x239975783302483301886
structured (396 KiB)65536-byte chunksevents750.7316.003.17x204416683286389002291
structured (396 KiB)4096-byte chunksevents10050.8713.673.72x285135091151789352310
structured (396 KiB)random 1-256 B chunksevents321622.3310.542.12x160135883904196372610
structured (396 KiB)random 1-16 B chunksevents4776742.7023.031.85x28424712120045138145958
structured (396 KiB)1-byte chunksevents406052172.8086.542.00x12967277117770545654634000
pretty (143 KiB)whole documentevents16.974.081.71x4298452980371067933
pretty (143 KiB)4096-byte chunksevents366.704.901.37x42012333078112621056
pretty (143 KiB)random 1-16 B chunksevents1722713.197.271.81x89069571601439282058
llm_tool (7 KiB)whole documentevents10.260.132.00x19772106839816
llm_tool (7 KiB)whole documentevents + string_deltas10.280.132.15x20507107819615
llm_tool (7 KiB)whole documentdecode_stream10.270.132.08x20444114599916
llm_tool (7 KiB)random 1-256 B chunksevents510.380.172.24x216671191311425
llm_tool (7 KiB)random 1-256 B chunksevents + string_deltas510.380.182.11x224771217511424
llm_tool (7 KiB)random 1-256 B chunksdecode_stream510.370.162.31x229861321711326
llm_tool (7 KiB)random 1-16 B chunksevents8600.790.481.65x462973722019990
llm_tool (7 KiB)random 1-16 B chunksevents + string_deltas8600.810.451.80x509233894920992
llm_tool (7 KiB)random 1-16 B chunksdecode_stream8600.840.491.71x5478845851214106
llm_tool (7 KiB)1-byte chunksevents73423.481.742.00x246833223910995589
llm_tool (7 KiB)1-byte chunksevents + string_deltas73423.611.712.11x2781012358731055613
llm_tool (7 KiB)1-byte chunksdecode_stream73423.711.822.04x3135282898001107724
long_string (332 KiB)whole documentevents111.654.462.61x9270154341824295720
long_string (332 KiB)whole documentevents + string_deltas112.744.332.94x9819224333924295720
long_string (332 KiB)4096-byte chunksevents8411.213.962.83x9906934324614312728
long_string (332 KiB)4096-byte chunksevents + string_deltas8411.183.952.83x9922834324184313728
long_string (332 KiB)random 1-16 B chunksevents3998231.2015.412.02x2051745166426388153946
long_string (332 KiB)random 1-16 B chunksevents + string_deltas3998229.5715.611.89x2214939173417091064188
escaped_unicode (89 KiB)whole documentevents13.021.312.31x3955223066851622381
escaped_unicode (89 KiB)random 1-16 B chunksevents108779.435.121.84x74655767206827541406
deep (3 KiB)whole documentevents19.103.132.91x863541449799623
deep (3 KiB)random 1-16 B chunksevents4264.173.211.30x91425352194173954

Reference rows, same document, complete value without events (after):

workloadchunkingdecoderms
structured (396 KiB)whole documentFormata.decode/36.4
structured (396 KiB)whole document:json.decode/13.1
structured (396 KiB)random 1-256 B chunks:json.decode_start/3 + decode_continue/24.3
llm_tool (7 KiB)random 1-16 B chunks:json.decode_start/3 + decode_continue/26.1
llm_tool (7 KiB)1-byte chunks:json.decode_start/3 + decode_continue/250.4
llm_tool (7 KiB)random 1-16 B chunksre-decode the buffer after each chunk57.9
long_string (332 KiB)whole documentFormata.decode/32.0
long_string (332 KiB)4096-byte chunks:json.decode_start/3 + decode_continue/248.4

Interpretation:

  • Large and medium chunks (4–64 KiB, 1–256-byte token deltas) of compact JSON improved 2.1–3.7×, long strings 2.6–2.9×, 1-byte chunks 2.0×, and reductions fell up to 19× (deep nesting).
  • Small token deltas (1–16 bytes) improved 1.65–2.0×. The remaining cost is a fixed ~0.3–0.4 µs per feed/2 (one machine call, one struct update, position, settling the token); fewer, larger chunks are cheaper.
  • Pretty-printed JSON improved less (1.4–1.8×): its cost is mostly whitespace, scanned byte by byte, and newlines, which take the per-byte path of position tracking.
  • A whole document in a single feed/2 is dominated by garbage collection of the event list returned at once (about 40,000 events for the 396 KiB document): with a pre-sized heap (min_heap_size: 2_000_000) the same run takes about 11 ms instead of 22 ms. Feeding chunks keeps the live event list small.
  • Event paths are forward lists, so each nested container needs its own copy of its path: memory for events is O(depth) per event and O(depth²) for a document nested depth levels deep (the deep workload, 400 levels: 2,001 events holding 652k words). Its time is dominated by heap growth for those paths, not by parsing (reductions fell 17–19×, time 1.3–2.9×). :max_depth bounds it.
  • The one-shot decoders remain faster when the whole document is available; the stream exists for input that arrives incrementally.

OTP :json as a backend

:json.decode_start/3 and :json.decode_continue/2 (OTP 27+) were evaluated as a replacement and rejected:

  • A pending token is re-parsed from its start on every continuation, so a long string streamed in small chunks is quadratic: a 256 KB string takes 1.5 s in 16-byte chunks versus 0.2 ms whole, and the 332 KiB long_string document takes 48 ms in 4 KiB chunks.
  • It emits nothing until a value completes (there is no key callback), so it cannot produce the current :key/:start_* events as input arrives.
  • It keeps duplicate keys ({"a":1,"a":2} decodes to %{"a" => 1}), reports errors as exceptions such as {:invalid_byte, 93} without a position, and supports neither comments nor trailing commas.

It is much faster for complete input, which is what Formata.decode/3 is for; a hybrid would add a second parser without serving the streaming case.

Phoenix JSON library

Phoenix requires decode!/1, encode!/1, and encode_to_iodata!/1 of its :json_library (the raising functions of Elixir's JSON); Plug's JSON parser calls decode!/1, or decode!/2 through an MFA tuple. Formata.JSON implements them:

  • Decoding shares the strict decoder with the portable codec, so request bodies and socket messages get the same defaults: RFC 8259, string keys, and duplicate keys rejected. decode/2 returns structured errors and decode!/2 raises the first Formata.Error.
  • Encoding delegates to Elixir's JSON rather than Formata's strict encoder. Phoenix and LiveView pass atom keys and values and application structs; the strict portable domain rejects those by design. Structs use the standard JSON.Encoder protocol, so existing @derive JSON.Encoder declarations keep working; the internal Formata.JSON.Backend fork and its protocol stay private.
  • Application-only decode options add floats: &Decimal.new/1 (or another one-argument callback) and objects: :ordered_objects. The number callback receives the original fractional/exponent token, without float conversion; integers remain integers and callback exceptions propagate. Decimal stays a consumer dependency, with no compile-time optional integration.
  • The existing ordered-object implementation is public as Formata.JSON.OrderedObject, with Access, Enumerable, and Elixir's JSON.Encoder. Duplicate detection still walks every object before returning either maps or ordered objects. Ordered encoding uses OTP's checked key-value-list encoder to reject encoded key collisions.
  • pretty: true formats Elixir JSON's encoded iodata with the existing JSON formatter, preserving its protocol support and ordered pairs. Unknown or invalid options raise Formata.Error (:unsupported / :invalid_option). The portable codec's value domain and accepted options are unchanged.
  • Checked end to end with Phoenix 1.8.16 and Plug 1.20.3 (Plug.Parsers, Phoenix.Controller.json/2, Phoenix.Socket.V2.JSONSerializer).

JSON5

  • Formata.Codec.JSON5 has its own recursive-descent parser. The JSON decoder was not extended with JSON5 switches, so JSON5 semantics (quotes, identifiers, hexadecimal, Infinity/NaN) cannot leak into :json. No regular-expression rewriting into JSON is used; the only regular expressions classify a single non-ASCII code point by Unicode general category for IdentifierName.
  • Special values map to :infinity, :negative_infinity, and :nan, the atoms already used by TOML and YAML. The encoder accepts them in addition to the JSON domain and writes Infinity, -Infinity, NaN.
  • Converting to :json keeps the JSON encoder's rule: non-finite numbers are :unrepresentable at their path unless a transform (for example :special_floats_to_strings) is requested. JSON Schema validation rejects them as :non_json_value.
  • :json5 takes no decode options; comments/trailing_commas are rejected because JSON5 always allows both.
  • Finite literals outside the float range are errors, not Infinity; underflow to 0.0 follows :erlang.binary_to_float/1, as in :json.