Incremental parser for a single JSON document that arrives in chunks.
It is built for streamed output such as LLM token deltas. Chunks may be cut at any byte: inside a UTF-8 character, an escape sequence, a string, a number, a key, a literal, a comment, or between structural characters. The parser keeps an immutable state value and, after each chunk, returns the syntax events it can already confirm. It never re-parses earlier input.
This is not a JSON Lines (NDJSON) parser: the input is one JSON text. To read
JSON Lines, split the records and decode each one with
Formata.decode(line, :json).
API
{:ok, parser} = Formata.Codec.JSON.Stream.new()
{:ok, events, parser} = Formata.Codec.JSON.Stream.feed(parser, ~s({"name": "Ad))
# events == [{:start_object, []}, {:key, ["name"], "name"}]
{:ok, events, parser} = Formata.Codec.JSON.Stream.feed(parser, ~s(a", "n": 4))
# events == [{:value, ["name"], "Ada"}, {:key, ["n"], "n"}]
{:ok, events} = Formata.Codec.JSON.Stream.finish(parser)
# events == [{:value, ["n"], 4}, {:end_object, []}]feed/2 returning {:ok, [], parser} means the input so far is a valid
prefix that needs more bytes. It does not mean the document is valid. Only
finish/1 checks that exactly one complete JSON value was received, followed
by nothing but whitespace (and comments, when enabled).
decode_stream/2 wraps new/1, feed/2, and finish/1 as a lazy stream.
Events
{:start_object, path}/{:end_object, path}{:start_array, path}/{:end_array, path}{:key, path, key}- a complete decoded key;pathis the path of the field it names{:value, path, value}- a complete scalar: string, integer, float, boolean, ornil{:string_delta, path, fragment}- only withstring_deltas: true
path starts at [] for the root and lists object keys (strings) and array
indexes (integers), as in Formata.Error.
A number is emitted only once a byte that cannot continue it arrives (or at
finish/1): 1 might still become 12 or 1.5. Literals and strings are
emitted as soon as they are complete.
With string_deltas: true, string values (not keys) also produce
{:string_delta, path, fragment} events with the decoded text received so
far, at the end of each feed/2 and before the final {:value, ...} event.
Fragments contain only complete characters and complete escape sequences;
joined, they equal the final string. Deltas are for previews: the string is
not complete until its {:value, path, string} event.
Events are provisional
Events report syntax the parser has recognized so far. Later bytes can
still make the document invalid, in which case feed/2 or finish/1
returns an error and every event of this parse must be discarded or marked
as failed. Do not start side effects (for example an LLM tool call) from
partial events. Wait for finish/1 to succeed, then validate the data
(Formata.validate/3) and apply your application's own confirmation.
Agreement with Formata.decode/3
For the same options, finish/1 succeeds exactly when
Formata.decode(input, :json, opts) succeeds on the concatenated input
(within the resource limits below), and the events describe the same values.
Duplicate keys (compared after unescaping) are rejected; the later value
never overrides the earlier one. Error codes can differ: the stream reports
the first problem in input order, and uses :incomplete for input that ends
early, where Formata.decode/3 reports :syntax.
Options
:comments- accept//and/* */comments (defaultfalse):trailing_commas- accept one trailing comma in arrays and objects (defaultfalse):string_deltas- emit:string_deltaevents (defaultfalse):max_depth- maximum nesting of arrays and objects (default1024):max_document_bytes- maximum total input size (default 64 MiB):max_token_bytes- maximum size of a single string (decoded) or number (default 8 MiB)
Comments and trailing commas behave exactly as in Formata.Codec.JSON.
JSON5 is not supported here.
Errors
Errors are Formata.Error values with kind: :parse, format: :json, and
backend: Formata.Codec.JSON.Stream:
:syntax- a byte that cannot continue a valid document:incomplete-finish/1was called on a valid but unfinished prefix (open string, container, escape, number, comment, or no value at all):duplicate_key- withdetails: %{key: key}and the object's path:limit_exceeded- a resource limit was hit, withdetails: %{limit: name}
location holds the byte offset from the start of the whole stream, and the
1-based line and column (in Unicode code points); chunk boundaries do not
reset them. path is the location in the data where the error occurred.
After an error, discard the parser.
Memory and cost
Each byte is scanned once. The parser keeps the container stack (with each
container's path), the token in progress (an unfinished string or number,
bounded by :max_token_bytes), at most 11 carried bytes of an incomplete
character, escape, or literal, and the keys already seen in each open object
(needed to reject duplicates). It does not keep the input. String and number
pieces that reference a chunk are copied, so a parser state does not keep
chunk binaries alive.
Each feed/2 call has a small fixed cost, so fewer, larger chunks are
cheaper than many tiny ones. All events of a chunk are returned together:
feeding a large document in one chunk materializes all of its events at
once. Event paths are separate lists, so their memory grows with nesting
depth. See docs/design.md for measurements.
Summary
Functions
Lazily parses an enumerable of binary chunks.
Parses the next chunk.
Ends the input and checks that a complete document was received.
Creates a parser. See the module documentation for options.
Types
@type errors() :: [Formata.Error.t(), ...]
@type path() :: [String.t() | non_neg_integer()]
@opaque t()
Opaque parser state.
Functions
@spec decode_stream(Enumerable.t(), keyword()) :: Enumerable.t()
Lazily parses an enumerable of binary chunks.
Returns a stream of {:ok, event} items. If the input is invalid or
incomplete, the last item is {:error, errors} and the stream stops; the
upstream enumerable is halted and not read further. Chunks are read one at a
time, only as the stream is consumed. The stream can be consumed once per
enumeration of chunks.
iex> ["[1, ", "2]"]
...> |> Formata.Codec.JSON.Stream.decode_stream()
...> |> Enum.to_list()
[
{:ok, {:start_array, []}},
{:ok, {:value, [0], 1}},
{:ok, {:value, [1], 2}},
{:ok, {:end_array, []}}
]
iex> [~s({"a": 1)] |> Formata.Codec.JSON.Stream.decode_stream() |> Enum.to_list()
...> |> List.last()
...> |> then(fn {:error, [error]} -> error.code end)
:incomplete
Parses the next chunk.
Returns the events completed by this chunk (possibly none) and the new parser, or an error if the input can no longer be a valid document.
iex> {:ok, parser} = Formata.Codec.JSON.Stream.new()
iex> {:ok, [{:start_array, []}], parser} = Formata.Codec.JSON.Stream.feed(parser, "[tr")
iex> {:ok, [{:value, [0], true}], _parser} = Formata.Codec.JSON.Stream.feed(parser, "ue,")
Ends the input and checks that a complete document was received.
Returns the remaining events (for example a root number, or the closing of the root container) or an error if the input is incomplete.
iex> {:ok, parser} = Formata.Codec.JSON.Stream.new()
iex> {:ok, [], parser} = Formata.Codec.JSON.Stream.feed(parser, "-12")
iex> Formata.Codec.JSON.Stream.finish(parser)
{:ok, [{:value, [], -12}]}
iex> {:ok, parser} = Formata.Codec.JSON.Stream.new()
iex> {:ok, _events, parser} = Formata.Codec.JSON.Stream.feed(parser, ~s({"a": "x))
iex> {:error, [%Formata.Error{code: :incomplete}]} = Formata.Codec.JSON.Stream.finish(parser)
Creates a parser. See the module documentation for options.
iex> {:ok, _parser} = Formata.Codec.JSON.Stream.new(comments: true)
iex> {:error, [%Formata.Error{code: :invalid_option}]} =
...> Formata.Codec.JSON.Stream.new(json5: true)