# `Formata.Codec.JSON.Stream`
[🔗](https://github.com/gsmlg-dev/formata/blob/v0.3.5/lib/formata/codec/json/stream.ex#L1)

Incremental parser for a single JSON document that arrives in chunks.

It is built for streamed output such as LLM token deltas. Chunks may be cut
at any byte: inside a UTF-8 character, an escape sequence, a string, a
number, a key, a literal, a comment, or between structural characters. The
parser keeps an immutable state value and, after each chunk, returns the
syntax events it can already confirm. It never re-parses earlier input.

This is not a JSON Lines (NDJSON) parser: the input is one JSON text. To read
JSON Lines, split the records and decode each one with
`Formata.decode(line, :json)`.

## API

    {:ok, parser} = Formata.Codec.JSON.Stream.new()
    {:ok, events, parser} = Formata.Codec.JSON.Stream.feed(parser, ~s({"name": "Ad))
    # events == [{:start_object, []}, {:key, ["name"], "name"}]
    {:ok, events, parser} = Formata.Codec.JSON.Stream.feed(parser, ~s(a", "n": 4))
    # events == [{:value, ["name"], "Ada"}, {:key, ["n"], "n"}]
    {:ok, events} = Formata.Codec.JSON.Stream.finish(parser)
    # events == [{:value, ["n"], 4}, {:end_object, []}]

`feed/2` returning `{:ok, [], parser}` means the input so far is a valid
prefix that needs more bytes. It does **not** mean the document is valid. Only
`finish/1` checks that exactly one complete JSON value was received, followed
by nothing but whitespace (and comments, when enabled).

`decode_stream/2` wraps `new/1`, `feed/2`, and `finish/1` as a lazy stream.

## Events

  * `{:start_object, path}` / `{:end_object, path}`
  * `{:start_array, path}` / `{:end_array, path}`
  * `{:key, path, key}` - a complete decoded key; `path` is the path of the
    field it names
  * `{:value, path, value}` - a complete scalar: string, integer, float,
    boolean, or `nil`
  * `{:string_delta, path, fragment}` - only with `string_deltas: true`

`path` starts at `[]` for the root and lists object keys (strings) and array
indexes (integers), as in `Formata.Error`.

A number is emitted only once a byte that cannot continue it arrives (or at
`finish/1`): `1` might still become `12` or `1.5`. Literals and strings are
emitted as soon as they are complete.

With `string_deltas: true`, string values (not keys) also produce
`{:string_delta, path, fragment}` events with the decoded text received so
far, at the end of each `feed/2` and before the final `{:value, ...}` event.
Fragments contain only complete characters and complete escape sequences;
joined, they equal the final string. Deltas are for previews: the string is
not complete until its `{:value, path, string}` event.

> #### Events are provisional {: .warning}
>
> Events report syntax the parser has recognized so far. Later bytes can
> still make the document invalid, in which case `feed/2` or `finish/1`
> returns an error and every event of this parse must be discarded or marked
> as failed. Do not start side effects (for example an LLM tool call) from
> partial events. Wait for `finish/1` to succeed, then validate the data
> (`Formata.validate/3`) and apply your application's own confirmation.

## Agreement with `Formata.decode/3`

For the same options, `finish/1` succeeds exactly when
`Formata.decode(input, :json, opts)` succeeds on the concatenated input
(within the resource limits below), and the events describe the same values.
Duplicate keys (compared after unescaping) are rejected; the later value
never overrides the earlier one. Error codes can differ: the stream reports
the first problem in input order, and uses `:incomplete` for input that ends
early, where `Formata.decode/3` reports `:syntax`.

## Options

  * `:comments` - accept `//` and `/* */` comments (default `false`)
  * `:trailing_commas` - accept one trailing comma in arrays and objects
    (default `false`)
  * `:string_deltas` - emit `:string_delta` events (default `false`)
  * `:max_depth` - maximum nesting of arrays and objects (default `1024`)
  * `:max_document_bytes` - maximum total input size (default 64 MiB)
  * `:max_token_bytes` - maximum size of a single string (decoded) or number
    (default 8 MiB)

Comments and trailing commas behave exactly as in `Formata.Codec.JSON`.
JSON5 is not supported here.

## Errors

Errors are `Formata.Error` values with `kind: :parse`, `format: :json`, and
`backend: Formata.Codec.JSON.Stream`:

  * `:syntax` - a byte that cannot continue a valid document
  * `:incomplete` - `finish/1` was called on a valid but unfinished prefix
    (open string, container, escape, number, comment, or no value at all)
  * `:duplicate_key` - with `details: %{key: key}` and the object's path
  * `:limit_exceeded` - a resource limit was hit, with `details: %{limit: name}`

`location` holds the byte `offset` from the start of the whole stream, and the
1-based `line` and `column` (in Unicode code points); chunk boundaries do not
reset them. `path` is the location in the data where the error occurred.

After an error, discard the parser.

## Memory and cost

Each byte is scanned once. The parser keeps the container stack (with each
container's path), the token in progress (an unfinished string or number,
bounded by `:max_token_bytes`), at most 11 carried bytes of an incomplete
character, escape, or literal, and the keys already seen in each open object
(needed to reject duplicates). It does not keep the input. String and number
pieces that reference a chunk are copied, so a parser state does not keep
chunk binaries alive.

Each `feed/2` call has a small fixed cost, so fewer, larger chunks are
cheaper than many tiny ones. All events of a chunk are returned together:
feeding a large document in one chunk materializes all of its events at
once. Event paths are separate lists, so their memory grows with nesting
depth. See `docs/design.md` for measurements.

# `errors`

```elixir
@type errors() :: [Formata.Error.t(), ...]
```

# `event`

```elixir
@type event() ::
  {:start_object, path()}
  | {:end_object, path()}
  | {:start_array, path()}
  | {:end_array, path()}
  | {:key, path(), String.t()}
  | {:value, path(), String.t() | number() | boolean() | nil}
  | {:string_delta, path(), String.t()}
```

# `path`

```elixir
@type path() :: [String.t() | non_neg_integer()]
```

# `t`

```elixir
@opaque t()
```

Opaque parser state.

# `decode_stream`

```elixir
@spec decode_stream(Enumerable.t(), keyword()) :: Enumerable.t()
```

Lazily parses an enumerable of binary chunks.

Returns a stream of `{:ok, event}` items. If the input is invalid or
incomplete, the last item is `{:error, errors}` and the stream stops; the
upstream enumerable is halted and not read further. Chunks are read one at a
time, only as the stream is consumed. The stream can be consumed once per
enumeration of `chunks`.

    iex> ["[1, ", "2]"]
    ...> |> Formata.Codec.JSON.Stream.decode_stream()
    ...> |> Enum.to_list()
    [
      {:ok, {:start_array, []}},
      {:ok, {:value, [0], 1}},
      {:ok, {:value, [1], 2}},
      {:ok, {:end_array, []}}
    ]

    iex> [~s({"a": 1)] |> Formata.Codec.JSON.Stream.decode_stream() |> Enum.to_list()
    ...> |> List.last()
    ...> |> then(fn {:error, [error]} -> error.code end)
    :incomplete

# `feed`

```elixir
@spec feed(t(), binary()) :: {:ok, [event()], t()} | {:error, errors()}
```

Parses the next chunk.

Returns the events completed by this chunk (possibly none) and the new
parser, or an error if the input can no longer be a valid document.

    iex> {:ok, parser} = Formata.Codec.JSON.Stream.new()
    iex> {:ok, [{:start_array, []}], parser} = Formata.Codec.JSON.Stream.feed(parser, "[tr")
    iex> {:ok, [{:value, [0], true}], _parser} = Formata.Codec.JSON.Stream.feed(parser, "ue,")

# `finish`

```elixir
@spec finish(t()) :: {:ok, [event()]} | {:error, errors()}
```

Ends the input and checks that a complete document was received.

Returns the remaining events (for example a root number, or the closing of
the root container) or an error if the input is incomplete.

    iex> {:ok, parser} = Formata.Codec.JSON.Stream.new()
    iex> {:ok, [], parser} = Formata.Codec.JSON.Stream.feed(parser, "-12")
    iex> Formata.Codec.JSON.Stream.finish(parser)
    {:ok, [{:value, [], -12}]}

    iex> {:ok, parser} = Formata.Codec.JSON.Stream.new()
    iex> {:ok, _events, parser} = Formata.Codec.JSON.Stream.feed(parser, ~s({"a": "x))
    iex> {:error, [%Formata.Error{code: :incomplete}]} = Formata.Codec.JSON.Stream.finish(parser)

# `new`

```elixir
@spec new(keyword()) :: {:ok, t()} | {:error, errors()}
```

Creates a parser. See the module documentation for options.

    iex> {:ok, _parser} = Formata.Codec.JSON.Stream.new(comments: true)
    iex> {:error, [%Formata.Error{code: :invalid_option}]} =
    ...>   Formata.Codec.JSON.Stream.new(json5: true)

---

*Consult [api-reference.md](api-reference.md) for complete listing*
