# Partial JSON Parser
Sometimes we need **LLM (Large Language Models)** to produce **structural information** instead of natural language. The easiest way is to use JSON.
But before receiving the last token of response, the JSON is broken, which means you can't use `JSON.parse` to decode it. But we still want to stream the data to the user.
Here comes `partial-json-parser`, a lightweight and customizable library for parsing partial JSON strings. Here is a [demo](https://promplate.dev/partial-json-parser).
(Note that there is [a JavaScript implementation](https://github.com/promplate/partial-json-parser-js) too)
## Installation
```sh
pip install partial-json-parser # or poetry / pdm / uv
```
`partial-json-parser` is implemented purely in Python, with good type hints. It is zero-dependency and works with Python 3.6+.
You can install run its demo playground by installing `rich` too or:
```sh
pip install partial-json-parser[playground]
```
Then run the `json-playground` in your terminal, and you can try the parser interactively.
## Usage
```py
from partial_json_parser import loads
>>> loads('{"key": "v') # {'key': 'v'}
```
Alternatively, you can use `ensure_json` to get the completed JSON string:
```py
from partial_json_parser import ensure_json
>>> ensure_json('{"key": "v') # '{"key": "v"}'
```
### Detailed Usage
You can import the `loads` function and the `Allow` object from the library like this:
```py
from partial_json_parser import loads, Allow
```
The `Allow` object is just an Enum for options. It determines what types can be partial. types not included in `allow` only appears after its completion can be ensured.
### Parsing complete / partial JSON strings
The `loads` function works just like the built-in `json.loads` when parsing a complete JSON string:
```py
result = loads('{"key":"value"}')
print(result) # Outputs: {'key': 'value'}
```
You can parse a partial JSON string by passing an additional parameter to the `loads` function. This parameter is a **bitwise OR** of the constants from the `Allow` flag:
(Note that you can directly import the constants you need from `partial-json-parser`)
```py
from partial_json_parser import loads, Allow, STR, OBJ
result = loads('{"key": "v', STR | OBJ)
print(result) # Outputs: {'key': 'v'}
```
In this example, `Allow.STR` tells the parser that it's okay if a string is incomplete, and `Allow.OBJ` tells the parser so as a dict. The parser then try to return as much data as it can.
If you don't allow partial strings, then it will not add `"key"` to the object because `"v` is not close:
```py
result = loads('{"key": "v', OBJ)
print(result) # Outputs: {}
result = loads('{"key": "value"', OBJ)
print(result) # Outputs: {'key': 'value'}
```
Similarity, you can parse partial lists or even partial special values if you allow it:
(Note that `allow` defaults to `Allow.ALL`)
```py
result = loads('[ {"key1": "value1", "key2": [ "value2')
print(result) # Outputs: [{'key1': 'value1', 'key2': ['value2']}]
result = loads("-Inf")
print(result) # Outputs: -inf
```
### Handling malformed JSON
If the JSON string is malformed, the `parse` function will throw an error:
```py
loads("wrong") # MalformedJSON: Malformed node or string on line 1
```
## API Reference
### loads(json_string, [allow_partial], [parser])
- `json_string` `<string>`: The (incomplete) JSON string to parse.
- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).
- `parser` `(str) -> JSON`: An ordinary JSON parser. Default is `json.loads`.
Complete the JSON string and parse it with `parser` function.
Returns the parsed Python value.
Alias: `decode`, `parse_json`.
### ensure_json(json_string, [allow_partial])
- `json_string` `<string>`: The (incomplete) JSON string to complete.
- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).
Returns the completed JSON string.
### fix(json_string, [allow_partial])
- `json_string` `<string>`: The (incomplete) JSON string to complete.
- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).
Returns a tuple of a slice of the input string and the completion.
Note that this is a low-level API, only useful for debugging and demonstration.
### Allow
Enum class that specifies what kind of partialness is allowed during JSON parsing. It has the following members:
- `STR`: Allow partial string.
- `NUM`: Allow partial number.
- `ARR`: Allow partial array.
- `OBJ`: Allow partial object.
- `NULL`: Allow partial null.
- `BOOL`: Allow partial boolean.
- `NAN`: Allow partial NaN.
- `INFINITY`: Allow partial Infinity.
- `_INFINITY`: Allow partial -Infinity.
- `INF`: Allow both partial Infinity and -Infinity.
- `SPECIAL`: Allow all special values.
- `ATOM`: Allow all atomic values.
- `COLLECTION`: Allow all collection values.
- `ALL`: Allow all values.
## Testing
To run the tests for this library, you should clone the repository and install the dependencies:
```sh
git clone https://github.com/promplate/partial-json-parser.git
cd partial-json-parser
pdm install
```
Then, you can run the tests using [Hypothesis](https://hypothesis.works/) and [Pytest](https://pytest.org/):
```sh
pdm test
```
Please note that while we strive to cover as many edge cases as possible, it's always possible that some cases might not be covered.
## License
This project is licensed under the MIT License.
Raw data
{
"_id": null,
"home_page": "https://promplate.dev/partial-json-parser",
"name": "partial-json-parser",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "JSON, parser, LLM, nlp",
"author": null,
"author_email": "Muspi Merol <me@promplate.dev>",
"download_url": "https://files.pythonhosted.org/packages/26/b7/1bea5eda490b2fbb2a5c7454da5827658f65306fc71d1053d06204f16c4c/partial_json_parser-0.2.1.1.post4.tar.gz",
"platform": null,
"description": "# Partial JSON Parser\n\nSometimes we need **LLM (Large Language Models)** to produce **structural information** instead of natural language. The easiest way is to use JSON.\n\nBut before receiving the last token of response, the JSON is broken, which means you can't use `JSON.parse` to decode it. But we still want to stream the data to the user.\n\nHere comes `partial-json-parser`, a lightweight and customizable library for parsing partial JSON strings. Here is a [demo](https://promplate.dev/partial-json-parser).\n\n(Note that there is [a JavaScript implementation](https://github.com/promplate/partial-json-parser-js) too)\n\n## Installation\n\n```sh\npip install partial-json-parser # or poetry / pdm / uv\n```\n\n`partial-json-parser` is implemented purely in Python, with good type hints. It is zero-dependency and works with Python 3.6+.\n\nYou can install run its demo playground by installing `rich` too or:\n\n```sh\npip install partial-json-parser[playground]\n```\n\nThen run the `json-playground` in your terminal, and you can try the parser interactively.\n\n## Usage\n\n```py\nfrom partial_json_parser import loads\n\n>>> loads('{\"key\": \"v') # {'key': 'v'}\n```\n\nAlternatively, you can use `ensure_json` to get the completed JSON string:\n\n```py\nfrom partial_json_parser import ensure_json\n\n>>> ensure_json('{\"key\": \"v') # '{\"key\": \"v\"}'\n```\n\n### Detailed Usage\n\nYou can import the `loads` function and the `Allow` object from the library like this:\n\n```py\nfrom partial_json_parser import loads, Allow\n```\n\nThe `Allow` object is just an Enum for options. It determines what types can be partial. types not included in `allow` only appears after its completion can be ensured.\n\n### Parsing complete / partial JSON strings\n\nThe `loads` function works just like the built-in `json.loads` when parsing a complete JSON string:\n\n```py\nresult = loads('{\"key\":\"value\"}')\nprint(result) # Outputs: {'key': 'value'}\n```\n\nYou can parse a partial JSON string by passing an additional parameter to the `loads` function. This parameter is a **bitwise OR** of the constants from the `Allow` flag:\n\n(Note that you can directly import the constants you need from `partial-json-parser`)\n\n```py\nfrom partial_json_parser import loads, Allow, STR, OBJ\n\nresult = loads('{\"key\": \"v', STR | OBJ)\nprint(result) # Outputs: {'key': 'v'}\n```\n\nIn this example, `Allow.STR` tells the parser that it's okay if a string is incomplete, and `Allow.OBJ` tells the parser so as a dict. The parser then try to return as much data as it can.\n\nIf you don't allow partial strings, then it will not add `\"key\"` to the object because `\"v` is not close:\n\n```py\nresult = loads('{\"key\": \"v', OBJ)\nprint(result) # Outputs: {}\n\nresult = loads('{\"key\": \"value\"', OBJ)\nprint(result) # Outputs: {'key': 'value'}\n```\n\nSimilarity, you can parse partial lists or even partial special values if you allow it:\n\n(Note that `allow` defaults to `Allow.ALL`)\n\n```py\nresult = loads('[ {\"key1\": \"value1\", \"key2\": [ \"value2')\nprint(result) # Outputs: [{'key1': 'value1', 'key2': ['value2']}]\n\nresult = loads(\"-Inf\")\nprint(result) # Outputs: -inf\n```\n\n### Handling malformed JSON\n\nIf the JSON string is malformed, the `parse` function will throw an error:\n\n```py\nloads(\"wrong\") # MalformedJSON: Malformed node or string on line 1\n```\n\n## API Reference\n\n### loads(json_string, [allow_partial], [parser])\n\n- `json_string` `<string>`: The (incomplete) JSON string to parse.\n- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).\n- `parser` `(str) -> JSON`: An ordinary JSON parser. Default is `json.loads`.\n\nComplete the JSON string and parse it with `parser` function.\n\nReturns the parsed Python value.\n\nAlias: `decode`, `parse_json`.\n\n### ensure_json(json_string, [allow_partial])\n\n- `json_string` `<string>`: The (incomplete) JSON string to complete.\n- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).\n\nReturns the completed JSON string.\n\n### fix(json_string, [allow_partial])\n\n- `json_string` `<string>`: The (incomplete) JSON string to complete.\n- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).\n\nReturns a tuple of a slice of the input string and the completion.\n\nNote that this is a low-level API, only useful for debugging and demonstration.\n\n### Allow\n\nEnum class that specifies what kind of partialness is allowed during JSON parsing. It has the following members:\n\n- `STR`: Allow partial string.\n- `NUM`: Allow partial number.\n- `ARR`: Allow partial array.\n- `OBJ`: Allow partial object.\n- `NULL`: Allow partial null.\n- `BOOL`: Allow partial boolean.\n- `NAN`: Allow partial NaN.\n- `INFINITY`: Allow partial Infinity.\n- `_INFINITY`: Allow partial -Infinity.\n- `INF`: Allow both partial Infinity and -Infinity.\n- `SPECIAL`: Allow all special values.\n- `ATOM`: Allow all atomic values.\n- `COLLECTION`: Allow all collection values.\n- `ALL`: Allow all values.\n\n## Testing\n\nTo run the tests for this library, you should clone the repository and install the dependencies:\n\n```sh\ngit clone https://github.com/promplate/partial-json-parser.git\ncd partial-json-parser\npdm install\n```\n\nThen, you can run the tests using [Hypothesis](https://hypothesis.works/) and [Pytest](https://pytest.org/):\n\n```sh\npdm test\n```\n\nPlease note that while we strive to cover as many edge cases as possible, it's always possible that some cases might not be covered.\n\n## License\n\nThis project is licensed under the MIT License.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Parse partial JSON generated by LLM",
"version": "0.2.1.1.post4",
"project_urls": {
"Homepage": "https://promplate.dev/partial-json-parser",
"Repository": "https://github.com/promplate/partial-json-parser"
},
"split_keywords": [
"json",
" parser",
" llm",
" nlp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9c93791cbea58b8dc27dc76621438d7b030741a4ad3bb5c222363dd01057175a",
"md5": "8b7663dcd08d9512301ba3ef54a86a78",
"sha256": "960144ff29ab9358b69c17dca6e867687438f0dd206ea113677040f14f9ccedd"
},
"downloads": -1,
"filename": "partial_json_parser-0.2.1.1.post4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8b7663dcd08d9512301ba3ef54a86a78",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 9867,
"upload_time": "2024-06-24T12:45:37",
"upload_time_iso_8601": "2024-06-24T12:45:37.733966Z",
"url": "https://files.pythonhosted.org/packages/9c/93/791cbea58b8dc27dc76621438d7b030741a4ad3bb5c222363dd01057175a/partial_json_parser-0.2.1.1.post4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "26b71bea5eda490b2fbb2a5c7454da5827658f65306fc71d1053d06204f16c4c",
"md5": "e12c7d6ef3bf65e8143720bf3ee8ea8d",
"sha256": "a38f11ceb89a86fbfbd0acad4170d417dcb317621c6ec9ab5f2b6652f3f460bf"
},
"downloads": -1,
"filename": "partial_json_parser-0.2.1.1.post4.tar.gz",
"has_sig": false,
"md5_digest": "e12c7d6ef3bf65e8143720bf3ee8ea8d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 9867,
"upload_time": "2024-06-24T12:45:39",
"upload_time_iso_8601": "2024-06-24T12:45:39.317247Z",
"url": "https://files.pythonhosted.org/packages/26/b7/1bea5eda490b2fbb2a5c7454da5827658f65306fc71d1053d06204f16c4c/partial_json_parser-0.2.1.1.post4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-24 12:45:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "promplate",
"github_project": "partial-json-parser",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "partial-json-parser"
}
JSON
