# Partial JSON Parser
Sometimes we need **LLM (Large Language Models)** to produce **structural information** instead of natural language. The easiest way is to use JSON.
But before receiving the last token of response, the JSON is broken, which means you can't use `JSON.parse` to decode it. But we still want to stream the data to the user.
Here comes `partial-json-parser`, a lightweight and customizable library for parsing partial JSON strings. Here is a [demo](https://promplate.dev/partial-json-parser).
(Note that there is [a JavaScript implementation](https://github.com/promplate/partial-json-parser-js) too)
## Installation
```sh
pip install partial-json-parser # or poetry / pdm / conda
```
`partial-json-parser` is implemented purely in Python, with good type hints.
## Usage
### Importing the library
You can import the `loads` function and the `Allow` object from the library like this:
```py
from partial_json_parser import loads, Allow
```
The `Allow` object is just an Enum for options. It determines what types can be partial. types not included in `allow` only appears after its completion can be ensured.
### Parsing complete / partial JSON strings
The `loads` function works just like the built-in `json.loads` when parsing a complete JSON string:
```py
result = loads('{"key":"value"}')
print(result) # Outputs: {'key': 'value'}
```
You can parse a partial JSON string by passing an additional parameter to the `loads` function. This parameter is a **bitwise OR** of the constants from the `Allow` flag:
(Note that you can directly import the constants you need from `partial-json-parser.options`)
```py
from partial_json_parser import loads, Allow
from partial_json_parser.options import STR, OBJ
result = loads('{"key": "v', STR | OBJ)
print(result) # Outputs: {'key': 'v'}
```
In this example, `Allow.STR` tells the parser that it's okay if a string is incomplete, and `Allow.OBJ` tells the parser so as a dict. The parser then try to return as much data as it can.
If you don't allow partial strings, then it will not add `"key"` to the object because `"v` is not close:
```py
result = loads('{"key": "v', OBJ)
print(result) # Outputs: {}
result = loads('{"key": "value"', OBJ)
print(result) # Outputs: {'key': 'value'}
```
Similarity, you can parse partial lists or even partial special values if you allow it:
(Note that `allow` defaults to `Allow.ALL`)
```py
result = loads('[ {"key1": "value1", "key2": [ "value2')
print(result) # Outputs: [{'key1': 'value1', 'key2': ['value2']}]
result = loads("-Inf")
print(result) # Outputs: -inf
```
### Handling malformed JSON
If the JSON string is malformed, the `parse` function will throw an error:
```py
loads("wrong") # MalformedJSON: Malformed node or string on line 1
```
## API Reference
### loads(json_string, [allow_partial])
- `json_string` `<string>`: The JSON string to parse.
- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).
Returns the parsed Python value.
### Allow
Enum class that specifies what kind of partialness is allowed during JSON parsing. It has the following members:
- `STR`: Allow partial string.
- `NUM`: Allow partial number.
- `ARR`: Allow partial array.
- `OBJ`: Allow partial object.
- `NULL`: Allow partial null.
- `BOOL`: Allow partial boolean.
- `NAN`: Allow partial NaN.
- `INFINITY`: Allow partial Infinity.
- `_INFINITY`: Allow partial -Infinity.
- `INF`: Allow both partial Infinity and -Infinity.
- `SPECIAL`: Allow all special values.
- `ATOM`: Allow all atomic values.
- `COLLECTION`: Allow all collection values.
- `ALL`: Allow all values.
## Testing
To run the tests for this library, you should clone the repository and install the dependencies:
```sh
git clone https://github.com/promplate/partial-json-parser.git
cd partial-json-parser
pdm install
```
Then, you can run the tests using [Hypothesis](https://hypothesis.works/) and [Pytest](https://pytest.org/):
```sh
pdm test
```
Please note that while we strive to cover as many edge cases as possible, it's always possible that some cases might not be covered.
## License
This project is licensed under the MIT License.
Raw data
{
"_id": null,
"home_page": "https://promplate.dev/partial-json-parser",
"name": "partial-json-parser",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "JSON parser LLM nlp",
"author": null,
"author_email": "Muspi Merol <me@promplate.dev>",
"download_url": "https://files.pythonhosted.org/packages/e6/ae/9bfaa043b4543f808686e8ef36e84495bbbd3d46e92d8cd7b170422c9649/partial_json_parser-0.2.0.tar.gz",
"platform": null,
"description": "# Partial JSON Parser\n\nSometimes we need **LLM (Large Language Models)** to produce **structural information** instead of natural language. The easiest way is to use JSON.\n\nBut before receiving the last token of response, the JSON is broken, which means you can't use `JSON.parse` to decode it. But we still want to stream the data to the user.\n\nHere comes `partial-json-parser`, a lightweight and customizable library for parsing partial JSON strings. Here is a [demo](https://promplate.dev/partial-json-parser).\n\n(Note that there is [a JavaScript implementation](https://github.com/promplate/partial-json-parser-js) too)\n\n## Installation\n\n```sh\npip install partial-json-parser # or poetry / pdm / conda\n```\n\n`partial-json-parser` is implemented purely in Python, with good type hints.\n\n## Usage\n\n### Importing the library\n\nYou can import the `loads` function and the `Allow` object from the library like this:\n\n```py\nfrom partial_json_parser import loads, Allow\n```\n\nThe `Allow` object is just an Enum for options. It determines what types can be partial. types not included in `allow` only appears after its completion can be ensured.\n\n### Parsing complete / partial JSON strings\n\nThe `loads` function works just like the built-in `json.loads` when parsing a complete JSON string:\n\n```py\nresult = loads('{\"key\":\"value\"}')\nprint(result) # Outputs: {'key': 'value'}\n```\n\nYou can parse a partial JSON string by passing an additional parameter to the `loads` function. This parameter is a **bitwise OR** of the constants from the `Allow` flag:\n\n(Note that you can directly import the constants you need from `partial-json-parser.options`)\n\n```py\nfrom partial_json_parser import loads, Allow\nfrom partial_json_parser.options import STR, OBJ\n\nresult = loads('{\"key\": \"v', STR | OBJ)\nprint(result) # Outputs: {'key': 'v'}\n```\n\nIn this example, `Allow.STR` tells the parser that it's okay if a string is incomplete, and `Allow.OBJ` tells the parser so as a dict. The parser then try to return as much data as it can.\n\nIf you don't allow partial strings, then it will not add `\"key\"` to the object because `\"v` is not close:\n\n```py\nresult = loads('{\"key\": \"v', OBJ)\nprint(result) # Outputs: {}\n\nresult = loads('{\"key\": \"value\"', OBJ)\nprint(result) # Outputs: {'key': 'value'}\n```\n\nSimilarity, you can parse partial lists or even partial special values if you allow it:\n\n(Note that `allow` defaults to `Allow.ALL`)\n\n```py\nresult = loads('[ {\"key1\": \"value1\", \"key2\": [ \"value2')\nprint(result) # Outputs: [{'key1': 'value1', 'key2': ['value2']}]\n\nresult = loads(\"-Inf\")\nprint(result) # Outputs: -inf\n```\n\n### Handling malformed JSON\n\nIf the JSON string is malformed, the `parse` function will throw an error:\n\n```py\nloads(\"wrong\") # MalformedJSON: Malformed node or string on line 1\n```\n\n## API Reference\n\n### loads(json_string, [allow_partial])\n\n- `json_string` `<string>`: The JSON string to parse.\n- `allow_partial` `<Allow | int>`: Specify what kind of partialness is allowed during JSON parsing (default: `Allow.ALL`).\n\nReturns the parsed Python value.\n\n### Allow\n\nEnum class that specifies what kind of partialness is allowed during JSON parsing. It has the following members:\n\n- `STR`: Allow partial string.\n- `NUM`: Allow partial number.\n- `ARR`: Allow partial array.\n- `OBJ`: Allow partial object.\n- `NULL`: Allow partial null.\n- `BOOL`: Allow partial boolean.\n- `NAN`: Allow partial NaN.\n- `INFINITY`: Allow partial Infinity.\n- `_INFINITY`: Allow partial -Infinity.\n- `INF`: Allow both partial Infinity and -Infinity.\n- `SPECIAL`: Allow all special values.\n- `ATOM`: Allow all atomic values.\n- `COLLECTION`: Allow all collection values.\n- `ALL`: Allow all values.\n\n## Testing\n\nTo run the tests for this library, you should clone the repository and install the dependencies:\n\n```sh\ngit clone https://github.com/promplate/partial-json-parser.git\ncd partial-json-parser\npdm install\n```\n\nThen, you can run the tests using [Hypothesis](https://hypothesis.works/) and [Pytest](https://pytest.org/):\n\n```sh\npdm test\n```\n\nPlease note that while we strive to cover as many edge cases as possible, it's always possible that some cases might not be covered.\n\n## License\n\nThis project is licensed under the MIT License.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Parse partial JSON generated by LLM",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://promplate.dev/partial-json-parser",
"Repository": "https://github.com/promplate/partial-json-parser"
},
"split_keywords": [
"json",
"parser",
"llm",
"nlp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fd6e99220d6d20ff25fc8b128cc9c299ec27ca828bc1b19c3b8cc6a67d204d9c",
"md5": "138d4a32eb094171f9fd864f1ba5d3fd",
"sha256": "9c80509d1d7a2ae84a627a245bd1a0a9c7eb6309fab02328b61f1c89d07379f8"
},
"downloads": -1,
"filename": "partial_json_parser-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "138d4a32eb094171f9fd864f1ba5d3fd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 5985,
"upload_time": "2024-03-19T23:48:27",
"upload_time_iso_8601": "2024-03-19T23:48:27.415405Z",
"url": "https://files.pythonhosted.org/packages/fd/6e/99220d6d20ff25fc8b128cc9c299ec27ca828bc1b19c3b8cc6a67d204d9c/partial_json_parser-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e6ae9bfaa043b4543f808686e8ef36e84495bbbd3d46e92d8cd7b170422c9649",
"md5": "b04c604e00bab2f7b9dd22ad05e3f680",
"sha256": "8e17255b0b7a0af565c42962fcf93ec64b0ef2c9b1987df1e6df401aca528c21"
},
"downloads": -1,
"filename": "partial_json_parser-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "b04c604e00bab2f7b9dd22ad05e3f680",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 6291,
"upload_time": "2024-03-19T23:48:29",
"upload_time_iso_8601": "2024-03-19T23:48:29.263205Z",
"url": "https://files.pythonhosted.org/packages/e6/ae/9bfaa043b4543f808686e8ef36e84495bbbd3d46e92d8cd7b170422c9649/partial_json_parser-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-19 23:48:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "promplate",
"github_project": "partial-json-parser",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "partial-json-parser"
}
JSON
