| Name | cels JSON |
| Version |
0.3.1
JSON |
| download |
| home_page | None |
| Summary | Patch your YAML and JSON files. |
| upload_time | 2024-10-26 14:30:09 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.7 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center">
<img src="https://raw.githubusercontent.com/pacha/cels/main/docs/cels-logo-header.png" alt="logo" width="100%">
</p>
cels
====
_Command line tool to patch your YAML and JSON files_
## Example
```yaml
# input.yaml
foo:
bar: 1
baz: 2
list:
- a
- b
level: 11
```
```yaml
# patch.yaml
foo:
bar: 100
list {insert}: c
level {delete}: null
```
```yaml
# command: cels patch input.yaml patch.yaml
foo:
bar: 100
baz: 2
list:
- a
- b
- c
```
## Description
Cels is a command-line tool and Python library that enables you to make multiple
modifications to YAML, JSON, or TOML documents. These modifications are based on
changes specified in a _patch_ file. The patch file is written in the same
format as the original data and mirrors its structure. For instance, if you want
to change the `bar` key value from `hello` to `bye` in this example:
```yaml
foo:
bar: hello
```
You simply need to create a patch like this:
```yaml
foo:
bar: bye
```
To run the tool and get the result, you just pass both files as arguments to
the `cels patch` command:
```shell
$ cels patch input.yaml output.yaml
```
For more complex modifications, you can annotate the keys of the patch
document with the operation to perform in the format: `{operation[@index1,
index2, …]}`. For example, to insert an element in the middle of a list, you
only need to specify the new location and the value to insert:
```yaml
# input
list:
- a
- c
# patch
list {insert@1}: b
# output
list:
- a
- b
- c
```
Working with JSON files in Cels is quite similar to working with YAML files. To
illustrate this, let's take a look at how the previous example would be
transformed:
Here's the input file:
```json
{
"list": ["a", "c"]
}
```
And here's the patch file:
```json
{
"list {insert@1}": "b"
}
```
After applying the patch, the result would be:
```json
{
"list": ["a", "b", "c"]
}
```
Cels supports a variety of operations (`set`, `delete`, `delete_value`,
`rename`, `insert`, `extend`, `use`, `link`, `render`), and most of them can
optionally take indices to work with any level of nested lists.
Cells can be beneficial when you possess a base configuration file or manifest
for a system and need to tailor it to various environments (eg. 'development'
and 'production'). Additionally, you can utilize Cells as a Python library and
integrate it into your application. This allows your users to modify their own
configuration in diverse setups (similar to `docker-compose` overrides, for
example).
Refer to [Usage](#usage) for a comprehensive description of all available
operations.
See [Similar Projects](#similar-projects) for a comparison between Cels and
other tools and specifications with similar objectives.
> _The project name_
>
> The name 'Cels' is inspired by the conventional, analog world of animation. In
> that context, 'cels' are transparent sheets featuring drawings that are layered
> atop one another to create the final image. Cels works in a similar way,
> creating the final result by layering the patch files over the input documents.
### Limitations
If you're considering using Cels, please take into account the following limitations:
* Cels is only compatible with documents where the top-level element is
a dictionary or map. That is, won't work with documents where the top-level
element is a list.
* Annotations can only be added to keys that are of the string type.
This isn't a problem with JSON documents, as all keys are inherently strings.
However, in YAML documents, keys aren't necessarily strings. While Cels can
handle YAML documents with non-string keys—and these keys can also appear in
patch files—they cannot be annotated. This means that advanced operations
cannot be applied to them.
## Installation
To install cells, simply use `pip`:
```shell
$ pip install cells
```
## Usage
### The 'patch' command
Cels' main command is `patch`. It takes an input file (the file that you want to
modify) and a patch file (the file that describes the changes to perform). Eg:
```shell
$ cels patch input.yaml patch.yaml
```
The result or running the command is sent to `stdout` by default. To save the
result to a file, just redirect the output or use the `-O` option:
```shell
$ cels patch input.yaml patch.yaml > output.yaml
$ cels patch input.yaml patch.yaml -O output.yaml
```
You can patch YAML, JSON and TOML files. Cels determines the correct format to
use from the file extensions, but you can explicitly set the format for each one
of the files with the `-i`, `-p`, and `-o` parameters (run `cels patch --help`
for more information).
> _Note_: It's possible to utilize various formats within the same command
> execution. For instance, you can use a JSON patch and a YAML input file
> simultaneously if you wish. Although there aren't many reasons to typically do
> this, Cels doesn't place any limitations on the combination of formats used.
### Basic operations
To override values of the input document, you don't typically need to specify
any operation in the patch file:
```yaml
# input
foo:
bar:
a: 1
b: 2
c: 3
# patch
foo:
bar:
b: 200
# output
foo:
bar:
a: 1
b: 200
c: 3
```
As shown in this example, dictionaries in the patch file get merged with
the ones in the input file. However, if you want to override an entire
dictionary, instead of merging it, you can use the operation `set`:
```yaml
# input
foo:
bar:
a: 1
b: 2
c: 3
# patch
foo:
bar {set}:
b: 200
# output
foo:
bar:
b: 200
```
Operations—in their most basic form—are specified by appending the operation
name between curly braces (`{<operation>}`) to the key string. The full
annotation format is:
```
key {<operation>[@<index1>,<index2>,...]}
```
where the indices are provided to modify lists (see below [Working with
lists](#working-with-lists)).
#### 'delete' and 'rename'
The other basic operations include `delete`, which removes a key, and `rename`,
which alters the key's name while preserving its value.
```yaml
# input
foo:
bar: "A Fair Field Full of Folk"
baz:
one: 1
two: 2
# patch
foo:
bar {rename}: newbar
baz {delete}: null
# output
foo:
newbar: "A Fair Field Full of Folk"
```
For the `delete` operation, you can pass any value (in this case `null`) as it
is just ignored by Cels.
## Working with Lists
The majority of operations can utilize an index following an `@` symbol. This
is to indicate that the action should be executed at a specific position within
a list (To know if a given operation supports indices, run `patch describe
operation OPERATION_NAME`, which will provide full information about it).
For example, to change the second element of a list, you can use the `set`
operation:
```yaml
# input
foo:
- a
- b
# patch
foo {set@1}: B
# output
foo:
- a
- B
```
Note that when working with lists:
* Indices start at 0 (meaning the first element is located at index 0).
* Negative indices can be used. For instance, -1 refers to the last element in
the list, -2 to the penultimate one, and so on.
* To specify elements within nested lists, you can provide multiple indices
separated by commas. For example, `1,0` refers to `c` in a nested list like
`[[a, b], [c, d]]`.
### 'insert' and 'extend'
The operations `insert` and `extend` are very useful when working with lists.
`insert` adds one element to a list:
```yaml
# input
foo:
- a
- b
# patch
foo {insert}: c
# output
foo:
- a
- b
- c
```
Whereas, in the case of `extend`, multiple elements are added simultaneously:
```yaml
# input
foo:
- a
- b
# patch
foo {extend}:
- c
- d
# output
foo:
- a
- b
- c
- d
```
In both scenarios, indices can be used to designate the position where you wish
to add elements:
* `{insert@NUM}` and `{extend@NUM}` place the elements _before_ the `NUM`
position. In other words, to insert them at the start of the list, before
any other element, use `{insert@0}` and `{extend@0}`.
* The unique index `_` can be utilized to signify that the elements should be
_appended_ to the list's end (i.e., `{insert@_}`). The `_` can be omitted when
operating on the top-level list, meaning `{insert}` and `{extend}` are
synonymous with `{insert@_}` and `{extend@_}`. However, if you need to append
elements to a nested list's end, it's necessary to use it.
For example, here's how you append an element to the end of a nested list:
```yaml
# input
foo:
- - a
- b
- - c
- d
# patch
foo {insert@1,_}: e
# output
- - a
- b
- - c
- d
- e
```
### 'delete_value'
The operation `delete_value` removes all occurrences of a given value from a list:
```yaml
# input
foo:
- a
- b
- a
# patch
foo {delete_value}: a
# output
foo:
- b
```
It is also possible to delete values from nested lists using indices and delete
entire objects (not only scalars):
```yaml
# input
foo:
- a
- b
- - c
- d
- x: 1
y: 2
# patch
foo {delete_value@2}:
x: 1
y: 2
# output
foo:
- a
- b
- - c
- d
```
### Using variables
It is possible to define variables and then reuse them at different places of
the patch file.
This is a quick example:
```yaml
# input
foo: 1
# patch
my_var {var}: World
foo {use}: my_var
bar {render}: "Hello {{ my_var }}!"
# output
foo: World
bar: "Hello Wold!"
```
As you can see, variables are defined with:
```yaml
key {var}: value
```
where `key` is the name of the variable and `value` is, well, its value.
Variable definitions, while not visible in the output document themselves, can
be utilized through the `use` and `render` operations to insert values in
various locations:
* The `use` operation simply inserts the variable as is. If the variable is a
list or a dictionary, `use` will incorporate it without any modifications.
* The `render` operation allows you to define a
[Jinja](https://jinja.palletsprojects.com/en/3.1.x/templates/) template
string that can reference one or more variables. If the variable is a list or
a dictionary, you can use the `.` or `[]` notation to pinpoint the exact
value you wish to use. Additionally, you can utilize any of the features
offered by the Jinja template language, such as filters or conditional
structures.
The following is a more elaborated example of using variables with the `render`
operation:
```yaml
# patch
my_var {var}:
one: a
two:
- b
- c
foo {render}: "{{ my_var.one }} lowercase, {{ my_var.two[0]|upper }} uppercase"
# output
foo: a lowercase, B uppercase
```
#### Variable scope
Please note that a variable comes with an associated scope:
* The `use` and `render` commands can only reference variables from the same
dictionary in which they are used, or from any parent or ancestor dictionary.
In other words, if you want a variable to be accessible throughout the entire
document, you should define it at its root dictionary.
* If a variable is redefined in a child dictionary, the value in the child
dictionary will take precedence over the one in the parent dictionary.
### Multiple changes for a same key
If you want to perform multiple changes to the same key, you can use the
`change` operation, that takes a list of the modifications to perform:
```yaml
# input
foo: 1
# patch
foo {change}:
- operation: set
value: 100
- operation: rename
value: bar
# output
bar: 100
```
The operations will be executed in the order they are presented. Each item in
the operations list should contain no more than three fields:
* `operation`: This refers to one of the possible operations (`set`, `delete`,
`rename`, `insert`, etc.).
* `value`: The value depends on the type of operation. It may be omitted for
keys that do not require a value, such as `rename`.
* `indices`: This is a list of integers (and `_` for `insert` and `extend`
operations) used to manipulate nested lists. This field can be omitted for
operations that do not intend to modify lists.
In summary, these three fields correspond to those that can be defined in a
standard annotation: `key {operation@indices}: value`.
> _Note_: the notation:
>
> ```
> foo {change}:
> - operation: set
> value: 100
> - operation: rename
> value: bar
> ```
>
> is different from listing the operations one after another:
>
> ```
> foo {set}: 100
> foo {rename}: bar
> ```
>
> In the latter scenario, the second operation would supersede the initial one.
> Therefore, there are minimal instances, if any, where you might want to do
> that.
### Repeating content from the input document
The `link` operation allows you to reference parts of the input document and
reuse them in other parts. For instance, here we use the `link` operation to
remove the `bar` level from the input document:
```yaml
# input
foo:
bar:
one: 1
two: 2
# patch
foo {delete}: null
new-foo {link}: .foo.bar
# result
new-foo:
one: 1
two: 2
```
The `link` operation always takes a path that employs the `.` and `[]` notation
to traverse through the dictionaries and lists within the input document. The
initial `.` symbol signifies the root dictionary of the document. It's important
to note that the path always refers to the unaltered input document, regardless
of any other operations being performed. This means that the value indicated by
the path will always be the original one.
One limitation of the `link` operation is that it just copies a part of the
input document as it is. If you need to reference a value from the input
document but you also require to modify it in some form, you can use a combination
of the `render` operation and `_get` template function. Here's an example:
```yaml
# input
foo:
- one
- two
# patch
foo {render@1}: "{{ _get('.foo[1]') | upper }}"
# output
foo:
- one
- TWO
```
`_get()` takes as parameter the path of the value in the input document that
you want to use (using the same dot notation than with `link`) and returns such
a value.
### Patching dictionaries that are nested in lists
If you need to modify a dictionary within a list, you can utilize the `patch`
operation. In most scenarios, explicitly using `patch` isn't required, as
dictionaries in the patch file are automatically merged with those in the input
document. However, when dealing with dictionaries within lists, it becomes
essential to use `patch` because you need to specify the index that indicates
the dictionary's position within the list. Here's an example:
```yaml
# input
foo:
- a: 1
b: 2
# patch
foo {patch@0}:
a: 100
# result
foo:
- a: 100
b: 2
```
## Changing the annotation format
By default, annotations in Cels appear as `<space>{operation@indices}`. However,
you can customize all the symbols used to better suit your needs by using the
following command parameters:
```shell
$ cels patch input.yaml patch.yaml \
--separator "_" \
--left-marker "(" \
--index-marker "%" \
--right-marker ")"
```
This example changes the format of annotations to: `_(operation%indices)`.
## Getting help
To list all available operations, you can use:
```shell
$ cels list operations
```
To show help for a given operation, including its description and usage
examples, you can use:
```shell
$ cels describe operation OPERATION_NAME
```
Additionally, with the `-v` flag, you can activate the verbose output.
This will display the operation that was used to generate each node in the
output document:
<img src="https://raw.githubusercontent.com/pacha/cels/main/docs/screenshot-verbose-output.png" alt="Cels screenshot" width="50%">
## Using Cels as a Python library
You can use Cels programmatically from your Python code. It provides two basic
functions: `patch_document` and `patch_dictionary`.
### patch_document
`patch_document` allows you to pass JSON, YAML or TOML text for the input and
patch documents, in the same way that they are passed to the `cels` command:
```python
from cels import patch_document
output = patch_document(input_format, input_text, patch_format, patch_text, output_format)
```
`input_format` and `patch_format` are string arguments and
can take any of the following values: `json`, `yaml` or `toml`.
`input_text` and `patch_text` are the raw texts to be used (of course, their
format should match with the parameters above).
Finally, it is possible to specify the format of the output text with
`output_format`, which doesn't necessarily have to match the input formats.
### patch_dictionary
`patch_dictionary` works exactly the same as `patch_document` but the data is
passed already in the format of python dictionaries:
```python
from cels import patch_dictionary
result = patch_dictionary(input_dict, patch_dict)
```
In both cases (`patch_document` and `patch_dictionary`), you can pass
`separator`, `left_marker`, `index_marker` and `right_marker` parameters to
define the format of the key annotations (see [Changing the annotation
format](#changing-the-annotation-format) for more information).
> _Deepcopy_
>
> For performance reasons, Cels' Python functions do not generate a complete copy of the
> output dictionary in memory. Instead, Cels interlaces the input and patch
> nodes, merging them to produce the final result.
>
> This approach works well if you only need to read the output. However, if you
> alter it in-place, you may inadvertently modify nodes of the input dictionary
> at the same time.
>
> If you need to change the output dictionary without affecting the input and
> patch dictionaries, simply create a _deep copy_ of the dictionary returned by
> the Cels functions:
>
> ```python
> from copy import deepcopy
> from cels import patch_dictionary
>
> result = deepcopy(patch_dictionary(input_dict, patch_dict))
> ```
## Similar Projects
### jq and yq
[jq](https://jqlang.github.io/jq/) and [yq](https://mikefarah.gitbook.io/yq/)
are widely used command-line tools for processing JSON and YAML. The key
distinction between these tools and Cels lies in their operational approach.
`jq` and `yq` operate using paths, whereas Cels utilizes patch files.
For example, if you want to alter the value of a key in `yq`, you would specify
it as follows:
```
.foo.bar.baz = "value"
```
In contrast, with Cels, you would write the actual YAML:
```
foo:
bar:
baz: "value"
```
`jq` and `yq` shine when it comes to making specific modifications to a
document. They allow you to pinpoint and alter a deeply nested key with a
single command line. However, their intuitiveness diminishes when multiple
changes are required within a document. In such scenarios, using a patch file
as in Cels may provide a better overall view of the modifications and their
interrelations.
In this discussion, we're primarily focusing on the patching capabilities of
these tools. However, they also offer additional features (like formatting and
data extraction) that Cels does not.
For more details, please refer to their respective documentation.
It's worth noting that `jq` is exclusively for JSON, while `yq` can handle both
JSON and YAML.
### Jsonnet, CUE, YTT
[Jsonnet](https://jsonnet.org/), [CUE](https://cuelang.org/), and
[YTT](https://carvel.dev/ytt/) are fully-fledged languages equipped with import
mechanisms, loops, conditionals, functions, and other programming language
constructs. They are designed as supersets of JSON, as seen in Jsonnet and CUE,
and a superset of YAML in the case of YTT (however, in this case, language
constructs are embedded within comments, ensuring compatibility as a YAML
document). These languages can be an excellent choice if you're tasked with
making complex modifications to JSON or YAML files.
Cels stands out from these solutions due to its simplicity. All you need to do
is annotate each key you wish to modify with the desired operation. While it
may not have the capabilities of Jsonnet, CUE, YTT, or any other configuration
programming language, it still covers a wide range of common use cases while
still being extremely simple to use.
### Starlark and Dhall
[Starlark](https://github.com/bazelbuild/starlark) and
[Dhall](https://dhall-lang.org) share similarities with Jsonnet, CUE, and YTT
as they are all fully-fledged configuration programming languages. However,
they don't extend JSON or YAML. Starlark is a subset of Python, while Dhall has
its own unique syntax (though it does allow exporting to any other format).
As in the case of, Jsonnet, CUE, and YTT, Cels sets itself apart from Starlark
and Dhall by being _not_ a comprehensive language, but rather a simple
collection of annotations. This makes it considerably less complex in
comparison.
### RFC 6902 and RFC 7396
[RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) and [RFC
7396](https://datatracker.ietf.org/doc/html/rfc7396) are both proposed
standards for patching JSON files.
`RFC 6902` defines a JSON structure for defining a list of operations to
be applied sequentially to the original document in order to patch it.
This is an example:
```
[
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
]
```
`RFC 6902` provides a broad spectrum of operations. However, it doesn't
replicate the original document, but merely outlines the operations in a list
format. On the other hand, the patch file in Cels can result in a more compact
document that is easier to read and potentially easier to maintain. Moreover,
Cels isn't limited to JSON support; it also accommodates YAML and TOML.
On the other hand, `RFC 7396` is very similar to Cels. Like Cels, it defines a
patch format that mirrors the original document.
For example, given the following example file:
```
{
"a": "b",
"c": {
"d": "e",
"f": "g"
}
}
```
A RFC 7396 patch may look like this:
```
{
"a":"z",
"c": {
"f": null
}
}
```
Which results in:
```
{
"a": "z",
"c": {
"d": "e",
}
}
```
(Setting a key to `null` deletes it).
For comparison, the equivalent Cels patch would look like:
```
{
"a":"z",
"c": {
"f {remove}": null
}
}
```
These examples illustrate that `RFC 7396` and Cels share many similarities, but
there are key differences:
* `RFC 7396` is limited to leaving a key as is, overwriting it, or deleting it.
In contrast, Cels provides a broader set of options.
* Cels has the capability to manipulate list elements, a feature that `RFC 7396`
lacks.
* `RFC 7396` employs `null` to remove keys from the original dictionary.
However, `null` is a perfectly valid value in a JSON file, which renders
`RFC 7396` incapable of representing certain valid JSON documents.
Specifically, it's impossible to assign a `null` value to a key as it
would be deleted instead.
* While `RFC 7396` only supports JSON, Cels can handle YAML and TOML as well.
The following is a (non-comprehensive) list of `RFC 6902` and `RFC7396` implementations:
* [json-patch](https://github.com/evanphx/json-patch): Go implementation of RFC 6902 and
RFC 7396.
* [python-json-patch](https://github.com/stefankoegl/python-json-patch): Python
implementation of RFC 6902.
* [yaml-diff-patch](https://github.com/grantila/yaml-diff-patch): Command line
and npm package that allows to apply RFC 6902 JSON patches to a YAML document.
* [chbrown/rfc6902](https://github.com/chbrown/rfc6902): TypeScript
implementation of RFC 6902.
## License
Cels is available under the MIT license.
Raw data
{
"_id": null,
"home_page": null,
"name": "cels",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Andr\u00e9s Sope\u00f1a P\u00e9rez <code@ehmm.org>",
"download_url": "https://files.pythonhosted.org/packages/18/82/14819c1cc67c67e909abdbd9824eb756adccd47a1dee1c846f7300716646/cels-0.3.1.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"https://raw.githubusercontent.com/pacha/cels/main/docs/cels-logo-header.png\" alt=\"logo\" width=\"100%\">\n</p>\n\ncels\n====\n\n\n\n\n\n\n_Command line tool to patch your YAML and JSON files_\n\n## Example\n\n```yaml\n# input.yaml\nfoo:\n bar: 1\n baz: 2\nlist:\n- a\n- b\nlevel: 11\n```\n\n```yaml\n# patch.yaml\nfoo:\n bar: 100\nlist {insert}: c\nlevel {delete}: null\n```\n\n```yaml\n# command: cels patch input.yaml patch.yaml\nfoo:\n bar: 100\n baz: 2\nlist:\n- a\n- b\n- c\n```\n\n## Description\n\nCels is a command-line tool and Python library that enables you to make multiple\nmodifications to YAML, JSON, or TOML documents. These modifications are based on\nchanges specified in a _patch_ file. The patch file is written in the same\nformat as the original data and mirrors its structure. For instance, if you want\nto change the `bar` key value from `hello` to `bye` in this example:\n```yaml\nfoo:\n bar: hello\n```\nYou simply need to create a patch like this:\n```yaml\nfoo:\n bar: bye\n```\n\nTo run the tool and get the result, you just pass both files as arguments to\nthe `cels patch` command:\n```shell\n$ cels patch input.yaml output.yaml\n```\n\nFor more complex modifications, you can annotate the keys of the patch\ndocument with the operation to perform in the format: `{operation[@index1,\nindex2, \u2026]}`. For example, to insert an element in the middle of a list, you\nonly need to specify the new location and the value to insert:\n```yaml\n\n# input\nlist:\n- a\n- c\n\n# patch\nlist {insert@1}: b\n\n# output\nlist:\n- a\n- b\n- c\n```\n\nWorking with JSON files in Cels is quite similar to working with YAML files. To\nillustrate this, let's take a look at how the previous example would be\ntransformed:\n\nHere's the input file:\n```json\n{\n \"list\": [\"a\", \"c\"]\n}\n```\nAnd here's the patch file:\n```json\n{\n \"list {insert@1}\": \"b\"\n}\n```\nAfter applying the patch, the result would be:\n```json\n{\n \"list\": [\"a\", \"b\", \"c\"]\n}\n```\n\nCels supports a variety of operations (`set`, `delete`, `delete_value`,\n`rename`, `insert`, `extend`, `use`, `link`, `render`), and most of them can\noptionally take indices to work with any level of nested lists.\n\nCells can be beneficial when you possess a base configuration file or manifest\nfor a system and need to tailor it to various environments (eg. 'development'\nand 'production'). Additionally, you can utilize Cells as a Python library and\nintegrate it into your application. This allows your users to modify their own\nconfiguration in diverse setups (similar to `docker-compose` overrides, for\nexample).\n\nRefer to [Usage](#usage) for a comprehensive description of all available\noperations.\n\nSee [Similar Projects](#similar-projects) for a comparison between Cels and\nother tools and specifications with similar objectives.\n\n> _The project name_\n>\n> The name 'Cels' is inspired by the conventional, analog world of animation. In\n> that context, 'cels' are transparent sheets featuring drawings that are layered\n> atop one another to create the final image. Cels works in a similar way,\n> creating the final result by layering the patch files over the input documents.\n\n### Limitations\n\nIf you're considering using Cels, please take into account the following limitations:\n\n* Cels is only compatible with documents where the top-level element is\n a dictionary or map. That is, won't work with documents where the top-level\n element is a list.\n* Annotations can only be added to keys that are of the string type.\n This isn't a problem with JSON documents, as all keys are inherently strings.\n However, in YAML documents, keys aren't necessarily strings. While Cels can\n handle YAML documents with non-string keys\u2014and these keys can also appear in\n patch files\u2014they cannot be annotated. This means that advanced operations\n cannot be applied to them.\n\n## Installation\n\nTo install cells, simply use `pip`:\n```shell\n$ pip install cells\n```\n\n## Usage\n\n### The 'patch' command\n\nCels' main command is `patch`. It takes an input file (the file that you want to\nmodify) and a patch file (the file that describes the changes to perform). Eg:\n```shell\n$ cels patch input.yaml patch.yaml\n```\nThe result or running the command is sent to `stdout` by default. To save the\nresult to a file, just redirect the output or use the `-O` option:\n```shell\n$ cels patch input.yaml patch.yaml > output.yaml\n$ cels patch input.yaml patch.yaml -O output.yaml\n```\nYou can patch YAML, JSON and TOML files. Cels determines the correct format to\nuse from the file extensions, but you can explicitly set the format for each one\nof the files with the `-i`, `-p`, and `-o` parameters (run `cels patch --help`\nfor more information).\n\n> _Note_: It's possible to utilize various formats within the same command\n> execution. For instance, you can use a JSON patch and a YAML input file\n> simultaneously if you wish. Although there aren't many reasons to typically do\n> this, Cels doesn't place any limitations on the combination of formats used.\n\n### Basic operations\n\nTo override values of the input document, you don't typically need to specify\nany operation in the patch file:\n```yaml\n# input\nfoo:\n bar:\n a: 1\n b: 2\n c: 3\n\n# patch\nfoo:\n bar:\n b: 200\n\n# output\nfoo:\n bar:\n a: 1\n b: 200\n c: 3\n```\nAs shown in this example, dictionaries in the patch file get merged with\nthe ones in the input file. However, if you want to override an entire\ndictionary, instead of merging it, you can use the operation `set`:\n```yaml\n# input\nfoo:\n bar:\n a: 1\n b: 2\n c: 3\n\n# patch\nfoo:\n bar {set}:\n b: 200\n\n# output\nfoo:\n bar:\n b: 200\n```\nOperations\u2014in their most basic form\u2014are specified by appending the operation\nname between curly braces (`{<operation>}`) to the key string. The full\nannotation format is:\n```\nkey {<operation>[@<index1>,<index2>,...]}\n```\nwhere the indices are provided to modify lists (see below [Working with\nlists](#working-with-lists)).\n\n#### 'delete' and 'rename'\n\nThe other basic operations include `delete`, which removes a key, and `rename`,\nwhich alters the key's name while preserving its value.\n```yaml\n# input\nfoo:\n bar: \"A Fair Field Full of Folk\"\n baz:\n one: 1\n two: 2\n\n# patch\nfoo:\n bar {rename}: newbar\n baz {delete}: null\n\n# output\nfoo:\n newbar: \"A Fair Field Full of Folk\"\n```\nFor the `delete` operation, you can pass any value (in this case `null`) as it\nis just ignored by Cels.\n\n## Working with Lists\n\nThe majority of operations can utilize an index following an `@` symbol. This\nis to indicate that the action should be executed at a specific position within\na list (To know if a given operation supports indices, run `patch describe\noperation OPERATION_NAME`, which will provide full information about it).\n\nFor example, to change the second element of a list, you can use the `set`\noperation:\n\n```yaml\n# input\nfoo:\n - a\n - b\n\n# patch\nfoo {set@1}: B\n\n# output\nfoo:\n - a\n - B\n```\n\nNote that when working with lists:\n\n* Indices start at 0 (meaning the first element is located at index 0).\n* Negative indices can be used. For instance, -1 refers to the last element in\n the list, -2 to the penultimate one, and so on.\n* To specify elements within nested lists, you can provide multiple indices\n separated by commas. For example, `1,0` refers to `c` in a nested list like\n `[[a, b], [c, d]]`.\n\n### 'insert' and 'extend'\n\nThe operations `insert` and `extend` are very useful when working with lists.\n\n`insert` adds one element to a list:\n```yaml\n# input\nfoo:\n - a\n - b\n\n# patch\nfoo {insert}: c\n\n# output\nfoo:\n - a\n - b\n - c\n```\nWhereas, in the case of `extend`, multiple elements are added simultaneously:\n```yaml\n# input\nfoo:\n - a\n - b\n\n# patch\nfoo {extend}:\n - c\n - d\n\n# output\nfoo:\n - a\n - b\n - c\n - d\n```\nIn both scenarios, indices can be used to designate the position where you wish\nto add elements:\n\n* `{insert@NUM}` and `{extend@NUM}` place the elements _before_ the `NUM`\n position. In other words, to insert them at the start of the list, before\n any other element, use `{insert@0}` and `{extend@0}`.\n* The unique index `_` can be utilized to signify that the elements should be\n _appended_ to the list's end (i.e., `{insert@_}`). The `_` can be omitted when\n operating on the top-level list, meaning `{insert}` and `{extend}` are\n synonymous with `{insert@_}` and `{extend@_}`. However, if you need to append\n elements to a nested list's end, it's necessary to use it.\n\nFor example, here's how you append an element to the end of a nested list:\n\n```yaml\n# input\nfoo:\n- - a\n - b\n- - c\n - d\n\n# patch\nfoo {insert@1,_}: e\n\n# output\n- - a\n - b\n- - c\n - d\n - e\n```\n\n### 'delete_value'\n\nThe operation `delete_value` removes all occurrences of a given value from a list:\n\n```yaml\n# input\nfoo:\n - a\n - b\n - a\n\n# patch\nfoo {delete_value}: a\n\n# output\nfoo:\n - b\n```\n\nIt is also possible to delete values from nested lists using indices and delete\nentire objects (not only scalars):\n\n```yaml\n# input\nfoo:\n - a\n - b\n - - c\n - d\n - x: 1\n y: 2\n\n\n# patch\nfoo {delete_value@2}:\n x: 1\n y: 2\n\n# output\nfoo:\n - a\n - b\n - - c\n - d\n```\n\n### Using variables\n\nIt is possible to define variables and then reuse them at different places of\nthe patch file.\n\nThis is a quick example:\n```yaml\n# input\nfoo: 1\n\n# patch\nmy_var {var}: World\nfoo {use}: my_var\nbar {render}: \"Hello {{ my_var }}!\"\n\n# output\nfoo: World\nbar: \"Hello Wold!\"\n```\n\nAs you can see, variables are defined with:\n```yaml\nkey {var}: value\n```\nwhere `key` is the name of the variable and `value` is, well, its value.\n\nVariable definitions, while not visible in the output document themselves, can\nbe utilized through the `use` and `render` operations to insert values in\nvarious locations:\n\n* The `use` operation simply inserts the variable as is. If the variable is a\n list or a dictionary, `use` will incorporate it without any modifications.\n* The `render` operation allows you to define a\n [Jinja](https://jinja.palletsprojects.com/en/3.1.x/templates/) template\n string that can reference one or more variables. If the variable is a list or\n a dictionary, you can use the `.` or `[]` notation to pinpoint the exact\n value you wish to use. Additionally, you can utilize any of the features\n offered by the Jinja template language, such as filters or conditional\n structures.\n\nThe following is a more elaborated example of using variables with the `render`\noperation:\n```yaml\n# patch\nmy_var {var}:\n one: a\n two:\n - b\n - c\nfoo {render}: \"{{ my_var.one }} lowercase, {{ my_var.two[0]|upper }} uppercase\"\n\n# output\nfoo: a lowercase, B uppercase\n```\n\n#### Variable scope\n\nPlease note that a variable comes with an associated scope:\n\n* The `use` and `render` commands can only reference variables from the same\n dictionary in which they are used, or from any parent or ancestor dictionary.\n In other words, if you want a variable to be accessible throughout the entire\n document, you should define it at its root dictionary.\n* If a variable is redefined in a child dictionary, the value in the child\n dictionary will take precedence over the one in the parent dictionary.\n\n### Multiple changes for a same key\n\nIf you want to perform multiple changes to the same key, you can use the\n`change` operation, that takes a list of the modifications to perform:\n```yaml\n# input\nfoo: 1\n\n# patch\nfoo {change}:\n - operation: set\n value: 100\n - operation: rename\n value: bar\n\n# output\nbar: 100\n```\nThe operations will be executed in the order they are presented. Each item in\nthe operations list should contain no more than three fields:\n\n* `operation`: This refers to one of the possible operations (`set`, `delete`,\n `rename`, `insert`, etc.).\n* `value`: The value depends on the type of operation. It may be omitted for\n keys that do not require a value, such as `rename`.\n* `indices`: This is a list of integers (and `_` for `insert` and `extend`\n operations) used to manipulate nested lists. This field can be omitted for\n operations that do not intend to modify lists.\n\nIn summary, these three fields correspond to those that can be defined in a\nstandard annotation: `key {operation@indices}: value`.\n\n> _Note_: the notation:\n> \n> ```\n> foo {change}:\n> - operation: set\n> value: 100\n> - operation: rename\n> value: bar\n> ```\n> \n> is different from listing the operations one after another:\n> \n> ```\n> foo {set}: 100\n> foo {rename}: bar\n> ```\n> \n> In the latter scenario, the second operation would supersede the initial one.\n> Therefore, there are minimal instances, if any, where you might want to do\n> that.\n\n### Repeating content from the input document\n\nThe `link` operation allows you to reference parts of the input document and\nreuse them in other parts. For instance, here we use the `link` operation to\nremove the `bar` level from the input document:\n```yaml\n# input\nfoo:\n bar:\n one: 1\n two: 2\n\n# patch\nfoo {delete}: null\nnew-foo {link}: .foo.bar\n\n# result\nnew-foo:\n one: 1\n two: 2\n```\nThe `link` operation always takes a path that employs the `.` and `[]` notation\nto traverse through the dictionaries and lists within the input document. The\ninitial `.` symbol signifies the root dictionary of the document. It's important\nto note that the path always refers to the unaltered input document, regardless\nof any other operations being performed. This means that the value indicated by\nthe path will always be the original one.\n\nOne limitation of the `link` operation is that it just copies a part of the\ninput document as it is. If you need to reference a value from the input\ndocument but you also require to modify it in some form, you can use a combination\nof the `render` operation and `_get` template function. Here's an example:\n\n```yaml\n# input\nfoo:\n - one\n - two\n\n# patch\nfoo {render@1}: \"{{ _get('.foo[1]') | upper }}\"\n\n# output\nfoo:\n - one\n - TWO\n```\n\n`_get()` takes as parameter the path of the value in the input document that\nyou want to use (using the same dot notation than with `link`) and returns such\na value.\n\n### Patching dictionaries that are nested in lists\n\nIf you need to modify a dictionary within a list, you can utilize the `patch`\noperation. In most scenarios, explicitly using `patch` isn't required, as\ndictionaries in the patch file are automatically merged with those in the input\ndocument. However, when dealing with dictionaries within lists, it becomes\nessential to use `patch` because you need to specify the index that indicates\nthe dictionary's position within the list. Here's an example:\n\n```yaml\n# input\nfoo:\n- a: 1\n b: 2\n\n# patch\nfoo {patch@0}:\n a: 100\n\n# result\nfoo:\n- a: 100\n b: 2\n```\n\n## Changing the annotation format\n\nBy default, annotations in Cels appear as `<space>{operation@indices}`. However,\nyou can customize all the symbols used to better suit your needs by using the\nfollowing command parameters:\n```shell\n$ cels patch input.yaml patch.yaml \\\n --separator \"_\" \\\n --left-marker \"(\" \\\n --index-marker \"%\" \\\n --right-marker \")\"\n```\nThis example changes the format of annotations to: `_(operation%indices)`.\n\n## Getting help\n\nTo list all available operations, you can use:\n```shell\n$ cels list operations\n```\n\nTo show help for a given operation, including its description and usage\nexamples, you can use:\n```shell\n$ cels describe operation OPERATION_NAME\n```\n\n\n\nAdditionally, with the `-v` flag, you can activate the verbose output.\nThis will display the operation that was used to generate each node in the\noutput document:\n\n<img src=\"https://raw.githubusercontent.com/pacha/cels/main/docs/screenshot-verbose-output.png\" alt=\"Cels screenshot\" width=\"50%\">\n\n## Using Cels as a Python library\n\nYou can use Cels programmatically from your Python code. It provides two basic\nfunctions: `patch_document` and `patch_dictionary`.\n\n### patch_document\n\n`patch_document` allows you to pass JSON, YAML or TOML text for the input and\npatch documents, in the same way that they are passed to the `cels` command:\n```python\nfrom cels import patch_document\n\noutput = patch_document(input_format, input_text, patch_format, patch_text, output_format)\n```\n\n`input_format` and `patch_format` are string arguments and\ncan take any of the following values: `json`, `yaml` or `toml`.\n\n`input_text` and `patch_text` are the raw texts to be used (of course, their\nformat should match with the parameters above).\n\nFinally, it is possible to specify the format of the output text with\n`output_format`, which doesn't necessarily have to match the input formats.\n\n### patch_dictionary\n\n`patch_dictionary` works exactly the same as `patch_document` but the data is\npassed already in the format of python dictionaries:\n```python\nfrom cels import patch_dictionary\n\nresult = patch_dictionary(input_dict, patch_dict)\n```\n\nIn both cases (`patch_document` and `patch_dictionary`), you can pass\n`separator`, `left_marker`, `index_marker` and `right_marker` parameters to\ndefine the format of the key annotations (see [Changing the annotation\nformat](#changing-the-annotation-format) for more information).\n\n> _Deepcopy_\n>\n> For performance reasons, Cels' Python functions do not generate a complete copy of the\n> output dictionary in memory. Instead, Cels interlaces the input and patch\n> nodes, merging them to produce the final result.\n>\n> This approach works well if you only need to read the output. However, if you\n> alter it in-place, you may inadvertently modify nodes of the input dictionary\n> at the same time.\n>\n> If you need to change the output dictionary without affecting the input and\n> patch dictionaries, simply create a _deep copy_ of the dictionary returned by\n> the Cels functions:\n>\n> ```python\n> from copy import deepcopy\n> from cels import patch_dictionary\n>\n> result = deepcopy(patch_dictionary(input_dict, patch_dict))\n> ```\n\n## Similar Projects\n\n### jq and yq\n\n[jq](https://jqlang.github.io/jq/) and [yq](https://mikefarah.gitbook.io/yq/)\nare widely used command-line tools for processing JSON and YAML. The key\ndistinction between these tools and Cels lies in their operational approach.\n`jq` and `yq` operate using paths, whereas Cels utilizes patch files.\n\nFor example, if you want to alter the value of a key in `yq`, you would specify\nit as follows:\n```\n.foo.bar.baz = \"value\"\n```\nIn contrast, with Cels, you would write the actual YAML:\n```\nfoo:\n bar:\n baz: \"value\"\n```\n`jq` and `yq` shine when it comes to making specific modifications to a\ndocument. They allow you to pinpoint and alter a deeply nested key with a\nsingle command line. However, their intuitiveness diminishes when multiple\nchanges are required within a document. In such scenarios, using a patch file\nas in Cels may provide a better overall view of the modifications and their\ninterrelations.\n\nIn this discussion, we're primarily focusing on the patching capabilities of\nthese tools. However, they also offer additional features (like formatting and\ndata extraction) that Cels does not.\nFor more details, please refer to their respective documentation.\n\nIt's worth noting that `jq` is exclusively for JSON, while `yq` can handle both\nJSON and YAML.\n\n### Jsonnet, CUE, YTT\n\n[Jsonnet](https://jsonnet.org/), [CUE](https://cuelang.org/), and\n[YTT](https://carvel.dev/ytt/) are fully-fledged languages equipped with import\nmechanisms, loops, conditionals, functions, and other programming language\nconstructs. They are designed as supersets of JSON, as seen in Jsonnet and CUE,\nand a superset of YAML in the case of YTT (however, in this case, language\nconstructs are embedded within comments, ensuring compatibility as a YAML\ndocument). These languages can be an excellent choice if you're tasked with\nmaking complex modifications to JSON or YAML files.\n\nCels stands out from these solutions due to its simplicity. All you need to do\nis annotate each key you wish to modify with the desired operation. While it\nmay not have the capabilities of Jsonnet, CUE, YTT, or any other configuration\nprogramming language, it still covers a wide range of common use cases while\nstill being extremely simple to use.\n\n### Starlark and Dhall\n\n[Starlark](https://github.com/bazelbuild/starlark) and\n[Dhall](https://dhall-lang.org) share similarities with Jsonnet, CUE, and YTT\nas they are all fully-fledged configuration programming languages. However,\nthey don't extend JSON or YAML. Starlark is a subset of Python, while Dhall has\nits own unique syntax (though it does allow exporting to any other format).\n\nAs in the case of, Jsonnet, CUE, and YTT, Cels sets itself apart from Starlark\nand Dhall by being _not_ a comprehensive language, but rather a simple\ncollection of annotations. This makes it considerably less complex in\ncomparison.\n\n### RFC 6902 and RFC 7396\n\n[RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) and [RFC\n7396](https://datatracker.ietf.org/doc/html/rfc7396) are both proposed\nstandards for patching JSON files.\n\n`RFC 6902` defines a JSON structure for defining a list of operations to\nbe applied sequentially to the original document in order to patch it.\n\nThis is an example:\n```\n[\n { \"op\": \"remove\", \"path\": \"/a/b/c\" },\n { \"op\": \"add\", \"path\": \"/a/b/c\", \"value\": [ \"foo\", \"bar\" ] },\n { \"op\": \"replace\", \"path\": \"/a/b/c\", \"value\": 42 },\n]\n```\n\n`RFC 6902` provides a broad spectrum of operations. However, it doesn't\nreplicate the original document, but merely outlines the operations in a list\nformat. On the other hand, the patch file in Cels can result in a more compact\ndocument that is easier to read and potentially easier to maintain. Moreover,\nCels isn't limited to JSON support; it also accommodates YAML and TOML.\n\nOn the other hand, `RFC 7396` is very similar to Cels. Like Cels, it defines a\npatch format that mirrors the original document.\n\nFor example, given the following example file:\n```\n{\n \"a\": \"b\",\n \"c\": {\n \"d\": \"e\",\n \"f\": \"g\"\n }\n}\n```\n\nA RFC 7396 patch may look like this:\n```\n{\n \"a\":\"z\",\n \"c\": {\n \"f\": null\n }\n}\n```\nWhich results in:\n```\n{\n \"a\": \"z\",\n \"c\": {\n \"d\": \"e\",\n }\n}\n```\n(Setting a key to `null` deletes it).\n\nFor comparison, the equivalent Cels patch would look like:\n```\n{\n \"a\":\"z\",\n \"c\": {\n \"f {remove}\": null\n }\n}\n```\n\nThese examples illustrate that `RFC 7396` and Cels share many similarities, but\nthere are key differences:\n\n* `RFC 7396` is limited to leaving a key as is, overwriting it, or deleting it.\n In contrast, Cels provides a broader set of options.\n* Cels has the capability to manipulate list elements, a feature that `RFC 7396`\n lacks.\n* `RFC 7396` employs `null` to remove keys from the original dictionary.\n However, `null` is a perfectly valid value in a JSON file, which renders\n `RFC 7396` incapable of representing certain valid JSON documents.\n Specifically, it's impossible to assign a `null` value to a key as it\n would be deleted instead.\n* While `RFC 7396` only supports JSON, Cels can handle YAML and TOML as well.\n\nThe following is a (non-comprehensive) list of `RFC 6902` and `RFC7396` implementations:\n\n* [json-patch](https://github.com/evanphx/json-patch): Go implementation of RFC 6902 and\n RFC 7396.\n* [python-json-patch](https://github.com/stefankoegl/python-json-patch): Python\n implementation of RFC 6902.\n* [yaml-diff-patch](https://github.com/grantila/yaml-diff-patch): Command line\n and npm package that allows to apply RFC 6902 JSON patches to a YAML document.\n* [chbrown/rfc6902](https://github.com/chbrown/rfc6902): TypeScript\n implementation of RFC 6902.\n\n## License\n\nCels is available under the MIT license.\n",
"bugtrack_url": null,
"license": null,
"summary": "Patch your YAML and JSON files.",
"version": "0.3.1",
"project_urls": {
"Bug Tracker": "https://github.com/pacha/cels/issues",
"Homepage": "https://github.com/pacha/cels"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "24a35b6f12f8327a301e1a82cf9a34dba9ca268d3c3765f6c845c0b17562697f",
"md5": "a2b57b5dd2710e5c1fc0678634da58f2",
"sha256": "197f47e0e8e8f80a92c25e55d8987afaa4ea9285829223247beeebc8edcf488e"
},
"downloads": -1,
"filename": "cels-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a2b57b5dd2710e5c1fc0678634da58f2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 43523,
"upload_time": "2024-10-26T14:30:07",
"upload_time_iso_8601": "2024-10-26T14:30:07.307584Z",
"url": "https://files.pythonhosted.org/packages/24/a3/5b6f12f8327a301e1a82cf9a34dba9ca268d3c3765f6c845c0b17562697f/cels-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "188214819c1cc67c67e909abdbd9824eb756adccd47a1dee1c846f7300716646",
"md5": "6d4dd0b4eee3e52e140fc191c5abb1eb",
"sha256": "ddea9582578fe963e08384ae073617f47f5839a88a2e7fd91cd66604446dbb92"
},
"downloads": -1,
"filename": "cels-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "6d4dd0b4eee3e52e140fc191c5abb1eb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 2901071,
"upload_time": "2024-10-26T14:30:09",
"upload_time_iso_8601": "2024-10-26T14:30:09.619538Z",
"url": "https://files.pythonhosted.org/packages/18/82/14819c1cc67c67e909abdbd9824eb756adccd47a1dee1c846f7300716646/cels-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-26 14:30:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pacha",
"github_project": "cels",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "cels"
}
