avro-preprocessor


Nameavro-preprocessor JSON
Version 0.5.0 PyPI version JSON
download
home_pagehttps://gitlab.com/Jaumo/avro-preprocessor
SummaryA preprocessor for Avro Schemata
upload_time2024-03-14 20:47:40
maintainer
docs_urlNone
authorJaumo GmbH
requires_python
licenseApache2
keywords avro preprocessor schema schemas schemata
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # Avro-Preprocessor




This Python project provides a preprocessor for Avro resources. 

The preprocessor is made up of a list of modules.
Each module, in the specified order, performs an operation over the entire list of Avro resources. 

The input is a directory tree (namespace-like) in which each file is either a `domain_message`s or a `type`.

The output is another directory tree, with the same structure as input, in which Avro resources
underwent the list of preprocessing operations. 

Input Avro resources (`domain_message` or `type`) have `exavsc` as file extension.
NEW: they can also be `yaml` files with the same Avro extensions as `exavsc`!
You can even mix `exavsc` and `yaml` files if passed `-ie exavsc,yexavsc`.
Output Avro resources have `avsc` as file extension.

## Schema definition

Two different kinds of data structures are used in this project: *domain messages* and *types*.

#### Domain messages

Domain messages are Avro schemas that can be sent over a Kafka topic.
In general, multiple messages can be associated to each Kafka topic. 
Domain messages are placed in

`schema/com/example/message_schema/domain/<domain_context>/<record_name>`

#### Types

Types are reusable structures that can be referenced in a domain message.
They are placed in

`schema/com/example/message_schema/type/<typename>`

Domain messages *cannot* be reused (i.e. referenced by other domain messages).

### Naming rules 

- Directory structure must match the namespaces used inside the message / type definitions
- Records are defined in camel case
- Properties are defined in snake case

### Kafka topics and Avro subjects

Each directory `<domain_context>` under `schema/com/example/message_schema/domain/`
represents a Kafka topic. All schemas under a specific topic directory (and only them)
will be available to be sent on the corresponding Kafka topic, i.e. they will be registered
on the schema registry.

For subject naming we follow the guidelines defined by Confluent to 
[put several event types in the same Kafka topic](
https://www.confluent.io/blog/put-several-event-types-kafka-topic/); 
in particular we use the strategy 
`io.confluent.kafka.serializers.subject.TopicRecordNameStrategy`.

This behaviour is defined in `avro_naming.py`. 

##### Topic name
```
domain.<domain_context>
```

##### Avro subject names
```
domain.<domain_context>-<fully_qualified_record_name>
```

##### Example
For schema directory:
```
schema/com/example/message_schema/domain/user/
```

Topic name:
```
domain.user
```

List of Avro subjects names:
```
domain.user-com.example.message_schema.domain.user.UserCreated
domain.user-com.example.message_schema.domain.user.UserUpdated
domain.user-com.example.message_schema.domain.user.UserDeleted
```


## List of available preprocessing modules

The following modules should be applied in the order shown as follows.
However, they are all optional.

- `DocumentationCondenser`: JSON strings don't support multi lines inside an editor. This makes
it inconvenient to write long documentation for some fields. This module adds support for
lists as `doc` field in Avro schemas (they will be joined in the output). 
- `NamespaceChecker`: Checks if a resource namespace corresponds to the directory tree structure.
- `DocumentationChecker`: Checks if the documentation corresponds to specific rules:
    1. (sub) schemas with a "name" property MUST have a "doc" property.
    2. (sub) schemas without a "name" property MUST NOT have a "doc" property.
    3. If two (sub) schemas have the same "doc", they must be identical.
- `DefaultValueNullChecker`: Makes sure no default values other than null are used.
- `NamespaceChecker`: Checks if a resource namespace corresponds to the directory tree structure.
properties in `snake_case`.
- `AvroOrderChecker`: Check if the input schemas are sorted according to a fixed order 
(see `avro_sorter.py`). Fails if they are not.
- `MetadataAdder`: Adds the `Metadata` `type` (specified as argument on the command line) as first
field of every domain schema. Use the `--metadata-exclude` option to exclude a comma-separated list
of namespaces.
- `SchemaDependenciesChecker`: Calculates all resources dependencies and assert no cycles are present.
- `NullableOptionalExpander`: Allows for a compact definition of `nullable_optionals`
for "partial entity updates", see its specific section below.
- `LogicalTypeChecker`: Checks for correct usage of logicalType property.
- `ReferencesExpander`: While Apache's `avro-tools` is able to deal with references to other
resources in different files, Confluent's schema registry can only take one JSON as input, for 
each schema. Therefore, references to `type`s in a `domain_message` have to be expanded. 
But with a caveat:
only the first reference ("depth-first", NOT "breadth-first"!) *must* be expanded. All subsequent
references *must not* be expanded, i.e. they have to remain references. 
- `AvroSorter`: For the sake of readability and clearness, JSON fields of Avro resources are
rearranged according to a fixed order (see `avro_sorter.py`). 
- `KeysGenerator`: Generates Avro Schemas for Kafka Keys automatically for every topic based
on the `partition-field` property of schema fields.
- `SchemaMappingGenerator`: Outputs a JSON map `fully qualified schema -> kafka topic` to the file 
`schema-mapping.json`. It also gathers the field names having custom property
`partition-field` and writes them in `schema-mapping.json`. This is done so that Kafka producers
know which set of fields to hash to decide which kafka partition to send a message to.
This module also checks that partition keys are consistent within a given Kafka topic.
Furthermore, gathers the field names of `logicalType == user-id-fields` and stores them in
`schema-mapping.json`. These are meant to be those fields related to users so that they
can be indexed separately (e.g. for GDPR.)
- `DeprecationMappingGenerator`: Outputs a JSON map with info about deprecated schemas/fields to the file 
`deprecation-mapping.json`. This is later used for code generation. Also removes "deprecated" and
"deprecated-symbols" properties from the final schemas.
- `JavaClassesCreator`: Creates Java classes using Apache's `avro-tools`. It is also a very 
convenient way to check correctness of Avro resources.
- `SchemaRegistryChecker`: Check if a schema is compatible with the latest version of the 
same schema in the Schema registry.
- `SchemaRegistrar`: Register an Avro resources (both `domain_message`s AND `type`s!)
for the appropriate
Kafka topic in the schema registry. `type`s are registered for a dummy topic so that we can
check compatibility for them as well (it would not be possible otherwise).
See the documentation in `schema_registrar.py` for further documentation of `subject` 
naming and topics.
Autogenerated keys or the generic key for all schemas (`-k` option from the command line) are
registered as well.


### Installation
```bash
pip install avro-preprocessor
```

### Example Usage

Download Avro tools JAR
```bash
sh download_avro-tools.sh
```

For help:
```bash
avropreprocessor.py -h
```

It is possible to define the `key_subject_name_strategy` with the `-ksns` switch.
Possible values are`RecordNameStrategy` and `TopicRecordNameStrategy` (default).
Only relevant if the `SchemaRegistryChecker` and/or `SchemaRegistrar` modules are active as well.

All modules usage
```bash
avropreprocessor.py -i schema/ -o extended-schema/ -v -s 'http://localhost:8081' -t com.jaumo.message_schema.type -d com.jaumo.message_schema.type.Metadata -p schema-mapping.json -ie 'exavsc' -a ./avro-tools-1.9.0.jar
```

No java classes, no schema registry
```bash
avropreprocessor.py -i schema/ -o build/schema/ -v -s 'http://localhost:8081' -t com.jaumo.message_schema.type -d com.jaumo.message_schema.type.Metadata -p build/schema-mapping.json -m DocumentationCondenser NamespaceChecker DocumentationChecker DefaultValueNullChecker NamesChecker AvroOrderChecker MetadataAdder SchemaDependenciesChecker NullableOptionalExpander ReferencesExpander AvroSorter SchemaMappingGenerator DeprecationMappingGenerator SchemaRegistryChecker
```

Only register to the schema registry
```bash
avropreprocessor.py -i build/schema/ -o /tmp/ -v -s 'http://localhost:8081' -t com.jaumo.message_schema.type -ie 'avsc' -m SchemaRegistryChecker SchemaRegistrar
```

Reorder the *input* schemas according to the order defined in `sort_avro.py`
```bash
avropreprocessor.py -i schema/ -o schema/ -v -t com.jaumo.message_schema.type -ie 'exavsc' -oe 'exavsc' -m AvroSorter
```

Convert input schemas from `exavsc` to `yaml`
```bash
avropreprocessor.py -i schema/ -o schema_yaml/ -v -t com.jaumo.message_schema.type -ie 'exavsc' -oe 'yexavsc' -if 'json' -of 'yaml' -m DocumentationCondenser
```

Then use the yaml schemas as input
```bash
avropreprocessor.py -i schema_yaml/ -o build/schema/ -ie 'yexavsc' -if 'yaml' ...
```

## `nullable_optional`s: a convenient solution to the `Set-as-null` and `null-because-not-present` issue

As `protobuf`, `avro` does not distinguish between these two cases, 
unless some sort of wrapper is used.
The simple solution is to send the entire content of an event every time 
(i.e. always complete updates, never partial updates).

However, since partial updates can still be useful, the module `NullableOptionalExpander` automatises 
the following process. Let's consider a simple Avro schema for user updates:

```json
{
    "namespace": "com.jaumo",
    "type": "record",
    "name": "UserUpdate",
    "doc": "Update user event",
    "fields": [
        {
            "name": "id",
            "doc": "The id",
            "type": "int"
        },
        {
            "name": "name",
            "doc": "The name of the user",
            "type": "string"
        },
        {
            "name": "age",
            "doc": "The age of the user",
            "type": "int"
        },
        {
            "name": "email",
            "doc": "The email of the user",
            "type": "string"
        }
    ]
}
```

This schema does not support `null` values at all. In `avro`, they can be supported as follows:

```json
{
    "namespace": "com.jaumo",
    "type": "record",
    "name": "UserUpdate",
    "doc": "Update user event",
    "fields": [
        {
            "name": "id",
            "doc": "The id",
            "type": "int"
        },
        {
            "name": "name",
            "doc": "The name of the user",
            "type": ["null", "string"],
            "default": null
        },
        {
            "name": "age",
            "doc": "The age of the user",
            "type": ["null", "int"],
            "default": null
        },
        {
            "name": "email",
            "doc": "The email of the user",
            "type": ["null", "string"],
            "default": null
        }
    ]
}
```

With the schema above, both `set-as-null` and `null-because-not-present` are just null,
so they are still ambiguous. To distinguish them, we can wrap around a record type 
(which in Java will be a specific class). Showing the `email` property only, this would be

```json
{
    "name": "email",
    "doc": "The email of the user",
    "default": null,
    "type": [
        "null",
        {
            "name": "OptionalString",
            "type": "record",
            "fields": [
                {
                    "name": "value",
                    "default": null,
                    "type": [
                        "null",
                        "string"
                    ]
                }
            ]
        }
    ]
}
```

Note how the name of the wrapper record (`"OptionalString"`, above)
comes from the prefix `Optional` and the capitalised version of the Avro type field (`string`).
The name of the inner property is always `"value"` (this is done automatically, see below).

Now we have three cases
- `email` set to null means `null-because-not-present`
- `email.value` set to null means `set-as-null`
- `email.value` set to string

Since there's a lot of boilerplate and thus this is error-prone, the `nullable_optional` 
is simple extension to the Avro syntax that adds the property `"nullable_optional": true`. 
If a field has this property set to 
`true`, the field is expanded with its wrapper. The schema above then becomes:

```json
{
    "name": "email",
    "doc": "The email of the user",
    "nullable_optional": true,
    "type": "string"
}
``` 

and it is saved to a `.exavsc` file.

Note how this step is completely **language independent**: its output is a 
completely standard Avro schema that can be used by any library or framework. 

            

Raw data

            {
    "_id": null,
    "home_page": "https://gitlab.com/Jaumo/avro-preprocessor",
    "name": "avro-preprocessor",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "avro,preprocessor,schema,schemas,schemata",
    "author": "Jaumo GmbH",
    "author_email": "nicola.bova@jaumo.com",
    "download_url": "https://files.pythonhosted.org/packages/81/68/8a24bef98d79259aec92533dba2389bb600238bb0b9401e44b37ee3b839a/avro-preprocessor-0.5.0.tar.gz",
    "platform": null,
    "description": "# Avro-Preprocessor\n\n\n\n\nThis Python project provides a preprocessor for Avro resources. \n\nThe preprocessor is made up of a list of modules.\nEach module, in the specified order, performs an operation over the entire list of Avro resources. \n\nThe input is a directory tree (namespace-like) in which each file is either a `domain_message`s or a `type`.\n\nThe output is another directory tree, with the same structure as input, in which Avro resources\nunderwent the list of preprocessing operations. \n\nInput Avro resources (`domain_message` or `type`) have `exavsc` as file extension.\nNEW: they can also be `yaml` files with the same Avro extensions as `exavsc`!\nYou can even mix `exavsc` and `yaml` files if passed `-ie exavsc,yexavsc`.\nOutput Avro resources have `avsc` as file extension.\n\n## Schema definition\n\nTwo different kinds of data structures are used in this project: *domain messages* and *types*.\n\n#### Domain messages\n\nDomain messages are Avro schemas that can be sent over a Kafka topic.\nIn general, multiple messages can be associated to each Kafka topic. \nDomain messages are placed in\n\n`schema/com/example/message_schema/domain/<domain_context>/<record_name>`\n\n#### Types\n\nTypes are reusable structures that can be referenced in a domain message.\nThey are placed in\n\n`schema/com/example/message_schema/type/<typename>`\n\nDomain messages *cannot* be reused (i.e. referenced by other domain messages).\n\n### Naming rules \n\n- Directory structure must match the namespaces used inside the message / type definitions\n- Records are defined in camel case\n- Properties are defined in snake case\n\n### Kafka topics and Avro subjects\n\nEach directory `<domain_context>` under `schema/com/example/message_schema/domain/`\nrepresents a Kafka topic. All schemas under a specific topic directory (and only them)\nwill be available to be sent on the corresponding Kafka topic, i.e. they will be registered\non the schema registry.\n\nFor subject naming we follow the guidelines defined by Confluent to \n[put several event types in the same Kafka topic](\nhttps://www.confluent.io/blog/put-several-event-types-kafka-topic/); \nin particular we use the strategy \n`io.confluent.kafka.serializers.subject.TopicRecordNameStrategy`.\n\nThis behaviour is defined in `avro_naming.py`. \n\n##### Topic name\n```\ndomain.<domain_context>\n```\n\n##### Avro subject names\n```\ndomain.<domain_context>-<fully_qualified_record_name>\n```\n\n##### Example\nFor schema directory:\n```\nschema/com/example/message_schema/domain/user/\n```\n\nTopic name:\n```\ndomain.user\n```\n\nList of Avro subjects names:\n```\ndomain.user-com.example.message_schema.domain.user.UserCreated\ndomain.user-com.example.message_schema.domain.user.UserUpdated\ndomain.user-com.example.message_schema.domain.user.UserDeleted\n```\n\n\n## List of available preprocessing modules\n\nThe following modules should be applied in the order shown as follows.\nHowever, they are all optional.\n\n- `DocumentationCondenser`: JSON strings don't support multi lines inside an editor. This makes\nit inconvenient to write long documentation for some fields. This module adds support for\nlists as `doc` field in Avro schemas (they will be joined in the output). \n- `NamespaceChecker`: Checks if a resource namespace corresponds to the directory tree structure.\n- `DocumentationChecker`: Checks if the documentation corresponds to specific rules:\n    1. (sub) schemas with a \"name\" property MUST have a \"doc\" property.\n    2. (sub) schemas without a \"name\" property MUST NOT have a \"doc\" property.\n    3. If two (sub) schemas have the same \"doc\", they must be identical.\n- `DefaultValueNullChecker`: Makes sure no default values other than null are used.\n- `NamespaceChecker`: Checks if a resource namespace corresponds to the directory tree structure.\nproperties in `snake_case`.\n- `AvroOrderChecker`: Check if the input schemas are sorted according to a fixed order \n(see `avro_sorter.py`). Fails if they are not.\n- `MetadataAdder`: Adds the `Metadata` `type` (specified as argument on the command line) as first\nfield of every domain schema. Use the `--metadata-exclude` option to exclude a comma-separated list\nof namespaces.\n- `SchemaDependenciesChecker`: Calculates all resources dependencies and assert no cycles are present.\n- `NullableOptionalExpander`: Allows for a compact definition of `nullable_optionals`\nfor \"partial entity updates\", see its specific section below.\n- `LogicalTypeChecker`: Checks for correct usage of logicalType property.\n- `ReferencesExpander`: While Apache's `avro-tools` is able to deal with references to other\nresources in different files, Confluent's schema registry can only take one JSON as input, for \neach schema. Therefore, references to `type`s in a `domain_message` have to be expanded. \nBut with a caveat:\nonly the first reference (\"depth-first\", NOT \"breadth-first\"!) *must* be expanded. All subsequent\nreferences *must not* be expanded, i.e. they have to remain references. \n- `AvroSorter`: For the sake of readability and clearness, JSON fields of Avro resources are\nrearranged according to a fixed order (see `avro_sorter.py`). \n- `KeysGenerator`: Generates Avro Schemas for Kafka Keys automatically for every topic based\non the `partition-field` property of schema fields.\n- `SchemaMappingGenerator`: Outputs a JSON map `fully qualified schema -> kafka topic` to the file \n`schema-mapping.json`. It also gathers the field names having custom property\n`partition-field` and writes them in `schema-mapping.json`. This is done so that Kafka producers\nknow which set of fields to hash to decide which kafka partition to send a message to.\nThis module also checks that partition keys are consistent within a given Kafka topic.\nFurthermore, gathers the field names of `logicalType == user-id-fields` and stores them in\n`schema-mapping.json`. These are meant to be those fields related to users so that they\ncan be indexed separately (e.g. for GDPR.)\n- `DeprecationMappingGenerator`: Outputs a JSON map with info about deprecated schemas/fields to the file \n`deprecation-mapping.json`. This is later used for code generation. Also removes \"deprecated\" and\n\"deprecated-symbols\" properties from the final schemas.\n- `JavaClassesCreator`: Creates Java classes using Apache's `avro-tools`. It is also a very \nconvenient way to check correctness of Avro resources.\n- `SchemaRegistryChecker`: Check if a schema is compatible with the latest version of the \nsame schema in the Schema registry.\n- `SchemaRegistrar`: Register an Avro resources (both `domain_message`s AND `type`s!)\nfor the appropriate\nKafka topic in the schema registry. `type`s are registered for a dummy topic so that we can\ncheck compatibility for them as well (it would not be possible otherwise).\nSee the documentation in `schema_registrar.py` for further documentation of `subject` \nnaming and topics.\nAutogenerated keys or the generic key for all schemas (`-k` option from the command line) are\nregistered as well.\n\n\n### Installation\n```bash\npip install avro-preprocessor\n```\n\n### Example Usage\n\nDownload Avro tools JAR\n```bash\nsh download_avro-tools.sh\n```\n\nFor help:\n```bash\navropreprocessor.py -h\n```\n\nIt is possible to define the `key_subject_name_strategy` with the `-ksns` switch.\nPossible values are`RecordNameStrategy` and `TopicRecordNameStrategy` (default).\nOnly relevant if the `SchemaRegistryChecker` and/or `SchemaRegistrar` modules are active as well.\n\nAll modules usage\n```bash\navropreprocessor.py -i schema/ -o extended-schema/ -v -s 'http://localhost:8081' -t com.jaumo.message_schema.type -d com.jaumo.message_schema.type.Metadata -p schema-mapping.json -ie 'exavsc' -a ./avro-tools-1.9.0.jar\n```\n\nNo java classes, no schema registry\n```bash\navropreprocessor.py -i schema/ -o build/schema/ -v -s 'http://localhost:8081' -t com.jaumo.message_schema.type -d com.jaumo.message_schema.type.Metadata -p build/schema-mapping.json -m DocumentationCondenser NamespaceChecker DocumentationChecker DefaultValueNullChecker NamesChecker AvroOrderChecker MetadataAdder SchemaDependenciesChecker NullableOptionalExpander ReferencesExpander AvroSorter SchemaMappingGenerator DeprecationMappingGenerator SchemaRegistryChecker\n```\n\nOnly register to the schema registry\n```bash\navropreprocessor.py -i build/schema/ -o /tmp/ -v -s 'http://localhost:8081' -t com.jaumo.message_schema.type -ie 'avsc' -m SchemaRegistryChecker SchemaRegistrar\n```\n\nReorder the *input* schemas according to the order defined in `sort_avro.py`\n```bash\navropreprocessor.py -i schema/ -o schema/ -v -t com.jaumo.message_schema.type -ie 'exavsc' -oe 'exavsc' -m AvroSorter\n```\n\nConvert input schemas from `exavsc` to `yaml`\n```bash\navropreprocessor.py -i schema/ -o schema_yaml/ -v -t com.jaumo.message_schema.type -ie 'exavsc' -oe 'yexavsc' -if 'json' -of 'yaml' -m DocumentationCondenser\n```\n\nThen use the yaml schemas as input\n```bash\navropreprocessor.py -i schema_yaml/ -o build/schema/ -ie 'yexavsc' -if 'yaml' ...\n```\n\n## `nullable_optional`s: a convenient solution to the `Set-as-null` and `null-because-not-present` issue\n\nAs `protobuf`, `avro` does not distinguish between these two cases, \nunless some sort of wrapper is used.\nThe simple solution is to send the entire content of an event every time \n(i.e. always complete updates, never partial updates).\n\nHowever, since partial updates can still be useful, the module `NullableOptionalExpander` automatises \nthe following process. Let's consider a simple Avro schema for user updates:\n\n```json\n{\n    \"namespace\": \"com.jaumo\",\n    \"type\": \"record\",\n    \"name\": \"UserUpdate\",\n    \"doc\": \"Update user event\",\n    \"fields\": [\n        {\n            \"name\": \"id\",\n            \"doc\": \"The id\",\n            \"type\": \"int\"\n        },\n        {\n            \"name\": \"name\",\n            \"doc\": \"The name of the user\",\n            \"type\": \"string\"\n        },\n        {\n            \"name\": \"age\",\n            \"doc\": \"The age of the user\",\n            \"type\": \"int\"\n        },\n        {\n            \"name\": \"email\",\n            \"doc\": \"The email of the user\",\n            \"type\": \"string\"\n        }\n    ]\n}\n```\n\nThis schema does not support `null` values at all. In `avro`, they can be supported as follows:\n\n```json\n{\n    \"namespace\": \"com.jaumo\",\n    \"type\": \"record\",\n    \"name\": \"UserUpdate\",\n    \"doc\": \"Update user event\",\n    \"fields\": [\n        {\n            \"name\": \"id\",\n            \"doc\": \"The id\",\n            \"type\": \"int\"\n        },\n        {\n            \"name\": \"name\",\n            \"doc\": \"The name of the user\",\n            \"type\": [\"null\", \"string\"],\n            \"default\": null\n        },\n        {\n            \"name\": \"age\",\n            \"doc\": \"The age of the user\",\n            \"type\": [\"null\", \"int\"],\n            \"default\": null\n        },\n        {\n            \"name\": \"email\",\n            \"doc\": \"The email of the user\",\n            \"type\": [\"null\", \"string\"],\n            \"default\": null\n        }\n    ]\n}\n```\n\nWith the schema above, both `set-as-null` and `null-because-not-present` are just null,\nso they are still ambiguous. To distinguish them, we can wrap around a record type \n(which in Java will be a specific class). Showing the `email` property only, this would be\n\n```json\n{\n    \"name\": \"email\",\n    \"doc\": \"The email of the user\",\n    \"default\": null,\n    \"type\": [\n        \"null\",\n        {\n            \"name\": \"OptionalString\",\n            \"type\": \"record\",\n            \"fields\": [\n                {\n                    \"name\": \"value\",\n                    \"default\": null,\n                    \"type\": [\n                        \"null\",\n                        \"string\"\n                    ]\n                }\n            ]\n        }\n    ]\n}\n```\n\nNote how the name of the wrapper record (`\"OptionalString\"`, above)\ncomes from the prefix `Optional` and the capitalised version of the Avro type field (`string`).\nThe name of the inner property is always `\"value\"` (this is done automatically, see below).\n\nNow we have three cases\n- `email` set to null means `null-because-not-present`\n- `email.value` set to null means `set-as-null`\n- `email.value` set to string\n\nSince there's a lot of boilerplate and thus this is error-prone, the `nullable_optional` \nis simple extension to the Avro syntax that adds the property `\"nullable_optional\": true`. \nIf a field has this property set to \n`true`, the field is expanded with its wrapper. The schema above then becomes:\n\n```json\n{\n    \"name\": \"email\",\n    \"doc\": \"The email of the user\",\n    \"nullable_optional\": true,\n    \"type\": \"string\"\n}\n``` \n\nand it is saved to a `.exavsc` file.\n\nNote how this step is completely **language independent**: its output is a \ncompletely standard Avro schema that can be used by any library or framework. \n",
    "bugtrack_url": null,
    "license": "Apache2",
    "summary": "A preprocessor for Avro Schemata",
    "version": "0.5.0",
    "project_urls": {
        "Homepage": "https://gitlab.com/Jaumo/avro-preprocessor"
    },
    "split_keywords": [
        "avro",
        "preprocessor",
        "schema",
        "schemas",
        "schemata"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "37a1acc819337a4967c2b9aaf8b8a5753f27f30727bd89f39ec12cc6440c2cfc",
                "md5": "a91c9c7e2ee085fcb6c48773ce29f638",
                "sha256": "3659a087b459f9a2d0c15077a4f0563dd4e83b83ec474daf512cde5cfb87914c"
            },
            "downloads": -1,
            "filename": "avro_preprocessor-0.5.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a91c9c7e2ee085fcb6c48773ce29f638",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 47241,
            "upload_time": "2024-03-14T20:47:38",
            "upload_time_iso_8601": "2024-03-14T20:47:38.919404Z",
            "url": "https://files.pythonhosted.org/packages/37/a1/acc819337a4967c2b9aaf8b8a5753f27f30727bd89f39ec12cc6440c2cfc/avro_preprocessor-0.5.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "81688a24bef98d79259aec92533dba2389bb600238bb0b9401e44b37ee3b839a",
                "md5": "e8e36f43c8a599599dc8e9555d7db93f",
                "sha256": "2e4a1cb52845c019ca585b1d4c74052a1261ff24bfd2daa3e52e5a2d22104009"
            },
            "downloads": -1,
            "filename": "avro-preprocessor-0.5.0.tar.gz",
            "has_sig": false,
            "md5_digest": "e8e36f43c8a599599dc8e9555d7db93f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 34286,
            "upload_time": "2024-03-14T20:47:40",
            "upload_time_iso_8601": "2024-03-14T20:47:40.839696Z",
            "url": "https://files.pythonhosted.org/packages/81/68/8a24bef98d79259aec92533dba2389bb600238bb0b9401e44b37ee3b839a/avro-preprocessor-0.5.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-03-14 20:47:40",
    "github": false,
    "gitlab": true,
    "bitbucket": false,
    "codeberg": false,
    "gitlab_user": "Jaumo",
    "gitlab_project": "avro-preprocessor",
    "lcname": "avro-preprocessor"
}
        
Elapsed time: 0.31915s