dsrecords

Name	dsrecords JSON
Version	0.4.18 JSON
	download
home_page
Summary	Dead-simple and easy-to-hack RecordIO-like data storing. Inspired by MXNet's RecordIO and Protobuf.
upload_time	2023-12-07 03:09:03
maintainer
docs_url	None
author
requires_python
license	MIT
keywords	recordio
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            
# Dead Simple Records


Easy to hack and dead simple RecordIO-like data storing. Inspired by MXNet's RecordIO and Protobuf.

- [Documentations](https://ndgnuh.gitlab.io/dsrecords/)

## (planned) features

- [x] Binary-based data format with index file
- [x] Easy custom serialization schemes
- [x] Common serialization schemes (more TBA)
- [x] Documentation
- [x] Append, update, delete data sample
- [ ] Pack/unpack form
- [ ] Schema serialization

## Quick start
See the overview section in the [documentations](https://ndgnuh.gitlab.io/dsrecords/).

Also, check out [this notebook](https://github.com/ndgnuh/ezrecords/blob/master/Examples.ipynb) for a quick example, this notebook has not been updated in a while though.

## How does it work?

#### Saving

1. Open an empty file, this is the data file.
2. Take whatever data you got, serialize it in a way (for example, write to PNG buffer)
3. Write the serialized data to the said file buffer, store the offset and size of the data
4. Repeat until all data are written
5. Write all the offsets and sizes to another file (index file)

#### Loading

1. Load the data file and the index file.
2. When given an index, use the index file to get the offset and size of the data.
3. Seek the file pointer to the offset, read the exact size and deserialize the data.

## Prebuilt format


## Customized serialization format

            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "dsrecords",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "RecordIO",
    "author": "",
    "author_email": "Hung Nguyen <ndgnuh@protonmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/46/fa/a35826e4d93024e15ec524e44a87db07193cd7a24a32fee2b54fa257aff0/dsrecords-0.4.18.tar.gz",
    "platform": null,
    "description": "\n# Dead Simple Records\n\n\nEasy to hack and dead simple RecordIO-like data storing. Inspired by MXNet's RecordIO and Protobuf.\n\n- [Documentations](https://ndgnuh.gitlab.io/dsrecords/)\n\n## (planned) features\n\n- [x] Binary-based data format with index file\n- [x] Easy custom serialization schemes\n- [x] Common serialization schemes (more TBA)\n- [x] Documentation\n- [x] Append, update, delete data sample\n- [ ] Pack/unpack form\n- [ ] Schema serialization\n\n## Quick start\nSee the overview section in the [documentations](https://ndgnuh.gitlab.io/dsrecords/).\n\nAlso, check out [this notebook](https://github.com/ndgnuh/ezrecords/blob/master/Examples.ipynb) for a quick example, this notebook has not been updated in a while though.\n\n## How does it work?\n\n#### Saving\n\n1. Open an empty file, this is the data file.\n2. Take whatever data you got, serialize it in a way (for example, write to PNG buffer)\n3. Write the serialized data to the said file buffer, store the offset and size of the data\n4. Repeat until all data are written\n5. Write all the offsets and sizes to another file (index file)\n\n#### Loading\n\n1. Load the data file and the index file.\n2. When given an index, use the index file to get the offset and size of the data.\n3. Seek the file pointer to the offset, read the exact size and deserialize the data.\n\n## Prebuilt format\n\n\n## Customized serialization format\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Dead-simple and easy-to-hack RecordIO-like data storing. Inspired by MXNet's RecordIO and Protobuf.",
    "version": "0.4.18",
    "project_urls": {
        "Bug tracker": "https://gitlab.com/ndgnuh/dsrecords/-/issues",
        "Documentation": "https://ndgnuh.gitlab.io/dsrecords/",
        "Homepage": "https://gitlab.com/ndgnuh/dsrecords"
    },
    "split_keywords": [
        "recordio"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "46faa35826e4d93024e15ec524e44a87db07193cd7a24a32fee2b54fa257aff0",
                "md5": "1afef3b5311104ddee15a064766a9fda",
                "sha256": "72ccec9342bc6ae619ddb0498b2f7832835f42fcc6351442d55af99394419f40"
            },
            "downloads": -1,
            "filename": "dsrecords-0.4.18.tar.gz",
            "has_sig": false,
            "md5_digest": "1afef3b5311104ddee15a064766a9fda",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 10143,
            "upload_time": "2023-12-07T03:09:03",
            "upload_time_iso_8601": "2023-12-07T03:09:03.289043Z",
            "url": "https://files.pythonhosted.org/packages/46/fa/a35826e4d93024e15ec524e44a87db07193cd7a24a32fee2b54fa257aff0/dsrecords-0.4.18.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-12-07 03:09:03",
    "github": false,
    "gitlab": true,
    "bitbucket": false,
    "codeberg": false,
    "gitlab_user": "ndgnuh",
    "gitlab_project": "dsrecords",
    "lcname": "dsrecords"
}