# ldv
Light Data Versioning
Ldv is a tool for version tracking big files on a remote storage and storing digest files together with code that uses the big files.
A digest file is a small file containing information about the file to version track. The information includes file size in bytes, timestamp of version, hexdigest (the version number), and local and remote filepaths.
Ldv can be run from the command line or from code, using the APIs of the package.
Ldv has no external dependencies on other programs, e.g. Git.
Currently, only AWS is supported as remote storage.
# Getting started
## Installation
This will install the package to version track on AWS remote storage.
```
pip install ldv
```
# Usage
## 1. Authentication
Authenticate with remote storage provider.
```
ldv auth aws -rn <region_name> [-pn <profile name>] [-ec]
```
* -rn is the region name where the bucket is located.
* -pn is the name of the profile specified in ~/.aws/config
* -ec Use credentials stored in environment variables. LDV_AWS_ACCESS_KEY_ID and LDV_AWS_SECRET_ACCESS_KEY must be set. Will not be used if profile is provided.
You either use -pn or -ec, not both.
Example:
```
ldv auth aws -rn eu-west-1 -pn dev-profile
```
## 2. Initialize
Initialize the local base path you want to version track files under. Also specify the remote storage url.
```
ldv init -p <path> -u <url> [-rta] [-du]
```
* -p specifies the local path that contains files to version track.
* -u is the remote storage url, e.g. s3://my_bucket/sub_folder.
* -rta is an optional flag will run version tracking of all files under the path.
* -du is an optional flag will prevent files from being uploaded to remote storage every time you version track them. Use this if you only want to create a digest files locally but not upload the files to remote storage.
Example:
```
ldv init -p local-data-folder -u s3://my_s3_bucket/sub_folder
```
## Version track file
```
ldv versioning track <file path> [-du]
```
* file path to version track. Can be absolute or relative path but must be under path provided when running 'init' command.
* -du is optional flag if you don't want to upload the file.
Example:
```
ldv versioning track local-data-folder/first_file.txt
```
## Version track all files
Version track all files under path specified in the 'init' command.
```
ldv versioning track-all [-du]
```
* -du is optional flag if you don't want to upload the files.
## Download file
Download file from remote storage to local storage.
```
ldv versioning download <file path to digest file> [-v <version>]
```
* -v is the optional version. If not provided, the version in the .digest file will be used.
## Download all files
Download all files under folder specified in 'init' command.
```
ldv versioning download-all
```
## List versions of file
```
ldv versioning list-versions <path to digest file>
```
Example:
```
ldv versioning list-versions local-data-folder/first_file.txt.digest
```
# Contact
Martin Neumann, ldv@martinneumann.se
Raw data
{
"_id": null,
"home_page": "https://github.com/martinnorin/ldv",
"name": "ldv",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Martin Neumann",
"author_email": "ldv@martinneumann.se",
"download_url": "https://files.pythonhosted.org/packages/60/78/606e2c17741fe6ebf82ecb8153e3870baa0ecf38aa2cadc74ce1936906ba/ldv-0.1.5.tar.gz",
"platform": null,
"description": "# ldv\nLight Data Versioning\n\nLdv is a tool for version tracking big files on a remote storage and storing digest files together with code that uses the big files.\nA digest file is a small file containing information about the file to version track. The information includes file size in bytes, timestamp of version, hexdigest (the version number), and local and remote filepaths.\n\nLdv can be run from the command line or from code, using the APIs of the package.\n\nLdv has no external dependencies on other programs, e.g. Git.\n\nCurrently, only AWS is supported as remote storage.\n\n# Getting started\n\n## Installation\nThis will install the package to version track on AWS remote storage.\n```\npip install ldv\n```\n\n# Usage\n\n## 1. Authentication\nAuthenticate with remote storage provider.\n\n```\nldv auth aws -rn <region_name> [-pn <profile name>] [-ec]\n```\n* -rn is the region name where the bucket is located.\n* -pn is the name of the profile specified in ~/.aws/config\n* -ec Use credentials stored in environment variables. LDV_AWS_ACCESS_KEY_ID and LDV_AWS_SECRET_ACCESS_KEY must be set. Will not be used if profile is provided.\n\nYou either use -pn or -ec, not both.\n\nExample:\n```\nldv auth aws -rn eu-west-1 -pn dev-profile\n```\n\n## 2. Initialize\nInitialize the local base path you want to version track files under. Also specify the remote storage url.\n\n```\nldv init -p <path> -u <url> [-rta] [-du]\n```\n* -p specifies the local path that contains files to version track.\n* -u is the remote storage url, e.g. s3://my_bucket/sub_folder.\n* -rta is an optional flag will run version tracking of all files under the path.\n* -du is an optional flag will prevent files from being uploaded to remote storage every time you version track them. Use this if you only want to create a digest files locally but not upload the files to remote storage.\n\nExample:\n```\nldv init -p local-data-folder -u s3://my_s3_bucket/sub_folder\n```\n\n\n\n## Version track file\n```\nldv versioning track <file path> [-du]\n```\n\n* file path to version track. Can be absolute or relative path but must be under path provided when running 'init' command.\n* -du is optional flag if you don't want to upload the file.\n\nExample:\n```\nldv versioning track local-data-folder/first_file.txt\n```\n## Version track all files\nVersion track all files under path specified in the 'init' command.\n```\nldv versioning track-all [-du]\n```\n* -du is optional flag if you don't want to upload the files.\n\n## Download file\nDownload file from remote storage to local storage.\n```\nldv versioning download <file path to digest file> [-v <version>]\n```\n* -v is the optional version. If not provided, the version in the .digest file will be used.\n\n## Download all files\nDownload all files under folder specified in 'init' command.\n```\nldv versioning download-all\n```\n\n## List versions of file\n```\nldv versioning list-versions <path to digest file>\n```\n\nExample:\n```\nldv versioning list-versions local-data-folder/first_file.txt.digest\n```\n\n# Contact\nMartin Neumann, ldv@martinneumann.se\n",
"bugtrack_url": null,
"license": null,
"summary": "Light Data Versioning",
"version": "0.1.5",
"project_urls": {
"Homepage": "https://github.com/martinnorin/ldv"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e567d2639fb8ae37b5bccfa3815dd867fe1bfedf2d3c69a567e2f9eb4f736f98",
"md5": "98d0964cd65b2d3f4d641a25092b9fdf",
"sha256": "24a7d468c02ed0eb39b1166002c97d3ec9a3bf8153241f9808d3c3d4112c39a1"
},
"downloads": -1,
"filename": "ldv-0.1.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "98d0964cd65b2d3f4d641a25092b9fdf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 32086,
"upload_time": "2024-09-03T09:54:09",
"upload_time_iso_8601": "2024-09-03T09:54:09.978513Z",
"url": "https://files.pythonhosted.org/packages/e5/67/d2639fb8ae37b5bccfa3815dd867fe1bfedf2d3c69a567e2f9eb4f736f98/ldv-0.1.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6078606e2c17741fe6ebf82ecb8153e3870baa0ecf38aa2cadc74ce1936906ba",
"md5": "142ac07a9e015e2f1794c1c159f4b211",
"sha256": "16ad02bfe277dc904d85623de729d558c05828c3786850d8fb15aab8d5dbca55"
},
"downloads": -1,
"filename": "ldv-0.1.5.tar.gz",
"has_sig": false,
"md5_digest": "142ac07a9e015e2f1794c1c159f4b211",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 22590,
"upload_time": "2024-09-03T09:54:11",
"upload_time_iso_8601": "2024-09-03T09:54:11.676201Z",
"url": "https://files.pythonhosted.org/packages/60/78/606e2c17741fe6ebf82ecb8153e3870baa0ecf38aa2cadc74ce1936906ba/ldv-0.1.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-03 09:54:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "martinnorin",
"github_project": "ldv",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "ldv"
}
JSON
