[](https://github.com/HEPData/hepdata-cli/actions?query=branch%3Amain)
[](https://coveralls.io/github/HEPData/hepdata-cli?branch=master)
[](https://github.com/HEPData/hepdata-cli/blob/main/LICENSE.txt)
[](https://github.com/HEPData/hepdata-cli/releases)
[](https://pypi.org/project/hepdata-cli/)
[](https://github.com/HEPData/hepdata-cli/issues)
# HEPData-CLI
## About
Command line interface (CLI) and application program interface (API) to allow users to search, download from and upload to [HEPData](https://www.hepdata.net).
The code is compatible with both Python 2 and Python 3. Inspiration from [arxiv-cli](https://github.com/jacquerie/arxiv-cli).
## Installation (for users)
Install from [PyPI](https://pypi.org/project/hepdata-cli/) using ```pip```:
```code
$ pip install --user hepdata-cli
$ hepdata-cli --help
```
With Python 3 (<3.7), if the `LANG` environment variable is not set, you might get an error like:
```
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.
```
In this case, you will need to export a Unicode locale as described in the
[Click documentation](https://click.palletsprojects.com/en/7.x/python3/#python-3-surrogate-handling).
## Installation (for developers)
Install from GitHub in a [virtual environment](https://docs.python.org/3/tutorial/venv.html):
```code
$ git clone https://github.com/HEPData/hepdata-cli.git
$ cd hepdata-cli
$ python3 -m venv ~/venv/hepdata-cli
$ source ~/venv/hepdata-cli/bin/activate
(hepdata-cli) $ pip install -e '.[tests]'
(hepdata-cli) $ hepdata-cli --help
(hepdata-cli) $ pytest --cov=hepdata_cli
```
## Usage
You can use HEPData-CLI both as a command-line interface (CLI) to search, download and upload records from/to the HEPData database, or as a Python library to perform the same operations via its application program interface (API).
## CLI
```code
$ hepdata-cli [-v/--version, --help]
$ hepdata-cli [--verbose] find [QUERY] [-kw/--keyword KEYWORD] [-i/--ids IDTYPE]
$ hepdata-cli [--verbose] download [IDS] [-f/--file-format FORMAT] [-i/--ids IDTYPE] [-t/--table-name TABLE-NAME] [-d/--download-dir DOWNLOAD-DIR]
$ hepdata-cli [--verbose] fetch-names [IDS] [-i/--ids IDTYPE]
$ hepdata-cli [--verbose] upload [PATH-TO-FILE-ARCHIVE] [-e/--email YOUR-EMAIL] [-r/--recid RECORD-ID] [-i/--invitation-cookie COOKIE] [-s/--sandbox TRUE/FALSE] [-p/--password PASSWORD]
```
The command ```find``` searches the [HEPData](https://www.hepdata.net/) database for matches of ```QUERY```. The advanced search syntax from the website can be used.
The command ```download``` downloads records from the database (see options below).
The command ```fetch-names``` returns the names of the data tables in the records whose ids are supplied.
The command ```upload``` uploads a file to the HEPData web site as either a sandbox or normal record.
The argument ```[-kw/--keyword KEYWORD]``` filters the search result dictionary for specific keywords.
An exact match of the keyword is first attempted, otherwise partial matches are accepted.
The argument ```[-i/--ids IDTYPE]``` accepts ```IDTYPE``` equal to ```arxiv```, ```hepdata``` or```inspire```.
The argument ```[-f/--file-format FORMAT]``` accepts ```FORMAT``` equal to ```csv```, ```root```, ```yaml```, ```yoda```, ```yoda1```, or ```json```.
In the first four cases a .tar.gz archive is downloaded and unpacked as a directory, whereas in the last case a .json file is downloaded.
The argument ```[-t/--table-name TABLE-NAME]``` accepts a string giving the table name as input.
In this case only the specified table is downloaded as a .csv, .root, .yaml, .yoda, .yoda1 or .json file.
The argument ```[-d/--download-dir DOWNLOAD-DIR]``` specifies the directory to download the files.
If not specified, the default download directory is ```./hepdata-downloads```.
The argument ```[-e/--email YOUR-EMAIL]``` is the uploader's email, needed to associate the submission to their HEPData account.
The argument ```[-i/--invitation-cookie COOKIE]``` must be supplied for non-sandbox submissions.
This can be found in the Uploader invitation email received at the beginning of the submission process.
The argument ```[-s/--sandbox TRUE/FALSE]``` is a boolean to decide whether to upload to the sandbox or not.
The argument ```[-p/--password PASSWORD``` is the password for the uploader's HEPData account (prompt if not specified).
Warning: do not store your password unencrypted in any code intended for shared use.
The ```hepdata-cli download/fetch-names``` and ```hepdata-cli find``` commands can be concatenated, if a ```IDTYPE``` is specified for ```find```.
It is also possible to concatenate ```arxiv download```, form [pypi/arxiv-cli](https://pypi.org/project/arxiv-cli/), with ```hepdata-cli find```, if ```arxiv``` is used as ```IDTYPE```.
## API
Equivalently to the above, these commands can be invoked by the API (in fact, the CLI is just a wrapper around the API).
```python
from hepdata_cli.api import Client
client = Client(verbose=True)
client.find(query, keyword, ids)
client.download(id_list, file_format, ids, table_name, download_dir)
client.fetch_names(id_list, ids)
client.upload(path_to_file, email, recid, invitation_cookie, sandbox, password)
```
## Examples
### Example 1 - a plain search:
```code
$ hepdata-cli --verbose find 'reactions:"P P--> LQ LQ X"'
```
or equivalently
```python
client.find('reactions:"P P--> LQ LQ X"')
```
matches a single entry and returns full metadata dictionary.
### Example 2 - search with keyword:
```code
$ hepdata-cli --verbose find 'reactions:"P P--> LQ LQ"' -kw year
```
or equivalently
```python
client.find('reactions:"P P--> LQ LQ"', keyword='year')
```
matches four entries and returns their publication years, as a dictionary.
### Example 3 - search for ids of records:
```code
$ hepdata-cli --verbose find 'reactions:"P P--> LQ LQ"' -i hepdata
```
or equivalently
```python
client.find('reactions:"P P--> LQ LQ"', ids='hepdata')
```
matches four entries and returns their hepdata ids, as a plain list.
### Example 4 - concatenate search with download using inspire ids:
```code
$ hepdata-cli --verbose download $(hepdata-cli find 'reactions:"P P--> LQ LQ"' -i inspire) -i inspire -f csv
```
or equivalently
```python
id_list = client.find('reactions:"P P--> LQ LQ"', ids='inspire')
client.download(id_list, ids='inspire', file_format='csv')
```
downloads four .tar.gz archives containing csv files and unpacks them in the default ```./hepdata-downloads``` directory.
### Example 5 - find table names in records:
```code
$ hepdata-cli fetch-names $(hepdata-cli find 'reactions:"P P--> LQ LQ"' -i hepdata) -i hepdata
```
or equivalently
```python
id_list = client.find('reactions:"P P--> LQ LQ"', ids='hepdata')
client.fetch_names(id_list, ids='hepdata')
```
returns all table names in the four matching records.
### Example 6 - concatenate search with download from arxiv-cli:
This example requires [arxiv-cli](https://github.com/jacquerie/arxiv-cli) to be installed, which is easily done via:
```code
$ pip install --user arxiv-cli
```
Note that arxiv-cli installs an older version of [click](https://pypi.org/project/click/) which changes the CLI command
in Example 5 above from ```fetch-names``` to ```fetch_names```.
Then,
```code
$ arxiv download $(hepdata-cli find 'reactions:"P P--> LQ LQ"' -i arxiv)
```
or equivalently
```python
import arxiv_cli
import hepdata_cli
arxiv_client = arxiv_cli.Client()
hepdata_client = hepdata_cli.Client()
id_list = hepdata_client.find('reactions:"P P--> LQ LQ"', ids='arxiv')
arxiv_client.download(id_list)
```
downloads two pdfs from the arXiv.
### Example 7 - upload record to the sandbox:
```code
$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -s True
```
or equivalently
```python
client.upload('/path/to/TestHEPSubmission.tar.gz', email='my@email.com', sandbox=True)
```
The uploaded submission can then be found from your [sandbox](https://www.hepdata.net/record/sandbox).
You will be prompted for the password associated with your HEPData account.
If your account was created with CERN or ORCID authentication, you will first need to [set a password](https://www.hepdata.net/lost-password/).
### Example 8 - replace a record in the sandbox:
```code
$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -r 1234567890 -s True
```
or equivalently
```python
client.upload('/path/to/TestHEPSubmission.tar.gz', email='my@email.com', recid='1234567890', sandbox=True)
```
Note that you must have uploaded the original sandbox record yourself and that you will be prompted for a password.
### Example 9 - upload a non-sandbox record:
```code
$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -r 123456 -i 8232e07f-d1d8-4883-bb1d-77fd9994ce4f -s False
```
or equivalently
```python
client.upload('/path/to/TestHEPSubmission.tar.gz', email='my@email.com', recid='123456', invitation_cookie='8232e07f-d1d8-4883-bb1d-77fd9994ce4f', sandbox=False)
```
The uploaded submission can then be found from your [Dashboard](https://www.hepdata.net/dashboard/).
The invitation cookie is sent in your original invitation email.
You must have already claimed permissions by clicking the link in that email or from your [Dashboard](https://www.hepdata.net/dashboard/).
Again, you will be prompted for a password, which must be [set](https://www.hepdata.net/lost-password/) if using CERN/ORCID login.
The password can alternatively be passed as an argument to the CLI (```-p PASSWORD```) or API (```password=PASSWORD```).
However, please be careful to keep your password secure, for example, by defining an encrypted environment variable for a CI/CD workflow.
Raw data
{
"_id": null,
"home_page": "https://github.com/HEPData/hepdata-cli",
"name": "hepdata-cli",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "hepdata cli api",
"author": "Giuseppe De Laurentis",
"author_email": "g.dl@hotmail.it",
"download_url": "https://files.pythonhosted.org/packages/1e/39/c0a96501dc28a47e0ccf90f0eb954535b32f72275b66b087af0241d88ca6/hepdata-cli-0.2.2.tar.gz",
"platform": "any",
"description": "[](https://github.com/HEPData/hepdata-cli/actions?query=branch%3Amain)\n[](https://coveralls.io/github/HEPData/hepdata-cli?branch=master)\n[](https://github.com/HEPData/hepdata-cli/blob/main/LICENSE.txt)\n[](https://github.com/HEPData/hepdata-cli/releases)\n[](https://pypi.org/project/hepdata-cli/)\n[](https://github.com/HEPData/hepdata-cli/issues)\n\n\n# HEPData-CLI\n\n## About\n\nCommand line interface (CLI) and application program interface (API) to allow users to search, download from and upload to [HEPData](https://www.hepdata.net).\n\nThe code is compatible with both Python 2 and Python 3. Inspiration from [arxiv-cli](https://github.com/jacquerie/arxiv-cli).\n\n## Installation (for users)\n\nInstall from [PyPI](https://pypi.org/project/hepdata-cli/) using ```pip```:\n\n```code\n$ pip install --user hepdata-cli\n$ hepdata-cli --help\n```\n\nWith Python 3 (<3.7), if the `LANG` environment variable is not set, you might get an error like:\n\n```\nRuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.\n```\n\nIn this case, you will need to export a Unicode locale as described in the\n[Click documentation](https://click.palletsprojects.com/en/7.x/python3/#python-3-surrogate-handling).\n\n## Installation (for developers)\n\nInstall from GitHub in a [virtual environment](https://docs.python.org/3/tutorial/venv.html):\n\n```code\n$ git clone https://github.com/HEPData/hepdata-cli.git\n$ cd hepdata-cli\n$ python3 -m venv ~/venv/hepdata-cli\n$ source ~/venv/hepdata-cli/bin/activate\n(hepdata-cli) $ pip install -e '.[tests]'\n(hepdata-cli) $ hepdata-cli --help\n(hepdata-cli) $ pytest --cov=hepdata_cli\n```\n\n## Usage\n\nYou can use HEPData-CLI both as a command-line interface (CLI) to search, download and upload records from/to the HEPData database, or as a Python library to perform the same operations via its application program interface (API).\n\n\n## CLI\n\n```code\n$ hepdata-cli [-v/--version, --help]\n$ hepdata-cli [--verbose] find [QUERY] [-kw/--keyword KEYWORD] [-i/--ids IDTYPE]\n$ hepdata-cli [--verbose] download [IDS] [-f/--file-format FORMAT] [-i/--ids IDTYPE] [-t/--table-name TABLE-NAME] [-d/--download-dir DOWNLOAD-DIR]\n$ hepdata-cli [--verbose] fetch-names [IDS] [-i/--ids IDTYPE]\n$ hepdata-cli [--verbose] upload [PATH-TO-FILE-ARCHIVE] [-e/--email YOUR-EMAIL] [-r/--recid RECORD-ID] [-i/--invitation-cookie COOKIE] [-s/--sandbox TRUE/FALSE] [-p/--password PASSWORD]\n```\n\nThe command ```find``` searches the [HEPData](https://www.hepdata.net/) database for matches of ```QUERY```. The advanced search syntax from the website can be used.\n\nThe command ```download``` downloads records from the database (see options below).\n\nThe command ```fetch-names``` returns the names of the data tables in the records whose ids are supplied.\n\nThe command ```upload``` uploads a file to the HEPData web site as either a sandbox or normal record.\n\nThe argument ```[-kw/--keyword KEYWORD]``` filters the search result dictionary for specific keywords.\nAn exact match of the keyword is first attempted, otherwise partial matches are accepted.\n\nThe argument ```[-i/--ids IDTYPE]``` accepts ```IDTYPE``` equal to ```arxiv```, ```hepdata``` or```inspire```.\n\nThe argument ```[-f/--file-format FORMAT]``` accepts ```FORMAT``` equal to ```csv```, ```root```, ```yaml```, ```yoda```, ```yoda1```, or ```json```.\nIn the first four cases a .tar.gz archive is downloaded and unpacked as a directory, whereas in the last case a .json file is downloaded.\n\nThe argument ```[-t/--table-name TABLE-NAME]``` accepts a string giving the table name as input.\nIn this case only the specified table is downloaded as a .csv, .root, .yaml, .yoda, .yoda1 or .json file.\n\nThe argument ```[-d/--download-dir DOWNLOAD-DIR]``` specifies the directory to download the files.\nIf not specified, the default download directory is ```./hepdata-downloads```.\n\nThe argument ```[-e/--email YOUR-EMAIL]``` is the uploader's email, needed to associate the submission to their HEPData account.\n\nThe argument ```[-i/--invitation-cookie COOKIE]``` must be supplied for non-sandbox submissions.\nThis can be found in the Uploader invitation email received at the beginning of the submission process.\n\nThe argument ```[-s/--sandbox TRUE/FALSE]``` is a boolean to decide whether to upload to the sandbox or not.\n\nThe argument ```[-p/--password PASSWORD``` is the password for the uploader's HEPData account (prompt if not specified).\nWarning: do not store your password unencrypted in any code intended for shared use.\n\nThe ```hepdata-cli download/fetch-names``` and ```hepdata-cli find``` commands can be concatenated, if a ```IDTYPE``` is specified for ```find```.\nIt is also possible to concatenate ```arxiv download```, form [pypi/arxiv-cli](https://pypi.org/project/arxiv-cli/), with ```hepdata-cli find```, if ```arxiv``` is used as ```IDTYPE```.\n\n## API\n\nEquivalently to the above, these commands can be invoked by the API (in fact, the CLI is just a wrapper around the API).\n\n```python\nfrom hepdata_cli.api import Client\nclient = Client(verbose=True)\nclient.find(query, keyword, ids)\nclient.download(id_list, file_format, ids, table_name, download_dir)\nclient.fetch_names(id_list, ids)\nclient.upload(path_to_file, email, recid, invitation_cookie, sandbox, password)\n\n```\n\n## Examples\n\n### Example 1 - a plain search:\n\n```code\n$ hepdata-cli --verbose find 'reactions:\"P P--> LQ LQ X\"'\n```\n\nor equivalently\n\n```python\nclient.find('reactions:\"P P--> LQ LQ X\"')\n```\n\nmatches a single entry and returns full metadata dictionary.\n\n### Example 2 - search with keyword:\n\n```code\n$ hepdata-cli --verbose find 'reactions:\"P P--> LQ LQ\"' -kw year\n```\n\nor equivalently\n\n```python\nclient.find('reactions:\"P P--> LQ LQ\"', keyword='year')\n```\n\nmatches four entries and returns their publication years, as a dictionary.\n\n### Example 3 - search for ids of records:\n\n```code\n$ hepdata-cli --verbose find 'reactions:\"P P--> LQ LQ\"' -i hepdata\n```\n\nor equivalently\n\n```python\nclient.find('reactions:\"P P--> LQ LQ\"', ids='hepdata')\n```\n\nmatches four entries and returns their hepdata ids, as a plain list.\n\n### Example 4 - concatenate search with download using inspire ids:\n\n```code\n$ hepdata-cli --verbose download $(hepdata-cli find 'reactions:\"P P--> LQ LQ\"' -i inspire) -i inspire -f csv\n```\n\nor equivalently\n\n```python\nid_list = client.find('reactions:\"P P--> LQ LQ\"', ids='inspire')\nclient.download(id_list, ids='inspire', file_format='csv')\n```\n\ndownloads four .tar.gz archives containing csv files and unpacks them in the default ```./hepdata-downloads``` directory.\n\n### Example 5 - find table names in records:\n\n```code\n$ hepdata-cli fetch-names $(hepdata-cli find 'reactions:\"P P--> LQ LQ\"' -i hepdata) -i hepdata\n```\n\nor equivalently\n\n```python\nid_list = client.find('reactions:\"P P--> LQ LQ\"', ids='hepdata')\nclient.fetch_names(id_list, ids='hepdata')\n```\n\nreturns all table names in the four matching records.\n\n### Example 6 - concatenate search with download from arxiv-cli:\n\nThis example requires [arxiv-cli](https://github.com/jacquerie/arxiv-cli) to be installed, which is easily done via:\n\n```code\n$ pip install --user arxiv-cli\n```\n\nNote that arxiv-cli installs an older version of [click](https://pypi.org/project/click/) which changes the CLI command\nin Example 5 above from ```fetch-names``` to ```fetch_names```.\n\nThen,\n\n```code\n$ arxiv download $(hepdata-cli find 'reactions:\"P P--> LQ LQ\"' -i arxiv)\n```\n\nor equivalently\n\n```python\nimport arxiv_cli\nimport hepdata_cli\narxiv_client = arxiv_cli.Client()\nhepdata_client = hepdata_cli.Client()\nid_list = hepdata_client.find('reactions:\"P P--> LQ LQ\"', ids='arxiv')\narxiv_client.download(id_list)\n```\n\ndownloads two pdfs from the arXiv.\n\n### Example 7 - upload record to the sandbox:\n\n```code\n$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -s True\n```\n\nor equivalently\n\n```python\nclient.upload('/path/to/TestHEPSubmission.tar.gz', email='my@email.com', sandbox=True)\n```\n\nThe uploaded submission can then be found from your [sandbox](https://www.hepdata.net/record/sandbox).\nYou will be prompted for the password associated with your HEPData account.\nIf your account was created with CERN or ORCID authentication, you will first need to [set a password](https://www.hepdata.net/lost-password/).\n\n### Example 8 - replace a record in the sandbox:\n\n```code\n$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -r 1234567890 -s True\n```\n\nor equivalently\n\n```python\nclient.upload('/path/to/TestHEPSubmission.tar.gz', email='my@email.com', recid='1234567890', sandbox=True)\n```\n\nNote that you must have uploaded the original sandbox record yourself and that you will be prompted for a password.\n\n### Example 9 - upload a non-sandbox record:\n\n```code\n$ hepdata-cli upload /path/to/TestHEPSubmission.tar.gz -e my@email.com -r 123456 -i 8232e07f-d1d8-4883-bb1d-77fd9994ce4f -s False \n```\n\nor equivalently\n\n```python\nclient.upload('/path/to/TestHEPSubmission.tar.gz', email='my@email.com', recid='123456', invitation_cookie='8232e07f-d1d8-4883-bb1d-77fd9994ce4f', sandbox=False)\n```\n\nThe uploaded submission can then be found from your [Dashboard](https://www.hepdata.net/dashboard/).\nThe invitation cookie is sent in your original invitation email.\nYou must have already claimed permissions by clicking the link in that email or from your [Dashboard](https://www.hepdata.net/dashboard/).\nAgain, you will be prompted for a password, which must be [set](https://www.hepdata.net/lost-password/) if using CERN/ORCID login.\nThe password can alternatively be passed as an argument to the CLI (```-p PASSWORD```) or API (```password=PASSWORD```).\nHowever, please be careful to keep your password secure, for example, by defining an encrypted environment variable for a CI/CD workflow.\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "CLI and API to allow users to search, download from and upload to HEPData",
"version": "0.2.2",
"project_urls": {
"Homepage": "https://github.com/HEPData/hepdata-cli"
},
"split_keywords": [
"hepdata",
"cli",
"api"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "816508e01ceb83a371343ec4b67ad6d5b5790cf91a3bc871c2e7e4a079c2350d",
"md5": "12336f4b5a79fe8053d18c00f84b76dd",
"sha256": "3adc0c765e3457d5f22996f3273ca1df87683509e7df074439cfed6f80d89526"
},
"downloads": -1,
"filename": "hepdata_cli-0.2.2-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "12336f4b5a79fe8053d18c00f84b76dd",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.6",
"size": 22834,
"upload_time": "2023-11-17T14:14:04",
"upload_time_iso_8601": "2023-11-17T14:14:04.696433Z",
"url": "https://files.pythonhosted.org/packages/81/65/08e01ceb83a371343ec4b67ad6d5b5790cf91a3bc871c2e7e4a079c2350d/hepdata_cli-0.2.2-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1e39c0a96501dc28a47e0ccf90f0eb954535b32f72275b66b087af0241d88ca6",
"md5": "5b7bc9cd917930ec8f7b82f1f2660330",
"sha256": "fabcff3085c057d806b7d21279cc4cec711417db65e87723993b4d001d86e27f"
},
"downloads": -1,
"filename": "hepdata-cli-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "5b7bc9cd917930ec8f7b82f1f2660330",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 24642,
"upload_time": "2023-11-17T14:14:06",
"upload_time_iso_8601": "2023-11-17T14:14:06.269711Z",
"url": "https://files.pythonhosted.org/packages/1e/39/c0a96501dc28a47e0ccf90f0eb954535b32f72275b66b087af0241d88ca6/hepdata-cli-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-17 14:14:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "HEPData",
"github_project": "hepdata-cli",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "hepdata-cli"
}
JSON
