## alien.py - Python interface to websocket endpoint of [ALICE](https://alice-collaboration.web.cern.ch) Grid Services
Quick containerized testing:
`singularity run library://adriansev/default/alienpy:latest [cmd]`
or
`singularity run oras://registry.cern.ch/asevcenc/alienpy:latest [cmd]`
`latest` _usually_ would point to master but not always.(if desired and needed, request by email a new latest tag)
see [here](https://cloud.sylabs.io/library/_container/6124dbd7d63fe43757facc7e) what tags are available and their dates of creation.
The docker images can be found @[DockerHub](https://hub.docker.com/r/adriansevcenco/alienpy.dock/tags)
```
docker run -it \
--user $(id -u):$(id -g) \
--workdir="/home/$USER" \
--env TMPDIR="${TMPDIR:-/tmp}" \
--volume="/etc/group:/etc/group:ro" \
--volume="/etc/passwd:/etc/passwd:ro" \
--volume="/etc/shadow:/etc/shadow:ro" \
--volume="/home:/home" \
adriansevcenco/alienpy.dock:latest [cmd]
```
if no cmd is passed, the shell form will start
### Basic usage
Can be used as command mode and interactive mode :
1. Command mode :
`alien.py <command> `
e.g :
`alien.py pwd `
**N.B.** command/arguments must be quoted to avoid being interpreted by the shell:
`alien.py 'rm my_alien_dir/*'`
2. Interactive/shell mode e.g :
```
alien.py
Welcome to the ALICE GRID
support mail: adrian.sevcenco@cern.ch
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >pwd
/alice/cern.ch/user/a/asevcenc/
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >whoami
asevcenc
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >
```
* For both command and shell mode multiple commands can be issued separated by `;`
* The interactive mode save the command history in `${HOME}/.alienpy_history` and it can be navigated with Up/Down keys
* `!` is understood as running into shell whatever command follows
* `|` pipe whatever output of AliEn command to a shell command (that follows after the first(only the first) `|`)
### Environment steering
There are a few environment variables that influence the mechanics of the script :
* JALIEN_TOKEN_CERT, JALIEN_TOKEN_KEY - will overwrite the defaults; they are either full path certificate,key token files OR their respective contents
* If set the following will be honored: X509_USER_CERT, X509_USER_KEY, X509_CERT_DIR or X509_CERT_FILE
* ALIENPY_TIMEOUT will change the interval for keep-alive mechanics.
* ALIENPY_CONNECT_TRIES - default = 3 : number of connect trials
* ALIENPY_CONNECT_TRIES_INTERVAL - default = 0.5 : seconds between connection trials
---
For debugging purposes there are a few environment toggles :
* ALIENPY_JSON - print the unprocessed json message from the server
* ALIENPY_JSONRAW - print the unprocessed byte stream message from the server
---
* ALIENPY_JCENTRAL - it will connect to this server, ignoring any other options
* ALIENPY_NO_STAGGER - disable staggered parallel host resolution and socket creation (see [RFC8305](https://tools.ietf.org/html/rfc8305))
---
* ALIENPY_DEBUG - detailed debug meesages will be found in ALIENPY_DEBUG_FILE
* ALIENPY_DEBUG_FILE - set the location of log file
* ALIENPY_DEBUG_APPEND - is set the output will be appended to the present log file. if not the file will be overwritten.
* ALIENPY_TIMECONNECT - if set will report time for websocket creation - e.g. `ALIENPY_TIMECONNECT=1 alien.py pwd`
* ALIENPY_TIMING - report detailed operation timing in the log file.
---
DEBUG file copy operations:
* ALIENPY_KEEP_META - keep the metafile generated for download operations. Can be directly used with xrdcp.
* XRD_LOGLEVEL='Dump'
* XRD_LOGFILE=xrdlog.txt
See also the native XRootD environment toggles: [docs](https://xrootd.slac.stanford.edu/doc/man/xrdcp.1.html#ENVIRONMENT "XRootD xrdcopy documentation")
## Authentication
The authentication process needs the presence of a X509 certificate (enrolled into ALICE VO, see [here](https://alien.web.cern.ch/content/vo/alice/userregistration))
and of a CA certificates directory for verification.
The default CA location that will be searched is within alice.cern.ch cvmfs repository
If not found, the CApath will default to /etc/grid-security/certificates
If these locations are not available, one _must_ set X509_CERT_DIR to a valid location
## Command usage and examples
The list of available commands can seen with: `?` or `help`
Command help can be listed with: `? command`, `help command`, `command -h`
### Storage related operations
This section refer to any copy to/from grid or file interactions.
`cat/more/less` will download the target lfn to a temporary file and will act upon it while
`vi/nano/mcedit/edit` will, after the modification of downloaded temporary, backup the existing lfn, and upload the modified file
The target file upload can support grid specifiers like those described in `cp` command e.g. `edit my_file@disk:2,SE1`
#### ```cp``` option
cp can take as arguments both files and directories and have the following options:
```
alien.py cp -h
Command format is of the form of (with the strict order of arguments):
cp <options> src dst
or
cp <options> -input input_file
location prefixes are: file: | file:// | alien: | alien://
if one prefix is specified the other operator is considered of the other kind (no local -> local, or grid->grid operations allowed)
if no prefix is specified, the src will be _first_ checked if local and then if remote.
-input argument is a file with >src dst< pairs
after each src,dst can be added comma separated specifiers in the form of: @disk:N,SE1,SE2,!SE3
where disk selects the number of replicas and the following specifiers add (or remove) storage endpoints from the received list
options are the following :
-h : print help
-f : replace destination file (if destination is local it will be replaced only if integrity check fails)
-P : enable persist on successful close semantic
-cksum : check hash sum of the file; for downloads the central catalogue md5 will be verified;
for uploads (for xrootd client > 4.12.0) a hash type will be negociated with remote and transfer will be validated
-y <nr_sources> : use up to the number of sources specified in parallel (N.B. Ignored as it breaks download of files stored in archives)
-S <aditional TPC streams> : uses num additional parallel streams to do the transfer. (max = 15)
-chunks <nr chunks> : number of chunks that should be requested in parallel
-chunksz <bytes> : chunk size (bytes)
-T <nr_copy_jobs> : number of parralel copy jobs from a set (for recursive copy); defaults to 8 for downloads
-noxrdzip: circumvent the XRootD mechanism of zip member copy and download the archive and locally extract the intended member.
N.B.!!! for recursive copy (all files) the same archive will be downloaded for each member.
If there are problems with native XRootD zip mechanism, download only the zip archive and locally extract the contents
for the recursive copy of directories the following options (of the find command) can be used:
-glob <globbing pattern> : this is the usual AliEn globbing format; N.B. this is NOT a REGEX!!! defaults to all "*"
-select <pattern> : select only these files to be copied; N.B. this is a REGEX applied to full path!!!
-name <pattern> : select only these files to be copied; N.B. this is a REGEX applied to a directory or file name!!!
-name <verb>_string : where verb = begin|contain|ends|ext and string is the text selection criteria.
verbs are aditive : -name begin_myf_contain_run1_ends_bla_ext_root
N.B. the text to be filtered cannont have underline <_> within!!!
-parent <parent depth> : in destination use this <parent depth> to add to destination ; defaults to 0
-a : copy also the hidden files .* (for recursive copy)
-j <queue_id> : select only the files created by the job with <queue_id> (for recursive copy)
-l <count> : copy only <count> nr of files (for recursive copy)
-o <offset> : skip first <offset> files found in the src directory (for recursive copy)
```
`.`, `..` are interpreted for all grid names (lfns)
`%ALIEN` is converted to user AliEn home directory
lfns that don't start with a `/` will have the current directory appended before being processed
### Miscellaneous
#### The shell prompt
It can show date and/or local directory:
* `prompt date` will toggle on/off the date
* `prompt pwd` will toggle on/off the local current directory
For permanent setting the following are env variables are available : ALIENPY_PROMPT_DATE, ALIENPY_PROMPT_CWD
```
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >prompt date
2020-02-07T16:49:05 AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >prompt pwd
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ local:/home.hdd/adrian/work-GRID/jalien_py >
```
---
##### CWD persistence
Default behaviour is to save (and then restore) the last used CWD.
This bevahiour can be disabled with the env var ALIENPY_NO_CWD_RESTORE
---
##### `ls` aliases
`ll`, `la`, `lla` are aliases to `ls -l`, `ls -a`, `ls -la`
---
##### Custom aliases
A fixed file `${HOME}/.alienpy_aliases` can be used to define alias=string pairs that will be used(translated) in the usage of alien.py. One can do `myalias=cmd1;cmd2;cmd3` and the `myalias` string will be replaced by it's value when used.
---
##### Python shell
`term` command will open an _Python_ shell within the context of alien.py and with a session object loaded
---
##### API usage
see examples directory
Raw data
{
"_id": null,
"home_page": "https://gitlab.cern.ch/jalien/xjalienfs",
"name": "alienpy",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "CERN ALICE JAliEn GRID",
"author": "Adrian Sevcenco",
"author_email": "adrian.sevcenco@cern.ch",
"download_url": "https://files.pythonhosted.org/packages/aa/d9/61fdb77e4db7a88e099c695987cf1ae85b4d72666f9ca396b303d108fa06/alienpy-1.6.1.tar.gz",
"platform": null,
"description": " \n\n## alien.py - Python interface to websocket endpoint of [ALICE](https://alice-collaboration.web.cern.ch) Grid Services \n\nQuick containerized testing: \n`singularity run library://adriansev/default/alienpy:latest [cmd]` \nor \n`singularity run oras://registry.cern.ch/asevcenc/alienpy:latest [cmd]` \n\n`latest` _usually_ would point to master but not always.(if desired and needed, request by email a new latest tag) \nsee [here](https://cloud.sylabs.io/library/_container/6124dbd7d63fe43757facc7e) what tags are available and their dates of creation. \n\nThe docker images can be found @[DockerHub](https://hub.docker.com/r/adriansevcenco/alienpy.dock/tags) \n```\ndocker run -it \\\n--user $(id -u):$(id -g) \\\n--workdir=\"/home/$USER\" \\\n--env TMPDIR=\"${TMPDIR:-/tmp}\" \\\n--volume=\"/etc/group:/etc/group:ro\" \\\n--volume=\"/etc/passwd:/etc/passwd:ro\" \\\n--volume=\"/etc/shadow:/etc/shadow:ro\" \\\n--volume=\"/home:/home\" \\\nadriansevcenco/alienpy.dock:latest [cmd]\n```\n\nif no cmd is passed, the shell form will start\n\n### Basic usage\nCan be used as command mode and interactive mode : \n1. Command mode : \n`alien.py <command> `\ne.g : \n`alien.py pwd ` \n**N.B.** command/arguments must be quoted to avoid being interpreted by the shell: \n`alien.py 'rm my_alien_dir/*'`\n\n2. Interactive/shell mode e.g : \n```\nalien.py\nWelcome to the ALICE GRID\nsupport mail: adrian.sevcenco@cern.ch\n\nAliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >pwd\n/alice/cern.ch/user/a/asevcenc/\nAliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >whoami\nasevcenc\nAliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >\n```\n* For both command and shell mode multiple commands can be issued separated by `;` \n* The interactive mode save the command history in `${HOME}/.alienpy_history` and it can be navigated with Up/Down keys \n* `!` is understood as running into shell whatever command follows \n* `|` pipe whatever output of AliEn command to a shell command (that follows after the first(only the first) `|`)\n\n### Environment steering\n\nThere are a few environment variables that influence the mechanics of the script : \n* JALIEN_TOKEN_CERT, JALIEN_TOKEN_KEY - will overwrite the defaults; they are either full path certificate,key token files OR their respective contents \n* If set the following will be honored: X509_USER_CERT, X509_USER_KEY, X509_CERT_DIR or X509_CERT_FILE \n* ALIENPY_TIMEOUT will change the interval for keep-alive mechanics.\n* ALIENPY_CONNECT_TRIES - default = 3 : number of connect trials\n* ALIENPY_CONNECT_TRIES_INTERVAL - default = 0.5 : seconds between connection trials\n\n---\nFor debugging purposes there are a few environment toggles : \n* ALIENPY_JSON - print the unprocessed json message from the server \n* ALIENPY_JSONRAW - print the unprocessed byte stream message from the server \n---\n* ALIENPY_JCENTRAL - it will connect to this server, ignoring any other options \n* ALIENPY_NO_STAGGER - disable staggered parallel host resolution and socket creation (see [RFC8305](https://tools.ietf.org/html/rfc8305))\n---\n* ALIENPY_DEBUG - detailed debug meesages will be found in ALIENPY_DEBUG_FILE\n* ALIENPY_DEBUG_FILE - set the location of log file\n* ALIENPY_DEBUG_APPEND - is set the output will be appended to the present log file. if not the file will be overwritten.\n* ALIENPY_TIMECONNECT - if set will report time for websocket creation - e.g. `ALIENPY_TIMECONNECT=1 alien.py pwd`\n* ALIENPY_TIMING - report detailed operation timing in the log file.\n\n---\nDEBUG file copy operations:\n* ALIENPY_KEEP_META - keep the metafile generated for download operations. Can be directly used with xrdcp.\n* XRD_LOGLEVEL='Dump'\n* XRD_LOGFILE=xrdlog.txt\n \nSee also the native XRootD environment toggles: [docs](https://xrootd.slac.stanford.edu/doc/man/xrdcp.1.html#ENVIRONMENT \"XRootD xrdcopy documentation\") \n\n## Authentication\nThe authentication process needs the presence of a X509 certificate (enrolled into ALICE VO, see [here](https://alien.web.cern.ch/content/vo/alice/userregistration))\nand of a CA certificates directory for verification.\nThe default CA location that will be searched is within alice.cern.ch cvmfs repository\nIf not found, the CApath will default to /etc/grid-security/certificates\nIf these locations are not available, one _must_ set X509_CERT_DIR to a valid location\n\n\n## Command usage and examples \n\nThe list of available commands can seen with: `?` or `help` \nCommand help can be listed with: `? command`, `help command`, `command -h` \n\n### Storage related operations\nThis section refer to any copy to/from grid or file interactions. \n\n`cat/more/less` will download the target lfn to a temporary file and will act upon it while \n`vi/nano/mcedit/edit` will, after the modification of downloaded temporary, backup the existing lfn, and upload the modified file \nThe target file upload can support grid specifiers like those described in `cp` command e.g. `edit my_file@disk:2,SE1` \n\n#### ```cp``` option \n\ncp can take as arguments both files and directories and have the following options: \n```\nalien.py cp -h\nCommand format is of the form of (with the strict order of arguments):\ncp <options> src dst\nor\ncp <options> -input input_file\n\nlocation prefixes are: file: | file:// | alien: | alien://\nif one prefix is specified the other operator is considered of the other kind (no local -> local, or grid->grid operations allowed)\nif no prefix is specified, the src will be _first_ checked if local and then if remote.\n\n-input argument is a file with >src dst< pairs\n\nafter each src,dst can be added comma separated specifiers in the form of: @disk:N,SE1,SE2,!SE3\nwhere disk selects the number of replicas and the following specifiers add (or remove) storage endpoints from the received list\noptions are the following :\n-h : print help\n-f : replace destination file (if destination is local it will be replaced only if integrity check fails)\n-P : enable persist on successful close semantic\n-cksum : check hash sum of the file; for downloads the central catalogue md5 will be verified;\nfor uploads (for xrootd client > 4.12.0) a hash type will be negociated with remote and transfer will be validated\n-y <nr_sources> : use up to the number of sources specified in parallel (N.B. Ignored as it breaks download of files stored in archives)\n-S <aditional TPC streams> : uses num additional parallel streams to do the transfer. (max = 15)\n-chunks <nr chunks> : number of chunks that should be requested in parallel\n-chunksz <bytes> : chunk size (bytes)\n-T <nr_copy_jobs> : number of parralel copy jobs from a set (for recursive copy); defaults to 8 for downloads\n-noxrdzip: circumvent the XRootD mechanism of zip member copy and download the archive and locally extract the intended member.\nN.B.!!! for recursive copy (all files) the same archive will be downloaded for each member.\nIf there are problems with native XRootD zip mechanism, download only the zip archive and locally extract the contents\n\nfor the recursive copy of directories the following options (of the find command) can be used:\n-glob <globbing pattern> : this is the usual AliEn globbing format; N.B. this is NOT a REGEX!!! defaults to all \"*\"\n-select <pattern> : select only these files to be copied; N.B. this is a REGEX applied to full path!!!\n-name <pattern> : select only these files to be copied; N.B. this is a REGEX applied to a directory or file name!!!\n-name <verb>_string : where verb = begin|contain|ends|ext and string is the text selection criteria.\nverbs are aditive : -name begin_myf_contain_run1_ends_bla_ext_root\nN.B. the text to be filtered cannont have underline <_> within!!!\n-parent <parent depth> : in destination use this <parent depth> to add to destination ; defaults to 0\n-a : copy also the hidden files .* (for recursive copy)\n-j <queue_id> : select only the files created by the job with <queue_id> (for recursive copy)\n-l <count> : copy only <count> nr of files (for recursive copy)\n-o <offset> : skip first <offset> files found in the src directory (for recursive copy)\n```\n\n`.`, `..` are interpreted for all grid names (lfns) \n`%ALIEN` is converted to user AliEn home directory \nlfns that don't start with a `/` will have the current directory appended before being processed\n\n### Miscellaneous\n\n#### The shell prompt\nIt can show date and/or local directory: \n* `prompt date` will toggle on/off the date \n* `prompt pwd` will toggle on/off the local current directory \nFor permanent setting the following are env variables are available : ALIENPY_PROMPT_DATE, ALIENPY_PROMPT_CWD \n```\nAliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >prompt date\n2020-02-07T16:49:05 AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >\n\nAliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >prompt pwd\nAliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ local:/home.hdd/adrian/work-GRID/jalien_py >\n```\n\n---\n##### CWD persistence\nDefault behaviour is to save (and then restore) the last used CWD. \nThis bevahiour can be disabled with the env var ALIENPY_NO_CWD_RESTORE \n\n---\n##### `ls` aliases\n`ll`, `la`, `lla` are aliases to `ls -l`, `ls -a`, `ls -la`\n\n---\n##### Custom aliases \nA fixed file `${HOME}/.alienpy_aliases` can be used to define alias=string pairs that will be used(translated) in the usage of alien.py. One can do `myalias=cmd1;cmd2;cmd3` and the `myalias` string will be replaced by it's value when used. \n\n---\n##### Python shell\n`term` command will open an _Python_ shell within the context of alien.py and with a session object loaded\n\n---\n##### API usage\nsee examples directory\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Websocket based cli interface for ALICE experiment GRID infrastructure",
"version": "1.6.1",
"project_urls": {
"CERN Mattermost/JAliEn": "https://mattermost.web.cern.ch/alice/channels/jalien",
"Changelog": "https://github.com/adriansev/jalien_py/commits/master",
"Dev git": "https://github.com/adriansev/jalien_py",
"Documentation": "https://jalien.docs.cern.ch",
"Homepage": "https://gitlab.cern.ch/jalien/xjalienfs",
"Issues": "https://github.com/adriansev/jalien_py/issues"
},
"split_keywords": [
"cern",
"alice",
"jalien",
"grid"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0d10bb86d2be11f48545e255d68a6712d13ee4dfdfdf500474227eae42e3b2a1",
"md5": "d3fdb8b9e0ce84ca60da80bfd7746203",
"sha256": "468b825f5fb5eee120bb95f833559627ac37af9bc7d9ea0bc5cdc953e8cbd4ef"
},
"downloads": -1,
"filename": "alienpy-1.6.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d3fdb8b9e0ce84ca60da80bfd7746203",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 94387,
"upload_time": "2024-04-10T12:35:06",
"upload_time_iso_8601": "2024-04-10T12:35:06.510757Z",
"url": "https://files.pythonhosted.org/packages/0d/10/bb86d2be11f48545e255d68a6712d13ee4dfdfdf500474227eae42e3b2a1/alienpy-1.6.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "aad961fdb77e4db7a88e099c695987cf1ae85b4d72666f9ca396b303d108fa06",
"md5": "b98191b36bac69f54107f8d228a2719b",
"sha256": "c917fda82129cf01a09a2b389141f3a9b753cea6dc0b06eec2811def17323944"
},
"downloads": -1,
"filename": "alienpy-1.6.1.tar.gz",
"has_sig": false,
"md5_digest": "b98191b36bac69f54107f8d228a2719b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 90223,
"upload_time": "2024-04-10T12:35:08",
"upload_time_iso_8601": "2024-04-10T12:35:08.232923Z",
"url": "https://files.pythonhosted.org/packages/aa/d9/61fdb77e4db7a88e099c695987cf1ae85b4d72666f9ca396b303d108fa06/alienpy-1.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-10 12:35:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "adriansev",
"github_project": "jalien_py",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "alienpy"
}
JSON
