Python iRODS Client (PRC)
=========================
[iRODS](https://www.irods.org) is an open source distributed data management system. This is a client API implemented in Python.
Currently supported:
- Python 2.7, 3.4 or newer
- Establish a connection to iRODS
- Authenticate via password, GSI, PAM
- iRODS connection over SSL
- Implement basic GenQueries (select columns and filtering)
- Support more advanced GenQueries with limits, offsets, and aggregations
- Query the collections and data objects within a collection
- Execute direct SQL queries
- Execute iRODS rules
- Support read, write, and seek operations for files
- Parallel PUT/GET data objects
- Create collections
- Rename collections
- Delete collections
- Create data objects
- Rename data objects
- Checksum data objects
- Delete data objects
- Register files and directories
- Query metadata for collections and data objects
- Add, edit, remove metadata
- Replicate data objects to different resource servers
- Connection pool management
- Implement GenQuery result sets as lazy queries
- Return empty result sets when CAT_NO_ROWS_FOUND is raised
- Manage permissions
- Manage users and groups
- Manage resources
- Unicode strings
- Ticket based access
Installing
----------
PRC requires Python 2.7 or 3.4+.
Canonically, to install with pip:
pip install python-irodsclient
or:
pip install git+https://github.com/irods/python-irodsclient.git[@branch|@commit|@tag]
Uninstalling
------------
pip uninstall python-irodsclient
Hazard: Outdated Python
-----------------------
With older versions of Python (as of this writing, the aforementioned
2.7 and 3.4), we can take preparatory steps toward securing workable
versions of pip and virtualenv by using these commands:
$ pip install --upgrade --user pip'<21.0'
$ python -m pip install --user virtualenv
We are then ready to use any of the following commands relevant to and
required for the installation:
$ python -m virtualenv ...
$ python -m pip install ...
Establishing a (secure) connection
----------------------------------
One way of starting a session is to pass iRODS credentials as keyword
arguments:
```python
>>> from irods.session import iRODSSession
>>> with iRODSSession(host='localhost', port=1247, user='bob', password='1234', zone='tempZone') as session:
... # workload
...
>>>
```
If you're an administrator acting on behalf of another user:
```python
>>> from irods.session import iRODSSession
>>> with iRODSSession(host='localhost', port=1247, user='rods', password='1234', zone='tempZone', client_user='bob',
client_zone='possibly_another_zone') as session:
... # workload
...
>>>
```
If no `client_zone` is provided, the `zone` parameter is used in its place.
Using environment files (including any SSL settings) in `~/.irods/`:
```python
>>> import os
>>> import ssl
>>> from irods.session import iRODSSession
>>> try:
... env_file = os.environ['IRODS_ENVIRONMENT_FILE']
... except KeyError:
... env_file = os.path.expanduser('~/.irods/irods_environment.json')
...
>>> ssl_settings = {} # Or, optionally: {'ssl_context': <user_customized_SSLContext>}
>>> with iRODSSession(irods_env_file=env_file, **ssl_settings) as session:
... # workload
...
>>>
```
In the above example, an SSL connection can be made even if no
'ssl_context' argument is specified, in which case the Python client
internally generates its own SSLContext object to best match the iRODS
SSL configuration parameters (such as
"irods_ssl_ca_certificate_file", etc.) used to initialize the
iRODSSession. Those parameters can be given either in the environment
file, or in the iRODSSession constructor call as shown by the next
example.
A pure Python SSL session (without a local `env_file` requires a few more things defined:
```python
>>> import ssl
>>> from irods.session import iRODSSession
>>> ssl_settings = {'client_server_negotiation': 'request_server_negotiation',
... 'client_server_policy': 'CS_NEG_REQUIRE',
... 'encryption_algorithm': 'AES-256-CBC',
... 'encryption_key_size': 32,
... 'encryption_num_hash_rounds': 16,
... 'encryption_salt_size': 8,
... 'ssl_context': ssl_context
... 'ssl_verify_server': 'cert',
... 'ssl_ca_certificate_file': '/etc/irods/ssl/irods.crt'
... }
```
If necessary, a user may provide a custom SSLContext object; although,
as of release v1.1.6, this will rarely be required:
```python
>>> ssl_settings ['ssl_context'] = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, # ... other options
... )
```
At this point, we are ready to instantiate and use the session:
```python
>>> with iRODSSession(host='irods-provider', port=1247, user='bob', password='1234', zone='tempZone', **ssl_settings) as session:
... # workload
```
Note that the `irods_` prefix is unnecessary when providing
the `encryption_*` and `ssl_*` options
directly to the constructor as keyword arguments, even though it is
required when they are placed in the environment file.
PAM logins
----------
Starting with v2.0.0, the python iRODS client is able to authenticate under PAM using the same file-based client environment as the
iCommands.
Caveat for iRODS 4.3+: when upgrading from 4.2, the "irods_authentication_scheme" setting must be changed from "pam" to "pam_password" in
`~/.irods/irods_environment.json` for all file-based client environments.
Maintaining a connection
------------------------
The default library timeout for a connection to an iRODS Server is 120
seconds.
This can be overridden by changing the session `connection_timeout` immediately after creation of the
session object:
```python
>>> session.connection_timeout = 300
```
This will set the timeout to five minutes for any associated
connections.
Session objects and cleanup
---------------------------
When iRODSSession objects are kept as state in an application, spurious
SYS_HEADER_READ_LEN_ERR errors can sometimes be seen in the
connected iRODS server's log file. This is frequently seen at program
exit because socket connections are terminated without having been
closed out by the session object's cleanup() method.
Starting with PRC Release 0.9.0, code has been included in the session
object's `__del__` method to call cleanup(), properly closing out
network connections. However, `__del__` cannot be relied to run under
all circumstances (Python2 being more problematic), so an alternative
may be to call session.cleanup() on any session variable which might not
be used again.
Simple PUTs and GETs
--------------------
We can use the just-created session object to put files to (or get them
from) iRODS.
```python
>>> logical_path = "/{0.zone}/home/{0.username}/{1}".format(session,"myfile.dat")
>>> session.data_objects.put( "myfile.dat", logical_path)
>>> session.data_objects.get( logical_path, "/tmp/myfile.dat.copy" )
```
Note that local file paths may be relative, but iRODS data objects must
always be referred to by their absolute paths. This is in contrast to
the `iput` and `iget` icommands, which keep track of the current working
collection (as modified by `icd`) for the unix shell.
Parallel Transfer
-----------------
Starting with release 0.9.0, data object transfers using put() and get()
will spawn a number of threads in order to optimize performance for
iRODS server versions 4.2.9+ and file sizes larger than a default
threshold value of 32 Megabytes.
Working with collections
------------------------
```python
>>> coll = session.collections.get("/tempZone/home/rods")
>>> coll.id
45798
>>> coll.path
/tempZone/home/rods
>>> for col in coll.subcollections:
>>> print(col)
<iRODSCollection /tempZone/home/rods/subcol1>
<iRODSCollection /tempZone/home/rods/subcol2>
>>> for obj in coll.data_objects:
>>> print(obj)
<iRODSDataObject /tempZone/home/rods/file.txt>
<iRODSDataObject /tempZone/home/rods/file2.txt>
```
Create a new collection:
```python
>>> coll = session.collections.create("/tempZone/home/rods/testdir")
>>> coll.id
45799
```
Working with data objects (files)
---------------------------------
Create a new data object:
```python
>>> obj = session.data_objects.create("/tempZone/home/rods/test1")
<iRODSDataObject /tempZone/home/rods/test1>
```
Get an existing data object:
```python
>>> obj = session.data_objects.get("/tempZone/home/rods/test1")
>>> obj.id 12345
>>> obj.name
test1
>>> obj.collection
<iRODSCollection /tempZone/home/rods>
>>> for replica in obj.replicas:
... print(replica.resource_name)
... print(replica.number)
... print(replica.path)
... print(replica.status)
...
demoResc
0
/var/lib/irods/Vault/home/rods/test1
1
```
Using the put() method rather than the create() method will trigger different policy enforcement points (PEPs) on the server.
Put an existing file as a new data object:
```python
>>> session.data_objects.put("test.txt", "/tempZone/home/rods/test2")
>>> obj2 = session.data_objects.get("/tempZone/home/rods/test2")
>>> obj2.id
56789
```
Specifying paths
----------------
Path strings for collection and data objects are usually expected to be
absolute in most contexts in the PRC. They must also be normalized to a
form including single slashes separating path elements and no slashes at
the string's end. If there is any doubt that a path string fulfills
this requirement, the wrapper class `irods.path.iRODSPath` (a subclass of `str`) may be used to normalize it:
if not session.collections.exists( iRODSPath( potentially_unnormalized_path )): #....
The wrapper serves also as a path joiner; thus:
iRODSPath( zone, "home", user )
may replace:
"/".join(["", zone, "home", user])
`iRODSPath` is available beginning with PRC release `v1.1.2`.
Reading and writing files
-------------------------
PRC provides [file-like
objects](http://docs.python.org/2/library/stdtypes.html#file-objects) for reading and writing files.
```python
>>> obj = session.data_objects.get("/tempZone/home/rods/test1")
>>> with obj.open('r+') as f:
... f.write('foonbarn')
... f.seek(0,0)
... for line in f:
... print(line)
...
foo
bar
```
As of v1.1.9, there is also an auto-close configuration setting for data
objects, set to `False` by default, which may be assigned
the value `True` for guaranteed auto-closing of open data
object handles at the proper time.
In a small but illustrative example, the following Python session does
not require an explicit call to `f.close()`:
```python
>>> import irods.client_configuration as config, irods.test.helpers as helpers
>>> config.data_objects.auto_close = True
>>> session = helpers.make_session()
>>> f = session.data_objects.open('/{0.zone}/home/{0.username}/new_object.txt'.format(session),'w')
>>> f.write(b'new content.')
```
This may be useful for Python programs in which frequent flushing of
write updates to data objects is undesirable -- with descriptors on such
objects possibly being held open for indeterminately long lifetimes --
yet the eventual application of those updates prior to the teardown of
the Python interpreter is required.
The current value of the setting is global in scope (i.e. applies to all
sessions, whenever created) and is always consulted for the creation of
any data object handle to govern that handle's cleanup behavior.
Python iRODS Client Settings File
---------------------------------
As of v1.1.9, Python iRODS client configuration can be saved in, and
loaded from, a settings file.
If the settings file exists, each of its lines contains (a) a dotted
name identifying a particular configuration setting to be assigned
within the PRC, potentially changing its runtime behavior; and (b) the
specific value, in Python "repr"-style format, that should be assigned
into it.
An example follows:
```
data_objects.auto_close True
```
New dotted names may be created following the example of the one valid
example created thus far, `data_objects.auto_close]`,
initialized in `irods/client_configuration/__init__.py`.
Each such name should correspond to a globally set value which the PRC
routinely checks when performing the affected library function.
The use of a settings file can be indicated, and the path to that file
determined, by setting the environment variable:
`PYTHON_IRODSCLIENT_CONFIGURATION_PATH`. If this variable
is present but empty, this denotes use of a default settings file path
of `~/.python-irodsclient`; if the variable's value is of
non-zero length, the value should be an absolute path to the settings
file whose use is desired. Also, if the variable is set, auto-load of
settings will be performed, meaning that the act of importing
`irods` or any of its submodules will cause the automatic
loading the settings from the settings file, assuming it exists.
(Failure to find the file at the indicated path will be logged as a
warning.)
Settings can also be saved and loaded manually using the `save()` and
`load()` functions in the `irods.client_configuration`
module. Each of these functions accepts an optional `file`
parameter which, if set to a non-empty string, will override the
settings file path currently "in force" (i.e., the
CONFIG_DEFAULT_PATH, as optionally overridden by the environment
variable PYTHON_IRODSCLIENT_CONFIGURATION_PATH).
Configuration settings may also be individually overridden by defining
certain environment variables. Here are relevant descriptions for each
one currently available, including the names of the environment
variables serving as overrides:
- Setting: Auto-close option for all data objects.
- Dotted Name: `data_objects.auto_close`
- Type: `bool`
- Default Value: `False`
- Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__DATA_OBJECTS__AUTO_CLOSE`
- Setting: Number of hours to request for the new password entry's TTL (Time To Live) when auto-renewing PAM-authenticated sessions.
- Dotted Name: `legacy_auth.pam.time_to_live_in_hours`
- Type: `int`
- Default Value: `0` (Meaning: conform to server's default TTL value.)
- Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__TIME_TO_LIVE_IN_HOURS`
- Setting: Plaintext PAM password value, to be used when auto-renewing PAM-authenticated sessions because TTL has expired.
- Dotted Name: `legacy_auth.pam.password_for_auto_renew`
- Type: `str`
- Default Value: `""` (Meaning: no password is set, and thus no automatic attempts will be made at auto-renewing PAM authentication.)
- Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__PASSWORD_FOR_AUTO_RENEW`. (But note that use of the environment variable could pose a threat to password security.)
- Setting: Whether to write the (native encoded) new hashed password to the iRODS password file. This step is only performed while auto-renewing PAM authenticated sessions.
- Dotted Name: `legacy_auth.pam.store_password_to_environment`
- Type: `bool`
- Default Value: `False`
- Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__STORE_PASSWORD_TO_ENVIRONMENT`
- Setting: Default choice of XML parser for all new threads.
- Dotted Name: `connections.xml_parser_default`
- Type: `str`
- Default Value: `"STANDARD_XML"`
- Possible Values: Any of `["STANDARD_XML", "QUASI_XML", "SECURE_XML"]`
- Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__CONNECTIONS__XML_PARSER_DEFAULT`
For example, if `~/python_irodsclient` contains the line :
```
connections.xml_parser_default "QUASI_XML"
```
then the session below illustrates the effect of defining the
appropriate environment variable. Note the value stored in the variable
must be a valid input for `ast.literal_eval()`; that is, a
primitive Pythonic value - and quoted, for instance, if a string.
```bash
$ PYTHON_IRODSCLIENT_CONFIGURATION_PATH="" \
PYTHON_IRODSCLIENT_CONFIG__CONNECTIONS__XML_PARSER_DEFAULT="'SECURE_XML'" \
python -c "import irods.message, irods.client_configuration as c; print (irods.message.default_XML_parser())"
XML_Parser_Type.SECURE_XML
$ PYTHON_IRODSCLIENT_CONFIGURATION_PATH="" \
python -c "import irods.message, irods.client_configuration as c; print (irods.message.default_XML_parser())"
XML_Parser_Type.QUASI_XML
```
Computing and Retrieving Checksums
----------------------------------
Each data object may be associated with a checksum by calling chksum()
on the object in question. Various behaviors can be elicited by passing
in combinations of keywords (for a description of which, please consult
the [header documentation](https://github.com/irods/irods/blob/4-2-stable/lib/api/include/dataObjChksum.h).)
As with most other iRODS APIs, it is straightforward to specify keywords
by adding them to an option dictionary:
```python
>>> data_object_1.chksum() # - computes the checksum if already in the catalog, otherwise computes and stores it
... # (i.e. default behavior with no keywords passed in.)
>>> from irods.manager.data_object_manager import Server_Checksum_Warning
>>> import irods.keywords as kw
>>> opts = { kw.VERIFY_CHKSUM_KW:'' }
>>> try:
... data_object_2.chksum( **opts ) # - Uses verification option. (Does not auto-vivify a checksum field).
... # or:
... opts[ kw.NO_COMPUTE_KW ] = ''
... data_object_2.chksum( **opts ) # - Uses both verification and no-compute options. (Like `ichksum -K --no-compute`)
... except Server_Checksum_Warning:
... print('some checksums are missing or wrong')
```
Additionally, if a freshly created `irods.message.RErrorStack` instance is
given, information can be returned and read by the client:
```python
>>> from irods.message import RErrorStack
>>> r_err_stk = RErrorStack()
>>> warn = None
>>> try: # Here, data_obj has one replica, not yet checksummed.
... data_obj.chksum( r_error = r_err_stk , **{kw.VERIFY_CHKSUM_KW:''} )
... except Server_Checksum_Warning as exc:
... warn = exc
>>> print(r_err_stk)
[RError<message = u'WARNING: No checksum available for replica [0].', status = -862000 CAT_NO_CHECKSUM_FOR_REPLICA>]
```
Working with metadata
---------------------
To enumerate AVUs on an object. With no metadata attached, the result
is an empty list:
```python
>>> from irods.meta import iRODSMeta
>>> obj = session.data_objects.get("/tempZone/home/rods/test1")
>>> print(obj.metadata.items())
[]
```
We then add some metadata. Just as with the icommand equivalent "imeta
add ...", we can add multiple AVUs with the same name field:
```python
>>> obj.metadata.add('key1', 'value1', 'units1')
>>> obj.metadata.add('key1', 'value2')
>>> obj.metadata.add('key2', 'value3')
>>> obj.metadata.add('key2', 'value4')
>>> print(obj.metadata.items())
[<iRODSMeta 13182 key1 value1 units1>, <iRODSMeta 13185 key2 value4 None>,
<iRODSMeta 13183 key1 value2 None>, <iRODSMeta 13184 key2 value3 None>]
```
We can also use Python's item indexing syntax to perform the equivalent
of an "imeta set \...", e.g. overwriting all AVUs with a name field
of "key2" in a single update:
```python
>>> new_meta = iRODSMeta('key2','value5','units2')
>>> obj.metadata\[new_meta.name\] = new_meta
>>> print(obj.metadata.items())
[<iRODSMeta 13182 key1 value1 units1>, <iRODSMeta 13183 key1 value2 None>,
<iRODSMeta 13186 key2 value5 units2>]
```
Now, with only one AVU on the object with a name of "key2", *get_one*
is assured of not throwing an exception:
```python
>>> print(obj.metadata.get_one('key2'))
<iRODSMeta 13186 key2 value5 units2>
```
However, the same is not true of "key1":
```python
>>> print(obj.metadata.get_one('key1'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/[...]/python-irodsclient/irods/meta.py", line 41, in get_one
raise KeyError
KeyError
```
Finally, to remove a specific AVU from an object:
```python
>>> obj.metadata.remove('key1', 'value1', 'units1')
>>> print(obj.metadata.items())
[<iRODSMeta 13186 key2 value5 units2>, <iRODSMeta 13183 key1 value2 None>]
Alternately, this form of the `remove()` method can also be useful:
```python
>>> for avu in obj.metadata.items():
... obj.metadata.remove(avu)
>>> print(obj.metadata.items())
[]
```
If we intended on deleting the data object anyway, we could have just
done this instead:
```
>>> obj.unlink(force=True)
```
But notice that the force option is important, since a data object in
the trash may still have AVUs attached.
At the end of a long session of AVU add/manipulate/delete operations,
one should make sure to delete all unused AVUs. We can in fact use any
`*Meta` data model in the queries below, since unattached AVUs are
not aware of the (type of) catalog object they once annotated:
```python
>>> from irods.models import (DataObjectMeta, ResourceMeta)
>>> len(list( session.query(ResourceMeta) ))
4
>>> from irods.test.helpers import remove_unused_metadata
>>> remove_unused_metadata(session)
>>> len(list( session.query(ResourceMeta) ))
0
```
When altering a fetched iRODSMeta, we must copy it first to avoid
errors, due to the fact the reference is cached by the iRODS object
reference. A shallow copy is sufficient:
```python
>>> meta = album.metadata.items()[0]
>>> meta.units
'quid'
>>> import copy; meta = copy.copy(meta); meta.units = 'pounds sterling'
>>> album.metadata[ meta.name ] = meta
```
Fortunately, as of PRC >= 1.1.4, we can simply do this instead:
```python
>>> album.metadata.set( meta )
```
In versions of iRODS 4.2.12 and later, we can also do:
```python
>>> album.metadata.set( meta, \*\*{kw.ADMIN_KW: ''} )
```
or even:
```python
>>> album.metadata(admin = True)\[meta.name\] = meta
```
In v1.1.5, the "timestamps" keyword is provided to enable the loading
of create and modify timestamps for every AVU returned from the server:
```python
>>> avus = album.metadata(timestamps = True).items()
>>> avus[0].create_time
datetime.datetime(2022, 9, 19, 15, 26, 7)
```
Atomic operations on metadata
-----------------------------
With release 4.2.8 of iRODS, the atomic metadata API was introduced to
allow a group of metadata add and remove operations to be performed
transactionally, within a single call to the server. This capability can
be leveraged in version 0.8.6 of the PRC.
So, for example, if 'obj' is a handle to an object in the iRODS
catalog (whether a data object, collection, user or storage resource),
we can send an arbitrary number of AVUOperation instances to be executed
together as one indivisible operation on that object:
```python
>>> from irods.meta import iRODSMeta, AVUOperation
>>> obj.metadata.apply_atomic_operations( AVUOperation(operation='remove', avu=iRODSMeta('a1','v1','these_units')),
... AVUOperation(operation='add', avu=iRODSMeta('a2','v2','those_units')),
... AVUOperation(operation='remove', avu=iRODSMeta('a3','v3')) \# , ...
... )
```
The list of operations will applied in the order given, so that a
"remove" followed by an "add" of the same AVU is, in effect, a
metadata "set" operation. Also note that a "remove" operation will
be ignored if the AVU value given does not exist on the target object at
that point in the sequence of operations.
We can also source from a pre-built list of AVUOperations using
Python's `f(*args_list)` syntax. For example, this
function uses the atomic metadata API to very quickly remove all AVUs
from an object:
```python
>>> def remove_all_avus( Object ):
... avus_on_Object = Object.metadata.items()
... Object.metadata.apply_atomic_operations( *[AVUOperation(operation='remove', avu=i) for i in avus_on_Object] )
```
Special Characters
------------------
Of course, it is fine to put Unicode characters into your collection and
data object names. However, certain non-printable ASCII characters, and
the backquote character as well, have historically presented problems
- especially for clients using iRODS's human readable XML protocol.
Consider this small, only slighly contrived, application:
```python
from irods.test.helpers import make_session
def create_notes( session, obj_name, content = u'' ):
get_home_coll = lambda ses: "/{0.zone}/home/{0.username}".format(ses)
path = get_home_coll(session) + "/" + obj_name
with session.data_objects.open(path,"a") as f:
f.seek(0, 2) # SEEK_END
f.write(content.encode('utf8'))
return session.data_objects.get(path)
with make_session() as session:
# Example 1 : exception thrown when name has non-printable character
try:
create_notes( session, "lucky\033.dat", content = u'test' )
except:
pass
# Example 2 (Ref. issue: irods/irods #4132, fixed for 4.2.9 release of iRODS)
print(
create_notes( session, "Alice's diary").name # note diff (' != ') in printed name
)
```
This creates two data objects, but with less than optimal success. The
first example object is created but receives no content because an
exception is thrown trying to query its name after creation. In the
second example, for iRODS 4.2.8 and before, a deficiency in packStruct
XML protocol causes the backtick to be read back as an apostrophe, which
could create problems manipulating or deleting the object later.
As of PRC v1.1.0, we can mitigate both problems by switching in the
QUASI_XML parser for the default one:
```
from irods.message import (XML_Parser_Type, ET)
ET( XML_Parser_Type.QUASI_XML, session.server_version )
```
Two dedicated environment variables may also be used to customize the
Python client's XML parsing behavior via the setting of global defaults
during start-up.
For example, we can set the default parser to QUASI_XML, optimized for
use with version 4.2.8 of the iRODS server, in the following manner:
```
Bash-Shell> export PYTHON_IRODSCLIENT_DEFAULT_XML=QUASI_XML PYTHON_IRODSCLIENT_QUASI_XML_SERVER_VERSION=4,2,8
```
Other alternatives for PYTHON_IRODSCLIENT_DEFAULT_XML are
"STANDARD_XML" and "SECURE_XML". These two latter options denote
use of the xml.etree and defusedxml modules, respectively.
Only the choice of "QUASI_XML" is affected by the specification of a
particular server version.
Finally, note that these global defaults, once set, may be overridden on
a per-thread basis using `ET(parser_type, server_version)`.
We can also revert the current thread's XML parser back to the global
default by calling `ET(None)`.
Rule Execution
--------------
A simple example of how to execute an iRODS rule from the Python client
is as follows. Suppose we have a rule file `native1.r`
which contains a rule in native iRODS Rule Language:
```
main() {
writeLine("*stream",
*X ++ " squared is " ++ str(double(*X)^2) )
}
INPUT *X="3", *stream="serverLog"
OUTPUT null
```
The following Python client code will run the rule and produce the
appropriate output in the irods server log:
```
r = irods.rule.Rule( session, rule_file = 'native1.r')
r.execute()
```
With release v1.1.1, not only can we target a specific rule engine
instance by name (which is useful when more than one is present), but we
can also use a file-like object for the `rule_file`
parameter:
```
Rule( session, rule_file = io.StringIO(u'''mainRule() { anotherRule(*x); writeLine('stdout',*x) }\n'''
u'''anotherRule(*OUT) {*OUT='hello world!'}\n\n'''
u'''OUTPUT ruleExecOut\n'''),
instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance' )
```
Incidentally, if we wanted to change the `native1.r` rule
code print to stdout also, we could set the `INPUT`
parameter, `*stream`, using the Rule constructor's
`params` keyword argument. Similarly, we can change the
`OUTPUT` parameter from `null` to
`ruleExecOut`, to accommodate the output stream, via the
`output` argument:
```
r = irods.rule.Rule( session, rule_file = 'native1.r',
instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance',
params={'*stream':'"stdout"'} , output = 'ruleExecOut' )
output = r.execute( )
if output and len(output.MsParam_PI):
buf = output.MsParam_PI[0].inOutStruct.stdoutBuf.buf
if buf: print(buf.rstrip(b'\0').decode('utf8'))
```
(Changing the input value to be squared in this example is left as an
exercise for the reader!)
To deal with errors resulting from rule execution failure, two
approaches can be taken. Suppose we have defined this in the
`/etc/irods/core.re` rule-base:
```
rule_that_fails_with_error_code(*x) {
*y = (if (*x!="") then int(*x) else 0)
# if (SOME_PROCEDURE_GOES_WRONG) {
if (*y < 0) { failmsg(*y,"-- my error message --"); } #-> throws an error code of int(*x) in REPF
else { fail(); } #-> throws FAIL_ACTION_ENCOUNTERED_ERR in REPF
# }
}
```
We can run the rule thus:
```python
>>> Rule( session, body='rule_that_fails_with_error_code(""), instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance',
... ).execute( r_error = (r_errs:= irods.message.RErrorStack()) )
```
Where we've used the Python 3.8 "walrus operator" for brevity. The
error will automatically be caught and translated to a returned-error
stack:
```python
>>> pprint.pprint([vars(r) for r in r_errs])
[{'raw_msg_': 'DEBUG: fail action encountered\n'
'line 14, col 15, rule base core\n'
' else { fail(); }\n'
' ^\n'
'\n',
'status_': -1220000}]
```
Note, if a stringized negative integer is given , i.e. as a special fail
code to be thrown within the rule, we must add this code into a special
parameter to have this automatically caught as well:
```python
>>> Rule( session, body='rule_that_fails_with_error_code("-2")',instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance'
... ).execute( acceptable_errors = ( FAIL_ACTION_ENCOUNTERED_ERR, -2),
... r_error = (r_errs := irods.message.RErrorStack()) )
```
Because the rule is written to emit a custom error message via `failmsg()`
in this case, the resulting r_error stack will now include that custom
error message as a substring:
```python
>>> pprint.pprint([vars(r) for r in r_errs])
[{'raw_msg_': 'DEBUG: -- my error message --\n'
'line 21, col 20, rule base core\n'
' if (*y < 0) { failmsg(*y,"-- my error message --"); } '
'#-> throws an error code of int(*x) in REPF\n'
' ^\n'
'\n',
'status_': -1220000}]
```
Alternatively, or in combination with the automatic catching of errors,
we may also catch errors as exceptions on the client side. For example,
if the Python rule engine is configured, and the following rule is
placed in `/etc/irods/core.py`:
```python
def python_rule(rule_args, callback, rei):
# if some operation fails():
raise RuntimeError
```
we can trap the error thus:
```python
try:
Rule( session, body = 'python_rule', instance_name = 'irods_rule_engine_plugin-python-instance' ).execute()
except irods.exception.RULE_ENGINE_ERROR:
print('Rule execution failed!')
exit(1)
print('Rule execution succeeded!')
```
As fail actions from native rules are not thrown by default (refer to
the help text for `Rule.execute`), if we anticipate these
and prefer to catch them as exceptions, we can do it this way:
```python
try:
Rule( session, body = 'python_rule', instance_name = 'irods_rule_engine_plugin-python-instance'
).execute( acceptable_errors = () )
except (irods.exception.RULE_ENGINE_ERROR,
irods.exception.FAIL_ACTION_ENCOUNTERED_ERR) as e:
print('Rule execution failed!')
exit(1)
print('Rule execution succeeded!')
```
Finally, keep in mind that rule code submitted through an
`irods.rule.Rule` object is processed by the
exec_rule_text function in the targeted plugin instance. This may be a
limitation for plugins not equipped to handle rule code in this way. In
a sort of middle-ground case, the iRODS Python Rule Engine Plugin is not
currently able to handle simple rule calls and the manipulation of iRODS
core primitives (like simple parameter passing and variable expansion')
as flexibly as the iRODS Rule Language.
Also, core.py rules may not be run directly (as is also true with
`irule`) by other than a rodsadmin user pending the
resolution of [this
issue](https://github.com/irods/irods_rule_engine_plugin_python/issues/105).
General Queries
---------------
```python
>>> import os
>>> from irods.session import iRODSSession
>>> from irods.models import Collection, DataObject
>>>
>>> env_file = os.path.expanduser('~/.irods/irods_environment.json')
>>> with iRODSSession(irods_env_file=env_file) as session:
... query = session.query(Collection.name, DataObject.id, DataObject.name, DataObject.size)
...
... for result in query:
... print('{}/{} id={} size={}'.format(result[Collection.name], result[DataObject.name], result[DataObject.id], result[DataObject.size]))
...
/tempZone/home/rods/manager/access_manager.py id=212665 size=2164
/tempZone/home/rods/manager/access_manager.pyc id=212668 size=2554
/tempZone/home/rods/manager/collection_manager.py id=212663 size=4472
/tempZone/home/rods/manager/collection_manager.pyc id=212664 size=4464
/tempZone/home/rods/manager/data_object_manager.py id=212662 size=10291
/tempZone/home/rods/manager/data_object_manager.pyc id=212667 size=8772
/tempZone/home/rods/manager/__init__.py id=212670 size=79
/tempZone/home/rods/manager/__init__.pyc id=212671 size=443
/tempZone/home/rods/manager/metadata_manager.py id=212660 size=4263
/tempZone/home/rods/manager/metadata_manager.pyc id=212659 size=4119
/tempZone/home/rods/manager/resource_manager.py id=212666 size=5329
/tempZone/home/rods/manager/resource_manager.pyc id=212661 size=4570
/tempZone/home/rods/manager/user_manager.py id=212669 size=5509
/tempZone/home/rods/manager/user_manager.pyc id=212658 size=5233
```
Query using other models:
```python
>>> from irods.column import Criterion
>>> from irods.models import DataObject, DataObjectMeta, Collection, CollectionMeta
>>> from irods.session import iRODSSession
>>> import os
>>> env_file = os.path.expanduser('~/.irods/irods_environment.json')
>>> with iRODSSession(irods_env_file=env_file) as session:
... # by metadata
... # equivalent to 'imeta qu -C type like Project'
... results = session.query(Collection, CollectionMeta).filter( \
... Criterion('=', CollectionMeta.name, 'type')).filter( \
... Criterion('like', CollectionMeta.value, '%Project%'))
... for r in results:
... print(r[Collection.name], r[CollectionMeta.name], r[CollectionMeta.value], r[CollectionMeta.units])
...
('/tempZone/home/rods', 'type', 'Project', None)
```
Beginning with version 0.8.3 of PRC, the 'in' genquery operator is
also available:
```python
>>> from irods.models import Resource
>>> from irods.column import In
>>> [ resc[Resource.id]for resc in session.query(Resource).filter(In(Resource.name, ['thisResc','thatResc'])) ]
[10037,10038]
```
Query with aggregation(min, max, sum, avg, count):
```python
>>> with iRODSSession(irods_env_file=env_file) as session:
... query = session.query(DataObject.owner_name).count(DataObject.id).sum(DataObject.size)
... print(next(query.get_results()))
{<irods.column.Column 411 D_OWNER_NAME>: 'rods', <irods.column.Column 407 DATA_SIZE>: 62262, <irods.column.Column 401 D_DATA_ID>: 14}
```
In this case since we are expecting only one row we can directly call
`query.execute()`:
```python
>>> with iRODSSession(irods_env_file=env_file) as session:
... query = session.query(DataObject.owner_name).count(DataObject.id).sum(DataObject.size)
... print(query.execute())
+--------------+-----------+-----------+
| D_OWNER_NAME | D_DATA_ID | DATA_SIZE |
+--------------+-----------+-----------+
| rods | 14 | 62262 |
+--------------+-----------+-----------+
```
For a case-insensitive query, add a `case_sensitive=False`
parameter to the query:
```python
>>> with iRODSSession(irods_env_file=env_file) as session:
... query = session.query(DataObject.name, case_sensitive=False).filter(Like(DataObject.name, "%oBjEcT"))
... print(query.all())
+---------------------+
| DATA_NAME |
+---------------------+
| caseSENSITIVEobject |
+---------------------+
```
Specific Queries
----------------
```python
>>> import os
>>> from irods.session import iRODSSession
>>> from irods.models import Collection, DataObject
>>> from irods.query import SpecificQuery
>>>
>>> env_file = os.path.expanduser('~/.irods/irods_environment.json')
>>> with iRODSSession(irods_env_file=env_file) as session:
... # define our query
... sql = "select data_name, data_id from r_data_main join r_coll_main using (coll_id) where coll_name = '/tempZone/home/rods/manager'"
... alias = 'list_data_name_id'
... columns = [DataObject.name, DataObject.id] # optional, if we want to get results by key
... query = SpecificQuery(session, sql, alias, columns)
...
... # register specific query in iCAT
... _ = query.register()
...
... for result in query:
... print('{} {}'.format(result[DataObject.name], result[DataObject.id]))
...
... # delete specific query
... _ = query.remove()
...
user_manager.pyc 212658
metadata_manager.pyc 212659
metadata_manager.py 212660
resource_manager.pyc 212661
data_object_manager.py 212662
collection_manager.py 212663
collection_manager.pyc 212664
access_manager.py 212665
resource_manager.py 212666
data_object_manager.pyc 212667
access_manager.pyc 212668
user_manager.py 212669
__init__.py 212670
__init__.pyc 212671
```
Recherché Queries
-----------------
In some cases you might like to use a GenQuery operator not directly
offered by this Python library, or even combine query filters in ways
GenQuery may not directly support.
As an example, the code below finds metadata value fields
lexicographically outside the range of decimal integers, while also
requiring that the data objects to which they are attached do not reside
in the trash.
```python
>>> search_tuple = (DataObject.name , Collection.name ,
... DataObjectMeta.name , DataObjectMeta.value)
>>> # "not like" : direct instantiation of Criterion (operator in literal string)
>>> not_in_trash = Criterion ('not like', Collection.name , '%/trash/%')
>>> # "not between"( column, X, Y) := column < X OR column > Y ("OR" done via chained iterators)
>>> res1 = session.query (* search_tuple).filter(not_in_trash).filter(DataObjectMeta.value < '0')
>>> res2 = session.query (* search_tuple).filter(not_in_trash).filter(DataObjectMeta.value > '9' * 9999 )
>>> chained_results = itertools.chain ( res1.get_results(), res2.get_results() )
>>> pprint( list( chained_results ) )
```
Instantiating iRODS objects from query results
----------------------------------------------
The General query works well for getting information out of the ICAT if
all we're interested in is information representable with primitive
types (i.e. object names, paths, and ID's, as strings or integers). But
Python's object orientation also allows us to create object references
to mirror the persistent entities (instances of *Collection*,
*DataObject*, *User*, or *Resource*, etc.) inhabiting the ICAT.
**Background:**
Certain iRODS object types can be instantiated easily
using the session object's custom type managers, particularly if some
parameter (often just the name or path) of the object is already known:
```python
>>> type(session.users)
<class 'irods.manager.user_manager.UserManager'>
>>> u = session.users.get('rods')
>>> u.id
10003
```
Type managers are good for specific operations, including object
creation and removal:
```python
>>> session.collections.create('/tempZone/home/rods/subColln')
>>> session.collections.remove('/tempZone/home/rods/subColln')
>>> session.data_objects.create('/tempZone/home/rods/dataObj')
>>> session.data_objects.unlink('/tempZone/home/rods/dataObj')
```
When we retrieve a reference to an existing collection using *get* :
```python
>>> c = session.collections.get('/tempZone/home/rods')
>>> c
<iRODSCollection 10011 rods>
```
we have, in that variable *c*, a reference to an iRODS *Collection*
object whose properties provide useful information:
```python
>>> [ x for x in dir(c) if not x.startswith('__') ]
['_meta', 'data_objects', 'id', 'manager', 'metadata', 'move', 'name', 'path', 'remove', 'subcollections', 'unregister', 'walk']
>>> c.name
'rods'
>>> c.path
'/tempZone/home/rods'
>>> c.data_objects
[<iRODSDataObject 10019 test1>]
>>> c.metadata.items()
[ <... list of AVUs attached to Collection c ... > ]
```
or whose methods can do useful things:
```python
>>> for sub_coll in c.walk(): print('---'); pprint( sub_coll )
[ ...< series of Python data structures giving the complete tree structure below collection 'c'> ...]
```
This approach of finding objects by name, or via their relations with
other objects (ie "contained by", or in the case of metadata,
"attached to"), is helpful if we know something about the location or
identity of what we're searching for, but we don't always have that
kind of a-priori knowledge.
So, although we can (as seen in the last example) walk an
*iRODSCollection* recursively to discover all subordinate collections
and their data objects, this approach will not always be best for a
given type of application or data discovery, especially in more advanced
use cases.
**A Different Approach:**
For the PRC to be sufficiently powerful for general use, we'll often need at least:
- general queries, and
- the capabilities afforded by the PRC's object-relational mapping.
Suppose, for example, we wish to enumerate all collections in the iRODS
catalog.
Again, the object managers are the answer, but they are now invoked
using a different scheme:
```python
>>> from irods.collection import iRODSCollection; from irods.models import Collection
>>> all_collns = [ iRODSCollection(session.collections,result) for result in session.query(Collection) ]
```
From there, we have the ability to do useful work, or filtering based on
the results of the enumeration. And, because *all_collns* is an
iterable of true objects, we can either use Python's list
comprehensions or execute more catalog queries to achieve further aims.
Note that, for similar system-wide queries of Data Objects (which, as it
happens, are inextricably joined to their parent Collection objects), a
bit more finesse is required. Let us query, for example, to find all
data objects in a particular zone with an AVU that matches the following
condition:
```
META_DATA_ATTR_NAME = "irods::alert_time" and META_DATA_ATTR_VALUE like '+0%'
```
```python
>>> import irods.keywords
>>> from irods.data_object import iRODSDataObject
>>> from irods.models import DataObjectMeta, DataObject
>>> from irods.column import Like
>>> q = session.query(DataObject).filter( DataObjectMeta.name == 'irods::alert_time',
Like(DataObjectMeta.value, '+0%') )
>>> zone_hint = "" # --> add a zone name in quotes to search another zone
>>> if zone_hint: q = q.add_keyword( irods.keywords.ZONE_KW, zone_hint )
>>> for res in q:
... colln_id = res [DataObject.collection_id]
... collObject = get_collection( colln_id, session, zone = zone_hint)
... dataObject = iRODSDataObject( session.data_objects, parent = collObject, results=[res])
... print( '{coll}/{data}'.format (coll = collObject.path, data = dataObject.name))
```
In the above loop we have used a helper function, *get_collection*, to
minimize the number of hits to the object catalog. Otherwise, me might
find within a typical application that some Collection objects are being
queried at a high rate of redundancy. *get_collection* can be
implemented thusly:
```python
import collections # of the Pythonic, not iRODS, kind
def makehash():
# see https://stackoverflow.com/questions/651794/whats-the-best-way-to-initialize-a-dict-of-dicts-in-python
return collections.defaultdict(makehash)
from irods.collection import iRODSCollection
from irods.models import Collection
def get_collection (Id, session, zone=None, memo = makehash()):
if not zone: zone = ""
c_obj = memo[session][zone].get(Id)
if c_obj is None:
q = session.query(Collection).filter(Collection.id==Id)
if zone != '': q = q.add_keyword( irods.keywords.ZONE_KW, zone )
c_id = q.one()
c_obj = iRODSCollection(session, result = c_id)
memo[session][zone][Id] = c_obj
return c_obj
```
Once instantiated, of course, any *iRODSDataObject*'s data to which we
have access permissions is available via its open() method.
As stated, this type of object discovery requires some extra study and
effort, but the ability to search arbitrary iRODS zones (to which we are
federated and have the user permissions) is powerful indeed.
Tickets
-------
The `irods.ticket.Ticket` class lets us issue "tickets"
which grant limited permissions for other users to access our own data
objects (or collections of data objects). As with the iticket client,
the access may be either "read" or "write". The recipient of the
ticket could be a rodsuser, or even an anonymous user.
Below is a demonstration of how to generate a new ticket for access to a
logical path - in this case, say a collection containing 1 or more data
objects. (We assume the creation of the granting_session and
receiving_session for the users respectively for the users providing
and consuming the ticket access.)
The user who wishes to provide an access may execute the following:
```python
>>> from irods.ticket import Ticket
>>> new_ticket = Ticket (granting_session)
>>> The_Ticket_String = new_ticket.issue('read',
... '/zone/home/my/collection_with_data_objects_for/somebody').string
```
at which point that ticket's unique string may be given to other users,
who can then apply the ticket to any existing session object in order to
gain access to the intended object(s):
```python
>>> from irods.models import Collection, DataObject
>>> ses = receiving_session
>>> Ticket(ses, The_Ticket_String).supply()
>>> c_result = ses.query(Collection).one()
>>> c = iRODSCollection( ses.collections, c_result)
>>> for dobj in (c.data_objects):
... ses.data_objects.get( dobj.path, '/tmp/' + dobj.name ) # download objects
```
In this case, however, modification will not be allowed because the
ticket is for read only:
```python
>>> c.data_objects[0].open('w').write( # raises
... b'new content') # CAT_NO_ACCESS_PERMISSION
```
In another example, we could generate a ticket that explicitly allows
'write' access on a specific data object, thus granting other users
the permissions to modify as well as read it:
```python
>>> ses = iRODSSession( user = 'anonymous', password = '', host = 'localhost',
port = 1247, zone = 'tempZone')
>>> Ticket(ses, write_data_ticket_string ).supply()
>>> d_result = ses.query(DataObject.name,Collection.name).one()
>>> d_path = ( d_result[Collection.name] + '/' +
... d_result[DataObject.name] )
>>> old_content = ses.data_objects.open(d_path,'r').read()
>>> with tempfile.NamedTemporaryFile() as f:
... f.write(b'blah'); f.flush()
... ses.data_objects.put(f.name,d_path)
```
As with iticket, we may set a time limit on the availability of a
ticket, either as a timestamp or in seconds since the epoch:
```python
>>> t=Ticket(ses); s = t.string
vIOQ6qzrWWPO9X7
>>> t.issue('read','/some/path')
>>> t.modify('expiry','2021-04-01.12:34:56') # timestamp assumed as UTC
```
To check the results of the above, we could invoke this icommand
elsewhere in a shell prompt:
```
iticket ls vIOQ6qzrWWPO9X7
```
and the server should report back the same expiration timestamp.
And, if we are the issuer of a ticket, we may also query, filter on, and
extract information based on a ticket's attributes and catalog
relations:
```python
>>> from irods.models import TicketQuery
>>> delay = lambda secs: int( time.time() + secs + 1)
>>> Ticket(ses).issue('read','/path/to/data_object').modify(
'expiry',delay(7*24*3600)) # lasts 1 week
>>> Q = ses.query (TicketQuery.Ticket, TicketQuery.DataObject).filter(
... TicketQuery.DataObject.name == 'data_object')
>>> print ([ _[TicketQuery.Ticket.expiry_ts] for _ in Q ])
['1636757427']
```
Tracking and manipulating replicas of Data Objects
--------------------------------------------------
Putting together the techniques we've seen so far, it's not hard to write client code to accomplish
useful, common tasks. Suppose, for instance, that a data object contains replicas on a given resource
or resource hierarchy (the "source"), and we want those replicas "moved" to a second resource
(the "destination"). This can be done by combining the replicate and trim operations, as in the following
code excerpt.
We'll assume, for our current purposes, that all pre-existing replicas are good (ie. they have a
`status` attribute of `'1'`); and that the nodes in question are named `src` and `dest`,
with `src` being the root node of a resource hierarchy and `dest` just a simple storage node.
Then we can accomplish the replica "move" thus:
```python
path = '/path/to/data/object'
data = session.data_objects.get('/path/to/data/object')
# Replicate the data object to the destination.
data.replicate(**{kw.DEST_RESC_NAME_KW: 'dest'})
# Find and trim replicas on the source resource hierarchy.
replica_numbers = [r.number for r in d.replicas if r.resc_hier.startswith('src;')]
for number in replica_numbers:
session.data_objects.trim(path, **{kw.DATA_REPL_NUM:number, kw.COPIES_KW:1})
```
Listing Users and Groups ; calculating Group Membership
-------------------------------------------------------
iRODS tracks groups and users using two tables, R_USER_MAIN and
R_USER_GROUP. Under this database schema, all "user groups" are also
users:
```python
>>> from irods.models import User, Group
>>> from pprint import pprint
>>> pprint(list((x[User.id], x[User.name]) for x in session.query(User)))
[(10048, 'alice'),
(10001, 'rodsadmin'),
(13187, 'bobby'),
(10045, 'collab'),
(10003, 'rods'),
(13193, 'empty'),
(10002, 'public')]
```
But it's also worth noting that the User.type field will be
'rodsgroup' for any user ID that iRODS internally recognizes as a
"Group":
```python
>>> groups = session.query(User).filter( User.type == 'rodsgroup' )
>>> [x[User.name] for x in groups]
['collab', 'public', 'rodsadmin', 'empty']
```
Since we can instantiate iRODSGroup and iRODSUser objects directly from
the rows of a general query on the corresponding tables, it is also
straightforward to trace out the groups' memberships:
```python
>>> from irods.user import iRODSUser, iRODSGroup
>>> grp_usr_mapping = [ (iRODSGroup(session.groups, result), iRODSUser(session.users, result)) \
... for result in session.query(Group,User) ]
>>> pprint( [ (x,y) for x,y in grp_usr_mapping if x.id != y.id ] )
[(<iRODSGroup 10045 collab>, <iRODSUser 10048 alice rodsuser tempZone>),
(<iRODSGroup 10001 rodsadmin>, <iRODSUser 10003 rods rodsadmin tempZone>),
(<iRODSGroup 10002 public>, <iRODSUser 10003 rods rodsadmin tempZone>),
(<iRODSGroup 10002 public>, <iRODSUser 10048 alice rodsuser tempZone>),
(<iRODSGroup 10045 collab>, <iRODSUser 13187 bobby rodsuser tempZone>),
(<iRODSGroup 10002 public>, <iRODSUser 13187 bobby rodsuser tempZone>)]
```
(Note that in general queries, fields cannot be compared to each other,
only to literal constants; thus the '!=' comparison in the Python list
comprehension.)
From the above, we can see that the group 'collab' (with user ID
10045) contains users 'bobby'(13187) and 'alice'(10048) but not
'rods'(10003), as the tuple (10045,10003) is not listed. Group
'rodsadmin'(10001) contains user 'rods'(10003) but no other users;
and group 'public'(10002) by default contains all canonical users
(those whose User.type is 'rodsadmin' or 'rodsuser'). The empty
group ('empty') has no users as members, so it doesn't show up in our
final list.
Group Administrator Capabilities
--------------------------------
With v1.1.7, PRC acquires the full range of abilities possessed by the
igroupadmin command.
Firstly, a groupadmin may invoke methods to create groups, and may add
users to, or remove them from, any group to which they themselves
belong:
```python
>>> session.groups.create('lab')
>>> session.groups.addmember('lab',session.username) # allow self to administer group
>>> session.groups.addmember('lab','otheruser')
>>> session.groups.removemember('lab','otheruser')
```
In addition, a groupadmin may also create accounts for new users and
enable their logins by initializing a native password for them:
```python
>>> session.users.create_with_password('alice', 'change_me')
```
iRODS Permissions (ACLs)
------------------------
The `iRODSAccess` class offers a convenient dictionary
interface mapping iRODS permission strings to the corresponding integer
codes:
```python
>>> from irods.access import iRODSAccess
>>> iRODSAccess.keys()
['null', 'read_metadata', 'read_object', 'create_metadata', 'modify_metadata', 'delete_metadata', 'create_object', 'modify_object', 'delete_object', 'own']
>>> WRITE = iRODSAccess.to_int('modify_object')
```
Armed with that, we can then query on all data objects with ACLs that
allow our user to write them:
```python
>>> from irods.models import (DataObject, User, DataAccess)
>>> data_objects_writable = list(session.query(DataObject, User, DataAccess).filter(User.name == session.username, DataAccess.type >= WRITE))
```
Finally, we can also access the list of permissions available through a
given session object via the `available_permissions`
property. Note that -- in keeping with changes in iRODS server 4.3 --
the permissions list will be longer, as appropriate, for session objects
connected to the more recent servers; and also that the embedded spaces
in some 4.2 permission strings will be replaced by underscores in 4.3
and later.
```python
>>> session.server_version
(4, 2, 11)
>>> session.available_permissions.items()
[('null', 1000), ('read object', 1050), ('modify object', 1120), ('own', 1200)]
```
Getting and setting permissions
-------------------------------
We can find the ID's of all the collections writable (ie having
"modify" ACL) by, but not owned by, alice (or even alice\#otherZone):
```python
>>> from irods.models import Collection,CollectionAccess,CollectionUser,User
>>> from irods.column import Like
>>> q = session.query (Collection,CollectionAccess).filter(
... CollectionUser.name == 'alice', # User.zone == 'otherZone', # zone optional
... Like(CollectionAccess.name, 'modify%') ) #defaults to current zone
```
If we then want to downgrade those permissions to read-only, we can do
the following:
```python
>>> from irods.access import iRODSAccess
>>> for c in q:
... session.acls.set( iRODSAccess('read', c[Collection.name], 'alice', # 'otherZone' # zone optional
... ))
```
A call to `session.acls.get(c)` -- with `c`
being the result of
`sessions.collections.get(c[Collection.name])` -- would
then verify the desired change had taken place (as well as list all ACLs
stored in the catalog for that collection).
One last note on permissions: The older access manager,
`<session>.permissions`, produced inconsistent results when
the `get()` method was invoked with the parameter
`report_raw_acls` set (or defaulting) to
`False`. Specifically, collections would exhibit the
"non-raw-ACL" behavior of reporting individual member users'
permissions as a by-product of group ACLs, whereas data objects would
not.
In release v1.1.6, we moved to correct this inconsistency by introducing
the synonym `<session>.acls` that acts almost identically
like `<session>.permissions`, except that the
`<session>.acls.get(...)` method does not accept the
`report_raw_acls` parameter. When we need to detect users'
permissions independent of their access to an object via group
membership, this can be achieved with another query.
`<session>.permissions` was therefore removed in v2.0.0
in favor of `<session>.acls`.
Quotas (v2.0.0)
---------------
Quotas may be set for a group:
```python
session.groups.set_quota('my_group', 50000, resource = 'my_limited_resource')
```
or per user, prior to iRODS 4.3.0:
```python
session.users.set_quota('alice', 100000)
```
(The default for the resource parameter is "total", denoting a general
quota usage not bound to a particular resource.)
The Quota model is also available for queries. So, to determine the
space remaining for a certain group on a given resource:
```python
from irods.models import Quota
session.groups.calculate_usage()
group, resource = ['my_group', 'my_limited_resource']
space_left_in_bytes = list(session.query(Quota.over).filter(Quota.user_id == session.groups.get(group).id,
Quota.resc_id == session.resources.get(resource).id))[0][Quota.over] * -1
```
And, to remove all quotas for a given group, one might (as a rodsadmin)
do the following:
```python
from irods.models import Resource, Quota
resc_map = dict([(x[Resource.id],x[Resource.name]) for x in sess.query(Resource)] + [(0,'total')])
group = sess.groups.get('my_group')
for quota in sess.query(Quota).filter(Quota.user_id == group.id):
sess.groups.remove_quota(group.name, resource = resc_map[quota.resc_id])
```
Managing users
--------------
You can create a user in the current zone (with an optional auth_str):
```python
>>> session.users.create('user', 'rodsuser', 'MyZone', auth_str)
```
If you want to create a user in a federated zone, use:
```python
>>> session.users.create('user', 'rodsuser', 'OtherZone', auth_str)
```
And more ...
------------
Additional code samples are available in the [test
directory](https://github.com/irods/python-irodsclient/tree/main/irods/test)
Testing
=======
Setting up and running tests
----------------------------
The Python iRODS Client comes with its own suite of tests. Some amount
of setting up may be necessary first:
1. Use `iinit` to specify the iRODS client environment.
For best results, point the client at a server running on the local
host.
2. Install the python-irodsclient along with the
`unittest unittest_xml_reporting` module or the older
`xmlrunner` equivalent.
- for PRC versions 1.1.1 and later:
- `pip install ./path-to-python-irodsclient-repo[tests]`
(when using a local Git repo); or,
- `pip install python-irodsclient[tests]'>=1.1.1'`
(when installing directly from PyPI).
- earlier releases (\<= 1.1.0) will install the outdated
`xmlrunner` module automatically
3. Follow further instructions in the [test
directory](https://github.com/irods/python-irodsclient/tree/main/irods/test)
Testing S3 parallel transfer
----------------------------
System requirements:
- Ubuntu 18 user with Docker installed.
- Local instance of iRODS server running.
- Logged in sudo privileges.
Run a MinIO service:
```
$ docker run -d -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"
```
Set up a bucket `s3://irods` under MinIO:
```
$ pip install awscli
$ aws configure
AWS Access Key ID [None]: minioadmin
AWS Secret Access Key [None]: minioadmin
Default region name [None]:
Default output format [None]:
$ aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://irods
```
Set up s3 credentials for the iRODS s3 storage resource:
```
$ sudo su - irods -c "/bin/echo -e 'minioadmin\nminioadmin' >/var/lib/irods/s3-credentials"
$ sudo chown 600 /var/lib/irods/s3-credentials
```
Create the s3 storage resource:
```
$ sudo apt install irods-resource-plugin-s3
```
As the 'irods' service account user:
```
$ iadmin mkresc s3resc s3 $(hostname):/irods/ \
"S3_DEFAULT_HOSTNAME=localhost:9000;"\
"S3_AUTH_FILE=/var/lib/irods/s3-credentials;"\
"S3_REGIONNAME=us-east-1;"\
"S3_RETRY_COUNT=1;"\
"S3_WAIT_TIME_SEC=3;"\
"S3_PROTO=HTTP;"\
"ARCHIVE_NAMING_POLICY=consistent;"\
"HOST_MODE=cacheless_attached"
$ dd if=/dev/urandom of=largefile count=40k bs=1k # create 40-megabyte test file
$ pip install 'python-irodsclient>=1.1.2'
$ python -c"from irods.test.helpers import make_session
import irods.keywords as kw
with make_session() as sess:
sess.data_objects.put( 'largefile',
'/tempZone/home/rods/largeFile1',
**{kw.DEST_RESC_NAME_KW:'s3resc'} )
sess.data_objects.get( '/tempZone/home/rods/largeFile1',
'/tmp/largefile')
```
Raw data
{
"_id": null,
"home_page": "https://github.com/irods/python-irodsclient",
"name": "python-irodsclient",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "irods",
"author": "iRODS Consortium",
"author_email": "support@irods.org",
"download_url": "https://files.pythonhosted.org/packages/94/3f/8c869c0a09fa28adb65f5543190f34b42ae98f8b7add3dbafdbcdf66dfac/python-irodsclient-2.0.1.tar.gz",
"platform": null,
"description": "Python iRODS Client (PRC)\n=========================\n\n[iRODS](https://www.irods.org) is an open source distributed data management system. This is a client API implemented in Python.\n\nCurrently supported:\n\n- Python 2.7, 3.4 or newer\n- Establish a connection to iRODS\n- Authenticate via password, GSI, PAM\n- iRODS connection over SSL\n- Implement basic GenQueries (select columns and filtering)\n- Support more advanced GenQueries with limits, offsets, and aggregations\n- Query the collections and data objects within a collection\n- Execute direct SQL queries\n- Execute iRODS rules\n- Support read, write, and seek operations for files\n- Parallel PUT/GET data objects\n- Create collections\n- Rename collections\n- Delete collections\n- Create data objects\n- Rename data objects\n- Checksum data objects\n- Delete data objects\n- Register files and directories\n- Query metadata for collections and data objects\n- Add, edit, remove metadata\n- Replicate data objects to different resource servers\n- Connection pool management\n- Implement GenQuery result sets as lazy queries\n- Return empty result sets when CAT_NO_ROWS_FOUND is raised\n- Manage permissions\n- Manage users and groups\n- Manage resources\n- Unicode strings\n- Ticket based access\n\nInstalling\n----------\n\nPRC requires Python 2.7 or 3.4+.\n\nCanonically, to install with pip:\n\n pip install python-irodsclient\n\nor:\n\n pip install git+https://github.com/irods/python-irodsclient.git[@branch|@commit|@tag]\n\nUninstalling\n------------\n\n pip uninstall python-irodsclient\n\nHazard: Outdated Python\n-----------------------\n\nWith older versions of Python (as of this writing, the aforementioned\n2.7 and 3.4), we can take preparatory steps toward securing workable\nversions of pip and virtualenv by using these commands:\n\n $ pip install --upgrade --user pip'<21.0'\n $ python -m pip install --user virtualenv\n\nWe are then ready to use any of the following commands relevant to and\nrequired for the installation:\n\n $ python -m virtualenv ... \n $ python -m pip install ...\n\nEstablishing a (secure) connection\n----------------------------------\n\nOne way of starting a session is to pass iRODS credentials as keyword\narguments:\n\n```python\n>>> from irods.session import iRODSSession\n>>> with iRODSSession(host='localhost', port=1247, user='bob', password='1234', zone='tempZone') as session:\n... # workload\n...\n>>>\n```\n\nIf you're an administrator acting on behalf of another user:\n\n```python\n>>> from irods.session import iRODSSession\n>>> with iRODSSession(host='localhost', port=1247, user='rods', password='1234', zone='tempZone', client_user='bob',\n client_zone='possibly_another_zone') as session:\n... # workload\n...\n>>>\n```\n\nIf no `client_zone` is provided, the `zone` parameter is used in its place.\n\nUsing environment files (including any SSL settings) in `~/.irods/`:\n\n```python\n>>> import os\n>>> import ssl\n>>> from irods.session import iRODSSession\n>>> try:\n... env_file = os.environ['IRODS_ENVIRONMENT_FILE']\n... except KeyError:\n... env_file = os.path.expanduser('~/.irods/irods_environment.json')\n...\n>>> ssl_settings = {} # Or, optionally: {'ssl_context': <user_customized_SSLContext>}\n>>> with iRODSSession(irods_env_file=env_file, **ssl_settings) as session:\n... # workload\n...\n>>>\n```\n\nIn the above example, an SSL connection can be made even if no\n'ssl_context' argument is specified, in which case the Python client\ninternally generates its own SSLContext object to best match the iRODS\nSSL configuration parameters (such as\n\"irods_ssl_ca_certificate_file\", etc.) used to initialize the\niRODSSession. Those parameters can be given either in the environment\nfile, or in the iRODSSession constructor call as shown by the next\nexample.\n\nA pure Python SSL session (without a local `env_file` requires a few more things defined:\n\n```python\n>>> import ssl\n>>> from irods.session import iRODSSession\n>>> ssl_settings = {'client_server_negotiation': 'request_server_negotiation',\n... 'client_server_policy': 'CS_NEG_REQUIRE',\n... 'encryption_algorithm': 'AES-256-CBC',\n... 'encryption_key_size': 32,\n... 'encryption_num_hash_rounds': 16,\n... 'encryption_salt_size': 8,\n... 'ssl_context': ssl_context\n... 'ssl_verify_server': 'cert',\n... 'ssl_ca_certificate_file': '/etc/irods/ssl/irods.crt'\n... }\n```\n\nIf necessary, a user may provide a custom SSLContext object; although,\nas of release v1.1.6, this will rarely be required:\n\n```python\n>>> ssl_settings ['ssl_context'] = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, # ... other options\n... )\n```\n\nAt this point, we are ready to instantiate and use the session:\n\n```python\n>>> with iRODSSession(host='irods-provider', port=1247, user='bob', password='1234', zone='tempZone', **ssl_settings) as session:\n...\t# workload\n```\n\nNote that the `irods_` prefix is unnecessary when providing\nthe `encryption_*` and `ssl_*` options\ndirectly to the constructor as keyword arguments, even though it is\nrequired when they are placed in the environment file.\n\nPAM logins\n----------\n\nStarting with v2.0.0, the python iRODS client is able to authenticate under PAM using the same file-based client environment as the\niCommands.\n\nCaveat for iRODS 4.3+: when upgrading from 4.2, the \"irods_authentication_scheme\" setting must be changed from \"pam\" to \"pam_password\" in\n`~/.irods/irods_environment.json` for all file-based client environments.\n\nMaintaining a connection\n------------------------\n\nThe default library timeout for a connection to an iRODS Server is 120\nseconds.\n\nThis can be overridden by changing the session `connection_timeout` immediately after creation of the\nsession object:\n\n```python\n>>> session.connection_timeout = 300\n```\n\nThis will set the timeout to five minutes for any associated\nconnections.\n\nSession objects and cleanup\n---------------------------\n\nWhen iRODSSession objects are kept as state in an application, spurious\nSYS_HEADER_READ_LEN_ERR errors can sometimes be seen in the\nconnected iRODS server's log file. This is frequently seen at program\nexit because socket connections are terminated without having been\nclosed out by the session object's cleanup() method.\n\nStarting with PRC Release 0.9.0, code has been included in the session\nobject's `__del__` method to call cleanup(), properly closing out\nnetwork connections. However, `__del__` cannot be relied to run under\nall circumstances (Python2 being more problematic), so an alternative\nmay be to call session.cleanup() on any session variable which might not\nbe used again.\n\nSimple PUTs and GETs\n--------------------\n\nWe can use the just-created session object to put files to (or get them\nfrom) iRODS.\n\n```python\n>>> logical_path = \"/{0.zone}/home/{0.username}/{1}\".format(session,\"myfile.dat\")\n>>> session.data_objects.put( \"myfile.dat\", logical_path)\n>>> session.data_objects.get( logical_path, \"/tmp/myfile.dat.copy\" )\n```\n\nNote that local file paths may be relative, but iRODS data objects must\nalways be referred to by their absolute paths. This is in contrast to\nthe `iput` and `iget` icommands, which keep track of the current working\ncollection (as modified by `icd`) for the unix shell.\n\nParallel Transfer\n-----------------\n\nStarting with release 0.9.0, data object transfers using put() and get()\nwill spawn a number of threads in order to optimize performance for\niRODS server versions 4.2.9+ and file sizes larger than a default\nthreshold value of 32 Megabytes.\n\nWorking with collections\n------------------------\n\n```python\n>>> coll = session.collections.get(\"/tempZone/home/rods\")\n\n>>> coll.id\n45798\n\n>>> coll.path\n/tempZone/home/rods\n\n>>> for col in coll.subcollections:\n>>> print(col)\n<iRODSCollection /tempZone/home/rods/subcol1>\n<iRODSCollection /tempZone/home/rods/subcol2>\n\n>>> for obj in coll.data_objects:\n>>> print(obj)\n<iRODSDataObject /tempZone/home/rods/file.txt>\n<iRODSDataObject /tempZone/home/rods/file2.txt>\n```\n\nCreate a new collection:\n\n```python\n>>> coll = session.collections.create(\"/tempZone/home/rods/testdir\")\n>>> coll.id\n45799\n```\n\nWorking with data objects (files)\n---------------------------------\n\nCreate a new data object:\n\n```python\n>>> obj = session.data_objects.create(\"/tempZone/home/rods/test1\")\n<iRODSDataObject /tempZone/home/rods/test1>\n```\n\nGet an existing data object:\n\n```python\n>>> obj = session.data_objects.get(\"/tempZone/home/rods/test1\")\n>>> obj.id 12345\n\n>>> obj.name\ntest1\n>>> obj.collection\n<iRODSCollection /tempZone/home/rods>\n\n>>> for replica in obj.replicas:\n... print(replica.resource_name)\n... print(replica.number)\n... print(replica.path)\n... print(replica.status)\n...\ndemoResc\n0\n/var/lib/irods/Vault/home/rods/test1\n1\n```\n\nUsing the put() method rather than the create() method will trigger different policy enforcement points (PEPs) on the server.\n\nPut an existing file as a new data object:\n\n```python\n>>> session.data_objects.put(\"test.txt\", \"/tempZone/home/rods/test2\")\n>>> obj2 = session.data_objects.get(\"/tempZone/home/rods/test2\")\n>>> obj2.id\n56789\n```\n\nSpecifying paths\n----------------\n\nPath strings for collection and data objects are usually expected to be\nabsolute in most contexts in the PRC. They must also be normalized to a\nform including single slashes separating path elements and no slashes at\nthe string's end. If there is any doubt that a path string fulfills\nthis requirement, the wrapper class `irods.path.iRODSPath` (a subclass of `str`) may be used to normalize it:\n\n if not session.collections.exists( iRODSPath( potentially_unnormalized_path )): #....\n\nThe wrapper serves also as a path joiner; thus:\n\n iRODSPath( zone, \"home\", user )\n\nmay replace:\n\n \"/\".join([\"\", zone, \"home\", user])\n\n`iRODSPath` is available beginning with PRC release `v1.1.2`.\n\nReading and writing files\n-------------------------\n\nPRC provides [file-like\nobjects](http://docs.python.org/2/library/stdtypes.html#file-objects) for reading and writing files.\n\n```python\n>>> obj = session.data_objects.get(\"/tempZone/home/rods/test1\")\n>>> with obj.open('r+') as f:\n... f.write('foonbarn')\n... f.seek(0,0)\n... for line in f:\n... print(line)\n...\nfoo\nbar\n```\n\nAs of v1.1.9, there is also an auto-close configuration setting for data\nobjects, set to `False` by default, which may be assigned\nthe value `True` for guaranteed auto-closing of open data\nobject handles at the proper time.\n\nIn a small but illustrative example, the following Python session does\nnot require an explicit call to `f.close()`:\n\n```python\n>>> import irods.client_configuration as config, irods.test.helpers as helpers\n>>> config.data_objects.auto_close = True\n>>> session = helpers.make_session()\n>>> f = session.data_objects.open('/{0.zone}/home/{0.username}/new_object.txt'.format(session),'w')\n>>> f.write(b'new content.')\n```\n\nThis may be useful for Python programs in which frequent flushing of\nwrite updates to data objects is undesirable -- with descriptors on such\nobjects possibly being held open for indeterminately long lifetimes --\nyet the eventual application of those updates prior to the teardown of\nthe Python interpreter is required.\n\nThe current value of the setting is global in scope (i.e. applies to all\nsessions, whenever created) and is always consulted for the creation of\nany data object handle to govern that handle's cleanup behavior.\n\nPython iRODS Client Settings File\n---------------------------------\n\nAs of v1.1.9, Python iRODS client configuration can be saved in, and\nloaded from, a settings file.\n\nIf the settings file exists, each of its lines contains (a) a dotted\nname identifying a particular configuration setting to be assigned\nwithin the PRC, potentially changing its runtime behavior; and (b) the\nspecific value, in Python \"repr\"-style format, that should be assigned\ninto it.\n\nAn example follows:\n\n```\ndata_objects.auto_close True\n```\n\nNew dotted names may be created following the example of the one valid\nexample created thus far, `data_objects.auto_close]`,\ninitialized in `irods/client_configuration/__init__.py`.\nEach such name should correspond to a globally set value which the PRC\nroutinely checks when performing the affected library function.\n\nThe use of a settings file can be indicated, and the path to that file\ndetermined, by setting the environment variable:\n`PYTHON_IRODSCLIENT_CONFIGURATION_PATH`. If this variable\nis present but empty, this denotes use of a default settings file path\nof `~/.python-irodsclient`; if the variable's value is of\nnon-zero length, the value should be an absolute path to the settings\nfile whose use is desired. Also, if the variable is set, auto-load of\nsettings will be performed, meaning that the act of importing\n`irods` or any of its submodules will cause the automatic\nloading the settings from the settings file, assuming it exists.\n(Failure to find the file at the indicated path will be logged as a\nwarning.)\n\nSettings can also be saved and loaded manually using the `save()` and\n`load()` functions in the `irods.client_configuration`\nmodule. Each of these functions accepts an optional `file`\nparameter which, if set to a non-empty string, will override the\nsettings file path currently \"in force\" (i.e., the\nCONFIG_DEFAULT_PATH, as optionally overridden by the environment\nvariable PYTHON_IRODSCLIENT_CONFIGURATION_PATH).\n\nConfiguration settings may also be individually overridden by defining\ncertain environment variables. Here are relevant descriptions for each\none currently available, including the names of the environment\nvariables serving as overrides:\n\n- Setting: Auto-close option for all data objects.\n - Dotted Name: `data_objects.auto_close`\n - Type: `bool`\n - Default Value: `False`\n - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__DATA_OBJECTS__AUTO_CLOSE`\n\n- Setting: Number of hours to request for the new password entry's TTL (Time To Live) when auto-renewing PAM-authenticated sessions.\n - Dotted Name: `legacy_auth.pam.time_to_live_in_hours`\n - Type: `int`\n - Default Value: `0` (Meaning: conform to server's default TTL value.)\n - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__TIME_TO_LIVE_IN_HOURS`\n\n- Setting: Plaintext PAM password value, to be used when auto-renewing PAM-authenticated sessions because TTL has expired.\n - Dotted Name: `legacy_auth.pam.password_for_auto_renew`\n - Type: `str`\n - Default Value: `\"\"` (Meaning: no password is set, and thus no automatic attempts will be made at auto-renewing PAM authentication.)\n - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__PASSWORD_FOR_AUTO_RENEW`. (But note that use of the environment variable could pose a threat to password security.)\n\n- Setting: Whether to write the (native encoded) new hashed password to the iRODS password file. This step is only performed while auto-renewing PAM authenticated sessions.\n - Dotted Name: `legacy_auth.pam.store_password_to_environment`\n - Type: `bool`\n - Default Value: `False`\n - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__STORE_PASSWORD_TO_ENVIRONMENT`\n\n- Setting: Default choice of XML parser for all new threads.\n - Dotted Name: `connections.xml_parser_default`\n - Type: `str`\n - Default Value: `\"STANDARD_XML\"`\n - Possible Values: Any of `[\"STANDARD_XML\", \"QUASI_XML\", \"SECURE_XML\"]`\n - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__CONNECTIONS__XML_PARSER_DEFAULT`\n\nFor example, if `~/python_irodsclient` contains the line :\n\n```\nconnections.xml_parser_default \"QUASI_XML\"\n```\n\nthen the session below illustrates the effect of defining the\nappropriate environment variable. Note the value stored in the variable\nmust be a valid input for `ast.literal_eval()`; that is, a\nprimitive Pythonic value - and quoted, for instance, if a string.\n\n```bash\n$ PYTHON_IRODSCLIENT_CONFIGURATION_PATH=\"\" \\\n PYTHON_IRODSCLIENT_CONFIG__CONNECTIONS__XML_PARSER_DEFAULT=\"'SECURE_XML'\" \\\n python -c \"import irods.message, irods.client_configuration as c; print (irods.message.default_XML_parser())\"\nXML_Parser_Type.SECURE_XML\n$ PYTHON_IRODSCLIENT_CONFIGURATION_PATH=\"\" \\\n python -c \"import irods.message, irods.client_configuration as c; print (irods.message.default_XML_parser())\"\nXML_Parser_Type.QUASI_XML\n```\n\nComputing and Retrieving Checksums\n----------------------------------\n\nEach data object may be associated with a checksum by calling chksum()\non the object in question. Various behaviors can be elicited by passing\nin combinations of keywords (for a description of which, please consult\nthe [header documentation](https://github.com/irods/irods/blob/4-2-stable/lib/api/include/dataObjChksum.h).)\n\nAs with most other iRODS APIs, it is straightforward to specify keywords\nby adding them to an option dictionary:\n\n```python\n>>> data_object_1.chksum() # - computes the checksum if already in the catalog, otherwise computes and stores it\n... # (i.e. default behavior with no keywords passed in.)\n>>> from irods.manager.data_object_manager import Server_Checksum_Warning\n>>> import irods.keywords as kw\n>>> opts = { kw.VERIFY_CHKSUM_KW:'' }\n>>> try:\n... data_object_2.chksum( **opts ) # - Uses verification option. (Does not auto-vivify a checksum field).\n... # or:\n... opts[ kw.NO_COMPUTE_KW ] = ''\n... data_object_2.chksum( **opts ) # - Uses both verification and no-compute options. (Like `ichksum -K --no-compute`)\n... except Server_Checksum_Warning:\n... print('some checksums are missing or wrong')\n```\n\nAdditionally, if a freshly created `irods.message.RErrorStack` instance is\ngiven, information can be returned and read by the client:\n\n```python\n>>> from irods.message import RErrorStack\n>>> r_err_stk = RErrorStack()\n>>> warn = None\n>>> try: # Here, data_obj has one replica, not yet checksummed.\n... data_obj.chksum( r_error = r_err_stk , **{kw.VERIFY_CHKSUM_KW:''} )\n... except Server_Checksum_Warning as exc:\n... warn = exc\n>>> print(r_err_stk)\n[RError<message = u'WARNING: No checksum available for replica [0].', status = -862000 CAT_NO_CHECKSUM_FOR_REPLICA>]\n```\n\nWorking with metadata\n---------------------\n\nTo enumerate AVUs on an object. With no metadata attached, the result\nis an empty list:\n\n```python\n>>> from irods.meta import iRODSMeta\n>>> obj = session.data_objects.get(\"/tempZone/home/rods/test1\")\n>>> print(obj.metadata.items())\n[]\n```\n\nWe then add some metadata. Just as with the icommand equivalent \"imeta\nadd ...\", we can add multiple AVUs with the same name field:\n\n```python\n>>> obj.metadata.add('key1', 'value1', 'units1')\n>>> obj.metadata.add('key1', 'value2')\n>>> obj.metadata.add('key2', 'value3')\n>>> obj.metadata.add('key2', 'value4')\n>>> print(obj.metadata.items())\n[<iRODSMeta 13182 key1 value1 units1>, <iRODSMeta 13185 key2 value4 None>,\n<iRODSMeta 13183 key1 value2 None>, <iRODSMeta 13184 key2 value3 None>]\n```\n\nWe can also use Python's item indexing syntax to perform the equivalent\nof an \"imeta set \\...\", e.g. overwriting all AVUs with a name field\nof \"key2\" in a single update:\n\n```python\n>>> new_meta = iRODSMeta('key2','value5','units2')\n>>> obj.metadata\\[new_meta.name\\] = new_meta\n>>> print(obj.metadata.items())\n[<iRODSMeta 13182 key1 value1 units1>, <iRODSMeta 13183 key1 value2 None>,\n<iRODSMeta 13186 key2 value5 units2>]\n```\n\nNow, with only one AVU on the object with a name of \"key2\", *get_one*\nis assured of not throwing an exception:\n\n```python\n>>> print(obj.metadata.get_one('key2'))\n<iRODSMeta 13186 key2 value5 units2>\n```\n\nHowever, the same is not true of \"key1\":\n\n```python\n>>> print(obj.metadata.get_one('key1'))\nTraceback (most recent call last):\n File \"<stdin>\", line 1, in <module>\n File \"/[...]/python-irodsclient/irods/meta.py\", line 41, in get_one\n raise KeyError\nKeyError\n```\n\nFinally, to remove a specific AVU from an object:\n\n```python\n>>> obj.metadata.remove('key1', 'value1', 'units1')\n>>> print(obj.metadata.items())\n[<iRODSMeta 13186 key2 value5 units2>, <iRODSMeta 13183 key1 value2 None>]\n\nAlternately, this form of the `remove()` method can also be useful:\n\n```python\n>>> for avu in obj.metadata.items():\n... obj.metadata.remove(avu)\n>>> print(obj.metadata.items())\n[]\n```\n\nIf we intended on deleting the data object anyway, we could have just\ndone this instead:\n\n```\n>>> obj.unlink(force=True)\n```\n\nBut notice that the force option is important, since a data object in\nthe trash may still have AVUs attached.\n\nAt the end of a long session of AVU add/manipulate/delete operations,\none should make sure to delete all unused AVUs. We can in fact use any\n`*Meta` data model in the queries below, since unattached AVUs are\nnot aware of the (type of) catalog object they once annotated:\n\n```python\n>>> from irods.models import (DataObjectMeta, ResourceMeta)\n>>> len(list( session.query(ResourceMeta) ))\n4\n>>> from irods.test.helpers import remove_unused_metadata\n>>> remove_unused_metadata(session)\n>>> len(list( session.query(ResourceMeta) ))\n0\n```\n\nWhen altering a fetched iRODSMeta, we must copy it first to avoid\nerrors, due to the fact the reference is cached by the iRODS object\nreference. A shallow copy is sufficient:\n\n```python\n>>> meta = album.metadata.items()[0]\n>>> meta.units\n'quid'\n>>> import copy; meta = copy.copy(meta); meta.units = 'pounds sterling'\n>>> album.metadata[ meta.name ] = meta\n```\n\nFortunately, as of PRC >= 1.1.4, we can simply do this instead:\n\n```python\n>>> album.metadata.set( meta )\n```\n\nIn versions of iRODS 4.2.12 and later, we can also do:\n\n```python\n>>> album.metadata.set( meta, \\*\\*{kw.ADMIN_KW: ''} )\n```\n\nor even:\n\n```python\n>>> album.metadata(admin = True)\\[meta.name\\] = meta\n```\n\nIn v1.1.5, the \"timestamps\" keyword is provided to enable the loading\nof create and modify timestamps for every AVU returned from the server:\n\n```python\n>>> avus = album.metadata(timestamps = True).items()\n>>> avus[0].create_time\ndatetime.datetime(2022, 9, 19, 15, 26, 7)\n```\n\nAtomic operations on metadata\n-----------------------------\n\nWith release 4.2.8 of iRODS, the atomic metadata API was introduced to\nallow a group of metadata add and remove operations to be performed\ntransactionally, within a single call to the server. This capability can\nbe leveraged in version 0.8.6 of the PRC.\n\nSo, for example, if 'obj' is a handle to an object in the iRODS\ncatalog (whether a data object, collection, user or storage resource),\nwe can send an arbitrary number of AVUOperation instances to be executed\ntogether as one indivisible operation on that object:\n\n```python\n>>> from irods.meta import iRODSMeta, AVUOperation\n>>> obj.metadata.apply_atomic_operations( AVUOperation(operation='remove', avu=iRODSMeta('a1','v1','these_units')),\n... AVUOperation(operation='add', avu=iRODSMeta('a2','v2','those_units')),\n... AVUOperation(operation='remove', avu=iRODSMeta('a3','v3')) \\# , ...\n... )\n```\n\nThe list of operations will applied in the order given, so that a\n\"remove\" followed by an \"add\" of the same AVU is, in effect, a\nmetadata \"set\" operation. Also note that a \"remove\" operation will\nbe ignored if the AVU value given does not exist on the target object at\nthat point in the sequence of operations.\n\nWe can also source from a pre-built list of AVUOperations using\nPython's `f(*args_list)` syntax. For example, this\nfunction uses the atomic metadata API to very quickly remove all AVUs\nfrom an object:\n\n```python\n>>> def remove_all_avus( Object ):\n... avus_on_Object = Object.metadata.items()\n... Object.metadata.apply_atomic_operations( *[AVUOperation(operation='remove', avu=i) for i in avus_on_Object] )\n```\n\nSpecial Characters\n------------------\n\nOf course, it is fine to put Unicode characters into your collection and\ndata object names. However, certain non-printable ASCII characters, and\nthe backquote character as well, have historically presented problems\n- especially for clients using iRODS's human readable XML protocol.\nConsider this small, only slighly contrived, application:\n\n```python\n from irods.test.helpers import make_session\n\n def create_notes( session, obj_name, content = u'' ):\n get_home_coll = lambda ses: \"/{0.zone}/home/{0.username}\".format(ses)\n path = get_home_coll(session) + \"/\" + obj_name\n with session.data_objects.open(path,\"a\") as f:\n f.seek(0, 2) # SEEK_END\n f.write(content.encode('utf8'))\n return session.data_objects.get(path)\n\n with make_session() as session:\n\n # Example 1 : exception thrown when name has non-printable character\n try:\n create_notes( session, \"lucky\\033.dat\", content = u'test' )\n except:\n pass\n\n # Example 2 (Ref. issue: irods/irods #4132, fixed for 4.2.9 release of iRODS)\n print(\n create_notes( session, \"Alice's diary\").name # note diff (' != ') in printed name\n )\n```\n\nThis creates two data objects, but with less than optimal success. The\nfirst example object is created but receives no content because an\nexception is thrown trying to query its name after creation. In the\nsecond example, for iRODS 4.2.8 and before, a deficiency in packStruct\nXML protocol causes the backtick to be read back as an apostrophe, which\ncould create problems manipulating or deleting the object later.\n\nAs of PRC v1.1.0, we can mitigate both problems by switching in the\nQUASI_XML parser for the default one:\n\n```\n from irods.message import (XML_Parser_Type, ET)\n ET( XML_Parser_Type.QUASI_XML, session.server_version )\n```\n\nTwo dedicated environment variables may also be used to customize the\nPython client's XML parsing behavior via the setting of global defaults\nduring start-up.\n\nFor example, we can set the default parser to QUASI_XML, optimized for\nuse with version 4.2.8 of the iRODS server, in the following manner:\n\n```\n Bash-Shell> export PYTHON_IRODSCLIENT_DEFAULT_XML=QUASI_XML PYTHON_IRODSCLIENT_QUASI_XML_SERVER_VERSION=4,2,8\n```\n\nOther alternatives for PYTHON_IRODSCLIENT_DEFAULT_XML are\n\"STANDARD_XML\" and \"SECURE_XML\". These two latter options denote\nuse of the xml.etree and defusedxml modules, respectively.\n\nOnly the choice of \"QUASI_XML\" is affected by the specification of a\nparticular server version.\n\nFinally, note that these global defaults, once set, may be overridden on\na per-thread basis using `ET(parser_type, server_version)`.\nWe can also revert the current thread's XML parser back to the global\ndefault by calling `ET(None)`.\n\nRule Execution\n--------------\n\nA simple example of how to execute an iRODS rule from the Python client\nis as follows. Suppose we have a rule file `native1.r`\nwhich contains a rule in native iRODS Rule Language:\n\n```\n main() {\n writeLine(\"*stream\",\n *X ++ \" squared is \" ++ str(double(*X)^2) )\n }\n\n INPUT *X=\"3\", *stream=\"serverLog\"\n OUTPUT null\n```\n\nThe following Python client code will run the rule and produce the\nappropriate output in the irods server log:\n\n```\n r = irods.rule.Rule( session, rule_file = 'native1.r')\n r.execute()\n```\n\nWith release v1.1.1, not only can we target a specific rule engine\ninstance by name (which is useful when more than one is present), but we\ncan also use a file-like object for the `rule_file`\nparameter:\n\n```\n Rule( session, rule_file = io.StringIO(u'''mainRule() { anotherRule(*x); writeLine('stdout',*x) }\\n'''\n u'''anotherRule(*OUT) {*OUT='hello world!'}\\n\\n'''\n u'''OUTPUT ruleExecOut\\n'''),\n instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance' )\n```\n\nIncidentally, if we wanted to change the `native1.r` rule\ncode print to stdout also, we could set the `INPUT`\nparameter, `*stream`, using the Rule constructor's\n`params` keyword argument. Similarly, we can change the\n`OUTPUT` parameter from `null` to\n`ruleExecOut`, to accommodate the output stream, via the\n`output` argument:\n\n```\n r = irods.rule.Rule( session, rule_file = 'native1.r',\n instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance',\n params={'*stream':'\"stdout\"'} , output = 'ruleExecOut' )\n output = r.execute( )\n if output and len(output.MsParam_PI):\n buf = output.MsParam_PI[0].inOutStruct.stdoutBuf.buf\n if buf: print(buf.rstrip(b'\\0').decode('utf8'))\n```\n\n(Changing the input value to be squared in this example is left as an\nexercise for the reader!)\n\nTo deal with errors resulting from rule execution failure, two\napproaches can be taken. Suppose we have defined this in the\n`/etc/irods/core.re` rule-base:\n\n```\n rule_that_fails_with_error_code(*x) {\n *y = (if (*x!=\"\") then int(*x) else 0)\n # if (SOME_PROCEDURE_GOES_WRONG) {\n if (*y < 0) { failmsg(*y,\"-- my error message --\"); } #-> throws an error code of int(*x) in REPF\n else { fail(); } #-> throws FAIL_ACTION_ENCOUNTERED_ERR in REPF\n # }\n }\n```\n\nWe can run the rule thus:\n\n```python\n>>> Rule( session, body='rule_that_fails_with_error_code(\"\"), instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance',\n... ).execute( r_error = (r_errs:= irods.message.RErrorStack()) )\n```\n\nWhere we've used the Python 3.8 \"walrus operator\" for brevity. The\nerror will automatically be caught and translated to a returned-error\nstack:\n\n```python\n>>> pprint.pprint([vars(r) for r in r_errs])\n[{'raw_msg_': 'DEBUG: fail action encountered\\n'\n 'line 14, col 15, rule base core\\n'\n ' else { fail(); }\\n'\n ' ^\\n'\n '\\n',\n 'status_': -1220000}]\n```\n\nNote, if a stringized negative integer is given , i.e. as a special fail\ncode to be thrown within the rule, we must add this code into a special\nparameter to have this automatically caught as well:\n\n```python\n>>> Rule( session, body='rule_that_fails_with_error_code(\"-2\")',instance_name = 'irods_rule_engine_plugin-irods_rule_language-instance'\n... ).execute( acceptable_errors = ( FAIL_ACTION_ENCOUNTERED_ERR, -2),\n... r_error = (r_errs := irods.message.RErrorStack()) )\n```\n\nBecause the rule is written to emit a custom error message via `failmsg()`\nin this case, the resulting r_error stack will now include that custom\nerror message as a substring:\n\n```python\n>>> pprint.pprint([vars(r) for r in r_errs])\n[{'raw_msg_': 'DEBUG: -- my error message --\\n'\n 'line 21, col 20, rule base core\\n'\n ' if (*y < 0) { failmsg(*y,\"-- my error message --\"); } '\n '#-> throws an error code of int(*x) in REPF\\n'\n ' ^\\n'\n '\\n',\n 'status_': -1220000}]\n```\n\nAlternatively, or in combination with the automatic catching of errors,\nwe may also catch errors as exceptions on the client side. For example,\nif the Python rule engine is configured, and the following rule is\nplaced in `/etc/irods/core.py`:\n\n```python\ndef python_rule(rule_args, callback, rei):\n# if some operation fails():\n raise RuntimeError\n```\n\nwe can trap the error thus:\n\n```python\ntry:\n Rule( session, body = 'python_rule', instance_name = 'irods_rule_engine_plugin-python-instance' ).execute()\nexcept irods.exception.RULE_ENGINE_ERROR:\n print('Rule execution failed!')\n exit(1)\nprint('Rule execution succeeded!')\n```\n\nAs fail actions from native rules are not thrown by default (refer to\nthe help text for `Rule.execute`), if we anticipate these\nand prefer to catch them as exceptions, we can do it this way:\n\n```python\ntry:\n Rule( session, body = 'python_rule', instance_name = 'irods_rule_engine_plugin-python-instance'\n ).execute( acceptable_errors = () )\nexcept (irods.exception.RULE_ENGINE_ERROR,\n irods.exception.FAIL_ACTION_ENCOUNTERED_ERR) as e:\n print('Rule execution failed!')\n exit(1)\nprint('Rule execution succeeded!')\n```\n\nFinally, keep in mind that rule code submitted through an\n`irods.rule.Rule` object is processed by the\nexec_rule_text function in the targeted plugin instance. This may be a\nlimitation for plugins not equipped to handle rule code in this way. In\na sort of middle-ground case, the iRODS Python Rule Engine Plugin is not\ncurrently able to handle simple rule calls and the manipulation of iRODS\ncore primitives (like simple parameter passing and variable expansion')\nas flexibly as the iRODS Rule Language.\n\nAlso, core.py rules may not be run directly (as is also true with\n`irule`) by other than a rodsadmin user pending the\nresolution of [this\nissue](https://github.com/irods/irods_rule_engine_plugin_python/issues/105).\n\n\nGeneral Queries\n---------------\n\n```python\n>>> import os\n>>> from irods.session import iRODSSession\n>>> from irods.models import Collection, DataObject\n>>>\n>>> env_file = os.path.expanduser('~/.irods/irods_environment.json')\n>>> with iRODSSession(irods_env_file=env_file) as session:\n... query = session.query(Collection.name, DataObject.id, DataObject.name, DataObject.size)\n...\n... for result in query:\n... print('{}/{} id={} size={}'.format(result[Collection.name], result[DataObject.name], result[DataObject.id], result[DataObject.size]))\n...\n/tempZone/home/rods/manager/access_manager.py id=212665 size=2164\n/tempZone/home/rods/manager/access_manager.pyc id=212668 size=2554\n/tempZone/home/rods/manager/collection_manager.py id=212663 size=4472\n/tempZone/home/rods/manager/collection_manager.pyc id=212664 size=4464\n/tempZone/home/rods/manager/data_object_manager.py id=212662 size=10291\n/tempZone/home/rods/manager/data_object_manager.pyc id=212667 size=8772\n/tempZone/home/rods/manager/__init__.py id=212670 size=79\n/tempZone/home/rods/manager/__init__.pyc id=212671 size=443\n/tempZone/home/rods/manager/metadata_manager.py id=212660 size=4263\n/tempZone/home/rods/manager/metadata_manager.pyc id=212659 size=4119\n/tempZone/home/rods/manager/resource_manager.py id=212666 size=5329\n/tempZone/home/rods/manager/resource_manager.pyc id=212661 size=4570\n/tempZone/home/rods/manager/user_manager.py id=212669 size=5509\n/tempZone/home/rods/manager/user_manager.pyc id=212658 size=5233\n```\n\nQuery using other models:\n\n```python\n>>> from irods.column import Criterion\n>>> from irods.models import DataObject, DataObjectMeta, Collection, CollectionMeta\n>>> from irods.session import iRODSSession\n>>> import os\n>>> env_file = os.path.expanduser('~/.irods/irods_environment.json')\n>>> with iRODSSession(irods_env_file=env_file) as session:\n... # by metadata\n... # equivalent to 'imeta qu -C type like Project'\n... results = session.query(Collection, CollectionMeta).filter( \\\n... Criterion('=', CollectionMeta.name, 'type')).filter( \\\n... Criterion('like', CollectionMeta.value, '%Project%'))\n... for r in results:\n... print(r[Collection.name], r[CollectionMeta.name], r[CollectionMeta.value], r[CollectionMeta.units])\n...\n('/tempZone/home/rods', 'type', 'Project', None)\n```\n\nBeginning with version 0.8.3 of PRC, the 'in' genquery operator is\nalso available:\n\n```python\n>>> from irods.models import Resource\n>>> from irods.column import In\n>>> [ resc[Resource.id]for resc in session.query(Resource).filter(In(Resource.name, ['thisResc','thatResc'])) ]\n[10037,10038]\n```\n\nQuery with aggregation(min, max, sum, avg, count):\n\n```python\n>>> with iRODSSession(irods_env_file=env_file) as session:\n... query = session.query(DataObject.owner_name).count(DataObject.id).sum(DataObject.size)\n... print(next(query.get_results()))\n{<irods.column.Column 411 D_OWNER_NAME>: 'rods', <irods.column.Column 407 DATA_SIZE>: 62262, <irods.column.Column 401 D_DATA_ID>: 14}\n```\n\nIn this case since we are expecting only one row we can directly call\n`query.execute()`:\n\n```python\n>>> with iRODSSession(irods_env_file=env_file) as session:\n... query = session.query(DataObject.owner_name).count(DataObject.id).sum(DataObject.size)\n... print(query.execute())\n+--------------+-----------+-----------+\n| D_OWNER_NAME | D_DATA_ID | DATA_SIZE |\n+--------------+-----------+-----------+\n| rods | 14 | 62262 |\n+--------------+-----------+-----------+\n```\n\nFor a case-insensitive query, add a `case_sensitive=False`\nparameter to the query:\n\n```python\n>>> with iRODSSession(irods_env_file=env_file) as session:\n... query = session.query(DataObject.name, case_sensitive=False).filter(Like(DataObject.name, \"%oBjEcT\"))\n... print(query.all())\n+---------------------+\n| DATA_NAME |\n+---------------------+\n| caseSENSITIVEobject |\n+---------------------+\n```\n\nSpecific Queries\n----------------\n\n```python\n>>> import os\n>>> from irods.session import iRODSSession\n>>> from irods.models import Collection, DataObject\n>>> from irods.query import SpecificQuery\n>>>\n>>> env_file = os.path.expanduser('~/.irods/irods_environment.json')\n>>> with iRODSSession(irods_env_file=env_file) as session:\n... # define our query\n... sql = \"select data_name, data_id from r_data_main join r_coll_main using (coll_id) where coll_name = '/tempZone/home/rods/manager'\"\n... alias = 'list_data_name_id'\n... columns = [DataObject.name, DataObject.id] # optional, if we want to get results by key\n... query = SpecificQuery(session, sql, alias, columns)\n...\n... # register specific query in iCAT\n... _ = query.register()\n...\n... for result in query:\n... print('{} {}'.format(result[DataObject.name], result[DataObject.id]))\n...\n... # delete specific query\n... _ = query.remove()\n...\nuser_manager.pyc 212658\nmetadata_manager.pyc 212659\nmetadata_manager.py 212660\nresource_manager.pyc 212661\ndata_object_manager.py 212662\ncollection_manager.py 212663\ncollection_manager.pyc 212664\naccess_manager.py 212665\nresource_manager.py 212666\ndata_object_manager.pyc 212667\naccess_manager.pyc 212668\nuser_manager.py 212669\n__init__.py 212670\n__init__.pyc 212671\n```\n\nRecherch\u00e9 Queries\n-----------------\n\nIn some cases you might like to use a GenQuery operator not directly\noffered by this Python library, or even combine query filters in ways\nGenQuery may not directly support.\n\nAs an example, the code below finds metadata value fields\nlexicographically outside the range of decimal integers, while also\nrequiring that the data objects to which they are attached do not reside\nin the trash.\n\n```python\n>>> search_tuple = (DataObject.name , Collection.name ,\n... DataObjectMeta.name , DataObjectMeta.value)\n\n>>> # \"not like\" : direct instantiation of Criterion (operator in literal string)\n>>> not_in_trash = Criterion ('not like', Collection.name , '%/trash/%')\n\n>>> # \"not between\"( column, X, Y) := column < X OR column > Y (\"OR\" done via chained iterators)\n>>> res1 = session.query (* search_tuple).filter(not_in_trash).filter(DataObjectMeta.value < '0')\n>>> res2 = session.query (* search_tuple).filter(not_in_trash).filter(DataObjectMeta.value > '9' * 9999 )\n\n>>> chained_results = itertools.chain ( res1.get_results(), res2.get_results() )\n>>> pprint( list( chained_results ) )\n```\n\nInstantiating iRODS objects from query results\n----------------------------------------------\n\nThe General query works well for getting information out of the ICAT if\nall we're interested in is information representable with primitive\ntypes (i.e. object names, paths, and ID's, as strings or integers). But\nPython's object orientation also allows us to create object references\nto mirror the persistent entities (instances of *Collection*,\n*DataObject*, *User*, or *Resource*, etc.) inhabiting the ICAT.\n\n**Background:**\n\nCertain iRODS object types can be instantiated easily\nusing the session object's custom type managers, particularly if some\nparameter (often just the name or path) of the object is already known:\n\n```python\n>>> type(session.users)\n<class 'irods.manager.user_manager.UserManager'>\n>>> u = session.users.get('rods')\n>>> u.id\n10003\n```\n\nType managers are good for specific operations, including object\ncreation and removal:\n\n```python\n>>> session.collections.create('/tempZone/home/rods/subColln')\n>>> session.collections.remove('/tempZone/home/rods/subColln')\n>>> session.data_objects.create('/tempZone/home/rods/dataObj')\n>>> session.data_objects.unlink('/tempZone/home/rods/dataObj')\n```\n\nWhen we retrieve a reference to an existing collection using *get* :\n\n```python\n>>> c = session.collections.get('/tempZone/home/rods')\n>>> c\n<iRODSCollection 10011 rods>\n```\n\nwe have, in that variable *c*, a reference to an iRODS *Collection*\nobject whose properties provide useful information:\n\n```python\n>>> [ x for x in dir(c) if not x.startswith('__') ]\n['_meta', 'data_objects', 'id', 'manager', 'metadata', 'move', 'name', 'path', 'remove', 'subcollections', 'unregister', 'walk']\n>>> c.name\n'rods'\n>>> c.path\n'/tempZone/home/rods'\n>>> c.data_objects\n[<iRODSDataObject 10019 test1>]\n>>> c.metadata.items()\n[ <... list of AVUs attached to Collection c ... > ]\n```\n\nor whose methods can do useful things:\n\n```python\n>>> for sub_coll in c.walk(): print('---'); pprint( sub_coll )\n[ ...< series of Python data structures giving the complete tree structure below collection 'c'> ...]\n```\n\nThis approach of finding objects by name, or via their relations with\nother objects (ie \"contained by\", or in the case of metadata,\n\"attached to\"), is helpful if we know something about the location or\nidentity of what we're searching for, but we don't always have that\nkind of a-priori knowledge.\n\nSo, although we can (as seen in the last example) walk an\n*iRODSCollection* recursively to discover all subordinate collections\nand their data objects, this approach will not always be best for a\ngiven type of application or data discovery, especially in more advanced\nuse cases.\n\n**A Different Approach:**\n\nFor the PRC to be sufficiently powerful for general use, we'll often need at least:\n\n- general queries, and\n- the capabilities afforded by the PRC's object-relational mapping.\n\nSuppose, for example, we wish to enumerate all collections in the iRODS\ncatalog.\n\nAgain, the object managers are the answer, but they are now invoked\nusing a different scheme:\n\n```python\n>>> from irods.collection import iRODSCollection; from irods.models import Collection\n>>> all_collns = [ iRODSCollection(session.collections,result) for result in session.query(Collection) ]\n```\n\nFrom there, we have the ability to do useful work, or filtering based on\nthe results of the enumeration. And, because *all_collns* is an\niterable of true objects, we can either use Python's list\ncomprehensions or execute more catalog queries to achieve further aims.\n\nNote that, for similar system-wide queries of Data Objects (which, as it\nhappens, are inextricably joined to their parent Collection objects), a\nbit more finesse is required. Let us query, for example, to find all\ndata objects in a particular zone with an AVU that matches the following\ncondition:\n\n```\n META_DATA_ATTR_NAME = \"irods::alert_time\" and META_DATA_ATTR_VALUE like '+0%'\n```\n\n```python\n>>> import irods.keywords\n>>> from irods.data_object import iRODSDataObject\n>>> from irods.models import DataObjectMeta, DataObject\n>>> from irods.column import Like\n>>> q = session.query(DataObject).filter( DataObjectMeta.name == 'irods::alert_time',\n Like(DataObjectMeta.value, '+0%') )\n>>> zone_hint = \"\" # --> add a zone name in quotes to search another zone\n>>> if zone_hint: q = q.add_keyword( irods.keywords.ZONE_KW, zone_hint )\n>>> for res in q:\n... colln_id = res [DataObject.collection_id]\n... collObject = get_collection( colln_id, session, zone = zone_hint)\n... dataObject = iRODSDataObject( session.data_objects, parent = collObject, results=[res])\n... print( '{coll}/{data}'.format (coll = collObject.path, data = dataObject.name))\n```\n\nIn the above loop we have used a helper function, *get_collection*, to\nminimize the number of hits to the object catalog. Otherwise, me might\nfind within a typical application that some Collection objects are being\nqueried at a high rate of redundancy. *get_collection* can be\nimplemented thusly:\n\n```python\nimport collections # of the Pythonic, not iRODS, kind\ndef makehash():\n # see https://stackoverflow.com/questions/651794/whats-the-best-way-to-initialize-a-dict-of-dicts-in-python\n return collections.defaultdict(makehash)\nfrom irods.collection import iRODSCollection\nfrom irods.models import Collection\ndef get_collection (Id, session, zone=None, memo = makehash()):\n if not zone: zone = \"\"\n c_obj = memo[session][zone].get(Id)\n if c_obj is None:\n q = session.query(Collection).filter(Collection.id==Id)\n if zone != '': q = q.add_keyword( irods.keywords.ZONE_KW, zone )\n c_id = q.one()\n c_obj = iRODSCollection(session, result = c_id)\n memo[session][zone][Id] = c_obj\n return c_obj\n```\n\nOnce instantiated, of course, any *iRODSDataObject*'s data to which we\nhave access permissions is available via its open() method.\n\nAs stated, this type of object discovery requires some extra study and\neffort, but the ability to search arbitrary iRODS zones (to which we are\nfederated and have the user permissions) is powerful indeed.\n\nTickets\n-------\n\nThe `irods.ticket.Ticket` class lets us issue \"tickets\"\nwhich grant limited permissions for other users to access our own data\nobjects (or collections of data objects). As with the iticket client,\nthe access may be either \"read\" or \"write\". The recipient of the\nticket could be a rodsuser, or even an anonymous user.\n\nBelow is a demonstration of how to generate a new ticket for access to a\nlogical path - in this case, say a collection containing 1 or more data\nobjects. (We assume the creation of the granting_session and\nreceiving_session for the users respectively for the users providing\nand consuming the ticket access.)\n\nThe user who wishes to provide an access may execute the following:\n\n```python\n>>> from irods.ticket import Ticket\n>>> new_ticket = Ticket (granting_session)\n>>> The_Ticket_String = new_ticket.issue('read', \n... '/zone/home/my/collection_with_data_objects_for/somebody').string\n```\n\nat which point that ticket's unique string may be given to other users,\nwho can then apply the ticket to any existing session object in order to\ngain access to the intended object(s):\n\n```python\n>>> from irods.models import Collection, DataObject\n>>> ses = receiving_session\n>>> Ticket(ses, The_Ticket_String).supply()\n>>> c_result = ses.query(Collection).one()\n>>> c = iRODSCollection( ses.collections, c_result)\n>>> for dobj in (c.data_objects):\n... ses.data_objects.get( dobj.path, '/tmp/' + dobj.name ) # download objects\n```\n\nIn this case, however, modification will not be allowed because the\nticket is for read only:\n\n```python\n>>> c.data_objects[0].open('w').write( # raises\n... b'new content') # CAT_NO_ACCESS_PERMISSION\n```\n\nIn another example, we could generate a ticket that explicitly allows\n'write' access on a specific data object, thus granting other users\nthe permissions to modify as well as read it:\n\n```python\n>>> ses = iRODSSession( user = 'anonymous', password = '', host = 'localhost',\n port = 1247, zone = 'tempZone')\n>>> Ticket(ses, write_data_ticket_string ).supply()\n>>> d_result = ses.query(DataObject.name,Collection.name).one()\n>>> d_path = ( d_result[Collection.name] + '/' +\n... d_result[DataObject.name] )\n>>> old_content = ses.data_objects.open(d_path,'r').read()\n>>> with tempfile.NamedTemporaryFile() as f:\n... f.write(b'blah'); f.flush()\n... ses.data_objects.put(f.name,d_path)\n```\n\nAs with iticket, we may set a time limit on the availability of a\nticket, either as a timestamp or in seconds since the epoch:\n\n```python\n>>> t=Ticket(ses); s = t.string\nvIOQ6qzrWWPO9X7\n>>> t.issue('read','/some/path')\n>>> t.modify('expiry','2021-04-01.12:34:56') # timestamp assumed as UTC\n```\n\nTo check the results of the above, we could invoke this icommand\nelsewhere in a shell prompt:\n\n```\niticket ls vIOQ6qzrWWPO9X7\n```\n\nand the server should report back the same expiration timestamp.\n\nAnd, if we are the issuer of a ticket, we may also query, filter on, and\nextract information based on a ticket's attributes and catalog\nrelations:\n\n```python\n>>> from irods.models import TicketQuery\n>>> delay = lambda secs: int( time.time() + secs + 1)\n>>> Ticket(ses).issue('read','/path/to/data_object').modify(\n 'expiry',delay(7*24*3600)) # lasts 1 week\n>>> Q = ses.query (TicketQuery.Ticket, TicketQuery.DataObject).filter(\n... TicketQuery.DataObject.name == 'data_object')\n>>> print ([ _[TicketQuery.Ticket.expiry_ts] for _ in Q ])\n['1636757427']\n```\n\nTracking and manipulating replicas of Data Objects\n--------------------------------------------------\n\nPutting together the techniques we've seen so far, it's not hard to write client code to accomplish\nuseful, common tasks. Suppose, for instance, that a data object contains replicas on a given resource\nor resource hierarchy (the \"source\"), and we want those replicas \"moved\" to a second resource\n(the \"destination\"). This can be done by combining the replicate and trim operations, as in the following\ncode excerpt.\n\nWe'll assume, for our current purposes, that all pre-existing replicas are good (ie. they have a\n`status` attribute of `'1'`); and that the nodes in question are named `src` and `dest`,\nwith `src` being the root node of a resource hierarchy and `dest` just a simple storage node.\n\nThen we can accomplish the replica \"move\" thus:\n\n```python\n path = '/path/to/data/object'\n data = session.data_objects.get('/path/to/data/object')\n\n # Replicate the data object to the destination.\n\n data.replicate(**{kw.DEST_RESC_NAME_KW: 'dest'})\n\n # Find and trim replicas on the source resource hierarchy.\n\n replica_numbers = [r.number for r in d.replicas if r.resc_hier.startswith('src;')]\n for number in replica_numbers:\n session.data_objects.trim(path, **{kw.DATA_REPL_NUM:number, kw.COPIES_KW:1})\n```\n\nListing Users and Groups ; calculating Group Membership\n-------------------------------------------------------\n\niRODS tracks groups and users using two tables, R_USER_MAIN and\nR_USER_GROUP. Under this database schema, all \"user groups\" are also\nusers:\n\n```python\n>>> from irods.models import User, Group\n>>> from pprint import pprint\n>>> pprint(list((x[User.id], x[User.name]) for x in session.query(User)))\n[(10048, 'alice'),\n (10001, 'rodsadmin'),\n (13187, 'bobby'),\n (10045, 'collab'),\n (10003, 'rods'),\n (13193, 'empty'),\n (10002, 'public')]\n```\n\nBut it's also worth noting that the User.type field will be\n'rodsgroup' for any user ID that iRODS internally recognizes as a\n\"Group\":\n\n```python\n>>> groups = session.query(User).filter( User.type == 'rodsgroup' )\n\n>>> [x[User.name] for x in groups]\n['collab', 'public', 'rodsadmin', 'empty']\n```\n\nSince we can instantiate iRODSGroup and iRODSUser objects directly from\nthe rows of a general query on the corresponding tables, it is also\nstraightforward to trace out the groups' memberships:\n\n```python\n>>> from irods.user import iRODSUser, iRODSGroup\n>>> grp_usr_mapping = [ (iRODSGroup(session.groups, result), iRODSUser(session.users, result)) \\\n... for result in session.query(Group,User) ]\n>>> pprint( [ (x,y) for x,y in grp_usr_mapping if x.id != y.id ] )\n[(<iRODSGroup 10045 collab>, <iRODSUser 10048 alice rodsuser tempZone>),\n (<iRODSGroup 10001 rodsadmin>, <iRODSUser 10003 rods rodsadmin tempZone>),\n (<iRODSGroup 10002 public>, <iRODSUser 10003 rods rodsadmin tempZone>),\n (<iRODSGroup 10002 public>, <iRODSUser 10048 alice rodsuser tempZone>),\n (<iRODSGroup 10045 collab>, <iRODSUser 13187 bobby rodsuser tempZone>),\n (<iRODSGroup 10002 public>, <iRODSUser 13187 bobby rodsuser tempZone>)]\n```\n\n(Note that in general queries, fields cannot be compared to each other,\nonly to literal constants; thus the '!=' comparison in the Python list\ncomprehension.)\n\nFrom the above, we can see that the group 'collab' (with user ID\n10045) contains users 'bobby'(13187) and 'alice'(10048) but not\n'rods'(10003), as the tuple (10045,10003) is not listed. Group\n'rodsadmin'(10001) contains user 'rods'(10003) but no other users;\nand group 'public'(10002) by default contains all canonical users\n(those whose User.type is 'rodsadmin' or 'rodsuser'). The empty\ngroup ('empty') has no users as members, so it doesn't show up in our\nfinal list.\n\nGroup Administrator Capabilities\n--------------------------------\n\nWith v1.1.7, PRC acquires the full range of abilities possessed by the\nigroupadmin command.\n\nFirstly, a groupadmin may invoke methods to create groups, and may add\nusers to, or remove them from, any group to which they themselves\nbelong:\n\n```python\n>>> session.groups.create('lab')\n>>> session.groups.addmember('lab',session.username) # allow self to administer group\n>>> session.groups.addmember('lab','otheruser')\n>>> session.groups.removemember('lab','otheruser')\n```\n\nIn addition, a groupadmin may also create accounts for new users and\nenable their logins by initializing a native password for them:\n\n```python\n>>> session.users.create_with_password('alice', 'change_me')\n```\n\niRODS Permissions (ACLs)\n------------------------\n\nThe `iRODSAccess` class offers a convenient dictionary\ninterface mapping iRODS permission strings to the corresponding integer\ncodes:\n\n```python\n>>> from irods.access import iRODSAccess\n>>> iRODSAccess.keys()\n['null', 'read_metadata', 'read_object', 'create_metadata', 'modify_metadata', 'delete_metadata', 'create_object', 'modify_object', 'delete_object', 'own']\n>>> WRITE = iRODSAccess.to_int('modify_object')\n```\n\nArmed with that, we can then query on all data objects with ACLs that\nallow our user to write them:\n\n```python\n>>> from irods.models import (DataObject, User, DataAccess)\n>>> data_objects_writable = list(session.query(DataObject, User, DataAccess).filter(User.name == session.username, DataAccess.type >= WRITE))\n```\n\nFinally, we can also access the list of permissions available through a\ngiven session object via the `available_permissions`\nproperty. Note that -- in keeping with changes in iRODS server 4.3 --\nthe permissions list will be longer, as appropriate, for session objects\nconnected to the more recent servers; and also that the embedded spaces\nin some 4.2 permission strings will be replaced by underscores in 4.3\nand later.\n\n```python\n>>> session.server_version\n(4, 2, 11)\n>>> session.available_permissions.items()\n[('null', 1000), ('read object', 1050), ('modify object', 1120), ('own', 1200)]\n```\n\nGetting and setting permissions\n-------------------------------\n\nWe can find the ID's of all the collections writable (ie having\n\"modify\" ACL) by, but not owned by, alice (or even alice\\#otherZone):\n\n```python\n>>> from irods.models import Collection,CollectionAccess,CollectionUser,User\n>>> from irods.column import Like\n>>> q = session.query (Collection,CollectionAccess).filter(\n... CollectionUser.name == 'alice', # User.zone == 'otherZone', # zone optional\n... Like(CollectionAccess.name, 'modify%') ) #defaults to current zone\n```\n\nIf we then want to downgrade those permissions to read-only, we can do\nthe following:\n\n```python\n>>> from irods.access import iRODSAccess\n>>> for c in q:\n... session.acls.set( iRODSAccess('read', c[Collection.name], 'alice', # 'otherZone' # zone optional\n... ))\n```\n\nA call to `session.acls.get(c)` -- with `c`\nbeing the result of\n`sessions.collections.get(c[Collection.name])` -- would\nthen verify the desired change had taken place (as well as list all ACLs\nstored in the catalog for that collection).\n\nOne last note on permissions: The older access manager,\n`<session>.permissions`, produced inconsistent results when\nthe `get()` method was invoked with the parameter\n`report_raw_acls` set (or defaulting) to\n`False`. Specifically, collections would exhibit the\n\"non-raw-ACL\" behavior of reporting individual member users'\npermissions as a by-product of group ACLs, whereas data objects would\nnot.\n\nIn release v1.1.6, we moved to correct this inconsistency by introducing\nthe synonym `<session>.acls` that acts almost identically\nlike `<session>.permissions`, except that the\n`<session>.acls.get(...)` method does not accept the\n`report_raw_acls` parameter. When we need to detect users'\npermissions independent of their access to an object via group\nmembership, this can be achieved with another query.\n\n`<session>.permissions` was therefore removed in v2.0.0\nin favor of `<session>.acls`.\n\nQuotas (v2.0.0)\n---------------\n\nQuotas may be set for a group:\n\n```python\nsession.groups.set_quota('my_group', 50000, resource = 'my_limited_resource')\n```\n\nor per user, prior to iRODS 4.3.0:\n\n```python\nsession.users.set_quota('alice', 100000)\n```\n\n(The default for the resource parameter is \"total\", denoting a general\nquota usage not bound to a particular resource.)\n\nThe Quota model is also available for queries. So, to determine the\nspace remaining for a certain group on a given resource:\n\n```python\nfrom irods.models import Quota\nsession.groups.calculate_usage()\ngroup, resource = ['my_group', 'my_limited_resource']\nspace_left_in_bytes = list(session.query(Quota.over).filter(Quota.user_id == session.groups.get(group).id,\n Quota.resc_id == session.resources.get(resource).id))[0][Quota.over] * -1\n```\n\nAnd, to remove all quotas for a given group, one might (as a rodsadmin)\ndo the following:\n\n```python\nfrom irods.models import Resource, Quota\nresc_map = dict([(x[Resource.id],x[Resource.name]) for x in sess.query(Resource)] + [(0,'total')])\ngroup = sess.groups.get('my_group')\nfor quota in sess.query(Quota).filter(Quota.user_id == group.id):\n sess.groups.remove_quota(group.name, resource = resc_map[quota.resc_id])\n```\n\nManaging users\n--------------\n\nYou can create a user in the current zone (with an optional auth_str):\n\n```python\n>>> session.users.create('user', 'rodsuser', 'MyZone', auth_str)\n```\n\nIf you want to create a user in a federated zone, use:\n\n```python\n>>> session.users.create('user', 'rodsuser', 'OtherZone', auth_str)\n```\n\nAnd more ...\n------------\n\nAdditional code samples are available in the [test\ndirectory](https://github.com/irods/python-irodsclient/tree/main/irods/test)\n\nTesting\n=======\n\nSetting up and running tests\n----------------------------\n\nThe Python iRODS Client comes with its own suite of tests. Some amount\nof setting up may be necessary first:\n\n1. Use `iinit` to specify the iRODS client environment.\n For best results, point the client at a server running on the local\n host.\n2. Install the python-irodsclient along with the\n `unittest unittest_xml_reporting` module or the older\n `xmlrunner` equivalent.\n - for PRC versions 1.1.1 and later:\n - `pip install ./path-to-python-irodsclient-repo[tests]`\n (when using a local Git repo); or,\n - `pip install python-irodsclient[tests]'>=1.1.1'`\n (when installing directly from PyPI).\n - earlier releases (\\<= 1.1.0) will install the outdated\n `xmlrunner` module automatically\n3. Follow further instructions in the [test\n directory](https://github.com/irods/python-irodsclient/tree/main/irods/test)\n\nTesting S3 parallel transfer\n----------------------------\n\nSystem requirements:\n\n - Ubuntu 18 user with Docker installed.\n - Local instance of iRODS server running.\n - Logged in sudo privileges.\n\nRun a MinIO service:\n\n```\n$ docker run -d -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address \":9001\"\n```\n\nSet up a bucket `s3://irods` under MinIO:\n\n```\n$ pip install awscli\n\n$ aws configure\nAWS Access Key ID [None]: minioadmin\nAWS Secret Access Key [None]: minioadmin\nDefault region name [None]:\nDefault output format [None]:\n\n$ aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://irods\n```\n\nSet up s3 credentials for the iRODS s3 storage resource:\n\n```\n$ sudo su - irods -c \"/bin/echo -e 'minioadmin\\nminioadmin' >/var/lib/irods/s3-credentials\"\n$ sudo chown 600 /var/lib/irods/s3-credentials\n```\n\nCreate the s3 storage resource:\n\n```\n$ sudo apt install irods-resource-plugin-s3\n```\n\nAs the 'irods' service account user:\n\n```\n$ iadmin mkresc s3resc s3 $(hostname):/irods/ \\\n \"S3_DEFAULT_HOSTNAME=localhost:9000;\"\\\n \"S3_AUTH_FILE=/var/lib/irods/s3-credentials;\"\\\n \"S3_REGIONNAME=us-east-1;\"\\\n \"S3_RETRY_COUNT=1;\"\\\n \"S3_WAIT_TIME_SEC=3;\"\\\n \"S3_PROTO=HTTP;\"\\\n \"ARCHIVE_NAMING_POLICY=consistent;\"\\\n \"HOST_MODE=cacheless_attached\"\n\n$ dd if=/dev/urandom of=largefile count=40k bs=1k # create 40-megabyte test file\n\n$ pip install 'python-irodsclient>=1.1.2'\n\n$ python -c\"from irods.test.helpers import make_session\n import irods.keywords as kw\n with make_session() as sess:\n sess.data_objects.put( 'largefile',\n '/tempZone/home/rods/largeFile1',\n **{kw.DEST_RESC_NAME_KW:'s3resc'} )\n sess.data_objects.get( '/tempZone/home/rods/largeFile1',\n '/tmp/largefile')\n```\n\n\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "A python API for iRODS",
"version": "2.0.1",
"project_urls": {
"Homepage": "https://github.com/irods/python-irodsclient"
},
"split_keywords": [
"irods"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "33bbd6adec2e95c5cb3b14ae159b43e0a99e8d3991739dc75829f5337fad5645",
"md5": "2322254887ab3317f85afb87a730fad5",
"sha256": "5c1da14d6f9d0137ddd49b3ff2be951a0f77d649df70336898f0d3bf215553ad"
},
"downloads": -1,
"filename": "python_irodsclient-2.0.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "2322254887ab3317f85afb87a730fad5",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 224300,
"upload_time": "2024-04-30T18:39:03",
"upload_time_iso_8601": "2024-04-30T18:39:03.168142Z",
"url": "https://files.pythonhosted.org/packages/33/bb/d6adec2e95c5cb3b14ae159b43e0a99e8d3991739dc75829f5337fad5645/python_irodsclient-2.0.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "943f8c869c0a09fa28adb65f5543190f34b42ae98f8b7add3dbafdbcdf66dfac",
"md5": "65c6f633817106b7861ee9dc5d6ad9bb",
"sha256": "5f312f5ee8c8b82288ab94fe0c5cb674193f4d4cd8cf2c38fc4505713d786ea4"
},
"downloads": -1,
"filename": "python-irodsclient-2.0.1.tar.gz",
"has_sig": false,
"md5_digest": "65c6f633817106b7861ee9dc5d6ad9bb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 247972,
"upload_time": "2024-04-30T18:39:05",
"upload_time_iso_8601": "2024-04-30T18:39:05.751466Z",
"url": "https://files.pythonhosted.org/packages/94/3f/8c869c0a09fa28adb65f5543190f34b42ae98f8b7add3dbafdbcdf66dfac/python-irodsclient-2.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-30 18:39:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "irods",
"github_project": "python-irodsclient",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "python-irodsclient"
}
JSON
