| Name | datahammer JSON |
| Version |
1.0.3
JSON |
| download |
| home_page | https://gitlab.com/n2vram/datahammer |
| Summary | This module provides an easy way to manipulate and inspect lists of data. It was designed to handle plain data types, especially the output from parsing JSON. It allows simple operations to be done on the items in parallel in a concise fashion. Many features will also work on other data types. |
| upload_time | 2023-08-05 02:15:35 |
| maintainer | |
| docs_url | None |
| author | NVRAM |
| requires_python | |
| license | MIT |
| keywords |
data
datasets
queries
json
resultsets
datahammer
hammer
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
datahammer
##########
`Version 1.0.3`
"When all you have is a hammer, everything looks like a nail." - *Anonymous*
----------
*Note that although the version is barely past 1.0, the code* is *production-ready.*
*It has 100% unit test coverage, is extensively document and has considerable examples.*
**TL;DR:** Too anxious to see just what *DataHammer* can do? Check out the `DataHammer Examples`_.
----------
.. contents:: **Index**
:depth: 2
:local:
.. style table { border: 2px solid red; font-family: fujimoto; }
Summary
=======
This module provides an easy way to filter, inspect, analyze and manipulate many similar data items. It was
designed to handle plain data types, especially the output from parsing JSON. It is designed to allow
operations to be done a concise fashion, and on all items in a simple parallel manner.
It works on many data types, but excels at handling of *list*, *tuple* and *dict* values - and was designed
primarily for handling similar result objects from parsing JSON or XML.
In the design, concise usages was favored over speed of performance - although for data sets of less than a
million records it is still quite performant. It was inspired by a need for a concise data manipulation
syntax and by the projects `jQuery <https://jquery.com/>`_ and `jq <https://stedolan.github.io/sjq/>`_,
and as such it also has no dependencies beyond a standar Python instance.
Details
=======
- Most operations on a *DataHammer* instance return a value or a new instance, they do not mutate the
contained data, although a returned ITEM could be mutated by the calling code.
- The contained data can be retrieved with the invert operator (**~**). It will be a **list**
unless constructed with a single ITEM, in which case that ITEM will be returned.
- In order to allow accessing arbitrary ITEM attributes uses the dot notation, **public functions start
with a single underscore**, in contrast to typical Python conventions. See `Functions`_.
- It uses a **list** as its top-level container, and will convert a **tuple** and some generators into a
**list**.
- When constructed with a single ITEM, that item will be wrapped in a **list** and *most* operations will
be identitical to having been constructed with a list with that single ITEM.
- It uses '.' to access *dict* members or object attributes, using **None** for items where there is no key or
attribute with the specified name, thus no *KeyError* or *AttributeError* will be raised.
- Almost all operations will silently ignore items that do not have a member with the "intended" key, attribute
or index.
- There is a **Mutator** class returned by the **_mutator()** function that is designed to allow modifying the
data in-place for some of the
`Augmented Assignment statements <https://docs.python.org/3/reference/simple_stmts.html#grammar-token-augmented_assignment_stmt>`_.
Build Status
------------
Current status:
.. image:: https://gitlab.com/n2vram/test-ci/badges/master/build.svg?rev=master
:alt: Build Status
:target: https://gitlab.com/n2vram/datahammer/pipelines
Known Issues
------------
- Using "*ITEM in OBJ*" works as you probably expect, but avoid using "*OBJ in OTHER*" for iterable
containers. [6]_
- By design and intent, the bitwise operators (`&`, `|`, `^`) actually create a new instance by applying
the `and`, `or` and `xor` operators, respectively. This is because those keyword operators cannot be
overridden to return an object as we wish.
- There are missing operators that could be added. Among these are **del** (attribute or key),
and the bitwise math operators.
- The ``OBJ._long()`` method was removed in version 0.9.6, since it was complicating the code and testing,
and is identical to using: ``OBJ._apply(long)`` Apologies for the breakage, if you hit that.
- The ``OBJ._toCSV()`` method changed in version 0.9.7, since it wasn't using the builtin *csv* package,
and was using a backslash to escape double-quotes within the quoted text, rather than a pair of
consecutive double-quotes. For example, the previous output ``"Derrial \"Preacher\" Book"`` is now
output as ``"Derrial ""Preacher"" Book"``.
Construction
------------
Creating a *DataHammer* can take several sources for its input. It is designed for use on a **list** of items
with the same schema.
+--------------------+----------------------------------------------------------------+
| **Parameters** | **Description** |
+====================+================================================================+
| ``data`` | This must be one of: |
| | |
| | * A `list` of ITEMS. |
| | * A single, non-`list` ITEM. |
| | |
| | If the **json** value is true, then `data` can be either of: |
| | |
| | * A `file` object, from which *all* data is read, and the |
| | results are treated as TEXT, or... |
| | * TEXT to be parsed as JSON. |
+--------------------+----------------------------------------------------------------+
| ``copy`` | If given and true, then a `deepcopy` will be made of `data`. |
+--------------------+----------------------------------------------------------------+
| ``json`` | If provided, it should either be `True` or a dict of arguments |
| | to be passed to *JSON.loads()* for when `data` is of either |
| | the `file` or `TEXT` forms. |
+--------------------+----------------------------------------------------------------+
Operations
^^^^^^^^^^
This is a list of supported operations, including applying builtin Python functions. [1]_
+------------------------------------------+---------------------------------------------------------------+
| **Operation** | **Description** |
+==========================================+===============================================================+
| ``~OBJ`` | Returns the contained data. |
+------------------------------------------+---------------------------------------------------------------+
| | ``OBJ.index`` | Creates a list by applying the *index* (an *int* for *list* |
| | ``OBJ._ind(index)`` | items, a key for *dict* items, or the name of an *attribute* |
| | ``OBJ._get(index)`` | or *property*), returning a *DataHammer* instance created |
| | using that list. [2]_ |
+------------------------------------------+---------------------------------------------------------------+
| | ``OBJ`` *op* ``OTHER`` | Return a *DataHammer* instance with a bool result from the |
| | *op* can be: ``< <= == != >= >`` | comparison of each ITEM with OTHER. [3]_ |
| | |
| | To test equality of contents, use: *~OBJ == OTHER* |
+------------------------------------------+---------------------------------------------------------------+
| | ``OBJ`` *bitop* ``OTHER`` | Return a *DataHammer* instance with the results of applying |
| | ``OTHER`` *bitop* ``OBJ`` | `and`, `or` and a "bool-xor" to each *ITEM* and *OTHER*, or |
| | *bitop* can be: ``& ^ |`` | (*OTHER* and *ITEM*). These are needed since those keywords |
| | cannot be overridden in the desired fashion. [4]_ |
+------------------------------------------+---------------------------------------------------------------+
| | ``OBJ`` *mathop* ``OTHER`` | Return a *DataHammer* instance with the results of applying |
| | *mathop* can be: ``+ - * / // ** %`` | a math operators in: *ITEM mathop OTHER*. [3]_ |
+------------------------------------------+---------------------------------------------------------------+
| | ``OTHER`` *mathop* ``OBJ`` | Return a *DataHammer* instance with the results of applying |
| | *mathop* can be: ``+ - * / // ** %`` | a math operators in: *OTHER mathop ITEM*. [3]_ |
+------------------------------------------+---------------------------------------------------------------+
| ``OBJ[indexes]`` | Depending on the argument, returns a *DataHammer* instance, a |
| | single contained ITEM, or a list of ITEMs. [4]_ |
| | See `Indexing`_, for more information. |
+------------------------------------------+---------------------------------------------------------------+
| | ``OBJ._bool()`` | Return a *DataHammer* instance with the results of applying |
| | ``OBJ._int()`` | the builtin type (*of the same name w/o the underscore*) to |
| | ``OBJ._float()`` | each item in the list. |
+------------------------------------------+---------------------------------------------------------------+
| ``reversed(OBJ)`` | Return a *DataHammer* instance with the contained data in |
| | reversed order. |
+------------------------------------------+---------------------------------------------------------------+
| ``len(OBJ)`` | Return an *int* for the number of contained data ITEMs. |
+------------------------------------------+---------------------------------------------------------------+
| ``hash(OBJ)`` | Return an *int* that is the hash of the tuple of the hash of |
| | every ITEM. |
| | This will raise an exception if *any* ITEM cannot be hashed. |
+------------------------------------------+---------------------------------------------------------------+
| ``ARG in OBJ`` | Return a bool, which is `True` if any *ITEM == OBJ*. |
| | With regard to limiting the items tested. [3]_ |
+------------------------------------------+---------------------------------------------------------------+
| ``OBJ in ARG`` | *This is almost never what you want!* Return a single bool, |
| | ignoring of contents of ARG or OBJ. The result is `True` if |
| | neither ARG nor OBJ are empty, and `False` if they both are. |
+------------------------------------------+---------------------------------------------------------------+
| ``-OBJ`` *(unary minus)* | Return a *DataHammer* instance with the results of applying |
| | *not ITEM* on each item. |
+------------------------------------------+---------------------------------------------------------------+
NOTE: The ``OBJ._long()`` method was removed in version 0.9.6, since it was complicating the code and testing,
and is identical to using: ``OBJ._apply(long)``
Functions
---------
This is a list of supported functions. [1]_
+-------------------------------------------+---------------------------------------------------------------+
| **Function** | **Description** |
+===========================================+===============================================================+
| | ``OBJ._ind(name)`` | Attribute, index or *dict* key dereference. [2]_ |
| | ``OBJ._get(name)`` | |
+-------------------------------------------+---------------------------------------------------------------+
| ``str(OBJ)`` | Returns a JSON dump of the contained data. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._contains(ARG)`` | Return a *DataHammer* instance with the results of applying |
| | *ARG in ITEM* for each item. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._in(ARG)`` | Return a *DataHammer* instance with the results of applying |
| | *ITEM in ARG* for each item. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._apply(FUNC, ARG, *ARGS, **KWDS)`` | Return a *DataHammer* instance with the results of applying |
| | ``FUNC(ITEM, ARG, *ARGS, **KWDS)`` to each item. [3]_ |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._strip(ARG)`` | Return a *DataHammer* instance with only the desired items. |
| | Based on the type of ARG given, the new instance has only the |
| | items for which the result is true of: |
| | 1. If ARG is not given: *bool(ITEM)* |
| | 2. If ARG is a callable: *ARG(ITEM)* |
| | 3. If ARG is a list, tuple or set: *(ITEM in ARG)* |
| | 4. Otherwise: *ITEM == ARG* |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._insert(INDEX, ITEM)`` | Return a *DataHammer* instance with ITEM inserted at INDEX. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._extend(INDEX, ITEMS)`` | Return a *DataHammer* instance with ITEMS added at the end. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._splice(INDEX, DELNUM, *ITEM)`` | Return a *DataHammer* instance with DELNUM items deleted at |
| | INDEX, and with ITEM(s) inserted there. [5]_ |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._slice(START [, END [, STEP ] ])`` | Return a *DataHammer* instance with the list sliced according |
| | to the given indices (like *list* slicing works). |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._flatten()`` | Return a *DataHammer* instance with contained items that are |
| | the result of flattening *this* instance's contained items by |
| | one level. Sub-items are added in iteration-order for items |
| | that are a *set*, *list* or *tuple* and for values from a |
| | *dict*. |
| | |
| | Other types are not flattened, and are added as-is. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._tuple(SELECTOR, SELECTOR, ...)`` | Return a tuple of results for each contained item, the result |
| | will be a tuple of values from the items, dereferenced by the |
| | *SELECTOR* parameters, in the same order. See [8]_ |
| | |
| | Only named *SELECTOR* parameters are allowed. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._toCSV(SELECTOR, SELECTOR, ...)`` | Return a tuple of `str` like a `Comma Separated Values` file, |
| | the first `str` represents the headers for each column, and |
| | each subsequent contains a CSV-style representation of the |
| | requested values from each item (which must be serializable). |
| | See [8]_ |
| | |
| | Both positional and named *SELECTOR* parameters are allowed. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._pick(SELECTOR, SELECTOR, ...)`` | Return a *DataHammer* instance of *dict* items made from one |
| | or more sub-items specified by the *SELECTOR*, as either |
| | positional or named parameters. |
| | Parameters dictate the keys in the resulting items. See [8]_ |
| | |
| | Both positional and named *SELECTOR* parameters are allowed. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._groupby(GRP, VALS [, POST])`` | Return a *DataHammer* instance of *dict* items made by taking |
| | all sub-items specified by `VALS` and combine them with other |
| | items with the same `GRP` values. It is similar to the `SQL` |
| | **GROUP BY** clause. See [8]_ and `Grouping`_. |
| | |
| | Both positional and named *SELECTOR* parameters are allowed. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._join(KEYS, OBJ [,FLAGS][,MERGE])`` | Return a *DataHammer* instance of *dict* items from merging |
| | items from this instance and **OBJ**, joining on the values |
| | corresponding to the `KEYS`. The `FLAGS` parameter controls |
| | specifics. Somewhat similar to the `SQL` **JOIN** operations. |
| | See `Joining`_ and the `Deeper Examples`_. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._unique(KEYS [,UNIQUE])`` | Return a *DataHammer* instance with items from this instance, |
| | based on the uniqueness of the values for `KEYS`. The |
| | `UNIQUE` parameter sets handling for items with duplicate key |
| | values. |
| | |
| | See `Unique`_ and the `Deeper Examples`_. |
+-------------------------------------------+---------------------------------------------------------------+
| ``OBJ._mutator()`` | Returns a *DataHammer.Mutator* instance to be used for making |
| | modifications to the contained data. See `Mutators`_. |
+-------------------------------------------+---------------------------------------------------------------+
Indexing
--------
Indexing a *DataHammer* instance with *[]* allows simple access to items from the contained data, but
there are various types of parameters types allowed. [4]_
1. Indexing with an **int** or an implicit or explicit **slice** object works like indexing **list**; the
result is identical to **(~OBJ)[...]**.
* A single item is returned with an **int** argument, and can raise an IndexError.
* A (possibly empty) list of items is returned with either:
* An explicit **slice** argument, eg: OBJ[slice(1, None, 5)]
* An implicit **slice** argument, eg: OBJ[1::5]
2. Indexing with a **list**, **tuple** or a *DataHammer* instance, will return another *DataHammer*
instance. [3]_ The parameter must either be all **bool** or all **int**, and they
dictate *which* items are used to construct the new instance:
* For **bool** indexes, each bool in the argument indicates if the corresponding item in the
*DataHammer* is included in the new instance.
* For **int** indexes, each int is used to index into the contained data, and which item is include
in the new instance. This allows both filtering and reordering of data.
Indexing Examples:
.. code:: python
>>> OBJ = DataHammer(list(range(10, 15)))
# Note that the following dereference the instance with "~" to show the contents:
>>> ~OBJ
[10, 11, 12, 13, 14]
>>> ~OBJ[(True, False, True, True, False, True)]
[10, 12, 13] # The last/6th `True` is ignored since len(OBJ)==5
>>> ~OBJ[(4, 2, 1, 40, -1, 3, 1)]
[14, 12, 11, 14, 13, 11] # 40 is ignored.
# Note these DO NOT dereference the result, they are not a DataHammer instance.
>>> type(OBJ[1])
<type 'int'>
>>> type(OBJ[:5])
<type 'list'>
>>> type(OBJ[slice(3)])
<type 'list'>
>>> OBJ[::3]
[10, 13]
Grouping
--------
The *_groupby(GROUP, VALUES [, POSTPROC])* method creates a new *DataHammer* instance, grouping values from
multiple source items. It functions somewhat like the **GROUP BY** feature of SQL, however rather than
necessarily combining column values, a the list of values is created.
The `GROUP` and `VALUES` parameters should be either a list/tuple or a dict.
- Strings in the list/tuple are treated like named `SELECTOR` parameters
- Items in a dict are treated like named `SELECTOR` parameters.
For each unique sets of values for the `GROUP` keys, one item will exist in the resulting instance. Each of
the new items will contain the grouping values and a value per `VALUES` key. The `GROUP` and `VALUES`
parameters may be either a list/tuple or a dict of `SELECTOR` parameters (see above).
For every key in the `VALUES` parameter, a list is built with the corresponding values, one list for each
set of `GROUP` values.
The `POSTPROC` parameter parameter, is optional and unless provided: each resulting item will contain the
corresponding list for each key in `VALUES`. If `FUNC` is provided, it will be called once per resulting
item. The lists are passed parameters in the same order as the keys in `VALUES`.
Note that the order of the resulting items will be the same as the order of the first occurence of that set
of `GROUP` keys in the source items. And the order of the list of values for each `VALUES` key is the same
as the order that those occurred in the source items.
Joining
-------
There is a method for joining two *DataHammer* instances, combining items for which the specified key values match,
this is partly inspired by the **JOIN** feature of SQL (`JOIN_PRODUCT`), and partly inspired by a use case where
one-to-one matches were needed.
With the two (2) flags [`JOIN_PRODUCT` or `JOIN_ORDERED`] for handling duplicates and the four (4) flags
[`JOIN_KEEP_NEITHER`, `JOIN_KEEP_LEFT`, `JOIN_KEEP_RIGHT` or `JOIN_KEEP_BOTH`] for handling unmatched items, there
are eight (8) different `flags` combinations.
**HANDLING OF ITEMS WITH DUPLICATE KEY VALUES**
Here, "duplicate" key-values means that a set of key-values occurs more than once in the same instance.
+-------------------+--------------------------------------------------------------------------------------+
| "Mode" Flag Name | Description |
+-------------------+--------------------------------------------------------------------------------------+
| `JOIN_PRODUCT` | Results are somewhat similar to SQL joins. The name comes from the "Cartesian |
| | Product" since the output contains an item produced from the each matching item the |
| | left input and the right input. |
+-------------------+--------------------------------------------------------------------------------------+
| `JOIN_ORDERED` | This pairs matching items from the left and the right, one-by-one. Pairing is in |
| | the same order as they were found in the input instances, and matching stops after |
| | exhausting the matching items in either the right input or left input. |
+-------------------+--------------------------------------------------------------------------------------+
If there are no duplicate entries in either input, then these modes function identically.
**HANDING OF UNMATCHED ITEMS**
An "unmatched" item is one whose key-values never occur in the items from the other instance.
Here, the INNER and OUTER join terminology is a remnant from SQL, the "KEEP" flags are equivalent and provided
since they describe the intended action. These can be summarized thus:
+----------------------+--------------------+-----------------------------------------------------------------+
| "Keep" Flag Name | Inner/Outer Name | Deescription |
+----------------------+--------------------+-----------------------------------------------------------------+
| `JOIN_KEEP_NEITHER` | `INNER_JOIN` | Discard unmatched items from left and from the right. |
+----------------------+--------------------+-----------------------------------------------------------------+
| `JOIN_KEEP_LEFT` | `LEFT_OUTER_JOIN` | Discard unmatched items from the right. |
+----------------------+--------------------+-----------------------------------------------------------------+
| `JOIN_KEEP_RIGHT` | `RIGHT_OUTER_JOIN` | Discard unmatched items from the left. |
+----------------------+--------------------+-----------------------------------------------------------------+
| `JOIN_KEEP_BOTH` | `FULL_OUTER_JOIN` | Keep unmatched items from the left and from the right. |
+----------------------+--------------------+-----------------------------------------------------------------+
**OUTPUT ORDER**
The order of items in the inputs dictates the order in the output. The algorithm simply iterates over the left
input, producing zero or more outputs depending on the flags and presence of any matching items in the right input.
It then appends unmatched items from the right, if desired.
See the examples, or use it for yourself, if this is not sufficiently clear.
Notes:
- With `JOIN_PRODUCT`, each matched item from the left will be paired with every matching item from the right, in
the order that the right items occurred.
- With `JOIN_ORDERED`, each item in the left will be paired with the corresponding order of the matching items in
the right input. After the items from the right are exhausted, the remaining items from the left input with that
set of key-values are considered unmatched. In addition, any items from the right input that are not consumed in
this way are also considered unmatched.
- With `JOIN_KEEP_LEFT` or `JOIN_KEEP_BOTH`, unmatched items from the left input will appear in the same order as
they are found in the left input.
- With `JOIN_KEEP_RIGHT` or `JOIN_KEEP_BOTH`, unmatched items from the right input will appear after all items
produced from items in the left input. They will be in the same order as they occurred the right input.
Unique
------------
The **_unique()** method allows eliminating items based on the uniqueness / duplication of key values.
Parameters are:
- `KEYS` should be a list/tuple of strings which are used as a `SELECTOR` into each item. The associated
values are used for the uniqueness test. (If `KEYS` is a single string, it is handled as expected.)
There is a special case when `KEYS` is **None**: if so, the hash of the item is used in lieu of key values.
Obviously, all items must be hashable.
- `UNIQUE` determines which items to keep, based on key values. `UNIQUE` may be:
- 0 = Keep only those items that are unique, with no duplicates.
- 1 = Keep the first item with key values, ignore subsequent duplicates.
- 2 = Keep all instances of items that have duplicate key values.
In each case, the order of the items is preserved from the original data.
Note that with **unique = 2**, there will be multiple items that have the same key values; to remove those you filter
them a second time with the same `KEYS`:
**OBJ._unique(KEYS, 2)._unique(KEYS)**
Mutators
--------
There is some support for making modifications to the data contained within a *DataHammer*, beyond
direct access. This is done with the *DataHammer._mutator* method on the instance.
Here **MUT** is used as a shorthand for **OBJ._mutator()** - which returns a *DataHammer.Mutator*
instance, and the name *Mutator* is also used for *DataHammer.Mutator*.
+-----------------------------------------+----------------------------------------------------------------+
| **Functions and Operation** | **Description** |
+=========================================+================================================================+
| ``MUT = OBJ._mutator()`` | Returns a new *Mutator* for the given *DataHammer* instance. |
+-----------------------------------------+----------------------------------------------------------------+
| ``~MUT`` | Returns the *DataHammer* instance for this *Mutator*. |
+-----------------------------------------+----------------------------------------------------------------+
| | ``MUT.index`` | Returns a new *Mutator* instance useful for modifying the |
| | ``MUT[index]`` | key, attribute or list item at *index*. [7]_ |
| | ``MUT._get(index)`` | |
| | ``MUT._ind(index)`` | Note that *all of these forms work identically*, though the |
| | first form can only be used with valid identifier names. This |
| | is in contrast with **[]** on a *DataHammer* instance where |
| | it returns an item from the contained data. |
+-----------------------------------------+----------------------------------------------------------------+
| | ``MUT`` *op* ``OTHER`` | Update the item member for the given *Mutator* instance, with |
| | *op* can be: ``+= -= *= /= **= //=`` | the given operation, which should be number (or object that |
| | supports that operation). |
+-----------------------------------------+----------------------------------------------------------------+
| ``MUT._set(OTHER)`` | Update the value designated by the given *Mutator* instance, |
| | overwriting with the given value(s). If *OTHER* is a list, |
| | tuple or *DataHammer* instance, then an interator is used, |
| | and application stops when the end is reached. [3]_ |
+-----------------------------------------+----------------------------------------------------------------+
| ``MUT._setall(OTHER)`` | Like ``MUT._set(OTHER)`` but regardless of the type, *OTHER* |
| | is used without iterating. Used to set all rows to the same |
| | *list* or *tuple* value, but can be used with any value/type. |
+-----------------------------------------+----------------------------------------------------------------+
| ``MUT._apply(FUNC, *ARGS, **KWDS)`` | Update the value designated by the given *Mutator* instance, |
| | overwriting with the the *return value* from calling: |
| | **``FUNC(VALUE, *ARGS, **KWDS)``**. |
+-----------------------------------------+----------------------------------------------------------------+
Installation
============
Install the package using **pip**, eg:
`pip install --user datahammer`
Or for a specific version of Python:
`python3 -m pip --user install datahammer`
To the source git repository, use:
`git clone https://gitlab.com/n2vram/datahammer.git`
Releases
--------
+-------------+--------------------------------------------------------+
| **Version** | **Description** |
+=============+========================================================+
| 0.9 | Initial release, documentation prototyping. |
+-------------+--------------------------------------------------------+
| 0.9.1 | Addition of "_pick" method. |
+-------------+--------------------------------------------------------+
| 0.9.2 | Addition of "_flatten" and "_toCSV" methods. |
+-------------+--------------------------------------------------------+
| 0.9.4 | Addition of "_groupby" and "_tuples" methods. |
+-------------+--------------------------------------------------------+
| 0.9.5 | Moved EXAMPLES into (and reorganized) the README file. |
| | Configured for tests, coverage and style on Travis CI. |
+-------------+--------------------------------------------------------+
| 0.9.6 | Removed 'OBJ._long()' method, as it was Python2-only. |
+-------------+--------------------------------------------------------+
| 0.9.7 | Added the 'OBJ._join()' and 'OBJ._fromCSV()' methods. |
+-------------+--------------------------------------------------------+
| 0.9.8 | Added the 'OBJ._unique()' and 'OBJ._in()' methods. |
+-------------+--------------------------------------------------------+
| 1.0 | Moved to gitlab.com, including GitLab-CI. |
+-------------+--------------------------------------------------------+
| 1.0.1 | Minor changes post move to GitLab. |
+-------------+--------------------------------------------------------+
| 1.0.2 | Final changes before pushing to PyPi. |
+-------------+--------------------------------------------------------+
| 1.0.3 | Documentation, whitespace only changes. |
+-------------+--------------------------------------------------------+
Reporting Issues, Contributing
------------------------------
As an open source project, *DataHammer* welcomes contributions and feedback.
1. Report any issues, including with the functionality or with the documentation
via the GitLab project: https://gitlab.com/n2vram/datahammer/issues
2. To contribute to the source code, please use a GitLab pull request for the
project, making sure to include full/extensive unit tests for any changes. Note
that if you cannot create a PR, then open an issue and attach a `diff` output
there. https://gitlab.com/n2vram/datahammer/
3. To translate the documentation, please follow the same process as for source
code contributions.
DataHammer Examples
===================
It is probably easier to show the utility of *DataHammer* with some examples.
Simple Examples
---------------
1. To construct a *DataHammer* instance you generally a list/tuple/iterable of items. Many builtin functions operate
on the *DataHammer* instance as it would on the list of objects. The original data can be returned using the tilde
operator (`~`).
See `Sample Data`_ for the **data** used here.
.. code:: python
>>> dh = DataHammer(data)
>>> len(dh)
8
>>> dh
<datahammer.DataHammer object at 0x7f258fac34e0>
>>> type(~dh)
<type 'list'>
>>> type(dh[0])
<type 'dict'>
>>> type(dh[:3])
<type 'list'>
>>> ~dh == dh[:]
True
>>> bool(dh)
True
2. Accessing the sub-items uses a simple dot notation. To allow irregular data, a `None` will represent a
member that was not present -- no `KeyError`, `AttributeError` or `IndexError` are raised.
.. code:: python
>>> ~dh.age
[45, 57, 33, 21, 24, 60, 63, 33]
>>> ~dh.name.last
['Stewart', 'Perry', 'Young', 'Lewis', 'Ward', 'Martinez', 'Evans', 'Moore']
# No KeyError
>>> ~dh.missingMember
[None, None, None, None, None, None, None, None]
3. Indexing into a list sub-item cannot be done with dot notation or slicing (eg: with `[]`), so the
*_ind()* method is provided for this reason. As for dot notation, if an index is out of range then the
value will be `None`.
.. code:: python
# This is not a DataHammer instance, it is just the `rank` member of the fourth item.
>>> dh.ranks[3]
[180, 190, 111]
# This is a DataHammer instance with the fourth item from each `rank` member, or `None`.
>>> ~dh.ranks._ind(3)
[None, 18, 155, None, None, 24, 64, None]
4. To avoid collisions with item members, the public methods of a *DataHammer* instance are all prefixed
with a single underscore, which may be confusing at first, but this is also done for
`collections.namedtuple` instances. Methods that begin with a double underscore are not public.
.. code:: python
# This 'mean' function is defined in the Sample Data section, below.
>>> ~dh.ranks._apply(mean)
[None, 70.33333333333333, 114.875, 160.33333333333334, 139.0, 40.2, 94.83333333333333, 97.0]
>>> ~dh._splice(2, 4).name.first
['Addison', 'Katherine', 'Grace', 'Sophia']
>>> print("\n".join(dh._toCSV(FIRST='name.first', LAST='name.last', AGE='age')))
"FIRST","LAST","AGE"
"Addison","Stewart",45
"Katherine","Perry",57
"Jack","Young",33
"Brianna","Lewis",21
"Logan","Ward",24
"Logan","Martinez",60
"Grace","Evans",63
"Sophia","Moore",33
5. Many operators are overridden to allow operating on the item with a simple syntax, returning a new *DataHammer*
instance with the results. Most operators work with another *DataHammer* instance, a list/tuple or scalar values.
In the case of a list/tuple, the length of the resulting instance will be the shorter of the two arguments.
.. code:: python
>>> ~(dh.gender == 'F')
[True, True, False, True, False, False, True, True]
>>> ~(dh.salary / 1000.0)
[10.0, 18.59, 28.64, 8.0, 8.0, 33.7, 26.22, 14.12]
>>> ~(dh.age > [50, 40, 30])
[False, True, True]
>>> ~(dh.salary * 1.0 / dh.age) # Avoid integer math.
[222.22222222222223, 326.140350877193, 867.8787878787879, 380.95238095238096,
333.3333333333333, 561.6666666666666, 416.1904761904762, 427.8787878787879]
6. Using many builtin operations work as you would expect, as if passing a list/tuple of the item data instead.
.. code:: python
>>> min(dh.age), max(dh.age)
(21, 63)
>>> sorted(dh.location.state)
['Maryland', 'Maryland', 'New Jersey', 'Oklahoma', 'Oregon', 'Oregon', 'Texas', 'Texas']
>>> sum(dh.salary)
147270
>>> min(dh.salary), mean(dh.salary), max(dh.salary)
(8000, 18408.75, 33700)
# This gives number of females, by counting occurences of `True`.
>>> sum(dh.gender == 'F')
5
7. Indexing with another *DataHammer* instance is another powerful feature. Also, indexing with integers allows
arbitrary keeping a subset of, or reordering of, the items.
.. code:: python
>>> len(dh.age < 30), sum(dh.age < 30)
(8, 2)
>>> twenties = (20 <= dh.age < 30)
>>> ~twenties
[False, False, False, True, True, False, False, False]
>>> ~dh[twenties].name
[{'first': 'Brianna', 'last': 'Lewis'}, {'first': 'Logan', 'last': 'Ward'}]
>>> ~dh.name.last
['Stewart', 'Perry', 'Young', 'Lewis', 'Ward', 'Martinez', 'Evans', 'Moore']
>>> ~dh[(0, 5, 3, 4)].name.last
['Stewart', 'Martinez', 'Lewis', 'Ward']
Deeper Examples
---------------
These demonstrate the extracting and manipulating power of *DataHammer* instances. Note that these examples and notes
are not trivial, so please read carefully so you can understand the functionality as it is designed.
8. There are methods for extracting parts of each item, including *_pick()*, *_tuples()* and *_toCSV()*. In addition
the *_groupby()* method allows extracting only certain parts `and` combining them across the items that share
certain values, similar to the **GROUP BY** syntax in SQL.
See the main README section for detailed *SELECTOR Syntax*, but the methods are demonstrated here:
a. The *_tuples(SELECTOR [, SELECTOR ...])* method returns a tuple of tuples with extracted values in the same order
as the names. Only positional `SELECTOR` parameters are allowed.
.. code:: python
>>> dh._tuples('location.city', 'name.last', 'age')
(('Baltimore', 'Stewart', 45),
('Baltimore', 'Perry', 57),
('Portland', 'Young', 33),
('San Antonio', 'Lewis', 21),
('Oklahoma ', 'Ward', 24),
('Portland', 'Martinez', 60),
('Jersey City', 'Evans', 63),
('San Antonio', 'Moore', 33))
b. The *_toCSV(SELECTOR [, SELECTOR ...])* method returns a tuple of strings in a `Comma Separated Values`
format. The first string is a header of the column names in order. Each subsequent string represents the
corresponding item in the data, in order. Both positional and named `SELECTOR` parameters are allowed.
.. code:: python
>>> dh._toCSV('location.city', lname='name.last', yrs='age')
('"city","lname","yrs"',
'"Baltimore","Stewart",45',
'"Baltimore","Perry",57',
'"Portland","Young",33',
'"San Antonio","Lewis",21',
'"Oklahoma ","Ward",24',
'"Portland","Martinez",60',
'"Jersey City","Evans",63',
'"San Antonio","Moore",33')
c. The *_pick(SELECTOR [, SELECTOR ...])* method returns a new *DataHammer* instance where each item is a dictionary
with only the requested members. Positional and named `SELECTOR` parameters are allowed.
.. code:: python
>>> ~dh._pick('location.state', ln='name.last', fn='name.first', years='age')
[{'state': 'Maryland', 'ln': 'Stewart', 'fn': 'Addison', 'years': 45},
{'state': 'Maryland', 'ln': 'Perry', 'fn': 'Katherine', 'years': 57},
{'state': 'Oregon', 'ln': 'Young', 'fn': 'Jack', 'years': 33},
{'state': 'Texas', 'ln': 'Lewis', 'fn': 'Brianna', 'years': 21},
{'state': 'Oklahoma', 'ln': 'Ward', 'fn': 'Logan', 'years': 24},
{'state': 'Oregon', 'ln': 'Martinez', 'fn': 'Logan', 'years': 60},
{'state': 'New Jersey', 'ln': 'Evans', 'fn': 'Grace', 'years': 63},
{'state': 'Texas', 'ln': 'Moore', 'fn': 'Sophia', 'years': 33}]
d. The *_groupby(GROUP, VALUES [, POSTPROC])* method returns a new *DataHammer* instance, using the first list of
keys for grouping by value, and the second list as the values to groupby. Like the **GROUP BY** functionality
in SQL, there will be one item in the resulting instance for each unique set of values of the `GROUP` keys.
Remember: even if passing a single key for `GROUP` or `VALUES`, it must be in a tuple or list.
.. code:: python
# An empty second parameter is allowed, too, the results is just the unique GROUP keys.
>>> ~dh._groupby(['gender', 'title'], [])
[{'gender': 'F', 'title': 'Systems Administrator'},
{'gender': 'F', 'title': 'Bookkeeper'},
{'gender': 'M', 'title': 'Controller'},
{'gender': 'F', 'title': 'UX Designer'},
{'gender': 'M', 'title': 'Web Developer'},
{'gender': 'M', 'title': 'Assessor'},
{'gender': 'F', 'title': 'Mobile Developer'}]
>>> ~dh._groupby(['gender'], ('age', 'salary'))
[{'gender': 'F', 'age': [45, 57, 21, 63, 33], 'salary': [10000, 18590, 8000, 26220, 14120]},
{'gender': 'M', 'age': [33, 24, 60], 'salary': [28640, 8000, 33700]}]
The third parameter is a callable that takes the constructed lists in `VALUES` key order, and
returns a tuple with same number of items, in the same order.
.. code:: python
>>> def reductor(ages, salaries):
... return (min(ages), max(ages)), (min(salaries), max(salaries))
>>> ~dh._groupby(['gender'], ('age', 'salary'), reductor)
[{'gender': 'F', 'age': (21, 63), 'salary': (8000, 26220)},
{'gender': 'M', 'age': (24, 60), 'salary': (8000, 33700)}]
9. There is a method for joining two *DataHammer* instances, combining items for which the specified
key values match. The `JOIN_PRODUCT` mode is inspired by the **JOIN** feature of SQL, whiel
`JOIN_ORDERED` was inspired by a use case where one-to-one matches were needed.
.. code:: python
>>> left = DataHammer([{"k": "A", "x": 1}, {"k": "B", "x": 2}, {"k": "C", "x": 3},
... {"k": "C", "x": 4}, {"k": "D", "x": 5}])
>>> right = DataHammer([{"k": "A", "y": 1}, {"k": "A", "y": 2}, {"k": "C", "y": 3},
... {"k": "C", "y": 4}, {"k": "E", "y": 5}])
# For JOIN_PRODUCT, each matched item from the left is paired with each the corresponding item
# from the right. Then the JOIN_KEEP_{...} flag determines unmatched item retention.
# Default is ORDERED + NEITHER
>>> ~left._join("k", right)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'A', 'x': 1, 'y': 2},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 3, 'y': 4},
{'k': 'C', 'x': 4, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4}]
>>> ~left._join("k", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_NEITHER)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'A', 'x': 1, 'y': 2},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 3, 'y': 4},
{'k': 'C', 'x': 4, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4}]
>>> ~left._join("k", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_RIGHT)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'A', 'x': 1, 'y': 2},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 3, 'y': 4},
{'k': 'C', 'x': 4, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4},
{'k': 'E', 'y': 5}]
>>> ~left._join("k", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_LEFT)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'A', 'x': 1, 'y': 2},
{'k': 'B', 'x': 2},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 3, 'y': 4},
{'k': 'C', 'x': 4, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4},
{'k': 'D', 'x': 5}]
>>> ~left._join("k", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_BOTH)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'A', 'x': 1, 'y': 2},
{'k': 'B', 'x': 2},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 3, 'y': 4},
{'k': 'C', 'x': 4, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4},
{'k': 'D', 'x': 5},
{'k': 'E', 'y': 5}]
# For JOIN_ORDERED, matched items from the left and right are paired, one-by-one, but only as
# until either side is exhausted, the remaining items are 'unmatched' and the JOIN_KEEP_{...}
# flag determines unmatched item retention.
>>> ~left._join("k", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_NEITHER)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4}]
>>> ~left._join("k", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_RIGHT)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4},
{'k': 'A', 'y': 2},
{'k': 'E', 'y': 5}]
>>> ~left._join("k", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_LEFT)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'B', 'x': 2},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4},
{'k': 'D', 'x': 5}]
>>> ~left._join("k", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_BOTH)
[{'k': 'A', 'x': 1, 'y': 1},
{'k': 'B', 'x': 2},
{'k': 'C', 'x': 3, 'y': 3},
{'k': 'C', 'x': 4, 'y': 4},
{'k': 'D', 'x': 5},
{'k': 'A', 'y': 2},
{'k': 'E', 'y': 5}]
(*Obviously, the outputs above were reformmated for clarity.*)
10. There is a method for easily dealing with duplicate values on particular keys.
Once again the idea of *key values* is used to determine what is considered.
Note that if KEYS is a string, it is handled correctly.
.. code:: python
>>> dh = DataHammer(data)
>>> keys = 'location.city'
>>> Counter(dh.location.city)
Counter({'Baltimore': 2, 'Portland': 2, 'San Antonio': 2, 'Oklahoma ': 1, 'Jersey City': 1})
# Zero (0) gives only the unique items, where count was 1 -- all but "Mobile Developer".
>>> ~dh._unique(keys, 0).location.city
['Oklahoma City', 'Jersey City']
# The default (one, 1) gives all key values, but only the first item with the value(s)
>>> ~dh._unique(keys).location.city
['Baltimore', 'Portland', 'San Antonio', 'Oklahoma City', 'Jersey City']
>>> ~dh._unique(keys, 1).name
[{'first': 'Addison', 'last': 'Stewart'}, {'first': 'Jack', 'last': 'Young'},
{'first': 'Brianna', 'last': 'Lewis'}, {'first': 'Logan', 'last': 'Ward'},
{'first': 'Grace', 'last': 'Evans'}]
# Two (2) gives only those items that have a duplicate (of 'location.city').
>>> ~dh._unique(keys, 2)._pick('name', keys)
[{'name': {'first': 'Addison', 'last': 'Stewart'}, 'city': 'Baltimore'},
{'name': {'first': 'Katherine', 'last': 'Perry'}, 'city': 'Baltimore'},
{'name': {'first': 'Jack', 'last': 'Young'}, 'city': 'Portland'},
{'name': {'first': 'Brianna', 'last': 'Lewis'}, 'city': 'San Antonio'},
{'name': {'first': 'Logan', 'last': 'Martinez'}, 'city': 'Portland'},
{'name': {'first': 'Sophia', 'last': 'Moore'}, 'city': 'San Antonio'}]
# To get the unique set of duplicated values for a given set of keys, you can use a second
# pass, using the default (1) for `unique`.
>>> ~dh._unique(keys, 2)._unique(keys).location
[{'city': 'Baltimore', 'state': 'Maryland'},
{'city': 'Portland', 'state': 'Oregon'},
{'city': 'San Antonio', 'state': 'Texas'}]
Formatting Specification
------------------------
11. An extension is provided for formatting, using the **j** `type`. Each item will be printed as JSON using
*json.dumps()*. In particular, the only allowed parts to the *format_spec* are:
a. A negative `sign` will cause a newline to be inserted between the item outputs.
b. A non-zero `width` causes the item JSON is used as the indent within the item output
c. The only `type` supported is "**j**".
.. code:: python
>>> dh.location[0:2]
[{'city': 'Baltimore', 'state': 'Maryland'}, {'city': 'Madison', 'state': 'Wisconsin'}]
>>> print("{:-j}".format(dh.location._slice(0,2)))
[{"city":"Baltimore","state":"Maryland"},
{"city":"Madison","state":"Wisconsin"}]
>>> print("{:-3j}".format(dh.location._slice(0,2)))
[{
"city":"Baltimore",
"state":"Maryland"
},
{
"city":"Madison",
"state":"Wisconsin"
}]
Warnings and Caveats
--------------------
12. Warning: To combine multiple instances with `bool` values you must use the `&` and `|`, and
*not* use `and` and `or` as you would with Python `bool` values.
.. code:: python
>>> dh1 = DataHammer([False, False, True, True])
>>> dh2 = DataHammer([False, True, False, True])
# These are item-wise correct results
>>> ~(dh1 & dh2)
[False, False, False, True]
>>> ~(dh1 | dh2)
[False, True, True, True]
# Since the objects are not empty, 'or' returns the first, 'and' returns the second:
>>> (dh1 or dh2) == dh1
True
>>> (dh1 and dh2) == dh2
True
Other Examples
--------------
13. Given a JSON file that has metadata separated from the data values, we can easily
combine these, and find the ones which match criteria we want.
.. code:: python
>>> from datahammer import DataHammer
>>> from six.moves.urllib import request
>>> from collections import Counter
>>> URL = 'https://data.ny.gov/api/views/pxa9-czw8/rows.json?accessType=DOWNLOAD'
>>> req = request.urlopen(URL)
>>> jobs = DataHammer(req, json=dict(encoding='utf-8'))
# Grab the contained data in order to find its keys.
>>> (~jobs).keys()
dict_keys(['meta', 'data'])
>>> names = jobs.meta.view.columns.name
>>> norm = DataHammer(dict(zip(names, row)) for row in jobs.data)
# Here 'norm' contains 1260 items, each a dict with the same schema.
>>> len(norm)
1260
>>> print(norm[0])
{'sid': 1, 'id': 'A0447302-02D8-4EFD-AB68-777680645F02', 'position': 1,
'created_at': 1437380960, 'created_meta': '707861', 'updated_at': 1437380960,
'updated_meta': '707861', 'meta': None, 'Year': '2012', 'Region': 'Capital Region',
'NAICS Code': '11', 'Industry': 'Agriculture, Forestry, Fishing and Hunting',
'Jobs': '2183'}
# Use collections.Counter to count the number of instances of values:
>>> Counter(norm.Year)
Counter({'2012': 210, '2013': 210, '2014': 210, '2015': 210, '2017': 210, '2016': 210})
>>> Counter(norm._get('NAICS Code'))
Counter({'11': 60, '21': 60, '22': 60, '23': 60, '42': 60, '51': 60, '52': 60,
'53': 60, '54': 60, '55': 60, '56': 60, '61': 60, '62': 60, '71': 60,
'72': 60, '81': 60, '90': 60, '99': 60, '31-33': 30, '44-45': 30,
'48-49': 30, '31': 30, '44': 30, '48': 30})
# Use '&' to require both conditions, it is a row-wise `and` of the separate tests.
>>> cap2013 = norm[(norm.Year == '2013') & norm.Region._contains('Capital Region')]
>>> len(cap2013)
21
>>> keepers = norm.Jobs._int() > 500000
>>> sum(keepers)
12
>>> large = norm[keepers]
>>> len(large)
12
>>> large[0]
{'sid': 121, 'id': '98A53A4E-712C-47A9-9106-C9062DB8CBBD', 'position': 121,
'created_at': 1437380961, 'created_meta': '707861', 'updated_at': 1437380961,
'updated_meta': '707861', 'meta': None, 'Year': '2012', 'Region': 'New York City',
'NAICS Code': '62', 'Industry': 'Health Care and Social Assistance', 'Jobs': '591686'}
>>> ~norm.Region._unique(None)
['Capital Region', 'Central New York', 'Finger Lakes', 'Long Island', 'Mid-Hudson',
'New York City', 'North Country', 'Southern Tier ', 'Western New York ', 'Mohawk Valley',
'Southern Tier', 'Western New York']
>>> Counter(norm.Region)
Counter({'Capital Region': 126, 'Central New York': 126, 'Finger Lakes': 126,
'Long Island': 126, 'Mid-Hudson': 126, 'New York City': 126, 'North Country': 126,
'Mohawk Valley': 126, 'Southern Tier ': 63, 'Western New York ': 63,
'Southern Tier': 63, 'Western New York': 63})
>>> sum(norm.Region._in(['Mohawk Valley', 'Southern Tier']))
189
SELECTOR Examples
-----------------
- The positional parameter **"b.b1"** would dererence a value like *OBJ.b.b1*, and the resulting key would be
the part after the last dot: **"b1"**.
- The named parameter **animal="b.b2"** would dererence like *OBJ.b.b2*, and the resulting key would be
**"animal"**.
.. code:: python
>>> dh = DataHammer([
{"a": 100, "b": {"b1": [101, 102, 103], "b2": "ape"}, "c": ["Apple", "Anise"]},
{"a": 200, "b": {"b1": [201, 202, 203], "b2": "bat"}, "c": ["Banana", "Basil"]},
{"a": 300, "b": {"b1": [301, 302, 303], "b2": "cat"}, "c": ["Cherry", "Cayenne"]}
])
>>> ~dh._pick('a', 'b.b1', animal='b.b2', food='c', nil='this.is.missing')
[{'a': 100, 'b1': [101, 102, 103], 'animal': 'ape', 'food': ['Apple', 'Anise'], 'nil': None},
{'a': 200, 'b1': [201, 202, 203], 'animal': 'bat', 'food': ['Banana', 'Basil'], 'nil': None},
{'a': 300, 'b1': [301, 302, 303], 'animal': 'cat', 'food': ['Cherry', 'Cayenne'], 'nil': None}]
#### Result is undefined due to the key collision.
>>> ~dh._pick('b.b1', b1='c')
## This '.0' syntax *might* change in future releases.
>>> ~dh._pick(animal='b.b2', fruit='c.0')
[{'animal': 'ape', 'fruit': 'Apple'},
{'animal': 'bat', 'fruit': 'Banana'},
{'animal': 'cat', 'fruit': 'Carmel'}]
Sample Data
-----------
Note that the data used here is randomly generated, no relationship to
anyone living, dead or undead is intended.
.. code:: python
>>> from datahammer import DataHammer
>>> mean = lambda nums: (sum(nums) * 1.0 / len(nums)) if nums else None
>>> data = [
{
"age":45,"gender":"F","location":{"city":"Baltimore","state":"Maryland"},
"name":{"first":"Addison","last":"Stewart"},"phone":"575-917-9109",
"ranks":[],"salary":10000,"title":"Systems Administrator"
},
{
"age":57,"gender":"F","location":{"city":"Baltimore","state":"Maryland"},
"name":{"first":"Katherine","last":"Perry"},"phone":"524-133-3495",
"ranks":[157,200,2,18,18,27],"salary":18590,"title":"Bookkeeper"
},
{
"age":33,"gender":"M","location":{"city":"Portland","state":"Oregon"},
"name":{"first":"Jack","last":"Young"},"phone":"803-435-5879",
"ranks":[9,157,197,155,190,56,58,97],"salary":28640,"title":"Controller"
},
{
"age":21,"gender":"F","location":{"city":"San Antonio","state":"Texas"},
"name":{"first":"Brianna","last":"Lewis"},"phone":"364-549-0753",
"ranks":[180,190,111],"salary":8000,"title":"UX Designer"
},
{
"age":24,"gender":"M","location":{"city":"Oklahoma City","state":"Oklahoma"},
"name":{"first":"Logan","last":"Ward"},"phone":"734-410-1116",
"ranks":[116,162],"salary":8000,"title":"Web Developer"
},
{
"age":60,"gender":"M","location":{"city":"Portland","state":"Oregon"},
"name":{"first":"Logan","last":"Martinez"},"phone":"652-193-9184",
"ranks":[70,16,59,24,32],"salary":33700,"title":"Assessor"
},
{
"age":63,"gender":"F","location":{"city":"Jersey City","state":"New Jersey"},
"name":{"first":"Grace","last":"Evans"},"phone":"955-466-6227",
"ranks":[123,126,118,64,110,28],"salary":26220,"title":"Mobile Developer"
},
{
"age":33,"gender":"F","location":{"city":"San Antonio","state":"Texas"},
"name":{"first":"Sophia","last":"Moore"},"phone":"636-269-3573",
"ranks":[97],"salary":14120,"title":"Mobile Developer"
}]
Foot Notes
==========
.. [1] Tokens
In these examples, *OBJ* refers to a *DataHammer* instance, *LIST* refers to the list of
contained items, and *ITEM* refers to an item in the contained list or directly in the *OBJ*.
.. [2] Dereferences
An attribute dereference (eg: *OBJ.index*) and the methods *OBJ._ind(index)* and *OBJ._get(index)* all
function identically, returning a new **DataHammer** instance. The latter are provided for use when
*index* is an *int* or otherwise not a valid string identifier.
.. [3] Scalars, Vectors and DataHammers
For most operations and functions that return a new instance, when a *DataHammer* instance is combined
with a list, tuple or other *DataHammer* instance, the length of the new instance will be limited by the
length of the shorter of the two operands. For example:
- Using a shorter operand, the result will be shortened as if the *DataHammer* instance had only that
many items.
- Using a longer operand, the result will be as if the *DataHammer* instance had only as many items as
that other operand.
.. code:: python
>>> dh1 = DataHammer(range(8))
>>> ~(dh1 + (10, 20))
[10, 21]
>>> dh2 = DataHammer((3, 1, 4))
>>> ~(dh1 == dh2)
[False, True, False]
>>> ~(dh1[dh2])
[3, 1, 4]
.. [4] Bracket Indexing
Because the **[]** syntax is used for `Indexing`_ and returns an ITEM or list, we cannot use this syntax
for chaining or to create another instance as we do for dotted-attribute access. This is why there is a
**_ind()** method, to allow
.. code:: python
>>> dh = DataHammer([[i, i*i] for i in range(10, 15)])
>>> ~dh
[[10, 100], [11, 121], [12, 144], [13, 169], [14, 196]]
>>> ~dh._ind(1)
[100, 121, 144, 169, 196]
>>> ~(dh._ind(1) > 125)
[False, False, True, True, True]
>>> ~dh[dh._ind(1) > 125]
[[12, 144], [13, 169], [14, 196]]
>>> dh = DataHammer([dict(a=i, b=tuple(range(i, i*2))) for i in range(6)])
# 'dh.b' returns a DataHammer of N-tuples, then '[3]' retrieves the 4th of these tuples as a `tuple`.
>>> dh.b[2]
(2, 3)
# Here 'dh.b' gives a DataHammer instance of N-tuples, but '_ind(2)' returns another DataHammer
# with the 3rd item from those N-tuples. Note the `None` for slots where the tuple length.
>>> dh.b._ind(2)
<datahammer.DataHammer object at 0x7f79eb1a9c10>
>>> ~dh.b._ind(2)
[None, None, None, 5, 6, 7]
.. [5] Slicing
This works similar to the *slice* method of the
`Javascript Array <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/slice>`_
class.
.. [6] In / Contains
Using "*ITEM in OBJ*" returns True if ITEM matches one of the items in OBJ, using the operator **==**
for the test. However, using *OBJ in OTHER* for an iterable containers *OTHER*, is useless.
useless.
Using "*OBJ in OTHER*" will evaluate the expression "**X == OBJ**" for each item X in OTHER,, resulting
in a list of bool. Unless either *OTHER* or *OBJ* are empty, this means a non-empty list will be
converted to **True** even if all of the comparisons fail.
.. [7] Mutator
*Mutator* operations dereference items based on the type of an item, regardless of the type of other items in
the contained data. Meaning: if a *DataHammer* with two items contains a `dict` with a key "foo" and an object
with an attribute "foo", then using **OBJ._mutator().foo** will update differently.
.. [8] *SELECTOR* Syntax.
The value of a *SELECTOR* must be a `str`, but depending on the method can be named or positional.
See `SELECTOR Examples`_.
1. For positional parameters, the text after the last dot, if any, is used for the resulting key.
2. For named parameters, the value will be used to fetch the value, and the parameter name will be used for
the key in the resulting item.
3. For both, a dot (`.`) indicates a sub-key, like normal dot notation and/or the *_ind()* method.
*Caveats*:
4. If there are multiple parameters that result in the same key, the result is undefined.
5. Currently, positional parameters are processed in order before the named parameters,
but that is not guaranteed to be true in future releases.
6. Currently, a bare int (in decimal form) is used to index into lists, but that syntax is not
guaranteed to be true in future releases. If a bare int is used as the last component of a
postitional parameter value, the resulting key will be a `str` - the decimal value.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/n2vram/datahammer",
"name": "datahammer",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "data,datasets,queries,JSON,resultsets,datahammer,hammer",
"author": "NVRAM",
"author_email": "nvram@users.sourceforge.net",
"download_url": "https://gitlab.com/n2vram/datahammer/archive/1.0.3",
"platform": "any",
"description": "datahammer\n##########\n\n`Version 1.0.3`\n\n\"When all you have is a hammer, everything looks like a nail.\" - *Anonymous*\n\n----------\n\n*Note that although the version is barely past 1.0, the code* is *production-ready.*\n*It has 100% unit test coverage, is extensively document and has considerable examples.*\n\n**TL;DR:** Too anxious to see just what *DataHammer* can do? Check out the `DataHammer Examples`_.\n\n----------\n\n.. contents:: **Index**\n :depth: 2\n :local:\n\n.. style table { border: 2px solid red; font-family: fujimoto; }\n\nSummary\n=======\n\nThis module provides an easy way to filter, inspect, analyze and manipulate many similar data items. It was\ndesigned to handle plain data types, especially the output from parsing JSON. It is designed to allow\noperations to be done a concise fashion, and on all items in a simple parallel manner.\n\nIt works on many data types, but excels at handling of *list*, *tuple* and *dict* values - and was designed\nprimarily for handling similar result objects from parsing JSON or XML.\n\nIn the design, concise usages was favored over speed of performance - although for data sets of less than a\nmillion records it is still quite performant. It was inspired by a need for a concise data manipulation\nsyntax and by the projects `jQuery <https://jquery.com/>`_ and `jq <https://stedolan.github.io/sjq/>`_,\nand as such it also has no dependencies beyond a standar Python instance.\n\n\nDetails\n=======\n\n- Most operations on a *DataHammer* instance return a value or a new instance, they do not mutate the\n contained data, although a returned ITEM could be mutated by the calling code.\n\n- The contained data can be retrieved with the invert operator (**~**). It will be a **list**\n unless constructed with a single ITEM, in which case that ITEM will be returned.\n\n- In order to allow accessing arbitrary ITEM attributes uses the dot notation, **public functions start\n with a single underscore**, in contrast to typical Python conventions. See `Functions`_.\n\n- It uses a **list** as its top-level container, and will convert a **tuple** and some generators into a\n **list**.\n\n- When constructed with a single ITEM, that item will be wrapped in a **list** and *most* operations will\n be identitical to having been constructed with a list with that single ITEM.\n\n- It uses '.' to access *dict* members or object attributes, using **None** for items where there is no key or\n attribute with the specified name, thus no *KeyError* or *AttributeError* will be raised.\n\n- Almost all operations will silently ignore items that do not have a member with the \"intended\" key, attribute\n or index.\n\n- There is a **Mutator** class returned by the **_mutator()** function that is designed to allow modifying the\n data in-place for some of the\n `Augmented Assignment statements <https://docs.python.org/3/reference/simple_stmts.html#grammar-token-augmented_assignment_stmt>`_.\n\n\nBuild Status\n------------\n\nCurrent status:\n\n.. image:: https://gitlab.com/n2vram/test-ci/badges/master/build.svg?rev=master\n :alt: Build Status\n :target: https://gitlab.com/n2vram/datahammer/pipelines\n\n\nKnown Issues\n------------\n\n- Using \"*ITEM in OBJ*\" works as you probably expect, but avoid using \"*OBJ in OTHER*\" for iterable\n containers. [6]_\n\n- By design and intent, the bitwise operators (`&`, `|`, `^`) actually create a new instance by applying\n the `and`, `or` and `xor` operators, respectively. This is because those keyword operators cannot be\n overridden to return an object as we wish.\n\n- There are missing operators that could be added. Among these are **del** (attribute or key),\n and the bitwise math operators.\n\n- The ``OBJ._long()`` method was removed in version 0.9.6, since it was complicating the code and testing,\n and is identical to using: ``OBJ._apply(long)`` Apologies for the breakage, if you hit that.\n\n- The ``OBJ._toCSV()`` method changed in version 0.9.7, since it wasn't using the builtin *csv* package,\n and was using a backslash to escape double-quotes within the quoted text, rather than a pair of\n consecutive double-quotes. For example, the previous output ``\"Derrial \\\"Preacher\\\" Book\"`` is now\n output as ``\"Derrial \"\"Preacher\"\" Book\"``.\n\n\nConstruction\n------------\n\nCreating a *DataHammer* can take several sources for its input. It is designed for use on a **list** of items\nwith the same schema.\n\n+--------------------+----------------------------------------------------------------+\n| **Parameters** | **Description** |\n+====================+================================================================+\n| ``data`` | This must be one of: |\n| | |\n| | * A `list` of ITEMS. |\n| | * A single, non-`list` ITEM. |\n| | |\n| | If the **json** value is true, then `data` can be either of: |\n| | |\n| | * A `file` object, from which *all* data is read, and the |\n| | results are treated as TEXT, or... |\n| | * TEXT to be parsed as JSON. |\n+--------------------+----------------------------------------------------------------+\n| ``copy`` | If given and true, then a `deepcopy` will be made of `data`. |\n+--------------------+----------------------------------------------------------------+\n| ``json`` | If provided, it should either be `True` or a dict of arguments |\n| | to be passed to *JSON.loads()* for when `data` is of either |\n| | the `file` or `TEXT` forms. |\n+--------------------+----------------------------------------------------------------+\n\n\nOperations\n^^^^^^^^^^\n\nThis is a list of supported operations, including applying builtin Python functions. [1]_\n\n+------------------------------------------+---------------------------------------------------------------+\n| **Operation** | **Description** |\n+==========================================+===============================================================+\n| ``~OBJ`` | Returns the contained data. |\n+------------------------------------------+---------------------------------------------------------------+\n| | ``OBJ.index`` | Creates a list by applying the *index* (an *int* for *list* |\n| | ``OBJ._ind(index)`` | items, a key for *dict* items, or the name of an *attribute* |\n| | ``OBJ._get(index)`` | or *property*), returning a *DataHammer* instance created |\n| | using that list. [2]_ |\n+------------------------------------------+---------------------------------------------------------------+\n| | ``OBJ`` *op* ``OTHER`` | Return a *DataHammer* instance with a bool result from the |\n| | *op* can be: ``< <= == != >= >`` | comparison of each ITEM with OTHER. [3]_ |\n| | |\n| | To test equality of contents, use: *~OBJ == OTHER* |\n+------------------------------------------+---------------------------------------------------------------+\n| | ``OBJ`` *bitop* ``OTHER`` | Return a *DataHammer* instance with the results of applying |\n| | ``OTHER`` *bitop* ``OBJ`` | `and`, `or` and a \"bool-xor\" to each *ITEM* and *OTHER*, or |\n| | *bitop* can be: ``& ^ |`` | (*OTHER* and *ITEM*). These are needed since those keywords |\n| | cannot be overridden in the desired fashion. [4]_ |\n+------------------------------------------+---------------------------------------------------------------+\n| | ``OBJ`` *mathop* ``OTHER`` | Return a *DataHammer* instance with the results of applying |\n| | *mathop* can be: ``+ - * / // ** %`` | a math operators in: *ITEM mathop OTHER*. [3]_ |\n+------------------------------------------+---------------------------------------------------------------+\n| | ``OTHER`` *mathop* ``OBJ`` | Return a *DataHammer* instance with the results of applying |\n| | *mathop* can be: ``+ - * / // ** %`` | a math operators in: *OTHER mathop ITEM*. [3]_ |\n+------------------------------------------+---------------------------------------------------------------+\n| ``OBJ[indexes]`` | Depending on the argument, returns a *DataHammer* instance, a |\n| | single contained ITEM, or a list of ITEMs. [4]_ |\n| | See `Indexing`_, for more information. |\n+------------------------------------------+---------------------------------------------------------------+\n| | ``OBJ._bool()`` | Return a *DataHammer* instance with the results of applying |\n| | ``OBJ._int()`` | the builtin type (*of the same name w/o the underscore*) to |\n| | ``OBJ._float()`` | each item in the list. |\n+------------------------------------------+---------------------------------------------------------------+\n| ``reversed(OBJ)`` | Return a *DataHammer* instance with the contained data in |\n| | reversed order. |\n+------------------------------------------+---------------------------------------------------------------+\n| ``len(OBJ)`` | Return an *int* for the number of contained data ITEMs. |\n+------------------------------------------+---------------------------------------------------------------+\n| ``hash(OBJ)`` | Return an *int* that is the hash of the tuple of the hash of |\n| | every ITEM. |\n| | This will raise an exception if *any* ITEM cannot be hashed. |\n+------------------------------------------+---------------------------------------------------------------+\n| ``ARG in OBJ`` | Return a bool, which is `True` if any *ITEM == OBJ*. |\n| | With regard to limiting the items tested. [3]_ |\n+------------------------------------------+---------------------------------------------------------------+\n| ``OBJ in ARG`` | *This is almost never what you want!* Return a single bool, |\n| | ignoring of contents of ARG or OBJ. The result is `True` if |\n| | neither ARG nor OBJ are empty, and `False` if they both are. |\n+------------------------------------------+---------------------------------------------------------------+\n| ``-OBJ`` *(unary minus)* | Return a *DataHammer* instance with the results of applying |\n| | *not ITEM* on each item. |\n+------------------------------------------+---------------------------------------------------------------+\n\nNOTE: The ``OBJ._long()`` method was removed in version 0.9.6, since it was complicating the code and testing,\nand is identical to using: ``OBJ._apply(long)``\n\nFunctions\n---------\n\nThis is a list of supported functions. [1]_\n\n+-------------------------------------------+---------------------------------------------------------------+\n| **Function** | **Description** |\n+===========================================+===============================================================+\n| | ``OBJ._ind(name)`` | Attribute, index or *dict* key dereference. [2]_ |\n| | ``OBJ._get(name)`` | |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``str(OBJ)`` | Returns a JSON dump of the contained data. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._contains(ARG)`` | Return a *DataHammer* instance with the results of applying |\n| | *ARG in ITEM* for each item. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._in(ARG)`` | Return a *DataHammer* instance with the results of applying |\n| | *ITEM in ARG* for each item. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._apply(FUNC, ARG, *ARGS, **KWDS)`` | Return a *DataHammer* instance with the results of applying |\n| | ``FUNC(ITEM, ARG, *ARGS, **KWDS)`` to each item. [3]_ |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._strip(ARG)`` | Return a *DataHammer* instance with only the desired items. |\n| | Based on the type of ARG given, the new instance has only the |\n| | items for which the result is true of: |\n| | 1. If ARG is not given: *bool(ITEM)* |\n| | 2. If ARG is a callable: *ARG(ITEM)* |\n| | 3. If ARG is a list, tuple or set: *(ITEM in ARG)* |\n| | 4. Otherwise: *ITEM == ARG* |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._insert(INDEX, ITEM)`` | Return a *DataHammer* instance with ITEM inserted at INDEX. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._extend(INDEX, ITEMS)`` | Return a *DataHammer* instance with ITEMS added at the end. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._splice(INDEX, DELNUM, *ITEM)`` | Return a *DataHammer* instance with DELNUM items deleted at |\n| | INDEX, and with ITEM(s) inserted there. [5]_ |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._slice(START [, END [, STEP ] ])`` | Return a *DataHammer* instance with the list sliced according |\n| | to the given indices (like *list* slicing works). |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._flatten()`` | Return a *DataHammer* instance with contained items that are |\n| | the result of flattening *this* instance's contained items by |\n| | one level. Sub-items are added in iteration-order for items |\n| | that are a *set*, *list* or *tuple* and for values from a |\n| | *dict*. |\n| | |\n| | Other types are not flattened, and are added as-is. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._tuple(SELECTOR, SELECTOR, ...)`` | Return a tuple of results for each contained item, the result |\n| | will be a tuple of values from the items, dereferenced by the |\n| | *SELECTOR* parameters, in the same order. See [8]_ |\n| | |\n| | Only named *SELECTOR* parameters are allowed. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._toCSV(SELECTOR, SELECTOR, ...)`` | Return a tuple of `str` like a `Comma Separated Values` file, |\n| | the first `str` represents the headers for each column, and |\n| | each subsequent contains a CSV-style representation of the |\n| | requested values from each item (which must be serializable). |\n| | See [8]_ |\n| | |\n| | Both positional and named *SELECTOR* parameters are allowed. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._pick(SELECTOR, SELECTOR, ...)`` | Return a *DataHammer* instance of *dict* items made from one |\n| | or more sub-items specified by the *SELECTOR*, as either |\n| | positional or named parameters. |\n| | Parameters dictate the keys in the resulting items. See [8]_ |\n| | |\n| | Both positional and named *SELECTOR* parameters are allowed. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._groupby(GRP, VALS [, POST])`` | Return a *DataHammer* instance of *dict* items made by taking |\n| | all sub-items specified by `VALS` and combine them with other |\n| | items with the same `GRP` values. It is similar to the `SQL` |\n| | **GROUP BY** clause. See [8]_ and `Grouping`_. |\n| | |\n| | Both positional and named *SELECTOR* parameters are allowed. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._join(KEYS, OBJ [,FLAGS][,MERGE])`` | Return a *DataHammer* instance of *dict* items from merging |\n| | items from this instance and **OBJ**, joining on the values |\n| | corresponding to the `KEYS`. The `FLAGS` parameter controls |\n| | specifics. Somewhat similar to the `SQL` **JOIN** operations. |\n| | See `Joining`_ and the `Deeper Examples`_. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._unique(KEYS [,UNIQUE])`` | Return a *DataHammer* instance with items from this instance, |\n| | based on the uniqueness of the values for `KEYS`. The |\n| | `UNIQUE` parameter sets handling for items with duplicate key |\n| | values. |\n| | |\n| | See `Unique`_ and the `Deeper Examples`_. |\n+-------------------------------------------+---------------------------------------------------------------+\n| ``OBJ._mutator()`` | Returns a *DataHammer.Mutator* instance to be used for making |\n| | modifications to the contained data. See `Mutators`_. |\n+-------------------------------------------+---------------------------------------------------------------+\n\n\nIndexing\n--------\n\nIndexing a *DataHammer* instance with *[]* allows simple access to items from the contained data, but\nthere are various types of parameters types allowed. [4]_\n\n1. Indexing with an **int** or an implicit or explicit **slice** object works like indexing **list**; the\n result is identical to **(~OBJ)[...]**.\n\n * A single item is returned with an **int** argument, and can raise an IndexError.\n * A (possibly empty) list of items is returned with either:\n\n * An explicit **slice** argument, eg: OBJ[slice(1, None, 5)]\n * An implicit **slice** argument, eg: OBJ[1::5]\n\n2. Indexing with a **list**, **tuple** or a *DataHammer* instance, will return another *DataHammer*\n instance. [3]_ The parameter must either be all **bool** or all **int**, and they\n dictate *which* items are used to construct the new instance:\n\n * For **bool** indexes, each bool in the argument indicates if the corresponding item in the\n *DataHammer* is included in the new instance.\n\n * For **int** indexes, each int is used to index into the contained data, and which item is include\n in the new instance. This allows both filtering and reordering of data.\n\nIndexing Examples:\n\n .. code:: python\n\n >>> OBJ = DataHammer(list(range(10, 15)))\n\n # Note that the following dereference the instance with \"~\" to show the contents:\n\n >>> ~OBJ\n [10, 11, 12, 13, 14]\n >>> ~OBJ[(True, False, True, True, False, True)]\n [10, 12, 13] # The last/6th `True` is ignored since len(OBJ)==5\n >>> ~OBJ[(4, 2, 1, 40, -1, 3, 1)]\n [14, 12, 11, 14, 13, 11] # 40 is ignored.\n\n # Note these DO NOT dereference the result, they are not a DataHammer instance.\n\n >>> type(OBJ[1])\n <type 'int'>\n >>> type(OBJ[:5])\n <type 'list'>\n >>> type(OBJ[slice(3)])\n <type 'list'>\n >>> OBJ[::3]\n [10, 13]\n\n\nGrouping\n--------\n\nThe *_groupby(GROUP, VALUES [, POSTPROC])* method creates a new *DataHammer* instance, grouping values from\nmultiple source items. It functions somewhat like the **GROUP BY** feature of SQL, however rather than\nnecessarily combining column values, a the list of values is created.\n\nThe `GROUP` and `VALUES` parameters should be either a list/tuple or a dict.\n\n- Strings in the list/tuple are treated like named `SELECTOR` parameters\n- Items in a dict are treated like named `SELECTOR` parameters.\n\nFor each unique sets of values for the `GROUP` keys, one item will exist in the resulting instance. Each of\nthe new items will contain the grouping values and a value per `VALUES` key. The `GROUP` and `VALUES`\nparameters may be either a list/tuple or a dict of `SELECTOR` parameters (see above).\n\nFor every key in the `VALUES` parameter, a list is built with the corresponding values, one list for each\nset of `GROUP` values.\n\nThe `POSTPROC` parameter parameter, is optional and unless provided: each resulting item will contain the\ncorresponding list for each key in `VALUES`. If `FUNC` is provided, it will be called once per resulting\nitem. The lists are passed parameters in the same order as the keys in `VALUES`.\n\nNote that the order of the resulting items will be the same as the order of the first occurence of that set\nof `GROUP` keys in the source items. And the order of the list of values for each `VALUES` key is the same\nas the order that those occurred in the source items.\n\n\nJoining\n-------\n\nThere is a method for joining two *DataHammer* instances, combining items for which the specified key values match,\nthis is partly inspired by the **JOIN** feature of SQL (`JOIN_PRODUCT`), and partly inspired by a use case where\none-to-one matches were needed.\n\nWith the two (2) flags [`JOIN_PRODUCT` or `JOIN_ORDERED`] for handling duplicates and the four (4) flags\n[`JOIN_KEEP_NEITHER`, `JOIN_KEEP_LEFT`, `JOIN_KEEP_RIGHT` or `JOIN_KEEP_BOTH`] for handling unmatched items, there\nare eight (8) different `flags` combinations.\n\n**HANDLING OF ITEMS WITH DUPLICATE KEY VALUES**\n\nHere, \"duplicate\" key-values means that a set of key-values occurs more than once in the same instance.\n\n+-------------------+--------------------------------------------------------------------------------------+\n| \"Mode\" Flag Name | Description |\n+-------------------+--------------------------------------------------------------------------------------+\n| `JOIN_PRODUCT` | Results are somewhat similar to SQL joins. The name comes from the \"Cartesian |\n| | Product\" since the output contains an item produced from the each matching item the |\n| | left input and the right input. |\n+-------------------+--------------------------------------------------------------------------------------+\n| `JOIN_ORDERED` | This pairs matching items from the left and the right, one-by-one. Pairing is in |\n| | the same order as they were found in the input instances, and matching stops after |\n| | exhausting the matching items in either the right input or left input. |\n+-------------------+--------------------------------------------------------------------------------------+\n\nIf there are no duplicate entries in either input, then these modes function identically.\n\n\n**HANDING OF UNMATCHED ITEMS**\n\nAn \"unmatched\" item is one whose key-values never occur in the items from the other instance.\n\nHere, the INNER and OUTER join terminology is a remnant from SQL, the \"KEEP\" flags are equivalent and provided\nsince they describe the intended action. These can be summarized thus:\n\n+----------------------+--------------------+-----------------------------------------------------------------+\n| \"Keep\" Flag Name | Inner/Outer Name | Deescription |\n+----------------------+--------------------+-----------------------------------------------------------------+\n| `JOIN_KEEP_NEITHER` | `INNER_JOIN` | Discard unmatched items from left and from the right. |\n+----------------------+--------------------+-----------------------------------------------------------------+\n| `JOIN_KEEP_LEFT` | `LEFT_OUTER_JOIN` | Discard unmatched items from the right. |\n+----------------------+--------------------+-----------------------------------------------------------------+\n| `JOIN_KEEP_RIGHT` | `RIGHT_OUTER_JOIN` | Discard unmatched items from the left. |\n+----------------------+--------------------+-----------------------------------------------------------------+\n| `JOIN_KEEP_BOTH` | `FULL_OUTER_JOIN` | Keep unmatched items from the left and from the right. |\n+----------------------+--------------------+-----------------------------------------------------------------+\n\n**OUTPUT ORDER**\n\nThe order of items in the inputs dictates the order in the output. The algorithm simply iterates over the left\ninput, producing zero or more outputs depending on the flags and presence of any matching items in the right input.\nIt then appends unmatched items from the right, if desired.\n\nSee the examples, or use it for yourself, if this is not sufficiently clear.\n\nNotes:\n\n - With `JOIN_PRODUCT`, each matched item from the left will be paired with every matching item from the right, in\n the order that the right items occurred.\n\n - With `JOIN_ORDERED`, each item in the left will be paired with the corresponding order of the matching items in\n the right input. After the items from the right are exhausted, the remaining items from the left input with that\n set of key-values are considered unmatched. In addition, any items from the right input that are not consumed in\n this way are also considered unmatched.\n\n - With `JOIN_KEEP_LEFT` or `JOIN_KEEP_BOTH`, unmatched items from the left input will appear in the same order as\n they are found in the left input.\n\n - With `JOIN_KEEP_RIGHT` or `JOIN_KEEP_BOTH`, unmatched items from the right input will appear after all items\n produced from items in the left input. They will be in the same order as they occurred the right input.\n\n\n\nUnique\n------------\n\nThe **_unique()** method allows eliminating items based on the uniqueness / duplication of key values.\n\nParameters are:\n\n- `KEYS` should be a list/tuple of strings which are used as a `SELECTOR` into each item. The associated\n values are used for the uniqueness test. (If `KEYS` is a single string, it is handled as expected.)\n\n There is a special case when `KEYS` is **None**: if so, the hash of the item is used in lieu of key values.\n Obviously, all items must be hashable.\n\n- `UNIQUE` determines which items to keep, based on key values. `UNIQUE` may be:\n\n - 0 = Keep only those items that are unique, with no duplicates.\n - 1 = Keep the first item with key values, ignore subsequent duplicates.\n - 2 = Keep all instances of items that have duplicate key values.\n \nIn each case, the order of the items is preserved from the original data.\n\nNote that with **unique = 2**, there will be multiple items that have the same key values; to remove those you filter\nthem a second time with the same `KEYS`:\n\n **OBJ._unique(KEYS, 2)._unique(KEYS)**\n\n\nMutators\n--------\n\nThere is some support for making modifications to the data contained within a *DataHammer*, beyond\ndirect access. This is done with the *DataHammer._mutator* method on the instance.\n\nHere **MUT** is used as a shorthand for **OBJ._mutator()** - which returns a *DataHammer.Mutator*\ninstance, and the name *Mutator* is also used for *DataHammer.Mutator*.\n\n\n+-----------------------------------------+----------------------------------------------------------------+\n| **Functions and Operation** | **Description** |\n+=========================================+================================================================+\n| ``MUT = OBJ._mutator()`` | Returns a new *Mutator* for the given *DataHammer* instance. |\n+-----------------------------------------+----------------------------------------------------------------+\n| ``~MUT`` | Returns the *DataHammer* instance for this *Mutator*. |\n+-----------------------------------------+----------------------------------------------------------------+\n| | ``MUT.index`` | Returns a new *Mutator* instance useful for modifying the |\n| | ``MUT[index]`` | key, attribute or list item at *index*. [7]_ |\n| | ``MUT._get(index)`` | |\n| | ``MUT._ind(index)`` | Note that *all of these forms work identically*, though the |\n| | first form can only be used with valid identifier names. This |\n| | is in contrast with **[]** on a *DataHammer* instance where |\n| | it returns an item from the contained data. |\n+-----------------------------------------+----------------------------------------------------------------+\n| | ``MUT`` *op* ``OTHER`` | Update the item member for the given *Mutator* instance, with |\n| | *op* can be: ``+= -= *= /= **= //=`` | the given operation, which should be number (or object that |\n| | supports that operation). |\n+-----------------------------------------+----------------------------------------------------------------+\n| ``MUT._set(OTHER)`` | Update the value designated by the given *Mutator* instance, |\n| | overwriting with the given value(s). If *OTHER* is a list, |\n| | tuple or *DataHammer* instance, then an interator is used, |\n| | and application stops when the end is reached. [3]_ |\n+-----------------------------------------+----------------------------------------------------------------+\n| ``MUT._setall(OTHER)`` | Like ``MUT._set(OTHER)`` but regardless of the type, *OTHER* |\n| | is used without iterating. Used to set all rows to the same |\n| | *list* or *tuple* value, but can be used with any value/type. |\n+-----------------------------------------+----------------------------------------------------------------+\n| ``MUT._apply(FUNC, *ARGS, **KWDS)`` | Update the value designated by the given *Mutator* instance, |\n| | overwriting with the the *return value* from calling: |\n| | **``FUNC(VALUE, *ARGS, **KWDS)``**. |\n+-----------------------------------------+----------------------------------------------------------------+\n\nInstallation\n============\n\nInstall the package using **pip**, eg:\n\n `pip install --user datahammer`\n\nOr for a specific version of Python:\n\n `python3 -m pip --user install datahammer`\n\n\nTo the source git repository, use:\n\n `git clone https://gitlab.com/n2vram/datahammer.git`\n\n\n\nReleases\n--------\n\n +-------------+--------------------------------------------------------+\n | **Version** | **Description** |\n +=============+========================================================+\n | 0.9 | Initial release, documentation prototyping. |\n +-------------+--------------------------------------------------------+\n | 0.9.1 | Addition of \"_pick\" method. |\n +-------------+--------------------------------------------------------+\n | 0.9.2 | Addition of \"_flatten\" and \"_toCSV\" methods. |\n +-------------+--------------------------------------------------------+\n | 0.9.4 | Addition of \"_groupby\" and \"_tuples\" methods. |\n +-------------+--------------------------------------------------------+\n | 0.9.5 | Moved EXAMPLES into (and reorganized) the README file. |\n | | Configured for tests, coverage and style on Travis CI. |\n +-------------+--------------------------------------------------------+\n | 0.9.6 | Removed 'OBJ._long()' method, as it was Python2-only. |\n +-------------+--------------------------------------------------------+\n | 0.9.7 | Added the 'OBJ._join()' and 'OBJ._fromCSV()' methods. |\n +-------------+--------------------------------------------------------+\n | 0.9.8 | Added the 'OBJ._unique()' and 'OBJ._in()' methods. |\n +-------------+--------------------------------------------------------+\n | 1.0 | Moved to gitlab.com, including GitLab-CI. |\n +-------------+--------------------------------------------------------+\n | 1.0.1 | Minor changes post move to GitLab. |\n +-------------+--------------------------------------------------------+\n | 1.0.2 | Final changes before pushing to PyPi. |\n +-------------+--------------------------------------------------------+\n | 1.0.3 | Documentation, whitespace only changes. |\n +-------------+--------------------------------------------------------+\n\n\nReporting Issues, Contributing\n------------------------------\n\nAs an open source project, *DataHammer* welcomes contributions and feedback.\n\n1. Report any issues, including with the functionality or with the documentation\n via the GitLab project: https://gitlab.com/n2vram/datahammer/issues\n\n2. To contribute to the source code, please use a GitLab pull request for the\n project, making sure to include full/extensive unit tests for any changes. Note\n that if you cannot create a PR, then open an issue and attach a `diff` output\n there. https://gitlab.com/n2vram/datahammer/\n\n3. To translate the documentation, please follow the same process as for source\n code contributions.\n\n\nDataHammer Examples\n===================\n\nIt is probably easier to show the utility of *DataHammer* with some examples.\n\n\nSimple Examples\n---------------\n\n\n1. To construct a *DataHammer* instance you generally a list/tuple/iterable of items. Many builtin functions operate\n on the *DataHammer* instance as it would on the list of objects. The original data can be returned using the tilde\n operator (`~`).\n\n See `Sample Data`_ for the **data** used here.\n\n.. code:: python\n \n >>> dh = DataHammer(data)\n >>> len(dh)\n 8\n >>> dh\n <datahammer.DataHammer object at 0x7f258fac34e0>\n >>> type(~dh)\n <type 'list'>\n >>> type(dh[0])\n <type 'dict'>\n >>> type(dh[:3])\n <type 'list'>\n >>> ~dh == dh[:]\n True\n >>> bool(dh)\n True\n\n\n2. Accessing the sub-items uses a simple dot notation. To allow irregular data, a `None` will represent a\n member that was not present -- no `KeyError`, `AttributeError` or `IndexError` are raised.\n\n.. code:: python\n \n >>> ~dh.age\n [45, 57, 33, 21, 24, 60, 63, 33]\n >>> ~dh.name.last\n ['Stewart', 'Perry', 'Young', 'Lewis', 'Ward', 'Martinez', 'Evans', 'Moore']\n # No KeyError\n >>> ~dh.missingMember\n [None, None, None, None, None, None, None, None]\n\n\n3. Indexing into a list sub-item cannot be done with dot notation or slicing (eg: with `[]`), so the\n *_ind()* method is provided for this reason. As for dot notation, if an index is out of range then the\n value will be `None`.\n\n.. code:: python\n \n # This is not a DataHammer instance, it is just the `rank` member of the fourth item.\n >>> dh.ranks[3]\n [180, 190, 111]\n\n # This is a DataHammer instance with the fourth item from each `rank` member, or `None`.\n >>> ~dh.ranks._ind(3)\n [None, 18, 155, None, None, 24, 64, None]\n\n\n4. To avoid collisions with item members, the public methods of a *DataHammer* instance are all prefixed\n with a single underscore, which may be confusing at first, but this is also done for\n `collections.namedtuple` instances. Methods that begin with a double underscore are not public.\n\n.. code:: python\n \n # This 'mean' function is defined in the Sample Data section, below.\n >>> ~dh.ranks._apply(mean)\n [None, 70.33333333333333, 114.875, 160.33333333333334, 139.0, 40.2, 94.83333333333333, 97.0]\n\n >>> ~dh._splice(2, 4).name.first\n ['Addison', 'Katherine', 'Grace', 'Sophia']\n\n >>> print(\"\\n\".join(dh._toCSV(FIRST='name.first', LAST='name.last', AGE='age')))\n \"FIRST\",\"LAST\",\"AGE\"\n \"Addison\",\"Stewart\",45\n \"Katherine\",\"Perry\",57\n \"Jack\",\"Young\",33\n \"Brianna\",\"Lewis\",21\n \"Logan\",\"Ward\",24\n \"Logan\",\"Martinez\",60\n \"Grace\",\"Evans\",63\n \"Sophia\",\"Moore\",33\n\n\n5. Many operators are overridden to allow operating on the item with a simple syntax, returning a new *DataHammer*\n instance with the results. Most operators work with another *DataHammer* instance, a list/tuple or scalar values.\n In the case of a list/tuple, the length of the resulting instance will be the shorter of the two arguments.\n\n.. code:: python\n\n\n >>> ~(dh.gender == 'F')\n [True, True, False, True, False, False, True, True]\n >>> ~(dh.salary / 1000.0)\n [10.0, 18.59, 28.64, 8.0, 8.0, 33.7, 26.22, 14.12]\n >>> ~(dh.age > [50, 40, 30])\n [False, True, True]\n >>> ~(dh.salary * 1.0 / dh.age) # Avoid integer math.\n [222.22222222222223, 326.140350877193, 867.8787878787879, 380.95238095238096,\n 333.3333333333333, 561.6666666666666, 416.1904761904762, 427.8787878787879]\n\n\n6. Using many builtin operations work as you would expect, as if passing a list/tuple of the item data instead.\n\n.. code:: python\n\n >>> min(dh.age), max(dh.age)\n (21, 63)\n >>> sorted(dh.location.state)\n ['Maryland', 'Maryland', 'New Jersey', 'Oklahoma', 'Oregon', 'Oregon', 'Texas', 'Texas']\n >>> sum(dh.salary)\n 147270\n >>> min(dh.salary), mean(dh.salary), max(dh.salary)\n (8000, 18408.75, 33700)\n\n # This gives number of females, by counting occurences of `True`.\n >>> sum(dh.gender == 'F')\n 5\n\n\n7. Indexing with another *DataHammer* instance is another powerful feature. Also, indexing with integers allows\n arbitrary keeping a subset of, or reordering of, the items.\n \n.. code:: python\n\n >>> len(dh.age < 30), sum(dh.age < 30)\n (8, 2)\n >>> twenties = (20 <= dh.age < 30)\n >>> ~twenties\n [False, False, False, True, True, False, False, False]\n >>> ~dh[twenties].name\n [{'first': 'Brianna', 'last': 'Lewis'}, {'first': 'Logan', 'last': 'Ward'}]\n >>> ~dh.name.last\n ['Stewart', 'Perry', 'Young', 'Lewis', 'Ward', 'Martinez', 'Evans', 'Moore']\n >>> ~dh[(0, 5, 3, 4)].name.last\n ['Stewart', 'Martinez', 'Lewis', 'Ward']\n \n\nDeeper Examples\n---------------\n\nThese demonstrate the extracting and manipulating power of *DataHammer* instances. Note that these examples and notes\nare not trivial, so please read carefully so you can understand the functionality as it is designed.\n\n\n8. There are methods for extracting parts of each item, including *_pick()*, *_tuples()* and *_toCSV()*. In addition\n the *_groupby()* method allows extracting only certain parts `and` combining them across the items that share\n certain values, similar to the **GROUP BY** syntax in SQL. \n\n See the main README section for detailed *SELECTOR Syntax*, but the methods are demonstrated here:\n\n\n a. The *_tuples(SELECTOR [, SELECTOR ...])* method returns a tuple of tuples with extracted values in the same order\n as the names. Only positional `SELECTOR` parameters are allowed.\n\n .. code:: python\n\n >>> dh._tuples('location.city', 'name.last', 'age')\n (('Baltimore', 'Stewart', 45),\n ('Baltimore', 'Perry', 57),\n ('Portland', 'Young', 33),\n ('San Antonio', 'Lewis', 21),\n ('Oklahoma ', 'Ward', 24),\n ('Portland', 'Martinez', 60),\n ('Jersey City', 'Evans', 63),\n ('San Antonio', 'Moore', 33))\n\n\n b. The *_toCSV(SELECTOR [, SELECTOR ...])* method returns a tuple of strings in a `Comma Separated Values`\n format. The first string is a header of the column names in order. Each subsequent string represents the\n corresponding item in the data, in order. Both positional and named `SELECTOR` parameters are allowed.\n\n .. code:: python\n\n >>> dh._toCSV('location.city', lname='name.last', yrs='age')\n ('\"city\",\"lname\",\"yrs\"',\n '\"Baltimore\",\"Stewart\",45',\n '\"Baltimore\",\"Perry\",57',\n '\"Portland\",\"Young\",33',\n '\"San Antonio\",\"Lewis\",21',\n '\"Oklahoma \",\"Ward\",24',\n '\"Portland\",\"Martinez\",60',\n '\"Jersey City\",\"Evans\",63',\n '\"San Antonio\",\"Moore\",33')\n\n\n c. The *_pick(SELECTOR [, SELECTOR ...])* method returns a new *DataHammer* instance where each item is a dictionary\n with only the requested members. Positional and named `SELECTOR` parameters are allowed.\n\n .. code:: python\n\n >>> ~dh._pick('location.state', ln='name.last', fn='name.first', years='age')\n [{'state': 'Maryland', 'ln': 'Stewart', 'fn': 'Addison', 'years': 45},\n {'state': 'Maryland', 'ln': 'Perry', 'fn': 'Katherine', 'years': 57},\n {'state': 'Oregon', 'ln': 'Young', 'fn': 'Jack', 'years': 33},\n {'state': 'Texas', 'ln': 'Lewis', 'fn': 'Brianna', 'years': 21},\n {'state': 'Oklahoma', 'ln': 'Ward', 'fn': 'Logan', 'years': 24},\n {'state': 'Oregon', 'ln': 'Martinez', 'fn': 'Logan', 'years': 60},\n {'state': 'New Jersey', 'ln': 'Evans', 'fn': 'Grace', 'years': 63},\n {'state': 'Texas', 'ln': 'Moore', 'fn': 'Sophia', 'years': 33}]\n\n\n d. The *_groupby(GROUP, VALUES [, POSTPROC])* method returns a new *DataHammer* instance, using the first list of\n keys for grouping by value, and the second list as the values to groupby. Like the **GROUP BY** functionality\n in SQL, there will be one item in the resulting instance for each unique set of values of the `GROUP` keys.\n\n Remember: even if passing a single key for `GROUP` or `VALUES`, it must be in a tuple or list.\n\n .. code:: python\n\n # An empty second parameter is allowed, too, the results is just the unique GROUP keys.\n >>> ~dh._groupby(['gender', 'title'], [])\n [{'gender': 'F', 'title': 'Systems Administrator'},\n {'gender': 'F', 'title': 'Bookkeeper'},\n {'gender': 'M', 'title': 'Controller'},\n {'gender': 'F', 'title': 'UX Designer'},\n {'gender': 'M', 'title': 'Web Developer'},\n {'gender': 'M', 'title': 'Assessor'},\n {'gender': 'F', 'title': 'Mobile Developer'}]\n\n >>> ~dh._groupby(['gender'], ('age', 'salary'))\n [{'gender': 'F', 'age': [45, 57, 21, 63, 33], 'salary': [10000, 18590, 8000, 26220, 14120]},\n {'gender': 'M', 'age': [33, 24, 60], 'salary': [28640, 8000, 33700]}]\n \n\n The third parameter is a callable that takes the constructed lists in `VALUES` key order, and\n returns a tuple with same number of items, in the same order.\n\n .. code:: python\n\n >>> def reductor(ages, salaries):\n ... return (min(ages), max(ages)), (min(salaries), max(salaries))\n\n >>> ~dh._groupby(['gender'], ('age', 'salary'), reductor)\n [{'gender': 'F', 'age': (21, 63), 'salary': (8000, 26220)},\n {'gender': 'M', 'age': (24, 60), 'salary': (8000, 33700)}]\n\n\n9. There is a method for joining two *DataHammer* instances, combining items for which the specified\n key values match. The `JOIN_PRODUCT` mode is inspired by the **JOIN** feature of SQL, whiel\n `JOIN_ORDERED` was inspired by a use case where one-to-one matches were needed.\n\n .. code:: python\n\n >>> left = DataHammer([{\"k\": \"A\", \"x\": 1}, {\"k\": \"B\", \"x\": 2}, {\"k\": \"C\", \"x\": 3},\n ... {\"k\": \"C\", \"x\": 4}, {\"k\": \"D\", \"x\": 5}])\n >>> right = DataHammer([{\"k\": \"A\", \"y\": 1}, {\"k\": \"A\", \"y\": 2}, {\"k\": \"C\", \"y\": 3},\n ... {\"k\": \"C\", \"y\": 4}, {\"k\": \"E\", \"y\": 5}])\n\n # For JOIN_PRODUCT, each matched item from the left is paired with each the corresponding item\n # from the right. Then the JOIN_KEEP_{...} flag determines unmatched item retention.\n\n # Default is ORDERED + NEITHER\n >>> ~left._join(\"k\", right)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'A', 'x': 1, 'y': 2},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 3, 'y': 4},\n {'k': 'C', 'x': 4, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4}]\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_NEITHER)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'A', 'x': 1, 'y': 2},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 3, 'y': 4},\n {'k': 'C', 'x': 4, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4}]\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_RIGHT)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'A', 'x': 1, 'y': 2},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 3, 'y': 4},\n {'k': 'C', 'x': 4, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4},\n {'k': 'E', 'y': 5}]\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_LEFT)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'A', 'x': 1, 'y': 2},\n {'k': 'B', 'x': 2},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 3, 'y': 4},\n {'k': 'C', 'x': 4, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4},\n {'k': 'D', 'x': 5}]\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_PRODUCT + DataHammer.JOIN_KEEP_BOTH)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'A', 'x': 1, 'y': 2},\n {'k': 'B', 'x': 2},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 3, 'y': 4},\n {'k': 'C', 'x': 4, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4},\n {'k': 'D', 'x': 5},\n {'k': 'E', 'y': 5}]\n\n # For JOIN_ORDERED, matched items from the left and right are paired, one-by-one, but only as\n # until either side is exhausted, the remaining items are 'unmatched' and the JOIN_KEEP_{...}\n # flag determines unmatched item retention.\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_NEITHER)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4}]\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_RIGHT)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4},\n {'k': 'A', 'y': 2},\n {'k': 'E', 'y': 5}]\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_LEFT)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'B', 'x': 2},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4},\n {'k': 'D', 'x': 5}]\n\n >>> ~left._join(\"k\", right, flags=DataHammer.JOIN_ORDERED + DataHammer.JOIN_KEEP_BOTH)\n [{'k': 'A', 'x': 1, 'y': 1},\n {'k': 'B', 'x': 2},\n {'k': 'C', 'x': 3, 'y': 3},\n {'k': 'C', 'x': 4, 'y': 4},\n {'k': 'D', 'x': 5},\n {'k': 'A', 'y': 2},\n {'k': 'E', 'y': 5}]\n\n (*Obviously, the outputs above were reformmated for clarity.*)\n\n\n10. There is a method for easily dealing with duplicate values on particular keys.\n Once again the idea of *key values* is used to determine what is considered.\n\n Note that if KEYS is a string, it is handled correctly.\n\n .. code:: python\n\n >>> dh = DataHammer(data)\n >>> keys = 'location.city'\n >>> Counter(dh.location.city)\n Counter({'Baltimore': 2, 'Portland': 2, 'San Antonio': 2, 'Oklahoma ': 1, 'Jersey City': 1})\n\n # Zero (0) gives only the unique items, where count was 1 -- all but \"Mobile Developer\".\n >>> ~dh._unique(keys, 0).location.city\n ['Oklahoma City', 'Jersey City']\n\n # The default (one, 1) gives all key values, but only the first item with the value(s)\n >>> ~dh._unique(keys).location.city\n ['Baltimore', 'Portland', 'San Antonio', 'Oklahoma City', 'Jersey City']\n >>> ~dh._unique(keys, 1).name\n [{'first': 'Addison', 'last': 'Stewart'}, {'first': 'Jack', 'last': 'Young'},\n {'first': 'Brianna', 'last': 'Lewis'}, {'first': 'Logan', 'last': 'Ward'},\n {'first': 'Grace', 'last': 'Evans'}]\n\n # Two (2) gives only those items that have a duplicate (of 'location.city').\n >>> ~dh._unique(keys, 2)._pick('name', keys)\n [{'name': {'first': 'Addison', 'last': 'Stewart'}, 'city': 'Baltimore'},\n {'name': {'first': 'Katherine', 'last': 'Perry'}, 'city': 'Baltimore'},\n {'name': {'first': 'Jack', 'last': 'Young'}, 'city': 'Portland'},\n {'name': {'first': 'Brianna', 'last': 'Lewis'}, 'city': 'San Antonio'},\n {'name': {'first': 'Logan', 'last': 'Martinez'}, 'city': 'Portland'},\n {'name': {'first': 'Sophia', 'last': 'Moore'}, 'city': 'San Antonio'}]\n\n # To get the unique set of duplicated values for a given set of keys, you can use a second\n # pass, using the default (1) for `unique`.\n >>> ~dh._unique(keys, 2)._unique(keys).location\n [{'city': 'Baltimore', 'state': 'Maryland'},\n {'city': 'Portland', 'state': 'Oregon'},\n {'city': 'San Antonio', 'state': 'Texas'}]\n\n\nFormatting Specification\n------------------------\n\n11. An extension is provided for formatting, using the **j** `type`. Each item will be printed as JSON using\n *json.dumps()*. In particular, the only allowed parts to the *format_spec* are:\n\n a. A negative `sign` will cause a newline to be inserted between the item outputs.\n b. A non-zero `width` causes the item JSON is used as the indent within the item output\n c. The only `type` supported is \"**j**\".\n\n.. code:: python \n\n >>> dh.location[0:2]\n [{'city': 'Baltimore', 'state': 'Maryland'}, {'city': 'Madison', 'state': 'Wisconsin'}]\n >>> print(\"{:-j}\".format(dh.location._slice(0,2)))\n [{\"city\":\"Baltimore\",\"state\":\"Maryland\"},\n {\"city\":\"Madison\",\"state\":\"Wisconsin\"}]\n >>> print(\"{:-3j}\".format(dh.location._slice(0,2)))\n [{\n \"city\":\"Baltimore\",\n \"state\":\"Maryland\"\n },\n {\n \"city\":\"Madison\",\n \"state\":\"Wisconsin\"\n }]\n\n\nWarnings and Caveats\n--------------------\n\n12. Warning: To combine multiple instances with `bool` values you must use the `&` and `|`, and\n *not* use `and` and `or` as you would with Python `bool` values.\n\n .. code:: python\n\n >>> dh1 = DataHammer([False, False, True, True])\n >>> dh2 = DataHammer([False, True, False, True])\n\n # These are item-wise correct results\n >>> ~(dh1 & dh2)\n [False, False, False, True]\n >>> ~(dh1 | dh2)\n [False, True, True, True]\n\n # Since the objects are not empty, 'or' returns the first, 'and' returns the second:\n >>> (dh1 or dh2) == dh1\n True\n >>> (dh1 and dh2) == dh2\n True\n\n\nOther Examples\n--------------\n\n13. Given a JSON file that has metadata separated from the data values, we can easily\n combine these, and find the ones which match criteria we want.\n\n .. code:: python\n\n >>> from datahammer import DataHammer\n >>> from six.moves.urllib import request\n >>> from collections import Counter\n\n >>> URL = 'https://data.ny.gov/api/views/pxa9-czw8/rows.json?accessType=DOWNLOAD'\n >>> req = request.urlopen(URL)\n >>> jobs = DataHammer(req, json=dict(encoding='utf-8'))\n\n # Grab the contained data in order to find its keys.\n >>> (~jobs).keys()\n dict_keys(['meta', 'data'])\n >>> names = jobs.meta.view.columns.name\n >>> norm = DataHammer(dict(zip(names, row)) for row in jobs.data)\n\n # Here 'norm' contains 1260 items, each a dict with the same schema.\n >>> len(norm)\n 1260\n >>> print(norm[0])\n {'sid': 1, 'id': 'A0447302-02D8-4EFD-AB68-777680645F02', 'position': 1,\n 'created_at': 1437380960, 'created_meta': '707861', 'updated_at': 1437380960,\n 'updated_meta': '707861', 'meta': None, 'Year': '2012', 'Region': 'Capital Region',\n 'NAICS Code': '11', 'Industry': 'Agriculture, Forestry, Fishing and Hunting',\n 'Jobs': '2183'}\n\n # Use collections.Counter to count the number of instances of values:\n >>> Counter(norm.Year)\n Counter({'2012': 210, '2013': 210, '2014': 210, '2015': 210, '2017': 210, '2016': 210})\n >>> Counter(norm._get('NAICS Code'))\n Counter({'11': 60, '21': 60, '22': 60, '23': 60, '42': 60, '51': 60, '52': 60,\n '53': 60, '54': 60, '55': 60, '56': 60, '61': 60, '62': 60, '71': 60,\n '72': 60, '81': 60, '90': 60, '99': 60, '31-33': 30, '44-45': 30,\n '48-49': 30, '31': 30, '44': 30, '48': 30})\n\n # Use '&' to require both conditions, it is a row-wise `and` of the separate tests.\n >>> cap2013 = norm[(norm.Year == '2013') & norm.Region._contains('Capital Region')]\n >>> len(cap2013)\n 21\n >>> keepers = norm.Jobs._int() > 500000\n >>> sum(keepers)\n 12\n >>> large = norm[keepers]\n >>> len(large)\n 12\n >>> large[0]\n {'sid': 121, 'id': '98A53A4E-712C-47A9-9106-C9062DB8CBBD', 'position': 121,\n 'created_at': 1437380961, 'created_meta': '707861', 'updated_at': 1437380961,\n 'updated_meta': '707861', 'meta': None, 'Year': '2012', 'Region': 'New York City',\n 'NAICS Code': '62', 'Industry': 'Health Care and Social Assistance', 'Jobs': '591686'}\n >>> ~norm.Region._unique(None)\n ['Capital Region', 'Central New York', 'Finger Lakes', 'Long Island', 'Mid-Hudson',\n 'New York City', 'North Country', 'Southern Tier ', 'Western New York ', 'Mohawk Valley',\n 'Southern Tier', 'Western New York']\n >>> Counter(norm.Region)\n Counter({'Capital Region': 126, 'Central New York': 126, 'Finger Lakes': 126,\n 'Long Island': 126, 'Mid-Hudson': 126, 'New York City': 126, 'North Country': 126,\n 'Mohawk Valley': 126, 'Southern Tier ': 63, 'Western New York ': 63,\n 'Southern Tier': 63, 'Western New York': 63})\n >>> sum(norm.Region._in(['Mohawk Valley', 'Southern Tier']))\n 189\n\n\nSELECTOR Examples\n-----------------\n \n- The positional parameter **\"b.b1\"** would dererence a value like *OBJ.b.b1*, and the resulting key would be\n the part after the last dot: **\"b1\"**.\n\n- The named parameter **animal=\"b.b2\"** would dererence like *OBJ.b.b2*, and the resulting key would be\n **\"animal\"**.\n\n.. code:: python\n\n >>> dh = DataHammer([\n {\"a\": 100, \"b\": {\"b1\": [101, 102, 103], \"b2\": \"ape\"}, \"c\": [\"Apple\", \"Anise\"]},\n {\"a\": 200, \"b\": {\"b1\": [201, 202, 203], \"b2\": \"bat\"}, \"c\": [\"Banana\", \"Basil\"]},\n {\"a\": 300, \"b\": {\"b1\": [301, 302, 303], \"b2\": \"cat\"}, \"c\": [\"Cherry\", \"Cayenne\"]}\n ])\n \n >>> ~dh._pick('a', 'b.b1', animal='b.b2', food='c', nil='this.is.missing')\n [{'a': 100, 'b1': [101, 102, 103], 'animal': 'ape', 'food': ['Apple', 'Anise'], 'nil': None},\n {'a': 200, 'b1': [201, 202, 203], 'animal': 'bat', 'food': ['Banana', 'Basil'], 'nil': None},\n {'a': 300, 'b1': [301, 302, 303], 'animal': 'cat', 'food': ['Cherry', 'Cayenne'], 'nil': None}] \n\n #### Result is undefined due to the key collision.\n >>> ~dh._pick('b.b1', b1='c')\n\n ## This '.0' syntax *might* change in future releases.\n >>> ~dh._pick(animal='b.b2', fruit='c.0')\n [{'animal': 'ape', 'fruit': 'Apple'},\n {'animal': 'bat', 'fruit': 'Banana'},\n {'animal': 'cat', 'fruit': 'Carmel'}]\n\n\nSample Data\n-----------\n\nNote that the data used here is randomly generated, no relationship to\nanyone living, dead or undead is intended.\n\n.. code:: python\n \n >>> from datahammer import DataHammer\n >>> mean = lambda nums: (sum(nums) * 1.0 / len(nums)) if nums else None\n >>> data = [\n {\n \"age\":45,\"gender\":\"F\",\"location\":{\"city\":\"Baltimore\",\"state\":\"Maryland\"},\n \"name\":{\"first\":\"Addison\",\"last\":\"Stewart\"},\"phone\":\"575-917-9109\",\n \"ranks\":[],\"salary\":10000,\"title\":\"Systems Administrator\"\n },\n {\n \"age\":57,\"gender\":\"F\",\"location\":{\"city\":\"Baltimore\",\"state\":\"Maryland\"},\n \"name\":{\"first\":\"Katherine\",\"last\":\"Perry\"},\"phone\":\"524-133-3495\",\n \"ranks\":[157,200,2,18,18,27],\"salary\":18590,\"title\":\"Bookkeeper\"\n },\n {\n \"age\":33,\"gender\":\"M\",\"location\":{\"city\":\"Portland\",\"state\":\"Oregon\"},\n \"name\":{\"first\":\"Jack\",\"last\":\"Young\"},\"phone\":\"803-435-5879\",\n \"ranks\":[9,157,197,155,190,56,58,97],\"salary\":28640,\"title\":\"Controller\"\n },\n {\n \"age\":21,\"gender\":\"F\",\"location\":{\"city\":\"San Antonio\",\"state\":\"Texas\"},\n \"name\":{\"first\":\"Brianna\",\"last\":\"Lewis\"},\"phone\":\"364-549-0753\",\n \"ranks\":[180,190,111],\"salary\":8000,\"title\":\"UX Designer\"\n },\n {\n \"age\":24,\"gender\":\"M\",\"location\":{\"city\":\"Oklahoma City\",\"state\":\"Oklahoma\"},\n \"name\":{\"first\":\"Logan\",\"last\":\"Ward\"},\"phone\":\"734-410-1116\",\n \"ranks\":[116,162],\"salary\":8000,\"title\":\"Web Developer\"\n },\n {\n \"age\":60,\"gender\":\"M\",\"location\":{\"city\":\"Portland\",\"state\":\"Oregon\"},\n \"name\":{\"first\":\"Logan\",\"last\":\"Martinez\"},\"phone\":\"652-193-9184\",\n \"ranks\":[70,16,59,24,32],\"salary\":33700,\"title\":\"Assessor\"\n },\n {\n \"age\":63,\"gender\":\"F\",\"location\":{\"city\":\"Jersey City\",\"state\":\"New Jersey\"},\n \"name\":{\"first\":\"Grace\",\"last\":\"Evans\"},\"phone\":\"955-466-6227\",\n \"ranks\":[123,126,118,64,110,28],\"salary\":26220,\"title\":\"Mobile Developer\"\n },\n {\n \"age\":33,\"gender\":\"F\",\"location\":{\"city\":\"San Antonio\",\"state\":\"Texas\"},\n \"name\":{\"first\":\"Sophia\",\"last\":\"Moore\"},\"phone\":\"636-269-3573\",\n \"ranks\":[97],\"salary\":14120,\"title\":\"Mobile Developer\"\n }]\n\n\nFoot Notes\n==========\n\n.. [1] Tokens\n\nIn these examples, *OBJ* refers to a *DataHammer* instance, *LIST* refers to the list of\ncontained items, and *ITEM* refers to an item in the contained list or directly in the *OBJ*.\n\n\n.. [2] Dereferences\n\nAn attribute dereference (eg: *OBJ.index*) and the methods *OBJ._ind(index)* and *OBJ._get(index)* all\nfunction identically, returning a new **DataHammer** instance. The latter are provided for use when\n*index* is an *int* or otherwise not a valid string identifier.\n\n\n.. [3] Scalars, Vectors and DataHammers\n\nFor most operations and functions that return a new instance, when a *DataHammer* instance is combined\nwith a list, tuple or other *DataHammer* instance, the length of the new instance will be limited by the\nlength of the shorter of the two operands. For example:\n\n - Using a shorter operand, the result will be shortened as if the *DataHammer* instance had only that\n many items.\n\n - Using a longer operand, the result will be as if the *DataHammer* instance had only as many items as\n that other operand.\n\n .. code:: python\n\n >>> dh1 = DataHammer(range(8))\n >>> ~(dh1 + (10, 20))\n [10, 21]\n >>> dh2 = DataHammer((3, 1, 4))\n >>> ~(dh1 == dh2)\n [False, True, False]\n >>> ~(dh1[dh2])\n [3, 1, 4]\n\n\n.. [4] Bracket Indexing\n\nBecause the **[]** syntax is used for `Indexing`_ and returns an ITEM or list, we cannot use this syntax\nfor chaining or to create another instance as we do for dotted-attribute access. This is why there is a\n**_ind()** method, to allow\n\n .. code:: python\n\n >>> dh = DataHammer([[i, i*i] for i in range(10, 15)])\n >>> ~dh\n [[10, 100], [11, 121], [12, 144], [13, 169], [14, 196]]\n >>> ~dh._ind(1)\n [100, 121, 144, 169, 196]\n >>> ~(dh._ind(1) > 125)\n [False, False, True, True, True]\n >>> ~dh[dh._ind(1) > 125]\n [[12, 144], [13, 169], [14, 196]]\n >>> dh = DataHammer([dict(a=i, b=tuple(range(i, i*2))) for i in range(6)])\n\n # 'dh.b' returns a DataHammer of N-tuples, then '[3]' retrieves the 4th of these tuples as a `tuple`.\n >>> dh.b[2]\n (2, 3)\n\n # Here 'dh.b' gives a DataHammer instance of N-tuples, but '_ind(2)' returns another DataHammer\n # with the 3rd item from those N-tuples. Note the `None` for slots where the tuple length.\n >>> dh.b._ind(2)\n <datahammer.DataHammer object at 0x7f79eb1a9c10>\n >>> ~dh.b._ind(2)\n [None, None, None, 5, 6, 7]\n\n\n.. [5] Slicing\n\nThis works similar to the *slice* method of the\n`Javascript Array <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/slice>`_\nclass.\n\n\n.. [6] In / Contains\n\nUsing \"*ITEM in OBJ*\" returns True if ITEM matches one of the items in OBJ, using the operator **==**\nfor the test. However, using *OBJ in OTHER* for an iterable containers *OTHER*, is useless.\nuseless.\n\nUsing \"*OBJ in OTHER*\" will evaluate the expression \"**X == OBJ**\" for each item X in OTHER,, resulting\nin a list of bool. Unless either *OTHER* or *OBJ* are empty, this means a non-empty list will be\nconverted to **True** even if all of the comparisons fail.\n\n\n.. [7] Mutator\n\n*Mutator* operations dereference items based on the type of an item, regardless of the type of other items in\nthe contained data. Meaning: if a *DataHammer* with two items contains a `dict` with a key \"foo\" and an object\nwith an attribute \"foo\", then using **OBJ._mutator().foo** will update differently.\n\n\n.. [8] *SELECTOR* Syntax.\n\nThe value of a *SELECTOR* must be a `str`, but depending on the method can be named or positional.\nSee `SELECTOR Examples`_.\n\n1. For positional parameters, the text after the last dot, if any, is used for the resulting key.\n2. For named parameters, the value will be used to fetch the value, and the parameter name will be used for\n the key in the resulting item.\n3. For both, a dot (`.`) indicates a sub-key, like normal dot notation and/or the *_ind()* method.\n\n*Caveats*:\n\n4. If there are multiple parameters that result in the same key, the result is undefined.\n5. Currently, positional parameters are processed in order before the named parameters,\n but that is not guaranteed to be true in future releases.\n6. Currently, a bare int (in decimal form) is used to index into lists, but that syntax is not\n guaranteed to be true in future releases. If a bare int is used as the last component of a\n postitional parameter value, the resulting key will be a `str` - the decimal value.\n\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "This module provides an easy way to manipulate and inspect lists of data. It was designed to handle plain data types, especially the output from parsing JSON. It allows simple operations to be done on the items in parallel in a concise fashion. Many features will also work on other data types.",
"version": "1.0.3",
"project_urls": {
"Download": "https://gitlab.com/n2vram/datahammer/archive/1.0.3",
"Homepage": "https://gitlab.com/n2vram/datahammer"
},
"split_keywords": [
"data",
"datasets",
"queries",
"json",
"resultsets",
"datahammer",
"hammer"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f5e0984f9892347b7c70d5225e977b28a7bf9c4f8ee9ac70dea7b70b53320fa8",
"md5": "234290419e67f4f8b26fc29541a045af",
"sha256": "2412eb08218747138d45bf4643c4fbbef9d0d9b3678cc7e526cbeb1166d9e6b6"
},
"downloads": -1,
"filename": "datahammer-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "234290419e67f4f8b26fc29541a045af",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 30743,
"upload_time": "2023-08-05T02:15:35",
"upload_time_iso_8601": "2023-08-05T02:15:35.423464Z",
"url": "https://files.pythonhosted.org/packages/f5/e0/984f9892347b7c70d5225e977b28a7bf9c4f8ee9ac70dea7b70b53320fa8/datahammer-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-05 02:15:35",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "n2vram",
"gitlab_project": "datahammer",
"lcname": "datahammer"
}
