splitutils


Namesplitutils JSON
Version 0.2.0 PyPI version JSON
download
home_pagehttps://github.com/reichelu/splitutils
SummaryGroup disjunct and stratified data splits
upload_time2023-12-01 08:57:56
maintainer
docs_urlNone
authorUwe Reichel
requires_python>=3.8
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # <a name="contents">Machine learning data partition tools</a>

- [Contents](#contents)
    - [Author](#author)
    - [Purpose](#purpose)
    - [Installation](#installation)
    - [Synopsis](#synopsis)
        - [optimize_traintest_split()](#otts)
        - [optimize_traindevtest_split()](#otdts)
        - [binning()](#binning)
    - [Usage](#usage)
        - [Example 1: Split dummy data into training and test partitions](#example1)
        - [Example 2: split dummy data into training, development, and test partitions](#example2)
        - [Example 3: split dummy data into training, development, and test partitions, the target and several stratification variables being numeric](#example3)
    - [Algorithm](#algorithm)
    - [How to interprete the returned info dict](#interpretation)
    - [Further documentation](#doc)

## <a name="author">Author</a>

Uwe Reichel, audEERING GmbH, Gilching, Germany

## <a name="purpose">Purpose</a>

* machine learning data splitting tool that allows for:
  * group-disjunct splits (e.g. different speakers in train, dev, and test partition)
  * stratification on multiple target and grouping variables (e.g. emotion, gender, language)

## <a name="installation">Installation</a>

### From PyPI

* set up a virtual environment `venv_splitutils`, activate it, and install `splitutils`. For Linux this works e.g. as follows:

```bash
$ virtualenv --python="/usr/bin/python3" venv_splitutils
$ source venv_splitutils/bin/activate
(venv_splitutils) $ pip install splitutils
```

### From GitHub

* project URL: https://github.com/reichelu/splitutils
* set up a virtual environment venv_copasul, activate it, and install requirements. For Linux this works e.g. as follows:

```bash
$ git clone git@github.com:reichelu/spliutils.git
$ cd splitutils/
$ virtualenv --python="/usr/bin/python3" venv_splitutils
$ source venv_splitutils/bin/activate
$ (venv_splitutils) $ pip install -r requirements.txt
```

## <a name="synopsis">Synopsis</a>

### <a name="otts">optimize_traintest_split()</a>

```python
def optimize_traintest_split(X, y, split_on, stratify_on, weight=None,
                             test_size=.1, k=30, seed=42):

    ''' optimize group-disjunct split into training and test set which is guided by:
    - disjunct split of values in SPLIT_ON
    - stratification by all keys in STRATIFY_ON (targets and groupings)
    - test set proportion in X should be close to test_size (which is the test
      proportion in set(split_on))

    Parameters:
    X: (np.array or pd.DataFrame) of features
    y: (np.array) of targets of length N
      if type(y[0]) in ["str", "int"]: y is assumed to be categorical, so that it is
      additionally tested that all partitions cover all classes.
      Else y is assumed to be numeric and no coverage test is done.
    split_on: (np.array) list of length N with grouping variable (e.g. speaker IDs),
      on which the group-disjunct split is to be performed. Must be categorical.
    stratify_on: (dict) Dict-keys are variable names (targets and/or further groupings)
      the split should be stratified on (groupings could e.g. be sex, age class, etc).
      Dict-Values are np.array-s of length N that contain the variable values. All
      variables must be categorical.
    weight: (dict) weight for each variable in stratify_on. Defines their amount of
      contribution to the optimization score. Uniform weighting by default. Additional
      key: "size_diff" defines how test size diff should be weighted.
    test_size: (float) test proportion in set(split_on), e.g. 10% of speakers to be
      held-out
    k: (int) number of different splits to be tried out
    seed: (int) random seed

    Returns:
    train_i: (np.array) train set indices in X
    test_i: (np.array) test set indices in X
    info: (dict) detail information about reference and achieved prob distributions
        "size_testset_in_spliton": intended test_size
        "size_testset_in_X": optimized test proportion in X
        "p_ref_{c}": reference class distribution calculated from stratify_on[c]
        "p_test_{c}": test set class distribution calculated from stratify_on[c][test_i]
    '''
```

### <a name="otdts">optimize_traindevtest_split()</a>


```python
def optimize_traindevtest_split(X, y, split_on, stratify_on, weight=None,
                                dev_size=.1, test_size=.1, k=30, seed=42):

    ''' optimize group-disjunct split into training, dev, and test set, which is
    guided by:
    - disjunct split of values in SPLIT_ON
    - stratification by all keys in STRATIFY_ON (targets and groupings)
    - test set proportion in X should be close to test_size (which is the test
      proportion in set(split_on))

    Parameters:
    X: (np.array or pd.DataFrame) of features
    y: (np.array) of targets of length N
      if type(y[0]) in ["str", "int"]: y is assumed to be categorical, so
         that it is additionally tested that all partitions cover all classes.
         Else y is assumed to be numeric and no coverage test is done.
    split_on: (np.array) list of length N with grouping variable (e.g. speaker IDs),
      on which the group-disjunct split is to be performed. Must be categorical.
    stratify_on: (dict) Dict-keys are variable names (targets and/or further groupings)
      the split should be stratified on (groupings could e.g. be sex, age class, etc).
      Dict-Values are np.array-s of length N that contain the variable values. All
      variables must be categorical.
    weight: (dict) weight for each variable in stratify_on. Defines their amount of
      contribution to the optimization score. Uniform weighting by default. Additional
      key: "size_diff" defines how the corresponding size differences should be weighted.
    dev_size: (float) proportion in set(split_on) for dev set, e.g. 10% of speakers
      to be held-out
    test_size: (float) test proportion in set(split_on) for test set
    k: (int) number of different splits to be tried out
    seed: (int) random seed

    Returns:
    train_i: (np.array) train set indices in X
    dev_i: (np.array) dev set indices in X
    test_i: (np.array) test set indices in X
    info: (dict) detail information about reference and achieved prob distributions
        "dev_size_in_spliton": intended grouping dev_size
        "dev_size_in_X": optimized dev proportion of observations in X
        "test_size_in_spliton": intended grouping test_size
        "test_size_in_X": optimized test proportion of observations in X
        "p_ref_{c}": reference class distribution calculated from stratify_on[c]
        "p_dev_{c}": dev set class distribution calculated from stratify_on[c][dev_i]
        "p_test_{c}": test set class distribution calculated from stratify_on[c][test_i]
    '''
```

### <a name="binning">binning()</a>

```python

def binning(x, nbins=2, lower_boundaries=None, seed=42):

    '''
    bins numeric data.

    If X is one-dimensional:
        binning is done either intrinsically into nbins classes
        based on an equidistant percentile split, or extrinsically
        by using the lower_boundaries values.
    If X is two-dimensional
        binning is done by kmeans clustering into nbins clusters

    Parameters:
    x: (list, np.array) with numeric data.
    nbins: (int) number of bins
    lower_boundaries: (list) of lower bin boundaries.
      If y is 1-dim and lower_boundaries is provided, nbins will be ignored
      and y is binned extrinsically. The first value of lower_boundaries
      is always corrected not to be higher than min(y).
    seed: (int) random seed for kmeans

    Returns:
    c: (np.array) integers as bin IDs
    '''
```

## <a name="usage">Usage</a>

### <a name="example1">Example 1: Split dummy data into training and test partitions</a>

* see `scripts/run_traintest_split.py`
* partitions are:
    * disjunct on categorical "split_var"
    * stratified on categorical "target", "strat_var1", "strat_var2"
    * each contain all levels of "target"

```python
import numpy as np
import os
import pandas as pd
import sys

# add this line if you have cloned the code from github to PROJECT_DIR
# sys.path.append(PROJECT_DIR)

from splitutils import optimize_traindevtest_split

# set seed
seed = 42
np.random.seed(seed)

# DUMMY DATA
# size
n = 100

# feature array
data = np.random.rand(100, 20)

# target variable
target = np.random.choice(["A", "B"], size=n, replace=True)

# array with variable on which to do a disjunct split
split_var = np.random.choice(["D", "E", "F", "G", "H", "I", "J", "K"],
                             size=n, replace=True)

# dict of variables to stratify on. Key names are arbitrary.
stratif_vars = {
    "target": target,
    "strat_var1": np.random.choice(["L", "M"], size=n, replace=True),
    "strat_var2": np.random.choice(["N", "O"], size=n, replace=True)
}

# ARGUMENTS
# weight importance of all stratification variables in stratify_in
# as well as of "size_diff", which punishes the deviation of intended
# and received partition sizes.
# Key names must match the names in stratif_vars.
weights = {
    "target": 2,
    "strat_var1": 1,
    "strat_var2": 1,
    "size_diff": 1
}

# test partition proportion (from 0 to 1)
test_size = .2

# number of disjunct splits to be tried out in brute force optimization
k = 30

# FIND OPTIMAL SPLIT
train_i, test_i, info = optimize_traintest_split(
    X=data,
    y=target,
    split_on=split_var,
    stratify_on=stratif_vars,
    weight=weights,
    test_size=test_size,
    k=k,
    seed=seed
)

# SOME OUTPUT
print("test levels of split_var:", sorted(set(split_var[test_i])))
print("goodness of split:", info)
```

### <a name="example2">Example 2: Split dummy data into training, development, and test partitionsy</a>

* see `scripts/run_traindevtest_split.py`
* Partitions are
    * disjunct on categorical "split_var"
    * stratified on categorical "target", "strat_var1", "strat_var2"
    * each contain all levels of "target"

```python
import numpy as np
import os
import pandas as pd
import sys

# add this line if you have cloned the code from github to PROJECT_DIR
# sys.path.append(PROJECT_DIR)

from splitutils import optimize_traindevtest_split

# set seed
seed = 42
np.random.seed(seed)

# DUMMY DATA
# size
n = 100

# feature array
data = np.random.rand(100, 20)

# target variable
target = np.random.choice(["A", "B"], size=n, replace=True)

# array with variable on which to do a disjunct split
split_var = np.random.choice(["D", "E", "F", "G", "H", "I", "J", "K"],
                             size=n, replace=True)

# dict of variables to stratify on. Key names are arbitrary.
stratif_vars = {
    "target": target,
    "strat_var1": np.random.choice(["F", "G"], size=n, replace=True),
    "strat_var2": np.random.choice(["H", "I"], size=n, replace=True)
}

# ARGUMENTS
# weight importance of all stratification variables in stratify_in
# as well as of "size_diff", which punishes the deviation of intended
# and received partition sizes.
# Key names must match the names in stratif_vars.
weights = {
    "target": 2,
    "strat_var1": 1,
    "strat_var2": 1,
    "size_diff": 1
}

# dev and test partition proportion (from 0 to 1)
dev_size = .1
test_size = .1

# number of disjunct splits to be tried out in brute force optimization
k = 30

# FIND OPTIMAL SPLIT
train_i, dev_i, test_i, info = optimize_traindevtest_split(
    X=data,
    y=target,
    split_on=split_var,
    stratify_on=stratif_vars,
    weight=weights,
    dev_size=dev_size,
    test_size=test_size,
    k=k,
    seed=seed
)

# SOME OUTPUT
print("test levels of split_var:", sorted(set(split_var[test_i])))
print("goodness of split:", info)
```

### <a name="example3">Example 3: Split dummy data into training, development, and test partitions, the target and several stratification variables being numeric</a>

* see `scripts/run_traindevtest_split_with_binning.py`
* Partitions are
    * disjunct on categorical "split_var"
    * stratified on numeric "target", and on 3 other numeric stratification variables 

```python
import numpy as np
import os
import pandas as pd
import sys

# add this line if you have cloned the code from github to PROJECT_DIR
# sys.path.append(PROJECT_DIR)

from splitutils import (
    binning,
    optimize_traindevtest_split
)

"""
example script how to split dummy data into training, development,
and test partitions that are
* disjunct on categorical "split_var"
* stratified on numeric "target", and on 3 other numeric stratification
  variables
"""

# set seed
seed = 42
np.random.seed(seed)

# DUMMY DATA
# size
n = 100

# features
data = np.random.rand(n, 20)

# numeric target variable
num_target = np.random.rand(n)

# array with variable on which to do a disjunct split
split_var = np.random.choice(["D", "E", "F", "G", "H", "I", "J", "K"],
                             size=n, replace=True)

# further numeric variables to stratify on
num_strat_vars = np.random.rand(n, 3)

# intrinsically bin target into 3 bins by equidistant
# percentile boundaries
binned_target = binning(num_target, nbins=3)

# ... alternatively, a variable can be extrinsically binned by
# specifying lower boundaries:
# binned_target = binning(target, lower_boundaries: [0, 0.33, 0.66])

# bin other stratification variables into a single variable with 6 bins
# (2-dim input is binned by StandardScaling and KMeans clustering)
binned_strat_var = binning(num_strat_vars, nbins=6)

# ... alternatively, each stratification variable could be binned
# individually - intrinsically or extrinsically the same way as num_target
# strat_var1 = binning(num_strat_vars[:,0], nbins=...) etc.

# now add the obtained categorical variable to stratification dict
stratif_vars = {
    "target": binned_target,
    "strat_var": binned_strat_var
}

# ARGUMENTS
# weight importance of all stratification variables in stratify_in
# as well as of "size_diff", which punishes the deviation of intended
# and received partition sizes
weights = {
    "target": 2,
    "strat_var": 1,
    "size_diff": 1
}

# dev and test partition proportion (from 0 to 1)
dev_size = .1
test_size = .1

# number of disjunct splits to be tried out in brute force optimization
k = 30

# FIND OPTIMAL SPLIT
train_i, dev_i, test_i, info = optimize_traindevtest_split(
    X=data,
    y=num_target,
    split_on=split_var,
    stratify_on=stratif_vars,
    weight=weights,
    dev_size=dev_size,
    test_size=test_size,
    k=k,
    seed=seed
)

# SOME OUTPUT
print("test levels of split_var:", sorted(set(split_var[test_i])))
print("goodness of split:", info)
```

## <a name="algorithm">Algorithm</a>

* find optimal train, dev, and test set split based on:
    * disjunct split of a categorical grouping variable *G* (e.g. speaker)
    * optimized joint stratification on an arbitrary amount of categorical target and grouping variables (e.g. emotion, gender, ...)
    * close match of partition proportions in *G* and underlying dataset *X*
* brute-force optimization on *k* disjunct splits of *G*
* **score to be minimzed for train/test set split:**

```
(sum_v[w(v) * irad(v)] + w(d) * d) / (sum_v[w(v)] + w(d))

v: variables to be stratified on
w(v): their weight
irad(v): information radius between reference and test set distribution of factor levels in v
d: absolute difference between test proportions of X and G, i.e. between the proportion of test
   samples and the proportion of groups (e.g. speakers) that go into the test set
w(d): its weight
```

* **score to be minimzed for train / dev / test set split:**

```
(sum_v[w(v) * max_irad(v)] + w(d) * max_d) / (sum_v[w(v)] + w(d))

v: variables to be stratified on
w(v): their weight
max_irad(v): maximum information radius of reference distribution of classes in v and
             - dev set distribution,
             - test set distribution
max_d: maximum of absolute difference between proportions of X and G (see above) calculated for
       the dev and test set
w(d): its weight
```

## <a name="interpretation">How to interprete the returned `info` dict</a>

* let's look at [Example 2](#example2) above. There `info` becomes:

```python
{
  'score': 0.030828359568603338,
  'size_devset_in_spliton': 0.1,
  'size_devset_in_X': 0.14,
  'size_testset_in_spliton': 0.1,
  'size_testset_in_X': 0.13,
  'p_target_ref': {'B': 0.49, 'A': 0.51},
  'p_target_dev': {'A': 0.5, 'B': 0.5},
  'p_target_test': {'A': 0.5384615384615384, 'B': 0.46153846153846156},
  'p_strat_var1_ref': {'G': 0.56, 'F': 0.44},
  'p_strat_var1_dev': {'G': 0.5714285714285714, 'F': 0.42857142857142855},
  'p_strat_var1_test': {'F': 0.5384615384615384, 'G': 0.46153846153846156},
  'p_strat_var2_ref': {'I': 0.48, 'H': 0.52},
  'p_strat_var2_dev': {'I': 0.5, 'H': 0.5},
  'p_strat_var2_test': {'I': 0.46153846153846156, 'H': 0.5384615384615384}
}
```

* **Explanations**
    * **score:** see above, **score to be minimzed for train / dev / test set split:**
    * **size_devset_in_spliton:** proportion of to-be-split-on variable levels in development set
    * **size_devset_in_X:** proportion of rows in X in development set
    * **size_testset_in_spliton:** proportion of to-be-split-on variable levels in test set
    * **size_testset_in_X:** proportion of rows in X in test set
    * **p_target_ref:** reference target class distribution over all data
    * **p_target_dev:** target class distribution in development set
    * **p_target_test:** target class distribution in test set
    * **p_strat_var1_ref:** first stratification variable's reference distribution over all data
    * **p_strat_var1_dev:** first stratification variable's class distribution in development set
    * **p_strat_var1_test:** first stratification variable's class distribution in test set
    * **p_strat_var2_ref:** second stratification variable's reference distribution over all data
    * **p_strat_var2_dev:** second stratification variable's class distribution in development set
    * **p_strat_var2_test:** second stratification variable's class distribution in test set
* **Remarks**
    * for `splitutils.optimize_traintest_split()` no development set results are reported
    * all `*_strat_var*` keys: key names derived from key names in `stratify_on` argument

## <a name="doc">Further documentation</a>

* Please find further details on the split scores and numeric variable binning in this [pdf](https://github.com/reichelu/splitutils/blob/main/docs/splitutils.pdf)


            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/reichelu/splitutils",
    "name": "splitutils",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": "",
    "keywords": "",
    "author": "Uwe Reichel",
    "author_email": "ureichel@audeering.com",
    "download_url": "https://files.pythonhosted.org/packages/3c/d0/130ad38c5a53eb5bb70ff1b90f515f353498813c71efa73eb842f0716154/splitutils-0.2.0.tar.gz",
    "platform": null,
    "description": "# <a name=\"contents\">Machine learning data partition tools</a>\n\n- [Contents](#contents)\n    - [Author](#author)\n    - [Purpose](#purpose)\n    - [Installation](#installation)\n    - [Synopsis](#synopsis)\n        - [optimize_traintest_split()](#otts)\n        - [optimize_traindevtest_split()](#otdts)\n        - [binning()](#binning)\n    - [Usage](#usage)\n        - [Example 1: Split dummy data into training and test partitions](#example1)\n        - [Example 2: split dummy data into training, development, and test partitions](#example2)\n        - [Example 3: split dummy data into training, development, and test partitions, the target and several stratification variables being numeric](#example3)\n    - [Algorithm](#algorithm)\n    - [How to interprete the returned info dict](#interpretation)\n    - [Further documentation](#doc)\n\n## <a name=\"author\">Author</a>\n\nUwe Reichel, audEERING GmbH, Gilching, Germany\n\n## <a name=\"purpose\">Purpose</a>\n\n* machine learning data splitting tool that allows for:\n  * group-disjunct splits (e.g. different speakers in train, dev, and test partition)\n  * stratification on multiple target and grouping variables (e.g. emotion, gender, language)\n\n## <a name=\"installation\">Installation</a>\n\n### From PyPI\n\n* set up a virtual environment `venv_splitutils`, activate it, and install `splitutils`. For Linux this works e.g. as follows:\n\n```bash\n$ virtualenv --python=\"/usr/bin/python3\" venv_splitutils\n$ source venv_splitutils/bin/activate\n(venv_splitutils) $ pip install splitutils\n```\n\n### From GitHub\n\n* project URL: https://github.com/reichelu/splitutils\n* set up a virtual environment venv_copasul, activate it, and install requirements. For Linux this works e.g. as follows:\n\n```bash\n$ git clone git@github.com:reichelu/spliutils.git\n$ cd splitutils/\n$ virtualenv --python=\"/usr/bin/python3\" venv_splitutils\n$ source venv_splitutils/bin/activate\n$ (venv_splitutils) $ pip install -r requirements.txt\n```\n\n## <a name=\"synopsis\">Synopsis</a>\n\n### <a name=\"otts\">optimize_traintest_split()</a>\n\n```python\ndef optimize_traintest_split(X, y, split_on, stratify_on, weight=None,\n                             test_size=.1, k=30, seed=42):\n\n    ''' optimize group-disjunct split into training and test set which is guided by:\n    - disjunct split of values in SPLIT_ON\n    - stratification by all keys in STRATIFY_ON (targets and groupings)\n    - test set proportion in X should be close to test_size (which is the test\n      proportion in set(split_on))\n\n    Parameters:\n    X: (np.array or pd.DataFrame) of features\n    y: (np.array) of targets of length N\n      if type(y[0]) in [\"str\", \"int\"]: y is assumed to be categorical, so that it is\n      additionally tested that all partitions cover all classes.\n      Else y is assumed to be numeric and no coverage test is done.\n    split_on: (np.array) list of length N with grouping variable (e.g. speaker IDs),\n      on which the group-disjunct split is to be performed. Must be categorical.\n    stratify_on: (dict) Dict-keys are variable names (targets and/or further groupings)\n      the split should be stratified on (groupings could e.g. be sex, age class, etc).\n      Dict-Values are np.array-s of length N that contain the variable values. All\n      variables must be categorical.\n    weight: (dict) weight for each variable in stratify_on. Defines their amount of\n      contribution to the optimization score. Uniform weighting by default. Additional\n      key: \"size_diff\" defines how test size diff should be weighted.\n    test_size: (float) test proportion in set(split_on), e.g. 10% of speakers to be\n      held-out\n    k: (int) number of different splits to be tried out\n    seed: (int) random seed\n\n    Returns:\n    train_i: (np.array) train set indices in X\n    test_i: (np.array) test set indices in X\n    info: (dict) detail information about reference and achieved prob distributions\n        \"size_testset_in_spliton\": intended test_size\n        \"size_testset_in_X\": optimized test proportion in X\n        \"p_ref_{c}\": reference class distribution calculated from stratify_on[c]\n        \"p_test_{c}\": test set class distribution calculated from stratify_on[c][test_i]\n    '''\n```\n\n### <a name=\"otdts\">optimize_traindevtest_split()</a>\n\n\n```python\ndef optimize_traindevtest_split(X, y, split_on, stratify_on, weight=None,\n                                dev_size=.1, test_size=.1, k=30, seed=42):\n\n    ''' optimize group-disjunct split into training, dev, and test set, which is\n    guided by:\n    - disjunct split of values in SPLIT_ON\n    - stratification by all keys in STRATIFY_ON (targets and groupings)\n    - test set proportion in X should be close to test_size (which is the test\n      proportion in set(split_on))\n\n    Parameters:\n    X: (np.array or pd.DataFrame) of features\n    y: (np.array) of targets of length N\n      if type(y[0]) in [\"str\", \"int\"]: y is assumed to be categorical, so\n         that it is additionally tested that all partitions cover all classes.\n         Else y is assumed to be numeric and no coverage test is done.\n    split_on: (np.array) list of length N with grouping variable (e.g. speaker IDs),\n      on which the group-disjunct split is to be performed. Must be categorical.\n    stratify_on: (dict) Dict-keys are variable names (targets and/or further groupings)\n      the split should be stratified on (groupings could e.g. be sex, age class, etc).\n      Dict-Values are np.array-s of length N that contain the variable values. All\n      variables must be categorical.\n    weight: (dict) weight for each variable in stratify_on. Defines their amount of\n      contribution to the optimization score. Uniform weighting by default. Additional\n      key: \"size_diff\" defines how the corresponding size differences should be weighted.\n    dev_size: (float) proportion in set(split_on) for dev set, e.g. 10% of speakers\n      to be held-out\n    test_size: (float) test proportion in set(split_on) for test set\n    k: (int) number of different splits to be tried out\n    seed: (int) random seed\n\n    Returns:\n    train_i: (np.array) train set indices in X\n    dev_i: (np.array) dev set indices in X\n    test_i: (np.array) test set indices in X\n    info: (dict) detail information about reference and achieved prob distributions\n        \"dev_size_in_spliton\": intended grouping dev_size\n        \"dev_size_in_X\": optimized dev proportion of observations in X\n        \"test_size_in_spliton\": intended grouping test_size\n        \"test_size_in_X\": optimized test proportion of observations in X\n        \"p_ref_{c}\": reference class distribution calculated from stratify_on[c]\n        \"p_dev_{c}\": dev set class distribution calculated from stratify_on[c][dev_i]\n        \"p_test_{c}\": test set class distribution calculated from stratify_on[c][test_i]\n    '''\n```\n\n### <a name=\"binning\">binning()</a>\n\n```python\n\ndef binning(x, nbins=2, lower_boundaries=None, seed=42):\n\n    '''\n    bins numeric data.\n\n    If X is one-dimensional:\n        binning is done either intrinsically into nbins classes\n        based on an equidistant percentile split, or extrinsically\n        by using the lower_boundaries values.\n    If X is two-dimensional\n        binning is done by kmeans clustering into nbins clusters\n\n    Parameters:\n    x: (list, np.array) with numeric data.\n    nbins: (int) number of bins\n    lower_boundaries: (list) of lower bin boundaries.\n      If y is 1-dim and lower_boundaries is provided, nbins will be ignored\n      and y is binned extrinsically. The first value of lower_boundaries\n      is always corrected not to be higher than min(y).\n    seed: (int) random seed for kmeans\n\n    Returns:\n    c: (np.array) integers as bin IDs\n    '''\n```\n\n## <a name=\"usage\">Usage</a>\n\n### <a name=\"example1\">Example 1: Split dummy data into training and test partitions</a>\n\n* see `scripts/run_traintest_split.py`\n* partitions are:\n    * disjunct on categorical \"split_var\"\n    * stratified on categorical \"target\", \"strat_var1\", \"strat_var2\"\n    * each contain all levels of \"target\"\n\n```python\nimport numpy as np\nimport os\nimport pandas as pd\nimport sys\n\n# add this line if you have cloned the code from github to PROJECT_DIR\n# sys.path.append(PROJECT_DIR)\n\nfrom splitutils import optimize_traindevtest_split\n\n# set seed\nseed = 42\nnp.random.seed(seed)\n\n# DUMMY DATA\n# size\nn = 100\n\n# feature array\ndata = np.random.rand(100, 20)\n\n# target variable\ntarget = np.random.choice([\"A\", \"B\"], size=n, replace=True)\n\n# array with variable on which to do a disjunct split\nsplit_var = np.random.choice([\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\", \"K\"],\n                             size=n, replace=True)\n\n# dict of variables to stratify on. Key names are arbitrary.\nstratif_vars = {\n    \"target\": target,\n    \"strat_var1\": np.random.choice([\"L\", \"M\"], size=n, replace=True),\n    \"strat_var2\": np.random.choice([\"N\", \"O\"], size=n, replace=True)\n}\n\n# ARGUMENTS\n# weight importance of all stratification variables in stratify_in\n# as well as of \"size_diff\", which punishes the deviation of intended\n# and received partition sizes.\n# Key names must match the names in stratif_vars.\nweights = {\n    \"target\": 2,\n    \"strat_var1\": 1,\n    \"strat_var2\": 1,\n    \"size_diff\": 1\n}\n\n# test partition proportion (from 0 to 1)\ntest_size = .2\n\n# number of disjunct splits to be tried out in brute force optimization\nk = 30\n\n# FIND OPTIMAL SPLIT\ntrain_i, test_i, info = optimize_traintest_split(\n    X=data,\n    y=target,\n    split_on=split_var,\n    stratify_on=stratif_vars,\n    weight=weights,\n    test_size=test_size,\n    k=k,\n    seed=seed\n)\n\n# SOME OUTPUT\nprint(\"test levels of split_var:\", sorted(set(split_var[test_i])))\nprint(\"goodness of split:\", info)\n```\n\n### <a name=\"example2\">Example 2: Split dummy data into training, development, and test partitionsy</a>\n\n* see `scripts/run_traindevtest_split.py`\n* Partitions are\n    * disjunct on categorical \"split_var\"\n    * stratified on categorical \"target\", \"strat_var1\", \"strat_var2\"\n    * each contain all levels of \"target\"\n\n```python\nimport numpy as np\nimport os\nimport pandas as pd\nimport sys\n\n# add this line if you have cloned the code from github to PROJECT_DIR\n# sys.path.append(PROJECT_DIR)\n\nfrom splitutils import optimize_traindevtest_split\n\n# set seed\nseed = 42\nnp.random.seed(seed)\n\n# DUMMY DATA\n# size\nn = 100\n\n# feature array\ndata = np.random.rand(100, 20)\n\n# target variable\ntarget = np.random.choice([\"A\", \"B\"], size=n, replace=True)\n\n# array with variable on which to do a disjunct split\nsplit_var = np.random.choice([\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\", \"K\"],\n                             size=n, replace=True)\n\n# dict of variables to stratify on. Key names are arbitrary.\nstratif_vars = {\n    \"target\": target,\n    \"strat_var1\": np.random.choice([\"F\", \"G\"], size=n, replace=True),\n    \"strat_var2\": np.random.choice([\"H\", \"I\"], size=n, replace=True)\n}\n\n# ARGUMENTS\n# weight importance of all stratification variables in stratify_in\n# as well as of \"size_diff\", which punishes the deviation of intended\n# and received partition sizes.\n# Key names must match the names in stratif_vars.\nweights = {\n    \"target\": 2,\n    \"strat_var1\": 1,\n    \"strat_var2\": 1,\n    \"size_diff\": 1\n}\n\n# dev and test partition proportion (from 0 to 1)\ndev_size = .1\ntest_size = .1\n\n# number of disjunct splits to be tried out in brute force optimization\nk = 30\n\n# FIND OPTIMAL SPLIT\ntrain_i, dev_i, test_i, info = optimize_traindevtest_split(\n    X=data,\n    y=target,\n    split_on=split_var,\n    stratify_on=stratif_vars,\n    weight=weights,\n    dev_size=dev_size,\n    test_size=test_size,\n    k=k,\n    seed=seed\n)\n\n# SOME OUTPUT\nprint(\"test levels of split_var:\", sorted(set(split_var[test_i])))\nprint(\"goodness of split:\", info)\n```\n\n### <a name=\"example3\">Example 3: Split dummy data into training, development, and test partitions, the target and several stratification variables being numeric</a>\n\n* see `scripts/run_traindevtest_split_with_binning.py`\n* Partitions are\n    * disjunct on categorical \"split_var\"\n    * stratified on numeric \"target\", and on 3 other numeric stratification variables \n\n```python\nimport numpy as np\nimport os\nimport pandas as pd\nimport sys\n\n# add this line if you have cloned the code from github to PROJECT_DIR\n# sys.path.append(PROJECT_DIR)\n\nfrom splitutils import (\n    binning,\n    optimize_traindevtest_split\n)\n\n\"\"\"\nexample script how to split dummy data into training, development,\nand test partitions that are\n* disjunct on categorical \"split_var\"\n* stratified on numeric \"target\", and on 3 other numeric stratification\n  variables\n\"\"\"\n\n# set seed\nseed = 42\nnp.random.seed(seed)\n\n# DUMMY DATA\n# size\nn = 100\n\n# features\ndata = np.random.rand(n, 20)\n\n# numeric target variable\nnum_target = np.random.rand(n)\n\n# array with variable on which to do a disjunct split\nsplit_var = np.random.choice([\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\", \"K\"],\n                             size=n, replace=True)\n\n# further numeric variables to stratify on\nnum_strat_vars = np.random.rand(n, 3)\n\n# intrinsically bin target into 3 bins by equidistant\n# percentile boundaries\nbinned_target = binning(num_target, nbins=3)\n\n# ... alternatively, a variable can be extrinsically binned by\n# specifying lower boundaries:\n# binned_target = binning(target, lower_boundaries: [0, 0.33, 0.66])\n\n# bin other stratification variables into a single variable with 6 bins\n# (2-dim input is binned by StandardScaling and KMeans clustering)\nbinned_strat_var = binning(num_strat_vars, nbins=6)\n\n# ... alternatively, each stratification variable could be binned\n# individually - intrinsically or extrinsically the same way as num_target\n# strat_var1 = binning(num_strat_vars[:,0], nbins=...) etc.\n\n# now add the obtained categorical variable to stratification dict\nstratif_vars = {\n    \"target\": binned_target,\n    \"strat_var\": binned_strat_var\n}\n\n# ARGUMENTS\n# weight importance of all stratification variables in stratify_in\n# as well as of \"size_diff\", which punishes the deviation of intended\n# and received partition sizes\nweights = {\n    \"target\": 2,\n    \"strat_var\": 1,\n    \"size_diff\": 1\n}\n\n# dev and test partition proportion (from 0 to 1)\ndev_size = .1\ntest_size = .1\n\n# number of disjunct splits to be tried out in brute force optimization\nk = 30\n\n# FIND OPTIMAL SPLIT\ntrain_i, dev_i, test_i, info = optimize_traindevtest_split(\n    X=data,\n    y=num_target,\n    split_on=split_var,\n    stratify_on=stratif_vars,\n    weight=weights,\n    dev_size=dev_size,\n    test_size=test_size,\n    k=k,\n    seed=seed\n)\n\n# SOME OUTPUT\nprint(\"test levels of split_var:\", sorted(set(split_var[test_i])))\nprint(\"goodness of split:\", info)\n```\n\n## <a name=\"algorithm\">Algorithm</a>\n\n* find optimal train, dev, and test set split based on:\n    * disjunct split of a categorical grouping variable *G* (e.g. speaker)\n    * optimized joint stratification on an arbitrary amount of categorical target and grouping variables (e.g. emotion, gender, ...)\n    * close match of partition proportions in *G* and underlying dataset *X*\n* brute-force optimization on *k* disjunct splits of *G*\n* **score to be minimzed for train/test set split:**\n\n```\n(sum_v[w(v) * irad(v)] + w(d) * d) / (sum_v[w(v)] + w(d))\n\nv: variables to be stratified on\nw(v): their weight\nirad(v): information radius between reference and test set distribution of factor levels in v\nd: absolute difference between test proportions of X and G, i.e. between the proportion of test\n   samples and the proportion of groups (e.g. speakers) that go into the test set\nw(d): its weight\n```\n\n* **score to be minimzed for train / dev / test set split:**\n\n```\n(sum_v[w(v) * max_irad(v)] + w(d) * max_d) / (sum_v[w(v)] + w(d))\n\nv: variables to be stratified on\nw(v): their weight\nmax_irad(v): maximum information radius of reference distribution of classes in v and\n             - dev set distribution,\n             - test set distribution\nmax_d: maximum of absolute difference between proportions of X and G (see above) calculated for\n       the dev and test set\nw(d): its weight\n```\n\n## <a name=\"interpretation\">How to interprete the returned `info` dict</a>\n\n* let's look at [Example 2](#example2) above. There `info` becomes:\n\n```python\n{\n  'score': 0.030828359568603338,\n  'size_devset_in_spliton': 0.1,\n  'size_devset_in_X': 0.14,\n  'size_testset_in_spliton': 0.1,\n  'size_testset_in_X': 0.13,\n  'p_target_ref': {'B': 0.49, 'A': 0.51},\n  'p_target_dev': {'A': 0.5, 'B': 0.5},\n  'p_target_test': {'A': 0.5384615384615384, 'B': 0.46153846153846156},\n  'p_strat_var1_ref': {'G': 0.56, 'F': 0.44},\n  'p_strat_var1_dev': {'G': 0.5714285714285714, 'F': 0.42857142857142855},\n  'p_strat_var1_test': {'F': 0.5384615384615384, 'G': 0.46153846153846156},\n  'p_strat_var2_ref': {'I': 0.48, 'H': 0.52},\n  'p_strat_var2_dev': {'I': 0.5, 'H': 0.5},\n  'p_strat_var2_test': {'I': 0.46153846153846156, 'H': 0.5384615384615384}\n}\n```\n\n* **Explanations**\n    * **score:** see above, **score to be minimzed for train / dev / test set split:**\n    * **size_devset_in_spliton:** proportion of to-be-split-on variable levels in development set\n    * **size_devset_in_X:** proportion of rows in X in development set\n    * **size_testset_in_spliton:** proportion of to-be-split-on variable levels in test set\n    * **size_testset_in_X:** proportion of rows in X in test set\n    * **p_target_ref:** reference target class distribution over all data\n    * **p_target_dev:** target class distribution in development set\n    * **p_target_test:** target class distribution in test set\n    * **p_strat_var1_ref:** first stratification variable's reference distribution over all data\n    * **p_strat_var1_dev:** first stratification variable's class distribution in development set\n    * **p_strat_var1_test:** first stratification variable's class distribution in test set\n    * **p_strat_var2_ref:** second stratification variable's reference distribution over all data\n    * **p_strat_var2_dev:** second stratification variable's class distribution in development set\n    * **p_strat_var2_test:** second stratification variable's class distribution in test set\n* **Remarks**\n    * for `splitutils.optimize_traintest_split()` no development set results are reported\n    * all `*_strat_var*` keys: key names derived from key names in `stratify_on` argument\n\n## <a name=\"doc\">Further documentation</a>\n\n* Please find further details on the split scores and numeric variable binning in this [pdf](https://github.com/reichelu/splitutils/blob/main/docs/splitutils.pdf)\n\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "Group disjunct and stratified data splits",
    "version": "0.2.0",
    "project_urls": {
        "Homepage": "https://github.com/reichelu/splitutils",
        "company": "https://www.audeering.com/",
        "documentation": "https://github.com/reichelu/splitutils",
        "repository": "https://github.com/reichelu/splitutils"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "faa45c3bdbdca9f7225c0c6493cd449e1169bb62c83299c1136d56dda900dde8",
                "md5": "44b03bea016e8d13e0db569fdc4164ac",
                "sha256": "3fe9eb7d64609b1797e6657dd3666cf736d541d3265a219f2620a60c57a7f59e"
            },
            "downloads": -1,
            "filename": "splitutils-0.2.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "44b03bea016e8d13e0db569fdc4164ac",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 11962,
            "upload_time": "2023-12-01T08:57:54",
            "upload_time_iso_8601": "2023-12-01T08:57:54.989897Z",
            "url": "https://files.pythonhosted.org/packages/fa/a4/5c3bdbdca9f7225c0c6493cd449e1169bb62c83299c1136d56dda900dde8/splitutils-0.2.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "3cd0130ad38c5a53eb5bb70ff1b90f515f353498813c71efa73eb842f0716154",
                "md5": "e1b64c868eeb291f8549d2ae3821af11",
                "sha256": "939aa27a7e76d77b22a4eef7a537e54f743b93a5d69097703dadf6643d91b6c8"
            },
            "downloads": -1,
            "filename": "splitutils-0.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "e1b64c868eeb291f8549d2ae3821af11",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 15152,
            "upload_time": "2023-12-01T08:57:56",
            "upload_time_iso_8601": "2023-12-01T08:57:56.511455Z",
            "url": "https://files.pythonhosted.org/packages/3c/d0/130ad38c5a53eb5bb70ff1b90f515f353498813c71efa73eb842f0716154/splitutils-0.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-12-01 08:57:56",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "reichelu",
    "github_project": "splitutils",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "requirements": [],
    "lcname": "splitutils"
}
        
Elapsed time: 0.55526s