This package enables core facilities for performing PPRL based on Bloom filters in Python.
It is mostly backed by the [bitarray](https://github.com/ilanschnell/bitarray) package which implements memory-efficient
arrays of bits in Python.
This package is composed of several submodules which implement different aspects of performing PPRL.
It is used by the [PPRL service package](https://github.com/ul-mds/pprl/tree/main/packages/pprl_service) under the hood
to power its PPRL capabilities.
# Bitarray primitives
`pprl_core.bits` contains functions for setting bits in a bitarray.
It implements the double hash, enhanced double hash, triple hash and random hash schemes for setting bits based on
a set number of initial hash values in a bitarray.
It also offers other utility functions for working on PPRL with Bloom filters.
```python
from bitarray import bitarray
from pprl_core import bits
ba = bitarray(20)
# These are all equivalent and result in the bit with the index 5 to be set.
bits.set_bit(ba, 5)
bits.set_bit(ba, 25)
bits.set_bit(ba, -6)
# These are also equivalent and will return True.
bits.test_bit(ba, 5)
bits.test_bit(ba, 25)
bits.test_bit(ba, -6)
# pprl_core.bits implements the double hash, enhanced double hash, random hash and triple hash schemes.
# Depending on chosen scheme, the corresponding functions require different initial hash values.
h0, h1, h2 = 13, 37, 42
k = 5
ba_double = bitarray(32)
bits.double_hash(ba_double, k, h0, h1)
print(ba_double)
# => bitarray('01000010000000000010000100001000')
ba_enhanced_double = bitarray(32)
bits.enhanced_double_hash(ba_enhanced_double, k, h0, h1)
print(ba_enhanced_double)
# => bitarray('10000000000100000010000010100000')
ba_triple = bitarray(32)
bits.triple_hash(ba_triple, k, h0, h1, h2)
print(ba_triple)
# => bitarray('01000000001000000010000000100100')
from random import Random
ba_random = bitarray(32)
bits.random_hash(ba_random, k, Random(h0))
print(ba_random)
# => bitarray('00000000010100101010000000000000')
# Compute the size of a bitarray such that a certain percentage of its bits are set after
# a number of bits are picked at random and set. In this example, the percentage is set to 50%
# and the amount of random bit sets is 100.
print(bits.optimal_size(.5, 100))
# => 145
# Serialize and deserialize a bitarray into a Base64-encoded form. The size of a deserialized
# bitarray may not always be the same size of the bitarray that generated the Base64 representation.
# This is because the deserialization will always return a bitarray whose length is a multiple of 8.
ba = bitarray("0010101110101001001010110101011101010010100000011101010100111100")
ba_b64_str = bits.to_base64(ba)
print(ba_b64_str)
# => "K6krV1KB1Tw="
ba_from_b64 = bits.from_base64(ba_b64_str)
print(ba == ba_from_b64)
# => True
```
# Hardening
`pprl_core.harden` contains factory functions for creating hardeners that can be applied to bitarrays.
These functions are guaranteed to always return a modified copy of the bitarrays they are supposed to harden.
They will never modify the input bitarrays.
```python
from pprl_core import harden
from random import Random
from bitarray import bitarray
# Create a new random bitarray.
rng = Random(727)
ba = bitarray([rng.random() < 0.5 for _ in range(64)])
print(ba)
# => bitarray('0000010100000100110010111001010101001000111110011011100100101000')
# Harden a bitarray by balancing its bits. With an original bitarray size of 64, the bitarray
# is expanded to a size of 128 in which 50% of its bits should be set. So the resulting bit
# count should be 64.
harden_balance = harden.balance()
ba_balance = harden_balance(ba.copy())
print(ba_balance)
# => bitarray('00000101000001001100101110010101010010001111100110111001001010001111101011111011001101000110101010110111000001100100011011010111')
print(ba.count(), ba_balance.count())
# => 27 64
# Harden a bitarray by performing XOR-folding. The resulting bitarray size should be half of the
# original bitarray.
harden_xor = harden.xor_fold()
ba_xor = harden_xor(ba.copy())
print(ba_xor)
# => bitarray('01001101111111010111001010111101')
print(len(ba), len(ba_xor))
# => 64 32
# Harden a bitarray by flipping bits using "randomized response". Performing an XOR of the resulting
# bitarray with the original bitarray will reveal the bits that have been modified as a result of this
# operation. This hardener requires a function that returns a random number generator and a probability
# with which a bit may be modified.
harden_rand_resp = harden.randomized_response(lambda: Random(727 * 2), .5)
ba_rand_resp = harden_rand_resp(ba.copy())
print(ba_rand_resp)
# => bitarray('0000010110000010110011001101110001001000111111011011101100001000')
print(ba ^ ba_rand_resp)
# => bitarray('0000000010000110000001110100100100000000000001000000001000100000')
# Harden a bitarray by randomly permuting its bits. This hardener requires a function that returns a
# random number generator for selecting bits to swap.
harden_permute = harden.permute(lambda: Random(727 * 3))
ba_permute = harden_permute(ba.copy())
print(ba_permute)
# => bitarray('0010110010110010101011110001010111000110001001001110000100001000')
# Harden a bitarray by having all bits be the result of an XOR of its left and right neighbors.
harden_rule_90 = harden.rule_90()
ba_rule_90 = harden_rule_90(ba.copy())
print(ba_rule_90)
# => bitarray('0000100010001011111100101110000000110101100011111010111011000100')
# Harden a bitarray by moving a sliding window over the bitarray which is used to instantiate a
# random number generator to draw random bits to mutate. In this example, the sliding window has
# a size of 8 bits and moves forward 4 bits after 2 random bits have been mutated.
harden_rehash = harden.rehash(8, 4, 2)
ba_rehash = harden_rehash(ba.copy())
print(ba_rehash)
# => bitarray('0000110101011110110110111001011111111010111111011011110100111000')
```
# Bitarray similarity
`pprl_core.similarity` contains functions for computing the similarity of bitarrays.
It implements the Dice coefficient, Cosine similarity and the Jaccard index.
```python
from pprl_core import similarity
from random import Random
from bitarray import bitarray
# Create new random bitarrays.
rng = Random(727)
ba_1 = bitarray([rng.random() < 0.5 for _ in range(32)])
ba_2 = bitarray([rng.random() < 0.5 for _ in range(32)])
print(ba_1)
# => bitarray('00000101000001001100101110010101')
print(ba_2)
# => bitarray('01001000111110011011100100101000')
# For all similarity functions, let n1 and n2 be the amount of set bits in ba_1 and ba_2 respectively,
# and let n12 be the amount of set bits in the intersection of ba_1 and ba_2.
# In ba_1 and ba_2, there are only 3 positions where bits are set in both bitarrays. Each similarity
# function will treat this a bit differently.
print((ba_1 & ba_2).count())
# => 3
# Dice coefficient (2 * n12 / (n1 + n2))
print(similarity.dice(ba_1, ba_2))
# => 0.2222222222222222
# Cosine similarity (n12 / sqrt(n1 * n2))
print(similarity.cosine(ba_1, ba_2))
# => 0.22360679774997896
# Jaccard index (n12 / (n1 + n2 - n12))
print(similarity.jaccard(ba_1, ba_2))
# => 0.125
```
# String transformation
`pprl_core.transform` contains factory functions for performing preprocessing on strings.
```python
from pprl_core import transform
# String normalization performs several steps. All non-ASCII characters are replaced with their
# closest ASCII variants. Unicode normalization in the NFKD form is performed. Non-ASCII characters
# are removed. All characters are converted to their lowercase counterparts and consecutive whitespaces
# are reduced to a single one.
normalize = transform.normalize()
print(normalize("Müller-Ludenscheidt"))
# => "muller-ludenscheidt"
# Character filtering allows for the definition of a sequence of characters which must be removed
# from a string.
character_filter = transform.character_filter("äöüß")
print(character_filter("Müller-Ludenscheidt"))
# => "Mller-Ludenscheidt"
# Number formatting takes in any numeric string and reduces or expands it to a set amount of decimal places.
number_zero_digits = transform.number(0)
number_six_digits = transform.number(6)
print(number_zero_digits("12.34"))
# => "12"
print(number_six_digits("12.34"))
# => "12.340000"
# Date time formatting takes in date and time strings in a set format and outputs in another.
date_time = transform.date_time("%Y-%m-%d", "%d.%m.%Y")
print(date_time("2024-06-29"))
# => "29.06.2024"
# Phonetic code transformation applies a phonetic code on an input string. It uses the pyphonetics
# library under the hood.
from pyphonetics import Soundex
phonetic_code = transform.phonetic_code(Soundex())
print(phonetic_code("Müller-Ludenscheidt"))
# => "M464"
# Mapping transformation allows for single characters or entire character sequences to be
# replaced with another.
mapping = transform.mapping({
"male": "m",
"female": "f"
})
print(mapping("male"))
# => "m"
# If no default value is set and no mapping is present, this will throw an error.
print(mapping("unknown"))
# => ValueError: value `unknown` has no mapping, or no default value is present
mapping_with_default = transform.mapping({
"male": "m",
"female": "f"
}, default_val="u")
# Setting a default value will prevent this error.
print(mapping("unknown"))
# => "u"
# By default, only entire strings are mapped. For inline transformations, set the corresponding
# parameter to True.
mapping_inline = transform.mapping({
"ä": "ae",
"ö": "oe",
"ü": "ue",
"ß": "ss"
}, inline=True)
print(mapping_inline("Müller-Ludenscheidt"))
# => "Mueller-Ludenscheidt"
```
# Additional phonetic codes
`pprl_core.phonetics_extra` contains additional phonetic code implementations that are compatible with
[pyphonetics](https://github.com/Lilykos/pyphonetics).
At the time, only the "Kölner Phonetik" is implemented, which is a phonetic code that is specialized
for the German language.
```python
from pprl_core import phonetics_extra
cologne = phonetics_extra.ColognePhonetics()
print(cologne.phonetics("Müller-Ludenscheidt"))
# => "65752682"
```
# License
MIT.
Raw data
{
"_id": null,
"home_page": "https://github.com/ul-mds/pprl",
"name": "pprl-core",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "record linkage, privacy, bloom filter, bitarray, cryptography",
"author": "Maximilian Jugl",
"author_email": "Maximilian.Jugl@medizin.uni-leipzig.de",
"download_url": "https://files.pythonhosted.org/packages/de/af/b06a93a684ef3efcc0ca53368e6bd9c755d5978cb0c1acb9c4bbd572384a/pprl_core-0.1.3.tar.gz",
"platform": null,
"description": "This package enables core facilities for performing PPRL based on Bloom filters in Python.\nIt is mostly backed by the [bitarray](https://github.com/ilanschnell/bitarray) package which implements memory-efficient\narrays of bits in Python.\nThis package is composed of several submodules which implement different aspects of performing PPRL.\nIt is used by the [PPRL service package](https://github.com/ul-mds/pprl/tree/main/packages/pprl_service) under the hood \nto power its PPRL capabilities.\n\n# Bitarray primitives\n\n`pprl_core.bits` contains functions for setting bits in a bitarray.\nIt implements the double hash, enhanced double hash, triple hash and random hash schemes for setting bits based on\na set number of initial hash values in a bitarray.\nIt also offers other utility functions for working on PPRL with Bloom filters.\n\n```python\nfrom bitarray import bitarray\nfrom pprl_core import bits\n\nba = bitarray(20)\n\n# These are all equivalent and result in the bit with the index 5 to be set.\nbits.set_bit(ba, 5)\nbits.set_bit(ba, 25)\nbits.set_bit(ba, -6)\n\n# These are also equivalent and will return True.\nbits.test_bit(ba, 5)\nbits.test_bit(ba, 25)\nbits.test_bit(ba, -6)\n\n# pprl_core.bits implements the double hash, enhanced double hash, random hash and triple hash schemes.\n# Depending on chosen scheme, the corresponding functions require different initial hash values.\nh0, h1, h2 = 13, 37, 42\nk = 5\n\nba_double = bitarray(32)\nbits.double_hash(ba_double, k, h0, h1)\nprint(ba_double)\n# => bitarray('01000010000000000010000100001000')\n\nba_enhanced_double = bitarray(32)\nbits.enhanced_double_hash(ba_enhanced_double, k, h0, h1)\nprint(ba_enhanced_double)\n# => bitarray('10000000000100000010000010100000')\n\nba_triple = bitarray(32)\nbits.triple_hash(ba_triple, k, h0, h1, h2)\nprint(ba_triple)\n# => bitarray('01000000001000000010000000100100')\n\nfrom random import Random\n\nba_random = bitarray(32)\nbits.random_hash(ba_random, k, Random(h0))\nprint(ba_random)\n# => bitarray('00000000010100101010000000000000')\n\n# Compute the size of a bitarray such that a certain percentage of its bits are set after\n# a number of bits are picked at random and set. In this example, the percentage is set to 50% \n# and the amount of random bit sets is 100.\nprint(bits.optimal_size(.5, 100))\n# => 145\n\n# Serialize and deserialize a bitarray into a Base64-encoded form. The size of a deserialized\n# bitarray may not always be the same size of the bitarray that generated the Base64 representation.\n# This is because the deserialization will always return a bitarray whose length is a multiple of 8.\nba = bitarray(\"0010101110101001001010110101011101010010100000011101010100111100\")\nba_b64_str = bits.to_base64(ba)\nprint(ba_b64_str)\n# => \"K6krV1KB1Tw=\"\nba_from_b64 = bits.from_base64(ba_b64_str)\nprint(ba == ba_from_b64)\n# => True\n```\n\n# Hardening\n\n`pprl_core.harden` contains factory functions for creating hardeners that can be applied to bitarrays.\nThese functions are guaranteed to always return a modified copy of the bitarrays they are supposed to harden.\nThey will never modify the input bitarrays.\n\n```python\nfrom pprl_core import harden\nfrom random import Random\nfrom bitarray import bitarray\n\n# Create a new random bitarray.\nrng = Random(727)\nba = bitarray([rng.random() < 0.5 for _ in range(64)])\nprint(ba)\n# => bitarray('0000010100000100110010111001010101001000111110011011100100101000')\n\n# Harden a bitarray by balancing its bits. With an original bitarray size of 64, the bitarray\n# is expanded to a size of 128 in which 50% of its bits should be set. So the resulting bit\n# count should be 64.\nharden_balance = harden.balance()\nba_balance = harden_balance(ba.copy())\nprint(ba_balance)\n# => bitarray('00000101000001001100101110010101010010001111100110111001001010001111101011111011001101000110101010110111000001100100011011010111')\nprint(ba.count(), ba_balance.count())\n# => 27 64\n\n# Harden a bitarray by performing XOR-folding. The resulting bitarray size should be half of the\n# original bitarray.\nharden_xor = harden.xor_fold()\nba_xor = harden_xor(ba.copy())\nprint(ba_xor)\n# => bitarray('01001101111111010111001010111101')\nprint(len(ba), len(ba_xor))\n# => 64 32\n\n# Harden a bitarray by flipping bits using \"randomized response\". Performing an XOR of the resulting\n# bitarray with the original bitarray will reveal the bits that have been modified as a result of this\n# operation. This hardener requires a function that returns a random number generator and a probability\n# with which a bit may be modified.\nharden_rand_resp = harden.randomized_response(lambda: Random(727 * 2), .5)\nba_rand_resp = harden_rand_resp(ba.copy())\nprint(ba_rand_resp)\n# => bitarray('0000010110000010110011001101110001001000111111011011101100001000')\nprint(ba ^ ba_rand_resp)\n# => bitarray('0000000010000110000001110100100100000000000001000000001000100000')\n\n# Harden a bitarray by randomly permuting its bits. This hardener requires a function that returns a\n# random number generator for selecting bits to swap.\nharden_permute = harden.permute(lambda: Random(727 * 3))\nba_permute = harden_permute(ba.copy())\nprint(ba_permute)\n# => bitarray('0010110010110010101011110001010111000110001001001110000100001000')\n\n# Harden a bitarray by having all bits be the result of an XOR of its left and right neighbors.\nharden_rule_90 = harden.rule_90()\nba_rule_90 = harden_rule_90(ba.copy())\nprint(ba_rule_90)\n# => bitarray('0000100010001011111100101110000000110101100011111010111011000100')\n\n# Harden a bitarray by moving a sliding window over the bitarray which is used to instantiate a \n# random number generator to draw random bits to mutate. In this example, the sliding window has \n# a size of 8 bits and moves forward 4 bits after 2 random bits have been mutated.\nharden_rehash = harden.rehash(8, 4, 2)\nba_rehash = harden_rehash(ba.copy())\nprint(ba_rehash)\n# => bitarray('0000110101011110110110111001011111111010111111011011110100111000')\n```\n\n# Bitarray similarity\n\n`pprl_core.similarity` contains functions for computing the similarity of bitarrays.\nIt implements the Dice coefficient, Cosine similarity and the Jaccard index.\n\n```python\nfrom pprl_core import similarity\nfrom random import Random\nfrom bitarray import bitarray\n\n# Create new random bitarrays.\nrng = Random(727)\nba_1 = bitarray([rng.random() < 0.5 for _ in range(32)])\nba_2 = bitarray([rng.random() < 0.5 for _ in range(32)])\nprint(ba_1)\n# => bitarray('00000101000001001100101110010101')\nprint(ba_2)\n# => bitarray('01001000111110011011100100101000')\n\n# For all similarity functions, let n1 and n2 be the amount of set bits in ba_1 and ba_2 respectively,\n# and let n12 be the amount of set bits in the intersection of ba_1 and ba_2.\n\n# In ba_1 and ba_2, there are only 3 positions where bits are set in both bitarrays. Each similarity\n# function will treat this a bit differently.\nprint((ba_1 & ba_2).count())\n# => 3\n\n# Dice coefficient (2 * n12 / (n1 + n2))\nprint(similarity.dice(ba_1, ba_2))\n# => 0.2222222222222222\n\n# Cosine similarity (n12 / sqrt(n1 * n2))\nprint(similarity.cosine(ba_1, ba_2))\n# => 0.22360679774997896\n\n# Jaccard index (n12 / (n1 + n2 - n12))\nprint(similarity.jaccard(ba_1, ba_2))\n# => 0.125\n```\n\n# String transformation\n\n`pprl_core.transform` contains factory functions for performing preprocessing on strings.\n\n```python\nfrom pprl_core import transform\n\n# String normalization performs several steps. All non-ASCII characters are replaced with their\n# closest ASCII variants. Unicode normalization in the NFKD form is performed. Non-ASCII characters\n# are removed. All characters are converted to their lowercase counterparts and consecutive whitespaces \n# are reduced to a single one.\nnormalize = transform.normalize()\nprint(normalize(\"M\u00fcller-Ludenscheidt\"))\n# => \"muller-ludenscheidt\"\n\n# Character filtering allows for the definition of a sequence of characters which must be removed\n# from a string.\ncharacter_filter = transform.character_filter(\"\u00e4\u00f6\u00fc\u00df\")\nprint(character_filter(\"M\u00fcller-Ludenscheidt\"))\n# => \"Mller-Ludenscheidt\"\n\n# Number formatting takes in any numeric string and reduces or expands it to a set amount of decimal places.\nnumber_zero_digits = transform.number(0)\nnumber_six_digits = transform.number(6)\nprint(number_zero_digits(\"12.34\"))\n# => \"12\"\nprint(number_six_digits(\"12.34\"))\n# => \"12.340000\"\n\n# Date time formatting takes in date and time strings in a set format and outputs in another.\ndate_time = transform.date_time(\"%Y-%m-%d\", \"%d.%m.%Y\")\nprint(date_time(\"2024-06-29\"))\n# => \"29.06.2024\"\n\n# Phonetic code transformation applies a phonetic code on an input string. It uses the pyphonetics\n# library under the hood.\nfrom pyphonetics import Soundex\n\nphonetic_code = transform.phonetic_code(Soundex())\nprint(phonetic_code(\"M\u00fcller-Ludenscheidt\"))\n# => \"M464\"\n\n# Mapping transformation allows for single characters or entire character sequences to be\n# replaced with another. \nmapping = transform.mapping({\n \"male\": \"m\",\n \"female\": \"f\"\n})\n\nprint(mapping(\"male\"))\n# => \"m\"\n\n# If no default value is set and no mapping is present, this will throw an error.\nprint(mapping(\"unknown\"))\n# => ValueError: value `unknown` has no mapping, or no default value is present\n\nmapping_with_default = transform.mapping({\n \"male\": \"m\",\n \"female\": \"f\"\n}, default_val=\"u\")\n\n# Setting a default value will prevent this error.\nprint(mapping(\"unknown\"))\n# => \"u\"\n\n# By default, only entire strings are mapped. For inline transformations, set the corresponding \n# parameter to True.\nmapping_inline = transform.mapping({\n \"\u00e4\": \"ae\",\n \"\u00f6\": \"oe\",\n \"\u00fc\": \"ue\",\n \"\u00df\": \"ss\"\n}, inline=True)\n\nprint(mapping_inline(\"M\u00fcller-Ludenscheidt\"))\n# => \"Mueller-Ludenscheidt\"\n```\n\n# Additional phonetic codes\n\n`pprl_core.phonetics_extra` contains additional phonetic code implementations that are compatible with\n[pyphonetics](https://github.com/Lilykos/pyphonetics).\nAt the time, only the \"K\u00f6lner Phonetik\" is implemented, which is a phonetic code that is specialized\nfor the German language.\n\n```python\nfrom pprl_core import phonetics_extra\n\ncologne = phonetics_extra.ColognePhonetics()\nprint(cologne.phonetics(\"M\u00fcller-Ludenscheidt\"))\n# => \"65752682\"\n```\n\n# License\n\nMIT.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Facilities for performing privacy-preserving record linkage with Bloom filters.",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/ul-mds/pprl",
"Repository": "https://github.com/ul-mds/pprl"
},
"split_keywords": [
"record linkage",
" privacy",
" bloom filter",
" bitarray",
" cryptography"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3cc8e682ad85b569c8c07c29ce3ef9e3175423751c3b30eccaad6ed61b8eec7e",
"md5": "6e7a6039f2c0dfbaf067900ed531c823",
"sha256": "7b416630bbd758b694aa690cc175f582232d3a435a0ac4680c9eb5ed6bbbfe31"
},
"downloads": -1,
"filename": "pprl_core-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6e7a6039f2c0dfbaf067900ed531c823",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 13489,
"upload_time": "2024-09-16T09:03:26",
"upload_time_iso_8601": "2024-09-16T09:03:26.820675Z",
"url": "https://files.pythonhosted.org/packages/3c/c8/e682ad85b569c8c07c29ce3ef9e3175423751c3b30eccaad6ed61b8eec7e/pprl_core-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "deafb06a93a684ef3efcc0ca53368e6bd9c755d5978cb0c1acb9c4bbd572384a",
"md5": "73cc5e19588e17e3c0c2d5de892a5238",
"sha256": "a82b05597430fef6711e934abbb072d89096160eea076658bc8379871d517689"
},
"downloads": -1,
"filename": "pprl_core-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "73cc5e19588e17e3c0c2d5de892a5238",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 14351,
"upload_time": "2024-09-16T09:03:28",
"upload_time_iso_8601": "2024-09-16T09:03:28.114496Z",
"url": "https://files.pythonhosted.org/packages/de/af/b06a93a684ef3efcc0ca53368e6bd9c755d5978cb0c1acb9c4bbd572384a/pprl_core-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-16 09:03:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ul-mds",
"github_project": "pprl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pprl-core"
}
JSON
