<div align="center">
PII Detection, Categorization, and Severity Assessment
[](https://www.python.org/)
[](https://GitHub.com/EdyVision/pii-codex/graphs/commit-activity)
[](https://codecov.io/gh/EdyVision/pii-codex)
[](https://opensource.org/licenses/BSD-3-Clause)
[](https://www.python.org/downloads/release/python-390/)
[](https://zenodo.org/badge/latestdoi/533554671)
[](https://joss.theoj.org/papers/5296a84bba0925e682dcddf14bec5880)
</div>
---
Author: Eidan Rosado - [@EdyVision](https://github.com/EdyVision) <br/>
Affiliation: Nova Southeastern University, College of Computing and Engineering
## Project Background
The <em>PII Codex</em> project was built as a core part of an ongoing research effort in Personal Identifiable Information (PII) detection and risk assessment (to be publicly released later in 2023). There was a need to not only detect PII in text, but also identify its severity, associated categorizations in cybersecurity research and policy documentation, and provide a way for others in similar research efforts to reproduce or extend the research. PII Codex is a combination of systematic research, conceptual frameworks, third-party open source software, and cloud service provider integrations. The categorizations are directly influenced by the research of Milne et al. (2016) while the ranking is a result of category severities on the scale provided by Schwartz and Solove (2012) from Non-Identifiable, Semi-Identifiable, and Identifiable.
The outputs of the primary PII Codex analysis and adapter functions are AnalysisResult or AnalysisResultSet objects that will provide a listing of detections, severities, mean risk scores for each string processed, and summary statistics on the analysis made. The final outputs do not contain the original texts but instead will provide where to find the detections should the end-user care for this information in their analysis.
### Statement of Need
The general knowledge base of identifiable data, the usage restrictions of this data, and the associated policies surrounding it have shifted drastically over the years. The tech industry has had to adjust to many policy changes regarding the tracking of individuals, the usage of data from online profiles and platforms, and the right to be forgotten entirely from a service or platform (GDPR). While the shift has provided data protections around the globe, the majority of technology users continue to have little to no control over their personal information with third-party data consumers (Trepte, 2020).
Understanding if identifiable data types exist in a data set can prevent accidental sharing of such data by allowing its detection in the first place and, in the case of this software package, present sanitized strings, the reasons to why the token was considered to be PII, and permit for the results to be publishable.
### Potential Usages
Potential usages include sanitizing of dataset strings (e.g. a collection of social media posts), presenting results to users for software examining their interactions (e.g. UX research on user-awareness in cybersecurity applications), etc.
<hr/>
## Running Locally with Poetry
This project uses Poetry. To run this project, install `poetry` and proceed to follow the instructions under `/docs/LOCAL_SETUP.md`.
`Note: This project has only been tested with Ubuntu and MacOS and with Python versions 3.9 and 3.10. You may need to upgrade pip ahead of installation.`
## Installing with PIP
Video capture of install provided in LOCAL_SETUP.md file. Make sure you set up a virtual environment with either python 3.9 or 3.10 and upgrade pip with:
```bash
pip install --upgrade pip
pip install -U pip setuptools wheel # only needed if you haven't already done so
```
Before adding `pii-codex` on your project, download the spaCy `en_core_web_lg` model:
```bash
pip install -U spacy
python3 -m spacy download en_core_web_lg
```
For more details on spaCy installation and usage, refer to their <a href="https://spacy.io/usage">docs</a>.
The repository releases are hosted on PyPi and can be installed with:
```bash
pip install pii-codex
pip install "pii-codex[detections]"
```
`Note: The extras installed with pii-codex[detections] are the spaCy, Micrisoft Presidio Analyzer, and Microsoft Anonymzer packages.`
Using Poetry:
```bash
poetry update
poetry add pii-codex
poetry install pii-codex --extras="detections"
```
For those using Google Collab, check out the example notebook:
[](https://colab.research.google.com/gist/EdyVision/802ce21aab21eb5d9afa9e43d301eef7/pii-codex-sample-notebook.ipynb)
## Usage
Video capture of usage provided in LOCAL_SETUP.md.
### Sample Input / Output
The built-in analyzer uses Microsoft Presidio. Feed in a collection of strings with analyze_collection() or just a single string with analyze_item(). Those analyzing a collection of strings will also be provided with statistics calculated on the risk scores for detected items.
```python
from pii_codex.services.analysis_service import PIIAnalysisService
PIIAnalysisService().analyze_collection(
texts=["your collection of strings"],
language_code="en",
collection_name="Data Set Label", # Optional Labeling
collection_type="SAMPLE" # Defaults to POPULATION, used stats calculations
)
```
You can also pass in a `data` param (dataframe) instead of simple text array with a text column and a metadata column to be analyzed for those analyzing social media posts. Current metadata supported are `URL`, `LOCATION`, and `SCREEN_NAME`.
Sample output (results object converted to `dict` from notebook):
```
{
"collection_name": "PII Collection 1",
"collection_type": "POPULATION",
"analyses": [
{
"analysis": [
{
"pii_type_detected": "PERSON",
"risk_level": 3,
"risk_level_definition": "Identifiable",
"cluster_membership_type": "Financial Information",
"hipaa_category": "Protected Health Information",
"dhs_category": "Linkable",
"nist_category": "Directly PII",
"entity_type": "PERSON",
"score": 0.85,
"start": 21,
"end": 24,
}
],
"index": 0,
"risk_score_mean": 3,
"sanitized_text: "Hi! My name is <REDACTED>",
},
...
],
"detection_count": 5,
"risk_scores": [3, 2.6666666666666665, 1, 2, 1],
"risk_score_mean": 1.9333333333333333,
"risk_score_mode": 1,
"risk_score_median": 2,
"risk_score_standard_deviation": 0.8273115763993905,
"risk_score_variance": 0.6844444444444444,
"detected_pii_types": {
"LOCATION",
"EMAIL_ADDRESS",
"URL",
"PHONE_NUMBER",
"PERSON",
},
"detected_pii_type_frequencies": {
"PERSON": 1,
"EMAIL_ADDRESS": 1,
"PHONE_NUMBER": 1,
"URL": 1,
"LOCATION": 1,
},
}
```
### Docs
For more information on usage, check out the respective documentation for guidance on using PII-Codex.
| Topic | Document | Description |
|-----------------------------|-----------------------------------------------------------------------|------------------------------------------------------------------------------------------|
| PII Type Mappings | [PII Mappings](docs/MAPPING.md) | Overview of how to perform mappings between PII types and how to review store PII types. |
| PII Detections and Analysis | [PII Detection and Analysis](docs/DETECTION_AND_ANALYSIS.md) | Overview of how to detect and analyze strings |
| Local Repo Setup | [Local Repo Setup](docs/LOCAL_SETUP.md) | Instructions for local repository setup |
| Example Analysis | [Example Analysis Notebook](notebooks/pii-analysis-ms-presidio.ipynb) | Notebook with example analysis using MSFT Presidio |
| PII-Codex Docs | docs/pii_codex/index.html | Autogenerated docs on classes, services, and models |
<hr/>
## Attributions
This project benefited greatly from a number of PII research works like that from Milne et al (2016) with the definition of the types and categories, Schwartz and Solove (2012) with the severity levels of Non-Identifiable, Semi-Identifiable, and Identifiable, and the documentation by NIST, DHS (2012), and HIPAA (full list of foundational publications provided below). A special thanks to all the open source projects, and frameworks that made the setup and structuring of this project much easier like Poetry, Microsoft Presidio, spaCy (2017), Jupyter, and several others.
### Foundational Publications
The following publications that inspired and provided a foundation for this repository:
| Concept | Document | Description |
|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
| PII Type Mappings | [Milne et al., (2016)](https://onlinelibrary.wiley.com/doi/abs/10.1111/joca.12111) | PII token categories and NIST and DHS categorizations. |
| Risk Continuum | [Schwartz & Solove, (2011)](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1909366) | Risk continuum concept and definition (what lead to the ranking in PII-Codex). |
| Privacy and Affordances | [Trepte, (2020)](https://academic.oup.com/ct/article-abstract/31/4/549/5828289?redirectedFrom=fulltext) | Third-Party data consumption and user control (lack thereof) background. |
| Social Media and Privacy | [Beigi & Liu, (2010)](https://dl.acm.org/doi/10.1145/3343038) | Privacy issues with social media and third-party data consumption. |
| Privacy Settings and Data Access | [Moura & Serrão, (2016)](https://www.researchgate.net/publication/332996823_Security_and_Privacy_Issues_of_Big_Data) | Privacy settings, data access, and unauthorized usage. |
| Information Privacy Review | [Bélanger & Crossler, (2011)](https://www.jstor.org/stable/41409971) | Concept of aggregation of data to identify individuals. |
| Big Data and Third Party Data Consumption | [Tene & Polonetsky, (2013)](https://www.researchgate.net/publication/256035043_Big_Data_for_All_Privacy_and_User_Control_in_the_Age_of_Analytics) | Third-party data usage, user control, and privacy. |
| PII and Confidentiality | [McCallister et al., (2010)](https://csrc.nist.gov/publications/detail/sp/800-122/final) | NIST guidance on PII confidentiality protections for federal agencies. |
| Data Capitalism and Privacy | [West, (2017)](https://journals.sagepub.com/doi/pdf/10.1177/0007650317718185) | Data capitalism, surveillance, and privacy. |
The remaining resources such as python library citations, cloud service provider docs, and cybersecurity guidelines are included in the paper.bib file.
## Community Guidelines
For community guidelines and contribution instructions, please view the [CONTRIBUTING.md](./CONTRIBUTING.md) file.
Raw data
{
"_id": null,
"home_page": "https://github.com/EdyVision/pii-codex",
"name": "pii-codex",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<3.11",
"maintainer_email": "",
"keywords": "PII,PII topology,risk categories,personal identifiable information,risk assessment",
"author": "Eidan J. Rosado",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/e2/0c/fc8f1be1e9b5f6c24b31805097ab227f7f58689f318f0162e328a78d98e9/pii_codex-0.4.6.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n\n\nPII Detection, Categorization, and Severity Assessment\n\n[](https://www.python.org/)\n\n[](https://GitHub.com/EdyVision/pii-codex/graphs/commit-activity)\n[](https://codecov.io/gh/EdyVision/pii-codex)\n[](https://opensource.org/licenses/BSD-3-Clause)\n[](https://www.python.org/downloads/release/python-390/)\n[](https://zenodo.org/badge/latestdoi/533554671)\n[](https://joss.theoj.org/papers/5296a84bba0925e682dcddf14bec5880)\n\n</div>\n\n---\n\nAuthor: Eidan Rosado - [@EdyVision](https://github.com/EdyVision) <br/>\nAffiliation: Nova Southeastern University, College of Computing and Engineering\n\n## Project Background\nThe <em>PII Codex</em> project was built as a core part of an ongoing research effort in Personal Identifiable Information (PII) detection and risk assessment (to be publicly released later in 2023). There was a need to not only detect PII in text, but also identify its severity, associated categorizations in cybersecurity research and policy documentation, and provide a way for others in similar research efforts to reproduce or extend the research. PII Codex is a combination of systematic research, conceptual frameworks, third-party open source software, and cloud service provider integrations. The categorizations are directly influenced by the research of Milne et al. (2016) while the ranking is a result of category severities on the scale provided by Schwartz and Solove (2012) from Non-Identifiable, Semi-Identifiable, and Identifiable.\n\nThe outputs of the primary PII Codex analysis and adapter functions are AnalysisResult or AnalysisResultSet objects that will provide a listing of detections, severities, mean risk scores for each string processed, and summary statistics on the analysis made. The final outputs do not contain the original texts but instead will provide where to find the detections should the end-user care for this information in their analysis.\n\n### Statement of Need\n\nThe general knowledge base of identifiable data, the usage restrictions of this data, and the associated policies surrounding it have shifted drastically over the years. The tech industry has had to adjust to many policy changes regarding the tracking of individuals, the usage of data from online profiles and platforms, and the right to be forgotten entirely from a service or platform (GDPR). While the shift has provided data protections around the globe, the majority of technology users continue to have little to no control over their personal information with third-party data consumers (Trepte, 2020). \n\nUnderstanding if identifiable data types exist in a data set can prevent accidental sharing of such data by allowing its detection in the first place and, in the case of this software package, present sanitized strings, the reasons to why the token was considered to be PII, and permit for the results to be publishable.\n\n### Potential Usages\nPotential usages include sanitizing of dataset strings (e.g. a collection of social media posts), presenting results to users for software examining their interactions (e.g. UX research on user-awareness in cybersecurity applications), etc.\n\n<hr/>\n\n## Running Locally with Poetry\nThis project uses Poetry. To run this project, install `poetry` and proceed to follow the instructions under `/docs/LOCAL_SETUP.md`.\n\n`Note: This project has only been tested with Ubuntu and MacOS and with Python versions 3.9 and 3.10. You may need to upgrade pip ahead of installation.`\n\n## Installing with PIP\nVideo capture of install provided in LOCAL_SETUP.md file. Make sure you set up a virtual environment with either python 3.9 or 3.10 and upgrade pip with:\n\n```bash\npip install --upgrade pip\npip install -U pip setuptools wheel # only needed if you haven't already done so \n```\n\nBefore adding `pii-codex` on your project, download the spaCy `en_core_web_lg` model:\n\n```bash\npip install -U spacy\npython3 -m spacy download en_core_web_lg\n```\n\nFor more details on spaCy installation and usage, refer to their <a href=\"https://spacy.io/usage\">docs</a>.\n\nThe repository releases are hosted on PyPi and can be installed with:\n\n```bash\npip install pii-codex\npip install \"pii-codex[detections]\"\n```\n\n`Note: The extras installed with pii-codex[detections] are the spaCy, Micrisoft Presidio Analyzer, and Microsoft Anonymzer packages.`\n\nUsing Poetry:\n\n```bash\npoetry update\npoetry add pii-codex\npoetry install pii-codex --extras=\"detections\"\n```\n\nFor those using Google Collab, check out the example notebook:\n\n[](https://colab.research.google.com/gist/EdyVision/802ce21aab21eb5d9afa9e43d301eef7/pii-codex-sample-notebook.ipynb)\n\n## Usage\nVideo capture of usage provided in LOCAL_SETUP.md.\n\n### Sample Input / Output\nThe built-in analyzer uses Microsoft Presidio. Feed in a collection of strings with analyze_collection() or just a single string with analyze_item(). Those analyzing a collection of strings will also be provided with statistics calculated on the risk scores for detected items.\n```python\nfrom pii_codex.services.analysis_service import PIIAnalysisService\nPIIAnalysisService().analyze_collection(\n texts=[\"your collection of strings\"],\n language_code=\"en\",\n collection_name=\"Data Set Label\", # Optional Labeling\n collection_type=\"SAMPLE\" # Defaults to POPULATION, used stats calculations\n)\n```\n\nYou can also pass in a `data` param (dataframe) instead of simple text array with a text column and a metadata column to be analyzed for those analyzing social media posts. Current metadata supported are `URL`, `LOCATION`, and `SCREEN_NAME`.\n\nSample output (results object converted to `dict` from notebook):\n```\n{\n \"collection_name\": \"PII Collection 1\",\n \"collection_type\": \"POPULATION\",\n \"analyses\": [\n {\n \"analysis\": [\n {\n \"pii_type_detected\": \"PERSON\",\n \"risk_level\": 3,\n \"risk_level_definition\": \"Identifiable\",\n \"cluster_membership_type\": \"Financial Information\",\n \"hipaa_category\": \"Protected Health Information\",\n \"dhs_category\": \"Linkable\",\n \"nist_category\": \"Directly PII\",\n \"entity_type\": \"PERSON\",\n \"score\": 0.85,\n \"start\": 21,\n \"end\": 24,\n }\n ],\n \"index\": 0,\n \"risk_score_mean\": 3,\n \"sanitized_text: \"Hi! My name is <REDACTED>\",\n },\n ...\n ],\n \"detection_count\": 5,\n \"risk_scores\": [3, 2.6666666666666665, 1, 2, 1],\n \"risk_score_mean\": 1.9333333333333333,\n \"risk_score_mode\": 1,\n \"risk_score_median\": 2,\n \"risk_score_standard_deviation\": 0.8273115763993905,\n \"risk_score_variance\": 0.6844444444444444,\n \"detected_pii_types\": {\n \"LOCATION\",\n \"EMAIL_ADDRESS\",\n \"URL\",\n \"PHONE_NUMBER\",\n \"PERSON\",\n },\n \"detected_pii_type_frequencies\": {\n \"PERSON\": 1,\n \"EMAIL_ADDRESS\": 1,\n \"PHONE_NUMBER\": 1,\n \"URL\": 1,\n \"LOCATION\": 1,\n },\n}\n```\n\n### Docs\nFor more information on usage, check out the respective documentation for guidance on using PII-Codex.\n\n| Topic | Document | Description |\n|-----------------------------|-----------------------------------------------------------------------|------------------------------------------------------------------------------------------|\n| PII Type Mappings | [PII Mappings](docs/MAPPING.md) | Overview of how to perform mappings between PII types and how to review store PII types. |\n| PII Detections and Analysis | [PII Detection and Analysis](docs/DETECTION_AND_ANALYSIS.md) | Overview of how to detect and analyze strings |\n| Local Repo Setup | [Local Repo Setup](docs/LOCAL_SETUP.md) | Instructions for local repository setup |\n| Example Analysis | [Example Analysis Notebook](notebooks/pii-analysis-ms-presidio.ipynb) | Notebook with example analysis using MSFT Presidio |\n| PII-Codex Docs | docs/pii_codex/index.html | Autogenerated docs on classes, services, and models |\n\n<hr/>\n\n## Attributions\nThis project benefited greatly from a number of PII research works like that from Milne et al (2016) with the definition of the types and categories, Schwartz and Solove (2012) with the severity levels of Non-Identifiable, Semi-Identifiable, and Identifiable, and the documentation by NIST, DHS (2012), and HIPAA (full list of foundational publications provided below). A special thanks to all the open source projects, and frameworks that made the setup and structuring of this project much easier like Poetry, Microsoft Presidio, spaCy (2017), Jupyter, and several others.\n\n### Foundational Publications\nThe following publications that inspired and provided a foundation for this repository:\n\n| Concept | Document | Description |\n|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|\n| PII Type Mappings | [Milne et al., (2016)](https://onlinelibrary.wiley.com/doi/abs/10.1111/joca.12111) | PII token categories and NIST and DHS categorizations. |\n| Risk Continuum | [Schwartz & Solove, (2011)](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1909366) | Risk continuum concept and definition (what lead to the ranking in PII-Codex). |\n| Privacy and Affordances | [Trepte, (2020)](https://academic.oup.com/ct/article-abstract/31/4/549/5828289?redirectedFrom=fulltext) | Third-Party data consumption and user control (lack thereof) background. |\n| Social Media and Privacy | [Beigi & Liu, (2010)](https://dl.acm.org/doi/10.1145/3343038) | Privacy issues with social media and third-party data consumption. |\n| Privacy Settings and Data Access | [Moura & Serr\u00e3o, (2016)](https://www.researchgate.net/publication/332996823_Security_and_Privacy_Issues_of_Big_Data) | Privacy settings, data access, and unauthorized usage. |\n| Information Privacy Review | [B\u00e9langer & Crossler, (2011)](https://www.jstor.org/stable/41409971) | Concept of aggregation of data to identify individuals. |\n| Big Data and Third Party Data Consumption | [Tene & Polonetsky, (2013)](https://www.researchgate.net/publication/256035043_Big_Data_for_All_Privacy_and_User_Control_in_the_Age_of_Analytics) | Third-party data usage, user control, and privacy. |\n| PII and Confidentiality | [McCallister et al., (2010)](https://csrc.nist.gov/publications/detail/sp/800-122/final) | NIST guidance on PII confidentiality protections for federal agencies. |\n| Data Capitalism and Privacy | [West, (2017)](https://journals.sagepub.com/doi/pdf/10.1177/0007650317718185) | Data capitalism, surveillance, and privacy. |\n\nThe remaining resources such as python library citations, cloud service provider docs, and cybersecurity guidelines are included in the paper.bib file.\n\n## Community Guidelines\nFor community guidelines and contribution instructions, please view the [CONTRIBUTING.md](./CONTRIBUTING.md) file.",
"bugtrack_url": null,
"license": "BSD 3-Clause",
"summary": "PII Detection, Categorization, and Severity Assessment",
"version": "0.4.6",
"project_urls": {
"Homepage": "https://github.com/EdyVision/pii-codex",
"Repository": "https://github.com/EdyVision/pii-codex"
},
"split_keywords": [
"pii",
"pii topology",
"risk categories",
"personal identifiable information",
"risk assessment"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "54985fb31b050c69ce7bcd9e1a4800571f4ee8b06e4d7dcd34b77f66d3907db4",
"md5": "321055c99940c08dcf6fb192bbebcdad",
"sha256": "8976b810a3ab933601d8ba8c65d60c6a3ee0f7dd4ecfd699ed984600b33a64e5"
},
"downloads": -1,
"filename": "pii_codex-0.4.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "321055c99940c08dcf6fb192bbebcdad",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<3.11",
"size": 34304,
"upload_time": "2023-06-18T23:38:41",
"upload_time_iso_8601": "2023-06-18T23:38:41.865870Z",
"url": "https://files.pythonhosted.org/packages/54/98/5fb31b050c69ce7bcd9e1a4800571f4ee8b06e4d7dcd34b77f66d3907db4/pii_codex-0.4.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e20cfc8f1be1e9b5f6c24b31805097ab227f7f58689f318f0162e328a78d98e9",
"md5": "bb824990700cc7547e5e9a438072b801",
"sha256": "e28eafe5daa6429172616820cf37789df0541047f8689849da2c7393c413aa08"
},
"downloads": -1,
"filename": "pii_codex-0.4.6.tar.gz",
"has_sig": false,
"md5_digest": "bb824990700cc7547e5e9a438072b801",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<3.11",
"size": 29936,
"upload_time": "2023-06-18T23:38:43",
"upload_time_iso_8601": "2023-06-18T23:38:43.231826Z",
"url": "https://files.pythonhosted.org/packages/e2/0c/fc8f1be1e9b5f6c24b31805097ab227f7f58689f318f0162e328a78d98e9/pii_codex-0.4.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-18 23:38:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EdyVision",
"github_project": "pii-codex",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pii-codex"
}
JSON
