| Name | allyanonimiser JSON |
| Version |
2.1.0
JSON |
| download |
| home_page | https://github.com/srepho/Allyanonimiser |
| Summary | Australian-focused PII detection and anonymization for the insurance industry |
| upload_time | 2025-03-03 04:22:48 |
| maintainer | None |
| docs_url | None |
| author | Stephen Oates |
| requires_python | >=3.10 |
| license | MIT License
Copyright (c) 2025 Stephen Oates
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
|
| keywords |
pii
anonymization
privacy
insurance
australia
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
spacy
presidio-analyzer
presidio-anonymizer
pytest
pytest-cov
black
isort
flake8
mypy
en-core-web-lg
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Allyanonimiser
[](https://pypi.org/project/allyanonimiser/2.1.0/)
[](https://pypi.org/project/allyanonimiser/)
[](https://github.com/srepho/Allyanonimiser/actions/workflows/tests.yml)
[](https://github.com/srepho/Allyanonimiser/actions/workflows/package.yml)
[](https://opensource.org/licenses/MIT)
Australian-focused PII detection and anonymization for the insurance industry with support for stream processing of very large files.
## Installation
```bash
# Basic installation
pip install allyanonimiser==2.1.0
# With stream processing support for large files
pip install "allyanonimiser[stream]==2.1.0"
# With LLM integration for advanced pattern generation
pip install "allyanonimiser[llm]==2.1.0"
# Complete installation with all optional dependencies
pip install "allyanonimiser[stream,llm]==2.1.0"
```
**Prerequisites:**
- Python 3.10 or higher
- A spaCy language model (recommended):
```bash
python -m spacy download en_core_web_lg # Recommended
# OR for limited resources:
python -m spacy download en_core_web_sm # Smaller model
```
## Quick Start
```python
from allyanonimiser import create_allyanonimiser
# Create the Allyanonimiser instance with default settings
ally = create_allyanonimiser()
# Analyze text
results = ally.analyze(
text="Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123."
)
# Print results
for result in results:
print(f"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}")
# Anonymize text
anonymized = ally.anonymize(
text="Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123.",
operators={
"POLICY_NUMBER": "mask", # Replace with asterisks
"VEHICLE_REGISTRATION": "replace" # Replace with <VEHICLE_REGISTRATION>
}
)
print(anonymized["text"])
# Output: "Please reference your policy ********** for claims related to your vehicle rego <VEHICLE_REGISTRATION>."
```
### Adding Custom Patterns
```python
from allyanonimiser import create_allyanonimiser
# Create an Allyanonimiser instance
ally = create_allyanonimiser()
# Add a custom pattern with regex
ally.add_pattern({
"entity_type": "REFERENCE_CODE",
"patterns": [r"REF-\d{6}-[A-Z]{2}", r"Reference\s+#\d{6}"],
"context": ["reference", "code", "ref"],
"name": "Reference Code"
})
# Generate a pattern from examples
ally.create_pattern_from_examples(
entity_type="EMPLOYEE_ID",
examples=["EMP00123", "EMP45678", "EMP98765"],
context=["employee", "staff", "id"],
generalization_level="medium"
)
# Test custom patterns
text = "Employee EMP12345 created REF-123456-AB for the project."
results = ally.analyze(text)
for result in results:
print(f"Found {result.entity_type}: {result.text}")
```
## New in Version 2.0.0: Comprehensive Reporting System
Allyanonimiser now includes a comprehensive reporting system that allows you to track, analyze, and visualize anonymization activities.
```python
from allyanonimiser import create_allyanonimiser
# Create instance
ally = create_allyanonimiser()
# Start a new report session
ally.start_new_report("my_session")
# Process multiple texts
texts = [
"Customer John Smith (DOB: 15/06/1980) called about claim CL-12345.",
"Jane Doe (email: jane.doe@example.com) requested policy information.",
"Claims assessor reviewed case for Robert Johnson (ID: 987654321)."
]
for i, text in enumerate(texts):
ally.anonymize(
text=text,
operators={
"PERSON": "replace",
"EMAIL_ADDRESS": "mask",
"DATE_OF_BIRTH": "age_bracket"
},
document_id=f"doc_{i+1}"
)
# Get report summary
report = ally.get_report()
summary = report.get_summary()
print(f"Total documents processed: {summary['total_documents']}")
print(f"Total entities detected: {summary['total_entities']}")
print(f"Entities per document: {summary['entities_per_document']:.2f}")
print(f"Anonymization rate: {summary['anonymization_rate']*100:.2f}%")
print(f"Average processing time: {summary['avg_processing_time']*1000:.2f} ms")
# Export report to different formats
report.export_report("report.html", "html") # Rich HTML visualization
report.export_report("report.json", "json") # Detailed JSON data
report.export_report("report.csv", "csv") # CSV statistics
# In Jupyter notebooks, display rich visualizations
# ally.display_report_in_notebook()
```
## Features
- **Australian-Focused PII Detection**: Specialized patterns for TFNs, Medicare numbers, vehicle registrations, addresses, and more
- **Insurance Industry Specialization**: Detect policy numbers, claim references, and other industry-specific identifiers
- **Multiple Entity Types**: Comprehensive detection of general and specialized PII
- **Flexible Anonymization**: Multiple anonymization operators (replace, mask, redact, hash, and more)
- **Stream Processing**: Memory-efficient processing of large files with chunking support
- **Reporting System**: Comprehensive tracking and visualization of anonymization activities
- **Jupyter Integration**: Rich visualization capabilities in notebook environments
- **DataFrame Support**: Process pandas DataFrames with batch processing and multi-processing support
## Built-in Pattern Reference
### Australian Patterns
| Entity Type | Description | Example Pattern | Pattern File |
|-------------|-------------|----------------|-------------|
| AU_TFN | Australian Tax File Number | `\b\d{3}\s*\d{3}\s*\d{3}\b` | au_patterns.py |
| AU_PHONE | Australian Phone Number | `\b(?:\+?61\|0)4\d{2}\s*\d{3}\s*\d{3}\b` | au_patterns.py |
| AU_MEDICARE | Australian Medicare Number | `\b\d{4}\s*\d{5}\s*\d{1}\b` | au_patterns.py |
| AU_DRIVERS_LICENSE | Australian Driver's License | Various formats including<br>`\b\d{8}\b` (NSW)<br>`\b\d{4}[a-zA-Z]{2}\b` (NSW legacy) | au_patterns.py |
| AU_ADDRESS | Australian Address | Address patterns with street names | au_patterns.py |
| AU_POSTCODE | Australian Postcode | `\b\d{4}\b` | au_patterns.py |
| AU_BSB_ACCOUNT | Australian BSB and Account Number | `\b\d{3}-\d{3}\s*\d{6,10}\b` | au_patterns.py |
| AU_ABN | Australian Business Number | `\b\d{2}\s*\d{3}\s*\d{3}\s*\d{3}\b` | au_patterns.py |
| AU_PASSPORT | Australian Passport Number | `\b[A-Za-z]\d{8}\b` | au_patterns.py |
### General Patterns
| Entity Type | Description | Example Pattern | Pattern File |
|-------------|-------------|----------------|-------------|
| CREDIT_CARD | Credit Card Number | `\b\d{4}[\s-]\d{4}[\s-]\d{4}[\s-]\d{4}\b` | general_patterns.py |
| PERSON | Person Name | Name patterns with context | general_patterns.py |
| EMAIL_ADDRESS | Email Address | `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z\|a-z]{2,}\b` | general_patterns.py |
| DATE_OF_BIRTH | Date of Birth | `\bDOB:\s*\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4}\b` | general_patterns.py |
| LOCATION | Location | City and location patterns | general_patterns.py |
| DATE | Date | `\b\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4}\b` | general_patterns.py |
| MONETARY_VALUE | Money Amount | `\$\d{1,3}(?:,\d{3})*(?:\.\d{2})?\b` | general_patterns.py |
| ORGANIZATION | Organization | Organization name patterns | general_patterns.py |
### Insurance Patterns
| Entity Type | Description | Example Pattern | Pattern File |
|-------------|-------------|----------------|-------------|
| INSURANCE_POLICY_NUMBER | Insurance Policy Number | `\b(?:POL\|P\|Policy)[- ]?\d{6,9}\b` | insurance_patterns.py |
| INSURANCE_CLAIM_NUMBER | Insurance Claim Number | `\b(?:CL\|C)[- ]?\d{6,9}\b` | insurance_patterns.py |
| INSURANCE_MEMBER_NUMBER | Insurance Member Number | Member ID patterns | insurance_patterns.py |
| INSURANCE_GROUP_NUMBER | Group Policy Number | Group policy patterns | insurance_patterns.py |
| VEHICLE_IDENTIFIER | Vehicle ID (VIN, plates) | `\b[A-HJ-NPR-Z0-9]{17}\b` | insurance_patterns.py |
| CASE_REFERENCE | Case Reference Numbers | Case ID patterns | insurance_patterns.py |
| VEHICLE_DETAILS | Vehicle Details | Make/model patterns | insurance_patterns.py |
## Usage Examples
### Pattern Management
```python
from allyanonimiser import create_allyanonimiser
# Create an instance
ally = create_allyanonimiser()
# 1. Adding pattern to an existing group in pattern files
# If you want to contribute a new pattern to the codebase,
# edit the appropriate file in patterns/ directory:
# - patterns/au_patterns.py: For Australian-specific patterns
# - patterns/general_patterns.py: For general PII patterns
# - patterns/insurance_patterns.py: For insurance-specific patterns
# 2. Using custom patterns without modifying code
# Add a custom pattern with detailed options
ally.add_pattern({
"entity_type": "COMPANY_PROJECT_ID",
"patterns": [
r"PRJ-\d{4}-[A-Z]{3}", # Format: PRJ-1234-ABC
r"Project\s+ID:\s*(\d{4})" # Format: Project ID: 1234
],
"context": ["project", "id", "identifier", "code"],
"name": "Company Project ID",
"score": 0.85, # Confidence score (0-1)
"language": "en", # Language code
"description": "Internal project identifier format"
})
# 3. Save patterns for reuse
ally.export_config("company_patterns.json")
# 4. Load saved patterns in another session
new_ally = create_allyanonimiser(settings_path="company_patterns.json")
# 5. Pattern testing and validation
from allyanonimiser.validators import test_pattern_against_examples
# Test if a pattern works against examples
results = test_pattern_against_examples(
pattern=r"PRJ-\d{4}-[A-Z]{3}",
positive_examples=["PRJ-1234-ABC", "PRJ-5678-XYZ"],
negative_examples=["PRJ-123-AB", "PROJECT-1234"]
)
print(f"Pattern is valid: {results['is_valid']}")
print(f"Diagnostic info: {results['message']}")
```
### Analyze Text for PII Entities
```python
from allyanonimiser import create_allyanonimiser
ally = create_allyanonimiser()
# Analyze text
results = ally.analyze(
text="Customer John Smith (TFN: 123 456 789) reported an incident on 15/06/2023 at his residence in Sydney NSW 2000."
)
# Print detected entities
for result in results:
print(f"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}")
```
### Anonymize Text with Different Operators
```python
from allyanonimiser import create_allyanonimiser, AnonymizationConfig
ally = create_allyanonimiser()
# Using configuration object
config = AnonymizationConfig(
operators={
"PERSON": "replace", # Replace with <PERSON>
"AU_TFN": "hash", # Replace with SHA-256 hash
"DATE": "redact", # Replace with [REDACTED]
"AU_ADDRESS": "mask", # Replace with *****
"DATE_OF_BIRTH": "age_bracket" # Replace with age bracket (e.g., <40-45>)
},
age_bracket_size=5 # Size of age brackets
)
# Anonymize text
anonymized = ally.anonymize(
text="Customer John Smith (TFN: 123 456 789) reported an incident on 15/06/2023. He was born on 05/08/1982 and lives at 42 Main St, Sydney NSW 2000.",
config=config
)
print(anonymized["text"])
```
### Process Text with Analysis and Anonymization
```python
from allyanonimiser import create_allyanonimiser, AnalysisConfig, AnonymizationConfig
ally = create_allyanonimiser()
# Configure analysis
analysis_config = AnalysisConfig(
active_entity_types=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "DATE_OF_BIRTH"],
min_score_threshold=0.7
)
# Configure anonymization
anonymization_config = AnonymizationConfig(
operators={
"PERSON": "replace",
"EMAIL_ADDRESS": "mask",
"PHONE_NUMBER": "redact",
"DATE_OF_BIRTH": "age_bracket"
}
)
# Process text (analyze + anonymize)
result = ally.process(
text="Customer Jane Doe (jane.doe@example.com) called on 0412-345-678 regarding her DOB: 22/05/1990.",
analysis_config=analysis_config,
anonymization_config=anonymization_config
)
# Access different parts of the result
print("Anonymized text:")
print(result["anonymized"])
print("\nDetected entities:")
for entity in result["analysis"]["entities"]:
print(f"{entity['entity_type']}: {entity['text']} (score: {entity['score']:.2f})")
print("\nPII-rich segments:")
for segment in result["segments"]:
print(f"Original: {segment['text']}")
print(f"Anonymized: {segment['anonymized']}")
```
## Working with DataFrames
```python
import pandas as pd
from allyanonimiser import create_allyanonimiser
# Create DataFrame
df = pd.DataFrame({
"id": [1, 2, 3],
"notes": [
"Customer John Smith (DOB: 15/6/1980) called about policy POL-123456.",
"Jane Doe (email: jane.doe@example.com) requested a refund.",
"Alex Johnson from Sydney NSW 2000 reported an incident."
]
})
# Create Allyanonimiser
ally = create_allyanonimiser()
# Anonymize a specific column
anonymized_df = ally.process_dataframe(
df,
column="notes",
operation="anonymize",
output_column="anonymized_notes", # New column for anonymized text
operators={
"PERSON": "replace",
"EMAIL_ADDRESS": "mask",
"PHONE_NUMBER": "redact"
}
)
# Display result
print(anonymized_df[["id", "notes", "anonymized_notes"]])
```
## Generating Reports
```python
from allyanonimiser import create_allyanonimiser
import os
# Create output directory
os.makedirs("output", exist_ok=True)
# Create an Allyanonimiser instance
ally = create_allyanonimiser()
# Start a new report session
ally.start_new_report(session_id="example_session")
# Configure anonymization
anonymization_config = {
"operators": {
"PERSON": "replace",
"EMAIL_ADDRESS": "mask",
"PHONE_NUMBER": "redact",
"AU_ADDRESS": "replace",
"DATE_OF_BIRTH": "age_bracket",
"AU_TFN": "hash",
"AU_MEDICARE": "mask"
},
"age_bracket_size": 10
}
# Process a batch of files
result = ally.process_files(
file_paths=["data/sample1.txt", "data/sample2.txt", "data/sample3.txt"],
output_dir="output/anonymized",
anonymize=True,
operators=anonymization_config["operators"],
report=True,
report_output="output/report.html",
report_format="html"
)
# Display summary
print(f"Processed {result['total_files']} files")
print(f"Detected {result['report']['total_entities']} entities")
print(f"Average processing time: {result['report']['avg_processing_time']*1000:.2f} ms")
```
## In Jupyter Notebooks
```python
from allyanonimiser import create_allyanonimiser
import pandas as pd
import matplotlib.pyplot as plt
# Create an Allyanonimiser instance
ally = create_allyanonimiser()
# Start a report session and process some texts
# ... processing code ...
# Display rich interactive report
ally.display_report_in_notebook()
# Access report data programmatically
report = ally.get_report()
summary = report.get_summary()
# Create custom visualizations
entity_counts = summary['entity_counts']
plt.figure(figsize=(10, 6))
plt.bar(entity_counts.keys(), entity_counts.values())
plt.title('Entity Type Distribution')
plt.xlabel('Entity Type')
plt.ylabel('Count')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
```
## Documentation
For complete documentation, examples, and advanced usage, visit the [GitHub repository](https://github.com/srepho/Allyanonimiser).
## License
This project is licensed under the MIT License - see the LICENSE file for details.
Raw data
{
"_id": null,
"home_page": "https://github.com/srepho/Allyanonimiser",
"name": "allyanonimiser",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "pii, anonymization, privacy, insurance, australia",
"author": "Stephen Oates",
"author_email": "Stephen Oates <stephen.j.a.oates@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/60/f2/99e863719ad25d7cc6a2349f48b08d1dae406aec5e1b4112f8a1c9268fa6/allyanonimiser-2.1.0.tar.gz",
"platform": null,
"description": "# Allyanonimiser\n\n[](https://pypi.org/project/allyanonimiser/2.1.0/)\n[](https://pypi.org/project/allyanonimiser/)\n[](https://github.com/srepho/Allyanonimiser/actions/workflows/tests.yml)\n[](https://github.com/srepho/Allyanonimiser/actions/workflows/package.yml)\n[](https://opensource.org/licenses/MIT)\n\nAustralian-focused PII detection and anonymization for the insurance industry with support for stream processing of very large files.\n\n## Installation\n\n```bash\n# Basic installation\npip install allyanonimiser==2.1.0\n\n# With stream processing support for large files\npip install \"allyanonimiser[stream]==2.1.0\"\n\n# With LLM integration for advanced pattern generation\npip install \"allyanonimiser[llm]==2.1.0\"\n\n# Complete installation with all optional dependencies\npip install \"allyanonimiser[stream,llm]==2.1.0\"\n```\n\n**Prerequisites:**\n- Python 3.10 or higher\n- A spaCy language model (recommended):\n ```bash\n python -m spacy download en_core_web_lg # Recommended\n # OR for limited resources:\n python -m spacy download en_core_web_sm # Smaller model\n ```\n\n## Quick Start\n\n```python\nfrom allyanonimiser import create_allyanonimiser\n\n# Create the Allyanonimiser instance with default settings\nally = create_allyanonimiser()\n\n# Analyze text\nresults = ally.analyze(\n text=\"Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123.\"\n)\n\n# Print results\nfor result in results:\n print(f\"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}\")\n\n# Anonymize text\nanonymized = ally.anonymize(\n text=\"Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123.\",\n operators={\n \"POLICY_NUMBER\": \"mask\", # Replace with asterisks\n \"VEHICLE_REGISTRATION\": \"replace\" # Replace with <VEHICLE_REGISTRATION>\n }\n)\n\nprint(anonymized[\"text\"])\n# Output: \"Please reference your policy ********** for claims related to your vehicle rego <VEHICLE_REGISTRATION>.\"\n```\n\n### Adding Custom Patterns\n\n```python\nfrom allyanonimiser import create_allyanonimiser\n\n# Create an Allyanonimiser instance\nally = create_allyanonimiser()\n\n# Add a custom pattern with regex\nally.add_pattern({\n \"entity_type\": \"REFERENCE_CODE\",\n \"patterns\": [r\"REF-\\d{6}-[A-Z]{2}\", r\"Reference\\s+#\\d{6}\"],\n \"context\": [\"reference\", \"code\", \"ref\"],\n \"name\": \"Reference Code\"\n})\n\n# Generate a pattern from examples\nally.create_pattern_from_examples(\n entity_type=\"EMPLOYEE_ID\",\n examples=[\"EMP00123\", \"EMP45678\", \"EMP98765\"],\n context=[\"employee\", \"staff\", \"id\"],\n generalization_level=\"medium\"\n)\n\n# Test custom patterns\ntext = \"Employee EMP12345 created REF-123456-AB for the project.\"\nresults = ally.analyze(text)\nfor result in results:\n print(f\"Found {result.entity_type}: {result.text}\")\n```\n\n## New in Version 2.0.0: Comprehensive Reporting System\n\nAllyanonimiser now includes a comprehensive reporting system that allows you to track, analyze, and visualize anonymization activities.\n\n```python\nfrom allyanonimiser import create_allyanonimiser\n\n# Create instance\nally = create_allyanonimiser()\n\n# Start a new report session\nally.start_new_report(\"my_session\")\n\n# Process multiple texts\ntexts = [\n \"Customer John Smith (DOB: 15/06/1980) called about claim CL-12345.\",\n \"Jane Doe (email: jane.doe@example.com) requested policy information.\",\n \"Claims assessor reviewed case for Robert Johnson (ID: 987654321).\"\n]\n\nfor i, text in enumerate(texts):\n ally.anonymize(\n text=text,\n operators={\n \"PERSON\": \"replace\",\n \"EMAIL_ADDRESS\": \"mask\",\n \"DATE_OF_BIRTH\": \"age_bracket\"\n },\n document_id=f\"doc_{i+1}\"\n )\n\n# Get report summary\nreport = ally.get_report()\nsummary = report.get_summary()\n\nprint(f\"Total documents processed: {summary['total_documents']}\")\nprint(f\"Total entities detected: {summary['total_entities']}\")\nprint(f\"Entities per document: {summary['entities_per_document']:.2f}\")\nprint(f\"Anonymization rate: {summary['anonymization_rate']*100:.2f}%\")\nprint(f\"Average processing time: {summary['avg_processing_time']*1000:.2f} ms\")\n\n# Export report to different formats\nreport.export_report(\"report.html\", \"html\") # Rich HTML visualization\nreport.export_report(\"report.json\", \"json\") # Detailed JSON data\nreport.export_report(\"report.csv\", \"csv\") # CSV statistics\n\n# In Jupyter notebooks, display rich visualizations\n# ally.display_report_in_notebook()\n```\n\n## Features\n\n- **Australian-Focused PII Detection**: Specialized patterns for TFNs, Medicare numbers, vehicle registrations, addresses, and more\n- **Insurance Industry Specialization**: Detect policy numbers, claim references, and other industry-specific identifiers\n- **Multiple Entity Types**: Comprehensive detection of general and specialized PII\n- **Flexible Anonymization**: Multiple anonymization operators (replace, mask, redact, hash, and more)\n- **Stream Processing**: Memory-efficient processing of large files with chunking support\n- **Reporting System**: Comprehensive tracking and visualization of anonymization activities\n- **Jupyter Integration**: Rich visualization capabilities in notebook environments\n- **DataFrame Support**: Process pandas DataFrames with batch processing and multi-processing support\n\n## Built-in Pattern Reference\n\n### Australian Patterns\n\n| Entity Type | Description | Example Pattern | Pattern File |\n|-------------|-------------|----------------|-------------|\n| AU_TFN | Australian Tax File Number | `\\b\\d{3}\\s*\\d{3}\\s*\\d{3}\\b` | au_patterns.py |\n| AU_PHONE | Australian Phone Number | `\\b(?:\\+?61\\|0)4\\d{2}\\s*\\d{3}\\s*\\d{3}\\b` | au_patterns.py |\n| AU_MEDICARE | Australian Medicare Number | `\\b\\d{4}\\s*\\d{5}\\s*\\d{1}\\b` | au_patterns.py |\n| AU_DRIVERS_LICENSE | Australian Driver's License | Various formats including<br>`\\b\\d{8}\\b` (NSW)<br>`\\b\\d{4}[a-zA-Z]{2}\\b` (NSW legacy) | au_patterns.py |\n| AU_ADDRESS | Australian Address | Address patterns with street names | au_patterns.py |\n| AU_POSTCODE | Australian Postcode | `\\b\\d{4}\\b` | au_patterns.py |\n| AU_BSB_ACCOUNT | Australian BSB and Account Number | `\\b\\d{3}-\\d{3}\\s*\\d{6,10}\\b` | au_patterns.py |\n| AU_ABN | Australian Business Number | `\\b\\d{2}\\s*\\d{3}\\s*\\d{3}\\s*\\d{3}\\b` | au_patterns.py |\n| AU_PASSPORT | Australian Passport Number | `\\b[A-Za-z]\\d{8}\\b` | au_patterns.py |\n\n### General Patterns\n\n| Entity Type | Description | Example Pattern | Pattern File |\n|-------------|-------------|----------------|-------------|\n| CREDIT_CARD | Credit Card Number | `\\b\\d{4}[\\s-]\\d{4}[\\s-]\\d{4}[\\s-]\\d{4}\\b` | general_patterns.py |\n| PERSON | Person Name | Name patterns with context | general_patterns.py |\n| EMAIL_ADDRESS | Email Address | `\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z\\|a-z]{2,}\\b` | general_patterns.py |\n| DATE_OF_BIRTH | Date of Birth | `\\bDOB:\\s*\\d{1,2}[/.-]\\d{1,2}[/.-]\\d{2,4}\\b` | general_patterns.py |\n| LOCATION | Location | City and location patterns | general_patterns.py |\n| DATE | Date | `\\b\\d{1,2}[/.-]\\d{1,2}[/.-]\\d{2,4}\\b` | general_patterns.py |\n| MONETARY_VALUE | Money Amount | `\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{2})?\\b` | general_patterns.py |\n| ORGANIZATION | Organization | Organization name patterns | general_patterns.py |\n\n### Insurance Patterns\n\n| Entity Type | Description | Example Pattern | Pattern File |\n|-------------|-------------|----------------|-------------|\n| INSURANCE_POLICY_NUMBER | Insurance Policy Number | `\\b(?:POL\\|P\\|Policy)[- ]?\\d{6,9}\\b` | insurance_patterns.py |\n| INSURANCE_CLAIM_NUMBER | Insurance Claim Number | `\\b(?:CL\\|C)[- ]?\\d{6,9}\\b` | insurance_patterns.py |\n| INSURANCE_MEMBER_NUMBER | Insurance Member Number | Member ID patterns | insurance_patterns.py |\n| INSURANCE_GROUP_NUMBER | Group Policy Number | Group policy patterns | insurance_patterns.py |\n| VEHICLE_IDENTIFIER | Vehicle ID (VIN, plates) | `\\b[A-HJ-NPR-Z0-9]{17}\\b` | insurance_patterns.py |\n| CASE_REFERENCE | Case Reference Numbers | Case ID patterns | insurance_patterns.py |\n| VEHICLE_DETAILS | Vehicle Details | Make/model patterns | insurance_patterns.py |\n\n## Usage Examples\n\n### Pattern Management\n\n```python\nfrom allyanonimiser import create_allyanonimiser\n\n# Create an instance\nally = create_allyanonimiser()\n\n# 1. Adding pattern to an existing group in pattern files\n# If you want to contribute a new pattern to the codebase,\n# edit the appropriate file in patterns/ directory:\n# - patterns/au_patterns.py: For Australian-specific patterns\n# - patterns/general_patterns.py: For general PII patterns \n# - patterns/insurance_patterns.py: For insurance-specific patterns\n\n# 2. Using custom patterns without modifying code\n# Add a custom pattern with detailed options\nally.add_pattern({\n \"entity_type\": \"COMPANY_PROJECT_ID\",\n \"patterns\": [\n r\"PRJ-\\d{4}-[A-Z]{3}\", # Format: PRJ-1234-ABC\n r\"Project\\s+ID:\\s*(\\d{4})\" # Format: Project ID: 1234\n ],\n \"context\": [\"project\", \"id\", \"identifier\", \"code\"],\n \"name\": \"Company Project ID\",\n \"score\": 0.85, # Confidence score (0-1)\n \"language\": \"en\", # Language code\n \"description\": \"Internal project identifier format\"\n})\n\n# 3. Save patterns for reuse\nally.export_config(\"company_patterns.json\")\n\n# 4. Load saved patterns in another session\nnew_ally = create_allyanonimiser(settings_path=\"company_patterns.json\")\n\n# 5. Pattern testing and validation\nfrom allyanonimiser.validators import test_pattern_against_examples\n\n# Test if a pattern works against examples\nresults = test_pattern_against_examples(\n pattern=r\"PRJ-\\d{4}-[A-Z]{3}\",\n positive_examples=[\"PRJ-1234-ABC\", \"PRJ-5678-XYZ\"],\n negative_examples=[\"PRJ-123-AB\", \"PROJECT-1234\"]\n)\nprint(f\"Pattern is valid: {results['is_valid']}\")\nprint(f\"Diagnostic info: {results['message']}\")\n```\n\n### Analyze Text for PII Entities\n\n```python\nfrom allyanonimiser import create_allyanonimiser\n\nally = create_allyanonimiser()\n\n# Analyze text\nresults = ally.analyze(\n text=\"Customer John Smith (TFN: 123 456 789) reported an incident on 15/06/2023 at his residence in Sydney NSW 2000.\"\n)\n\n# Print detected entities\nfor result in results:\n print(f\"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}\")\n```\n\n### Anonymize Text with Different Operators\n\n```python\nfrom allyanonimiser import create_allyanonimiser, AnonymizationConfig\n\nally = create_allyanonimiser()\n\n# Using configuration object\nconfig = AnonymizationConfig(\n operators={\n \"PERSON\": \"replace\", # Replace with <PERSON>\n \"AU_TFN\": \"hash\", # Replace with SHA-256 hash\n \"DATE\": \"redact\", # Replace with [REDACTED]\n \"AU_ADDRESS\": \"mask\", # Replace with *****\n \"DATE_OF_BIRTH\": \"age_bracket\" # Replace with age bracket (e.g., <40-45>)\n },\n age_bracket_size=5 # Size of age brackets\n)\n\n# Anonymize text\nanonymized = ally.anonymize(\n text=\"Customer John Smith (TFN: 123 456 789) reported an incident on 15/06/2023. He was born on 05/08/1982 and lives at 42 Main St, Sydney NSW 2000.\",\n config=config\n)\n\nprint(anonymized[\"text\"])\n```\n\n### Process Text with Analysis and Anonymization\n\n```python\nfrom allyanonimiser import create_allyanonimiser, AnalysisConfig, AnonymizationConfig\n\nally = create_allyanonimiser()\n\n# Configure analysis\nanalysis_config = AnalysisConfig(\n active_entity_types=[\"PERSON\", \"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"DATE_OF_BIRTH\"],\n min_score_threshold=0.7\n)\n\n# Configure anonymization\nanonymization_config = AnonymizationConfig(\n operators={\n \"PERSON\": \"replace\",\n \"EMAIL_ADDRESS\": \"mask\",\n \"PHONE_NUMBER\": \"redact\",\n \"DATE_OF_BIRTH\": \"age_bracket\"\n }\n)\n\n# Process text (analyze + anonymize)\nresult = ally.process(\n text=\"Customer Jane Doe (jane.doe@example.com) called on 0412-345-678 regarding her DOB: 22/05/1990.\",\n analysis_config=analysis_config,\n anonymization_config=anonymization_config\n)\n\n# Access different parts of the result\nprint(\"Anonymized text:\")\nprint(result[\"anonymized\"])\n\nprint(\"\\nDetected entities:\")\nfor entity in result[\"analysis\"][\"entities\"]:\n print(f\"{entity['entity_type']}: {entity['text']} (score: {entity['score']:.2f})\")\n\nprint(\"\\nPII-rich segments:\")\nfor segment in result[\"segments\"]:\n print(f\"Original: {segment['text']}\")\n print(f\"Anonymized: {segment['anonymized']}\")\n```\n\n## Working with DataFrames\n\n```python\nimport pandas as pd\nfrom allyanonimiser import create_allyanonimiser\n\n# Create DataFrame\ndf = pd.DataFrame({\n \"id\": [1, 2, 3],\n \"notes\": [\n \"Customer John Smith (DOB: 15/6/1980) called about policy POL-123456.\",\n \"Jane Doe (email: jane.doe@example.com) requested a refund.\",\n \"Alex Johnson from Sydney NSW 2000 reported an incident.\"\n ]\n})\n\n# Create Allyanonimiser\nally = create_allyanonimiser()\n\n# Anonymize a specific column\nanonymized_df = ally.process_dataframe(\n df, \n column=\"notes\", \n operation=\"anonymize\",\n output_column=\"anonymized_notes\", # New column for anonymized text\n operators={\n \"PERSON\": \"replace\",\n \"EMAIL_ADDRESS\": \"mask\",\n \"PHONE_NUMBER\": \"redact\"\n }\n)\n\n# Display result\nprint(anonymized_df[[\"id\", \"notes\", \"anonymized_notes\"]])\n```\n\n## Generating Reports\n\n```python\nfrom allyanonimiser import create_allyanonimiser\nimport os\n\n# Create output directory\nos.makedirs(\"output\", exist_ok=True)\n\n# Create an Allyanonimiser instance\nally = create_allyanonimiser()\n\n# Start a new report session\nally.start_new_report(session_id=\"example_session\")\n\n# Configure anonymization\nanonymization_config = {\n \"operators\": {\n \"PERSON\": \"replace\",\n \"EMAIL_ADDRESS\": \"mask\",\n \"PHONE_NUMBER\": \"redact\",\n \"AU_ADDRESS\": \"replace\",\n \"DATE_OF_BIRTH\": \"age_bracket\",\n \"AU_TFN\": \"hash\",\n \"AU_MEDICARE\": \"mask\"\n },\n \"age_bracket_size\": 10\n}\n\n# Process a batch of files\nresult = ally.process_files(\n file_paths=[\"data/sample1.txt\", \"data/sample2.txt\", \"data/sample3.txt\"],\n output_dir=\"output/anonymized\",\n anonymize=True,\n operators=anonymization_config[\"operators\"],\n report=True,\n report_output=\"output/report.html\",\n report_format=\"html\"\n)\n\n# Display summary\nprint(f\"Processed {result['total_files']} files\")\nprint(f\"Detected {result['report']['total_entities']} entities\")\nprint(f\"Average processing time: {result['report']['avg_processing_time']*1000:.2f} ms\")\n```\n\n## In Jupyter Notebooks\n\n```python\nfrom allyanonimiser import create_allyanonimiser\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n# Create an Allyanonimiser instance\nally = create_allyanonimiser()\n\n# Start a report session and process some texts\n# ... processing code ...\n\n# Display rich interactive report\nally.display_report_in_notebook()\n\n# Access report data programmatically\nreport = ally.get_report()\nsummary = report.get_summary()\n\n# Create custom visualizations\nentity_counts = summary['entity_counts']\nplt.figure(figsize=(10, 6))\nplt.bar(entity_counts.keys(), entity_counts.values())\nplt.title('Entity Type Distribution')\nplt.xlabel('Entity Type')\nplt.ylabel('Count')\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()\n```\n\n## Documentation\n\nFor complete documentation, examples, and advanced usage, visit the [GitHub repository](https://github.com/srepho/Allyanonimiser).\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n",
"bugtrack_url": null,
"license": "MIT License\n \n Copyright (c) 2025 Stephen Oates\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.\n ",
"summary": "Australian-focused PII detection and anonymization for the insurance industry",
"version": "2.1.0",
"project_urls": {
"Bug Tracker": "https://github.com/srepho/Allyanonimiser/issues",
"Documentation": "https://github.com/srepho/Allyanonimiser#readme",
"Homepage": "https://github.com/srepho/Allyanonimiser"
},
"split_keywords": [
"pii",
" anonymization",
" privacy",
" insurance",
" australia"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "b7ede2b54fd770530fd9872011968839dec152e739245338237b0051e2840b04",
"md5": "5efa2f25d00cd556b99bdc681fcf6fe8",
"sha256": "5e9d28e5baf3dac4c1cfc36de296d0e7f5e82a1a789e13e0a1243f09da249dba"
},
"downloads": -1,
"filename": "allyanonimiser-2.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5efa2f25d00cd556b99bdc681fcf6fe8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 205735,
"upload_time": "2025-03-03T04:22:47",
"upload_time_iso_8601": "2025-03-03T04:22:47.114255Z",
"url": "https://files.pythonhosted.org/packages/b7/ed/e2b54fd770530fd9872011968839dec152e739245338237b0051e2840b04/allyanonimiser-2.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "60f299e863719ad25d7cc6a2349f48b08d1dae406aec5e1b4112f8a1c9268fa6",
"md5": "a542b9c678932f788776801d2dab2960",
"sha256": "d731414ef2bc2597900a639f29c5e5d417c798f6d14dfe3c01d84af2f5093101"
},
"downloads": -1,
"filename": "allyanonimiser-2.1.0.tar.gz",
"has_sig": false,
"md5_digest": "a542b9c678932f788776801d2dab2960",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 180269,
"upload_time": "2025-03-03T04:22:48",
"upload_time_iso_8601": "2025-03-03T04:22:48.715797Z",
"url": "https://files.pythonhosted.org/packages/60/f2/99e863719ad25d7cc6a2349f48b08d1dae406aec5e1b4112f8a1c9268fa6/allyanonimiser-2.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-03-03 04:22:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "srepho",
"github_project": "Allyanonimiser",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "spacy",
"specs": [
[
">=",
"3.5.0"
]
]
},
{
"name": "presidio-analyzer",
"specs": [
[
">=",
"2.2.0"
]
]
},
{
"name": "presidio-anonymizer",
"specs": [
[
">=",
"2.2.0"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"7.0.0"
]
]
},
{
"name": "pytest-cov",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "black",
"specs": [
[
">=",
"22.0.0"
]
]
},
{
"name": "isort",
"specs": [
[
">=",
"5.0.0"
]
]
},
{
"name": "flake8",
"specs": [
[
">=",
"5.0.0"
]
]
},
{
"name": "mypy",
"specs": [
[
">=",
"0.9.0"
]
]
},
{
"name": "en-core-web-lg",
"specs": []
}
],
"lcname": "allyanonimiser"
}
