# MSTIC Jupyter and Python Security Tools
[](https://dev.azure.com/mstic-detections/mstic-jupyter/_build/latest?definitionId=14&branchName=main)
[](https://pepy.tech/project/msticpy)
[](https://www.blackhat.com/us-20/arsenal/schedule/#msticpy-the-security-analysis-swiss-army-knife-19872)
Microsoft Threat Intelligence Python Security Tools.
**msticpy** is a library for InfoSec investigation and hunting
in Jupyter Notebooks. It includes functionality to:
- query log data from multiple sources
- enrich the data with Threat Intelligence, geolocations and Azure
resource data
- extract Indicators of Activity (IoA) from logs and unpack encoded data
- perform sophisticated analysis such as anomalous session detection and
time series decomposition
- visualize data using interactive timelines, process trees and
multi-dimensional Morph Charts
It also includes some time-saving notebook tools such as widgets to
set query time boundaries, select and display items from lists, and
configure the notebook environment.
<img src="./docs/source/visualization/_static/Timeline-08.png"
alt="Timeline" title="Msticpy Timeline Control" height="300" />
The **msticpy** package was initially developed to support
[Jupyter Notebooks](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/)
authoring for
[Azure Sentinel](https://azure.microsoft.com/en-us/services/azure-sentinel/).
While Azure Sentinel is still a big focus of our work, we are
extending the data query/acquisition components to pull log data from
other sources (currently Splunk, Microsoft Defender for Endpoint and
Microsoft Graph are supported but we
are actively working on support for data from other SIEM platforms).
Most of the components can also be used with data from any source. Pandas
DataFrames are used as the ubiquitous input and output format of almost
all components. There is also a data provider to make it easy to and process
data from local CSV files and pickled DataFrames.
The package addresses three central needs for security investigators
and hunters:
- Acquiring and enriching data
- Analyzing data
- Visualizing data
We welcome feedback, bug reports, suggestions for new features and contributions.
## Installing
For core install:
`pip install msticpy`
If you are using *MSTICPy* with Azure Sentinel you should install with
the "azsentinel" extra package:
`pip install msticpy[azsentinel]`
or for the latest dev build
`pip install git+https://github.com/microsoft/msticpy`
## Upgrading
To upgrade msticpy to the latest public non-beta release, run:
`pip install --upgrade msticpy`
Note it is good practice to copy your msticpyconfig.yaml and store it on your disk but outside of your msticpy folder, referencing it in an environment variable. This prevents you from losing your configurations every time you update your msticpy installation.
## Documentation
Full documentation is at [ReadTheDocs](https://msticpy.readthedocs.io/en/latest/)
Sample notebooks for many of the modules are in the
[docs/notebooks](https://github.com/microsoft/msticpy/blob/master/docs/notebooks)
folder and accompanying notebooks.
You can also browse through the sample notebooks referenced at the end of this document
to see some of the functionality used in context. You can play with some of the package
functions in this interactive demo on mybinder.org.
[](https://mybinder.org/v2/gh/Azure/Azure-Sentinel-Notebooks/master?filepath=A%20Tour%20of%20Cybersec%20notebook%20features.ipynb)
---
## Log Data Acquisition
QueryProvider is an extensible query library targeting Azure Sentinel/Log Analytics,
Splunk, OData
and other log data sources. It also has special support for
[Mordor](https://github.com/OTRF/mordor) data sets and using local data.
Built-in parameterized queries allow complex queries to be run
from a single function call. Add your own queries using a simple YAML
schema.
[Data Queries Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Data_Queries.ipynb)
## Data Enrichment
### Threat Intelligence providers
The TILookup class can lookup IoCs across multiple TI providers. built-in
providers include AlienVault OTX, IBM XForce, VirusTotal and Azure Sentinel.
The input can be a single IoC observable or a pandas DataFrame containing
multiple observables. Depending on the provider, you may require an account
and an API key. Some providers also enforce throttling (especially for free
tiers), which might affect performing bulk lookups.
[TIProviders](https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html)
and
[TILookup Usage Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/TIProviders.ipynb)
### GeoLocation Data
The GeoIP lookup classes allow you to match the geo-locations of IP addresses
using either:
- GeoLiteLookup - Maxmind Geolite (see <https://www.maxmind.com>)
- IPStackLookup - IPStack (see <https://ipstack.com>)
<img src="./docs/source/visualization/_static/folium_sf_zoom.png"
alt="Folium map"
title="Plotting Geo IP Location" height="200" />
[GeoIP Lookup](https://msticpy.readthedocs.io/en/latest/data_acquisition/GeoIPLookups.html)
and
[GeoIP Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/GeoIPLookups.ipynb)
### Azure Resource Data, Storage and Azure Sentinel API
The AzureData module contains functionality for enriching data regarding Azure host
details with additional host details exposed via the Azure API. The AzureSentinel
module allows you to query incidents, retrieve detector and hunting
queries. AzureBlogStorage lets you read and write data from blob storage.
[Azure Resource APIs](https://msticpy.readthedocs.io/en/latest/data_acquisition/AzureData.html),
[Azure Sentinel APIs](https://msticpy.readthedocs.io/en/latest/data_acquisition/Sentinel.html),
[Azure Storage](https://msticpy.readthedocs.io/en/latest/data_acquisition/AzureBlobStorage.html)
## Security Analysis
This subpackage contains several modules helpful for working on security investigations and hunting:
### Anomalous Sequence Detection
Detect unusual sequences of events in your Office, Active Directory or other log data.
You can extract sessions (e.g. activity initiated by the same account) and identify and
visualize unusual sequences of activity. For example, detecting an attacker setting
a mail forwarding rule on someone's mailbox.
[Anomalous Sessions](https://msticpy.readthedocs.io/en/latest/data_analysis/AnomalousSequence.html)
and
[Anomalous Sequence Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/AnomalousSequence.ipynb)
### Time Series Analysis
Time series analysis allows you to identify unusual patterns in your log data
taking into account normal seasonal variations (e.g. the regular ebb and flow of
events over hours of the day, days of the week, etc.). Using both analysis and
visualization highlights unusual traffic flows or event activity for any data
set.
<img src="./docs/source/visualization/_static/TimeSeriesAnomalieswithRangeTool.png"
alt="Time Series anomalies" title="Time Series anomalies" height="300" />
[Time Series](https://msticpy.readthedocs.io/en/latest/visualization/TimeSeriesAnomalies.html)
## Visualization
### Event Timelines
Display any log events on an interactive timeline. Using the
[Bokeh Visualization Library](https://bokeh.org/) the timeline control enables
you to visualize one or more event streams, interactively zoom into specific time
slots and view event details for plotted events.
<img src="./docs/source/visualization/_static/TimeLine-01.png"
alt="Timeline" title="Msticpy Timeline Control" height="300" />
[Timeline](https://msticpy.readthedocs.io/en/latest/visualization/EventTimeline.html)
and
[Timeline Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/EventTimeline.ipynb)
### Process Trees
The process tree functionality has two main components:
- Process Tree creation - taking a process creation log from a host and building
the parent-child relationships between processes in the data set.
- Process Tree visualization - this takes the processed output displays an interactive process tree using Bokeh plots.
There are a set of utility functions to extract individual and partial trees from the processed data set.
<img src="./docs/source/visualization/_static/process_tree3.png"
alt="Process Tree"
title="Interactive Process Tree" height="400" />
[Process Tree](https://msticpy.readthedocs.io/en/latest/visualization/ProcessTree.html)
and
[Process Tree Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/ProcessTree.ipynb)
## Data Manipulation and Utility functions
### Pivot Functions
Lets you use *MSTICPy* functionality in an "entity-centric" way.
All functions, queries and lookups that relate to a particular entity type
(e.g. Host, IpAddress, Url) are collected together as methods of that
entity class. So, if you want to do things with an IP address, just load
the IpAddress entity and browse its methods.
[Pivot Functions](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html)
and
[Pivot Functions Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/PivotFunctions.ipynb)
### base64unpack
Base64 and archive (gz, zip, tar) extractor. It will try to identify any base64 encoded
strings and try decode them. If the result looks like one of the supported archive types it
will unpack the contents. The results of each decode/unpack are rechecked for further
base64 content and up to a specified depth.
[Base64 Decoding](https://msticpy.readthedocs.io/en/latest/data_analysis/Base64Unpack.html)
and
[Base64Unpack Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Base64Unpack.ipynb)
### iocextract
Uses regular expressions to look for Indicator of Compromise (IoC) patterns - IP Addresses, URLs,
DNS domains, Hashes, file paths.
Input can be a single string or a pandas dataframe.
[IoC Extraction](https://msticpy.readthedocs.io/en/latest/data_analysis/IoCExtract.html)
and
[IoCExtract Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/IoCExtract.ipynb)
### eventcluster (experimental)
This module is intended to be used to summarize large numbers of
events into clusters of different patterns. High volume repeating
events can often make it difficult to see unique and interesting items.
<img src="./docs/source/data_analysis/_static/EventClustering_2a.png"
alt="Clustering"
title="Clustering based on command-line variability" height="400" />
This is an unsupervised learning module implemented using SciKit Learn DBScan.
[Event Clustering](https://msticpy.readthedocs.io/en/latest/data_analysis/EventClustering.html)
and
[Event Clustering Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/EventClustering.ipynb)
### auditdextract
Module to load and decode Linux audit logs. It collapses messages sharing the same
message ID into single events, decodes hex-encoded data fields and performs some
event-specific formatting and normalization (e.g. for process start events it will
re-assemble the process command line arguments into a single string).
### syslog_utils
Module to support an investigation of a Linux host with only syslog logging enabled.
This includes functions for collating host data, clustering logon events and detecting
user sessions containing suspicious activity.
### cmd_line
A module to support the detection of known malicious command line activity or suspicious
patterns of command line activity.
### domain_utils
A module to support investigation of domain names and URLs with functions to
validate a domain name and screenshot a URL.
### Notebook widgets
These are built from the [Jupyter ipywidgets](https://ipywidgets.readthedocs.io/) collection
and group common functionality useful in InfoSec tasks such as list pickers,
query time boundary settings and event display into an easy-to-use format.
<img src="./docs/source/visualization/_static/Widgets1.png"
alt="Time span Widget"
title="Query time setter" height="100" />
<img src="./docs/source/visualization/_static/Widgets4.png"
alt="Alert browser"
title="Alert browser" height="300" />
---
## Example MSTICPy notebooks
[MSTICPy Notebooks](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/)
## More Notebooks on Azure Sentinel Notebooks GitHub
[Azure Sentinel Notebooks](https://github.com/Azure/Azure-Sentinel-Notebooks)
Example notebooks:
- [Tour of MSTICPy Features](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/A%20Tour%20of%20Cybersec%20notebook%20features.ipynb)
- [Account Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Account.ipynb)
- [Domain and URL Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Domain%20and%20URL.ipynb)
- [IP Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20IP%20Address.ipynb)
- [Linux Host Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Linux%20Host.ipynb)
- [Windows Host Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Windows%20Host.ipynb)
- [Getting started in Azure Sentinel notebooks](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/A%20Getting%20Started%20Guide%20For%20Azure%20Sentinel%20ML%20Notebooks.ipynb)
View directly on GitHub or copy and paste the link into [nbviewer.org](https://nbviewer.jupyter.org/)
## Notebook examples with saved data
See the following notebooks for more examples of the use of this package in practice:
- [Simple Machine Learning](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Machine%20Learning%20in%20Notebooks%20Examples.ipynb)
## Supported Platforms and Packages
- msticpy is OS-independent
- Requires [Python 3.8 or later](https://www.python.org/dev/peps/pep-0494/)
- See [requirements.txt](requirements.txt) for more details and version requirements.
---
## Contributing
For (brief) developer guidelines, see this wiki article
[Contributor Guidelines](https://github.com/microsoft/msticpy/wiki/Contributor-guidelines)
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit <https://cla.microsoft.com>.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
Raw data
{
"_id": null,
"home_page": "https://github.com/microsoft/msticpy",
"name": "msticpy",
"maintainer": "Pete Bryan",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "peter.bryan@microsoft.com",
"keywords": "security, azure, sentinel, mstic, cybersec, infosec, cyber, cybersecurity, jupyter, notebooks, SOC, hunting",
"author": "Ian Hellen",
"author_email": "ianhelle@microsoft.com",
"download_url": "https://files.pythonhosted.org/packages/62/a8/c8f736dafbb7346b08fe3c5afe897236714c27aad7a44dd31123ac8ce8b4/msticpy-2.12.0.tar.gz",
"platform": null,
"description": "# MSTIC Jupyter and Python Security Tools\n\n\n[](https://dev.azure.com/mstic-detections/mstic-jupyter/_build/latest?definitionId=14&branchName=main)\n[](https://pepy.tech/project/msticpy)\n[](https://www.blackhat.com/us-20/arsenal/schedule/#msticpy-the-security-analysis-swiss-army-knife-19872)\n\nMicrosoft Threat Intelligence Python Security Tools.\n\n**msticpy** is a library for InfoSec investigation and hunting\nin Jupyter Notebooks. It includes functionality to:\n\n- query log data from multiple sources\n- enrich the data with Threat Intelligence, geolocations and Azure\n resource data\n- extract Indicators of Activity (IoA) from logs and unpack encoded data\n- perform sophisticated analysis such as anomalous session detection and\n time series decomposition\n- visualize data using interactive timelines, process trees and\n multi-dimensional Morph Charts\n\nIt also includes some time-saving notebook tools such as widgets to\nset query time boundaries, select and display items from lists, and\nconfigure the notebook environment.\n\n<img src=\"./docs/source/visualization/_static/Timeline-08.png\"\nalt=\"Timeline\" title=\"Msticpy Timeline Control\" height=\"300\" />\n\nThe **msticpy** package was initially developed to support\n[Jupyter Notebooks](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/)\nauthoring for\n[Azure Sentinel](https://azure.microsoft.com/en-us/services/azure-sentinel/).\nWhile Azure Sentinel is still a big focus of our work, we are\nextending the data query/acquisition components to pull log data from\nother sources (currently Splunk, Microsoft Defender for Endpoint and\nMicrosoft Graph are supported but we\nare actively working on support for data from other SIEM platforms).\nMost of the components can also be used with data from any source. Pandas\nDataFrames are used as the ubiquitous input and output format of almost\nall components. There is also a data provider to make it easy to and process\ndata from local CSV files and pickled DataFrames.\n\nThe package addresses three central needs for security investigators\nand hunters:\n\n- Acquiring and enriching data\n- Analyzing data\n- Visualizing data\n\nWe welcome feedback, bug reports, suggestions for new features and contributions.\n\n## Installing\n\nFor core install:\n\n`pip install msticpy`\n\nIf you are using *MSTICPy* with Azure Sentinel you should install with\nthe \"azsentinel\" extra package:\n\n`pip install msticpy[azsentinel]`\n\nor for the latest dev build\n\n`pip install git+https://github.com/microsoft/msticpy`\n\n## Upgrading\n\nTo upgrade msticpy to the latest public non-beta release, run:\n\n`pip install --upgrade msticpy`\n\nNote it is good practice to copy your msticpyconfig.yaml and store it on your disk but outside of your msticpy folder, referencing it in an environment variable. This prevents you from losing your configurations every time you update your msticpy installation.\n\n## Documentation\n\nFull documentation is at [ReadTheDocs](https://msticpy.readthedocs.io/en/latest/)\n\nSample notebooks for many of the modules are in the\n[docs/notebooks](https://github.com/microsoft/msticpy/blob/master/docs/notebooks)\nfolder and accompanying notebooks.\n\nYou can also browse through the sample notebooks referenced at the end of this document\nto see some of the functionality used in context. You can play with some of the package\nfunctions in this interactive demo on mybinder.org.\n\n[](https://mybinder.org/v2/gh/Azure/Azure-Sentinel-Notebooks/master?filepath=A%20Tour%20of%20Cybersec%20notebook%20features.ipynb)\n\n---\n\n## Log Data Acquisition\n\nQueryProvider is an extensible query library targeting Azure Sentinel/Log Analytics,\nSplunk, OData\nand other log data sources. It also has special support for\n[Mordor](https://github.com/OTRF/mordor) data sets and using local data.\n\nBuilt-in parameterized queries allow complex queries to be run\nfrom a single function call. Add your own queries using a simple YAML\nschema.\n\n[Data Queries Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Data_Queries.ipynb)\n\n## Data Enrichment\n\n### Threat Intelligence providers\n\nThe TILookup class can lookup IoCs across multiple TI providers. built-in\nproviders include AlienVault OTX, IBM XForce, VirusTotal and Azure Sentinel.\n\nThe input can be a single IoC observable or a pandas DataFrame containing\nmultiple observables. Depending on the provider, you may require an account\nand an API key. Some providers also enforce throttling (especially for free\ntiers), which might affect performing bulk lookups.\n\n[TIProviders](https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html)\nand\n[TILookup Usage Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/TIProviders.ipynb)\n\n### GeoLocation Data\n\nThe GeoIP lookup classes allow you to match the geo-locations of IP addresses\nusing either:\n\n- GeoLiteLookup - Maxmind Geolite (see <https://www.maxmind.com>)\n- IPStackLookup - IPStack (see <https://ipstack.com>)\n\n<img src=\"./docs/source/visualization/_static/folium_sf_zoom.png\"\n alt=\"Folium map\"\n title=\"Plotting Geo IP Location\" height=\"200\" />\n\n[GeoIP Lookup](https://msticpy.readthedocs.io/en/latest/data_acquisition/GeoIPLookups.html)\nand\n[GeoIP Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/GeoIPLookups.ipynb)\n\n### Azure Resource Data, Storage and Azure Sentinel API\n\nThe AzureData module contains functionality for enriching data regarding Azure host\ndetails with additional host details exposed via the Azure API. The AzureSentinel\nmodule allows you to query incidents, retrieve detector and hunting\nqueries. AzureBlogStorage lets you read and write data from blob storage.\n\n[Azure Resource APIs](https://msticpy.readthedocs.io/en/latest/data_acquisition/AzureData.html),\n[Azure Sentinel APIs](https://msticpy.readthedocs.io/en/latest/data_acquisition/Sentinel.html),\n[Azure Storage](https://msticpy.readthedocs.io/en/latest/data_acquisition/AzureBlobStorage.html)\n\n## Security Analysis\n\nThis subpackage contains several modules helpful for working on security investigations and hunting:\n\n### Anomalous Sequence Detection\n\nDetect unusual sequences of events in your Office, Active Directory or other log data.\nYou can extract sessions (e.g. activity initiated by the same account) and identify and\nvisualize unusual sequences of activity. For example, detecting an attacker setting\na mail forwarding rule on someone's mailbox.\n\n[Anomalous Sessions](https://msticpy.readthedocs.io/en/latest/data_analysis/AnomalousSequence.html)\nand\n[Anomalous Sequence Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/AnomalousSequence.ipynb)\n\n### Time Series Analysis\n\nTime series analysis allows you to identify unusual patterns in your log data\ntaking into account normal seasonal variations (e.g. the regular ebb and flow of\nevents over hours of the day, days of the week, etc.). Using both analysis and\nvisualization highlights unusual traffic flows or event activity for any data\nset.\n\n<img src=\"./docs/source/visualization/_static/TimeSeriesAnomalieswithRangeTool.png\"\nalt=\"Time Series anomalies\" title=\"Time Series anomalies\" height=\"300\" />\n\n[Time Series](https://msticpy.readthedocs.io/en/latest/visualization/TimeSeriesAnomalies.html)\n\n## Visualization\n\n### Event Timelines\n\nDisplay any log events on an interactive timeline. Using the\n[Bokeh Visualization Library](https://bokeh.org/) the timeline control enables\nyou to visualize one or more event streams, interactively zoom into specific time\nslots and view event details for plotted events.\n\n<img src=\"./docs/source/visualization/_static/TimeLine-01.png\"\nalt=\"Timeline\" title=\"Msticpy Timeline Control\" height=\"300\" />\n\n[Timeline](https://msticpy.readthedocs.io/en/latest/visualization/EventTimeline.html)\nand\n[Timeline Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/EventTimeline.ipynb)\n\n### Process Trees\n\nThe process tree functionality has two main components:\n\n- Process Tree creation - taking a process creation log from a host and building\n the parent-child relationships between processes in the data set.\n- Process Tree visualization - this takes the processed output displays an interactive process tree using Bokeh plots.\n\nThere are a set of utility functions to extract individual and partial trees from the processed data set.\n\n<img src=\"./docs/source/visualization/_static/process_tree3.png\"\nalt=\"Process Tree\"\ntitle=\"Interactive Process Tree\" height=\"400\" />\n\n[Process Tree](https://msticpy.readthedocs.io/en/latest/visualization/ProcessTree.html)\nand\n[Process Tree Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/ProcessTree.ipynb)\n\n## Data Manipulation and Utility functions\n\n### Pivot Functions\n\nLets you use *MSTICPy* functionality in an \"entity-centric\" way.\nAll functions, queries and lookups that relate to a particular entity type\n(e.g. Host, IpAddress, Url) are collected together as methods of that\nentity class. So, if you want to do things with an IP address, just load\nthe IpAddress entity and browse its methods.\n\n[Pivot Functions](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html)\nand\n[Pivot Functions Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/PivotFunctions.ipynb)\n\n### base64unpack\n\nBase64 and archive (gz, zip, tar) extractor. It will try to identify any base64 encoded\nstrings and try decode them. If the result looks like one of the supported archive types it\nwill unpack the contents. The results of each decode/unpack are rechecked for further\nbase64 content and up to a specified depth.\n\n[Base64 Decoding](https://msticpy.readthedocs.io/en/latest/data_analysis/Base64Unpack.html)\nand\n[Base64Unpack Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Base64Unpack.ipynb)\n\n### iocextract\n\nUses regular expressions to look for Indicator of Compromise (IoC) patterns - IP Addresses, URLs,\nDNS domains, Hashes, file paths.\nInput can be a single string or a pandas dataframe.\n\n[IoC Extraction](https://msticpy.readthedocs.io/en/latest/data_analysis/IoCExtract.html)\nand\n[IoCExtract Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/IoCExtract.ipynb)\n\n### eventcluster (experimental)\n\nThis module is intended to be used to summarize large numbers of\nevents into clusters of different patterns. High volume repeating\nevents can often make it difficult to see unique and interesting items.\n\n<img src=\"./docs/source/data_analysis/_static/EventClustering_2a.png\"\n alt=\"Clustering\"\n title=\"Clustering based on command-line variability\" height=\"400\" />\n\nThis is an unsupervised learning module implemented using SciKit Learn DBScan.\n\n[Event Clustering](https://msticpy.readthedocs.io/en/latest/data_analysis/EventClustering.html)\nand\n[Event Clustering Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/EventClustering.ipynb)\n\n### auditdextract\n\nModule to load and decode Linux audit logs. It collapses messages sharing the same\nmessage ID into single events, decodes hex-encoded data fields and performs some\nevent-specific formatting and normalization (e.g. for process start events it will\nre-assemble the process command line arguments into a single string).\n\n### syslog_utils\n\nModule to support an investigation of a Linux host with only syslog logging enabled.\nThis includes functions for collating host data, clustering logon events and detecting\nuser sessions containing suspicious activity.\n\n### cmd_line\n\nA module to support the detection of known malicious command line activity or suspicious\npatterns of command line activity.\n\n### domain_utils\n\nA module to support investigation of domain names and URLs with functions to\nvalidate a domain name and screenshot a URL.\n\n### Notebook widgets\n\nThese are built from the [Jupyter ipywidgets](https://ipywidgets.readthedocs.io/) collection\nand group common functionality useful in InfoSec tasks such as list pickers,\nquery time boundary settings and event display into an easy-to-use format.\n\n<img src=\"./docs/source/visualization/_static/Widgets1.png\"\n alt=\"Time span Widget\"\n title=\"Query time setter\" height=\"100\" />\n\n<img src=\"./docs/source/visualization/_static/Widgets4.png\"\n alt=\"Alert browser\"\n title=\"Alert browser\" height=\"300\" />\n\n---\n\n## Example MSTICPy notebooks\n\n[MSTICPy Notebooks](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/)\n\n## More Notebooks on Azure Sentinel Notebooks GitHub\n\n[Azure Sentinel Notebooks](https://github.com/Azure/Azure-Sentinel-Notebooks)\n\nExample notebooks:\n\n- [Tour of MSTICPy Features](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/A%20Tour%20of%20Cybersec%20notebook%20features.ipynb)\n- [Account Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Account.ipynb)\n- [Domain and URL Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Domain%20and%20URL.ipynb)\n- [IP Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20IP%20Address.ipynb)\n- [Linux Host Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Linux%20Host.ipynb)\n- [Windows Host Explorer](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Entity%20Explorer%20-%20Windows%20Host.ipynb)\n- [Getting started in Azure Sentinel notebooks](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/A%20Getting%20Started%20Guide%20For%20Azure%20Sentinel%20ML%20Notebooks.ipynb)\n\nView directly on GitHub or copy and paste the link into [nbviewer.org](https://nbviewer.jupyter.org/)\n\n## Notebook examples with saved data\n\nSee the following notebooks for more examples of the use of this package in practice:\n\n- [Simple Machine Learning](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/Machine%20Learning%20in%20Notebooks%20Examples.ipynb)\n\n## Supported Platforms and Packages\n\n- msticpy is OS-independent\n- Requires [Python 3.8 or later](https://www.python.org/dev/peps/pep-0494/)\n- See [requirements.txt](requirements.txt) for more details and version requirements.\n\n---\n\n## Contributing\n\nFor (brief) developer guidelines, see this wiki article\n[Contributor Guidelines](https://github.com/microsoft/msticpy/wiki/Contributor-guidelines)\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit <https://cla.microsoft.com>.\n\nWhen you submit a pull request, a CLA-bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "MSTIC Security Tools",
"version": "2.12.0",
"project_urls": {
"Code": "https://github.com/microsoft/msticpy",
"Documentation": "https://msticpy.readthedocs.io",
"Homepage": "https://github.com/microsoft/msticpy",
"Issue tracker": "https://github.com/microsoft/msticpy/issues"
},
"split_keywords": [
"security",
" azure",
" sentinel",
" mstic",
" cybersec",
" infosec",
" cyber",
" cybersecurity",
" jupyter",
" notebooks",
" soc",
" hunting"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d1c2db381862d0d3e3bf0cf11651383b55f9977462f687e5e858a6730aabdc9e",
"md5": "f92b00144469a632f773c8319cdd54de",
"sha256": "c7ae3dec1116879194d8413e3cc9e1ce01d4d1534ebc88cb66313054e078eb42"
},
"downloads": -1,
"filename": "msticpy-2.12.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f92b00144469a632f773c8319cdd54de",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 906313,
"upload_time": "2024-05-10T23:26:19",
"upload_time_iso_8601": "2024-05-10T23:26:19.005495Z",
"url": "https://files.pythonhosted.org/packages/d1/c2/db381862d0d3e3bf0cf11651383b55f9977462f687e5e858a6730aabdc9e/msticpy-2.12.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "62a8c8f736dafbb7346b08fe3c5afe897236714c27aad7a44dd31123ac8ce8b4",
"md5": "e446ee1e9287efbb8c4bcafac7a2ac7c",
"sha256": "5a0f4b2e0969fc62ea6718936e4b73e4d50bab0bc0e71593a08c7cbcf185a8cf"
},
"downloads": -1,
"filename": "msticpy-2.12.0.tar.gz",
"has_sig": false,
"md5_digest": "e446ee1e9287efbb8c4bcafac7a2ac7c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 676606,
"upload_time": "2024-05-10T23:26:22",
"upload_time_iso_8601": "2024-05-10T23:26:22.222654Z",
"url": "https://files.pythonhosted.org/packages/62/a8/c8f736dafbb7346b08fe3c5afe897236714c27aad7a44dd31123ac8ce8b4/msticpy-2.12.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-10 23:26:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "microsoft",
"github_project": "msticpy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "attrs",
"specs": [
[
">=",
"18.2.0"
]
]
},
{
"name": "azure-common",
"specs": [
[
">=",
"1.1.18"
]
]
},
{
"name": "azure-core",
"specs": [
[
">=",
"1.24.0"
]
]
},
{
"name": "azure-identity",
"specs": [
[
">=",
"1.10.0"
]
]
},
{
"name": "azure-keyvault-secrets",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "azure-kusto-data",
"specs": [
[
">=",
"4.0.0"
],
[
"<=",
"5.0.0"
]
]
},
{
"name": "azure-mgmt-keyvault",
"specs": [
[
">=",
"2.0.0"
]
]
},
{
"name": "azure-mgmt-subscription",
"specs": [
[
">=",
"3.0.0"
]
]
},
{
"name": "azure-monitor-query",
"specs": [
[
">=",
"1.0.0"
],
[
"<=",
"2.0.0"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "bokeh",
"specs": [
[
">=",
"1.4.0"
],
[
"<",
"3.4.0"
]
]
},
{
"name": "cryptography",
"specs": [
[
">=",
"3.1"
]
]
},
{
"name": "deprecated",
"specs": [
[
">=",
"1.2.4"
]
]
},
{
"name": "dnspython",
"specs": [
[
">=",
"2.0.0"
],
[
"<",
"3.0.0"
]
]
},
{
"name": "folium",
"specs": [
[
">=",
"0.9.0"
]
]
},
{
"name": "geoip2",
"specs": [
[
">=",
"2.9.0"
]
]
},
{
"name": "httpx",
"specs": [
[
">=",
"0.23.0"
],
[
"<",
"1.0.0"
]
]
},
{
"name": "html5lib",
"specs": []
},
{
"name": "ipython",
"specs": [
[
">=",
"7.1.1"
]
]
},
{
"name": "ipython",
"specs": [
[
">=",
"7.23.1"
]
]
},
{
"name": "ipywidgets",
"specs": [
[
">=",
"7.4.2"
],
[
"<",
"9.0.0"
]
]
},
{
"name": "keyring",
"specs": [
[
">=",
"13.2.1"
]
]
},
{
"name": "lxml",
"specs": [
[
">=",
"4.6.5"
]
]
},
{
"name": "msal",
"specs": [
[
">=",
"1.12.0"
]
]
},
{
"name": "msal_extensions",
"specs": [
[
">=",
"0.3.0"
]
]
},
{
"name": "msrest",
"specs": [
[
">=",
"0.6.0"
]
]
},
{
"name": "msrestazure",
"specs": [
[
">=",
"0.6.0"
]
]
},
{
"name": "nest_asyncio",
"specs": [
[
">=",
"1.4.0"
]
]
},
{
"name": "networkx",
"specs": [
[
">=",
"2.2"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.15.4"
]
]
},
{
"name": "pandas",
"specs": [
[
">=",
"1.4.0"
],
[
"<",
"3.0.0"
]
]
},
{
"name": "pydantic",
"specs": [
[
">=",
"1.8.0"
],
[
"<",
"3.0.0"
]
]
},
{
"name": "pygments",
"specs": [
[
">=",
"2.0.0"
]
]
},
{
"name": "pyjwt",
"specs": [
[
">=",
"2.3.0"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
">=",
"2.8.1"
]
]
},
{
"name": "pytz",
"specs": [
[
">=",
"2019.2"
]
]
},
{
"name": "pyyaml",
"specs": [
[
">=",
"3.13"
]
]
},
{
"name": "setuptools",
"specs": [
[
">=",
"40.6.3"
]
]
},
{
"name": "tldextract",
"specs": [
[
">=",
"2.2.2"
]
]
},
{
"name": "tqdm",
"specs": [
[
">=",
"4.36.1"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
">=",
"4.2.0"
]
]
},
{
"name": "urllib3",
"specs": [
[
">=",
"1.23"
]
]
}
],
"lcname": "msticpy"
}
JSON
