# mft2es
[](LICENSE)
[](https://badge.fury.io/py/mft2es)
[](https://pypi.org/project/mft2es/)
Fast import of Windows Master File Table(\$MFT) into Elasticsearch.
mft2es uses Rust library [pymft-rs](https://github.com/omerbenamram/pymft-rs), so it runs much faster than traditional software.
## Usage
**mft2es** can be executed from the command line or incorporated into a Python script.
```bash
$ mft2es /path/to/your/$MFT
```
or
```python
from mft2es import mft2es
if __name__ == '__main__':
filepath = '/path/to/your/$MFT'
mft2es(filepath)
```
### Args
mft2es supports simultaneous import of multiple files.
```bash
$ mft2es foo/MFT bar/MFT
```
Additionally, it also allows for recursive import under the specified directory.
```bash
$ tree .
mftfiles/
├── MFT
└── subdirectory/
├── MFT
└── subsubdirectory/
├── MFT
└── $MFT
$ mft2es /mftfiles/ # The Path is recursively expanded to all MFT, and $MFT.
```
### Options
```
--version, -v
--help, -h
--quiet, -q
Flag to suppress standard output
(default: False)
--multiprocess, -m:
Enable multiprocessing for faster execution
(default: False)
--size:
Chunk size for processing (default: 500)
--host:
ElasticSearch host address (default: localhost)
--port:
ElasticSearch port number (default: 9200)
--index:
Destination index name for importing (default: mft2es)
--scheme:
Protocol scheme to use (http or https) (default: http)
--pipeline
Elasticsearch Ingest Pipeline to use (default: )
--login:
The login to use if Elastic Security is enabled (default: )
--pwd:
The password associated with the provided login (default: )
```
### Examples
When using from the commandline interface:
```
$ mft2es /path/to/your/$MFT --host=localhost --port=9200 --index=foobar --size=500
```
When using from the python-script:
```py
if __name__ == '__main__':
mft2es('/path/to/your/$MFT', host=localhost, port=9200, index='foobar', size=500)
```
With credentials for Elastic Security:
```
$ mft2es /path/to/your/$MFT --host=localhost --port=9200 --index=foobar --login=elastic --pwd=******
```
Note: The current version does not verify the certificate.
## Appendix
### Mft2json
An additional feature: :sushi: :sushi: :sushi:
Convert Windows MFT to a JSON file.
```bash
$ mft2json /path/to/your/$MFT -o /path/to/output/target.json
```
Convert Windows Event Logs to a Python List[dict] object.
```python
from mft2es import mft2json
if __name__ == '__main__':
filepath = '/path/to/your/$MFT'
result: List[dict] = mft2json(filepath)
```
## Output Format
The structures is not well optimized for searchable with Elasticsearch. I'm waiting for your PR!!
```json
[
{
"header": {
"signature": [
70,
73,
76,
69
],
"usa_offset": 48,
"usa_size": 3,
"metadata_transaction_journal": 172848302,
"sequence": 1,
"hard_link_count": 1,
"first_attribute_record_offset": 56,
"flags": "ALLOCATED",
"used_entry_size": 416,
"total_entry_size": 1024,
"base_reference": {
"entry": 0,
"sequence": 0
},
"first_attribute_id": 6,
"record_number": 0
},
"attributes": {
"StandardInformation": {
"header": {
"type_code": "StandardInformation",
"record_length": 96,
"form_code": 0,
"residential_header": {
"index_flag": 0
},
"name_size": 0,
"name_offset": null,
"data_flags": "(empty)",
"instance": 0,
"name": ""
},
"data": {
"created": "2019-03-11T16:42:33.593750Z",
"modified": "2019-03-11T16:42:33.593750Z",
"mft_modified": "2019-03-11T16:42:33.593750Z",
"accessed": "2019-03-11T16:42:33.593750Z",
"file_flags": "FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM",
"max_version": 0,
"version": 0,
"class_id": 0,
"owner_id": 0,
"security_id": 256,
"quota": 0,
"usn": 0
}
},
"FileName": {
"header": {
"type_code": "FileName",
"record_length": 104,
"form_code": 0,
"residential_header": {
"index_flag": 1
},
"name_size": 0,
"name_offset": null,
"data_flags": "(empty)",
"instance": 3,
"name": ""
},
"data": {
"parent": {
"entry": 5,
"sequence": 5
},
"created": "2019-03-11T16:42:33.593750Z",
"modified": "2019-03-11T16:42:33.593750Z",
"mft_modified": "2019-03-11T16:42:33.593750Z",
"accessed": "2019-03-11T16:42:33.593750Z",
"logical_size": 16384,
"physical_size": 16384,
"flags": "FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM",
"reparse_value": 0,
"name_length": 4,
"namespace": "Win32AndDos",
"name": "$MFT",
"path": "$MFT"
}
},
"DATA": {
"header": {
"type_code": "DATA",
"record_length": 72,
"form_code": 1,
"residential_header": {
"vnc_first": 0,
"vnc_last": "0x198f",
"unit_compression_size": 0,
"allocated_length": 62390272,
"file_size": 62390272,
"valid_data_length": 62390272,
"total_allocated": null
},
"name_size": 0,
"name_offset": null,
"data_flags": "(empty)",
"instance": 1,
"name": ""
},
"data": null
},
"BITMAP": {
"header": {
"type_code": "BITMAP",
"record_length": 80,
"form_code": 1,
"residential_header": {
"vnc_first": 0,
"vnc_last": 0,
"unit_compression_size": 0,
"allocated_length": 12288,
"file_size": 8200,
"valid_data_length": 8200,
"total_allocated": null
},
"name_size": 0,
"name_offset": null,
"data_flags": "(empty)",
"instance": 5,
"name": ""
},
"data": null
}
}
}
...
]
````
## Installation
### from PyPI
```
$ pip install mft2es
```
### from GitHub Releases
The version compiled into a binary using Nuitka is also available for use.
```bash
$ chmod +x ./mft2es
$ ./mft2es {{options...}}
```
```powershell
> mft2es.exe {{options...}}
```
## Contributing
The source code for mft2es is hosted at GitHub, and you may download, fork, and review it from this repository(https://github.com/sumeshi/mft2es).
Please report issues and feature requests. :sushi: :sushi: :sushi:
## License
mft2es is released under the [MIT](https://github.com/sumeshi/mft2es/blob/master/LICENSE) License.
Powered by following libraries:
- [pymft-rs](https://github.com/omerbenamram/pymft-rs)
- [Nuitka](https://github.com/Nuitka/Nuitka)
Raw data
{
"_id": null,
"home_page": "https://github.com/sumeshi/mft2es",
"name": "mft2es",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.11,<4.0",
"maintainer_email": "",
"keywords": "mft,elasticsearch,json",
"author": "sumeshi",
"author_email": "sum3sh1@protonmail.com",
"download_url": "https://files.pythonhosted.org/packages/cd/fe/fc12f4bb23ce0c0ef3de39e0a0a60e06a8f4cdbe38162f997e6e33bd26e0/mft2es-1.4.0.tar.gz",
"platform": null,
"description": "# mft2es\n\n[](LICENSE)\n[](https://badge.fury.io/py/mft2es)\n[](https://pypi.org/project/mft2es/)\n\n\n\nFast import of Windows Master File Table(\\$MFT) into Elasticsearch.\n\nmft2es uses Rust library [pymft-rs](https://github.com/omerbenamram/pymft-rs), so it runs much faster than traditional software.\n\n## Usage\n**mft2es** can be executed from the command line or incorporated into a Python script.\n\n```bash\n$ mft2es /path/to/your/$MFT\n```\n\nor\n\n```python\nfrom mft2es import mft2es\n\nif __name__ == '__main__':\n filepath = '/path/to/your/$MFT'\n mft2es(filepath)\n```\n\n### Args\n\nmft2es supports simultaneous import of multiple files.\n\n```bash\n$ mft2es foo/MFT bar/MFT\n```\n\nAdditionally, it also allows for recursive import under the specified directory.\n\n```bash\n$ tree .\nmftfiles/\n \u251c\u2500\u2500 MFT\n \u2514\u2500\u2500 subdirectory/\n \u251c\u2500\u2500 MFT\n \u2514\u2500\u2500 subsubdirectory/\n \u251c\u2500\u2500 MFT\n \u2514\u2500\u2500 $MFT\n\n$ mft2es /mftfiles/ # The Path is recursively expanded to all MFT, and $MFT.\n```\n\n### Options\n\n```\n--version, -v\n\n--help, -h\n\n--quiet, -q\n Flag to suppress standard output\n (default: False)\n\n--multiprocess, -m:\n Enable multiprocessing for faster execution\n (default: False)\n\n--size:\n Chunk size for processing (default: 500)\n\n--host:\n ElasticSearch host address (default: localhost)\n\n--port:\n ElasticSearch port number (default: 9200)\n\n--index:\n Destination index name for importing (default: mft2es)\n\n--scheme:\n Protocol scheme to use (http or https) (default: http)\n\n--pipeline\n Elasticsearch Ingest Pipeline to use (default: )\n\n--login:\n The login to use if Elastic Security is enabled (default: )\n\n--pwd:\n The password associated with the provided login (default: )\n```\n\n### Examples\n\nWhen using from the commandline interface:\n\n```\n$ mft2es /path/to/your/$MFT --host=localhost --port=9200 --index=foobar --size=500\n```\n\nWhen using from the python-script:\n\n```py\nif __name__ == '__main__':\n mft2es('/path/to/your/$MFT', host=localhost, port=9200, index='foobar', size=500)\n```\n\nWith credentials for Elastic Security:\n\n```\n$ mft2es /path/to/your/$MFT --host=localhost --port=9200 --index=foobar --login=elastic --pwd=******\n```\n\nNote: The current version does not verify the certificate.\n\n## Appendix\n\n### Mft2json\n\nAn additional feature: :sushi: :sushi: :sushi:\n\nConvert Windows MFT to a JSON file.\n\n```bash\n$ mft2json /path/to/your/$MFT -o /path/to/output/target.json\n```\n\nConvert Windows Event Logs to a Python List[dict] object.\n\n```python\nfrom mft2es import mft2json\n\nif __name__ == '__main__':\n filepath = '/path/to/your/$MFT'\n result: List[dict] = mft2json(filepath)\n```\n\n## Output Format\n\nThe structures is not well optimized for searchable with Elasticsearch. I'm waiting for your PR!!\n\n```json\n[\n {\n \"header\": {\n \"signature\": [\n 70,\n 73,\n 76,\n 69\n ],\n \"usa_offset\": 48,\n \"usa_size\": 3,\n \"metadata_transaction_journal\": 172848302,\n \"sequence\": 1,\n \"hard_link_count\": 1,\n \"first_attribute_record_offset\": 56,\n \"flags\": \"ALLOCATED\",\n \"used_entry_size\": 416,\n \"total_entry_size\": 1024,\n \"base_reference\": {\n \"entry\": 0,\n \"sequence\": 0\n },\n \"first_attribute_id\": 6,\n \"record_number\": 0\n },\n \"attributes\": {\n \"StandardInformation\": {\n \"header\": {\n \"type_code\": \"StandardInformation\",\n \"record_length\": 96,\n \"form_code\": 0,\n \"residential_header\": {\n \"index_flag\": 0\n },\n \"name_size\": 0,\n \"name_offset\": null,\n \"data_flags\": \"(empty)\",\n \"instance\": 0,\n \"name\": \"\"\n },\n \"data\": {\n \"created\": \"2019-03-11T16:42:33.593750Z\",\n \"modified\": \"2019-03-11T16:42:33.593750Z\",\n \"mft_modified\": \"2019-03-11T16:42:33.593750Z\",\n \"accessed\": \"2019-03-11T16:42:33.593750Z\",\n \"file_flags\": \"FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM\",\n \"max_version\": 0,\n \"version\": 0,\n \"class_id\": 0,\n \"owner_id\": 0,\n \"security_id\": 256,\n \"quota\": 0,\n \"usn\": 0\n }\n },\n \"FileName\": {\n \"header\": {\n \"type_code\": \"FileName\",\n \"record_length\": 104,\n \"form_code\": 0,\n \"residential_header\": {\n \"index_flag\": 1\n },\n \"name_size\": 0,\n \"name_offset\": null,\n \"data_flags\": \"(empty)\",\n \"instance\": 3,\n \"name\": \"\"\n },\n \"data\": {\n \"parent\": {\n \"entry\": 5,\n \"sequence\": 5\n },\n \"created\": \"2019-03-11T16:42:33.593750Z\",\n \"modified\": \"2019-03-11T16:42:33.593750Z\",\n \"mft_modified\": \"2019-03-11T16:42:33.593750Z\",\n \"accessed\": \"2019-03-11T16:42:33.593750Z\",\n \"logical_size\": 16384,\n \"physical_size\": 16384,\n \"flags\": \"FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM\",\n \"reparse_value\": 0,\n \"name_length\": 4,\n \"namespace\": \"Win32AndDos\",\n \"name\": \"$MFT\",\n \"path\": \"$MFT\"\n }\n },\n \"DATA\": {\n \"header\": {\n \"type_code\": \"DATA\",\n \"record_length\": 72,\n \"form_code\": 1,\n \"residential_header\": {\n \"vnc_first\": 0,\n \"vnc_last\": \"0x198f\",\n \"unit_compression_size\": 0,\n \"allocated_length\": 62390272,\n \"file_size\": 62390272,\n \"valid_data_length\": 62390272,\n \"total_allocated\": null\n },\n \"name_size\": 0,\n \"name_offset\": null,\n \"data_flags\": \"(empty)\",\n \"instance\": 1,\n \"name\": \"\"\n },\n \"data\": null\n },\n \"BITMAP\": {\n \"header\": {\n \"type_code\": \"BITMAP\",\n \"record_length\": 80,\n \"form_code\": 1,\n \"residential_header\": {\n \"vnc_first\": 0,\n \"vnc_last\": 0,\n \"unit_compression_size\": 0,\n \"allocated_length\": 12288,\n \"file_size\": 8200,\n \"valid_data_length\": 8200,\n \"total_allocated\": null\n },\n \"name_size\": 0,\n \"name_offset\": null,\n \"data_flags\": \"(empty)\",\n \"instance\": 5,\n \"name\": \"\"\n },\n \"data\": null\n }\n }\n }\n ...\n]\n````\n\n## Installation\n\n### from PyPI\n```\n$ pip install mft2es\n```\n\n### from GitHub Releases\nThe version compiled into a binary using Nuitka is also available for use.\n\n```bash\n$ chmod +x ./mft2es\n$ ./mft2es {{options...}}\n```\n\n```powershell\n> mft2es.exe {{options...}}\n```\n\n## Contributing\n\nThe source code for mft2es is hosted at GitHub, and you may download, fork, and review it from this repository(https://github.com/sumeshi/mft2es). \nPlease report issues and feature requests. :sushi: :sushi: :sushi:\n\n## License\n\nmft2es is released under the [MIT](https://github.com/sumeshi/mft2es/blob/master/LICENSE) License.\n\nPowered by following libraries:\n- [pymft-rs](https://github.com/omerbenamram/pymft-rs)\n- [Nuitka](https://github.com/Nuitka/Nuitka)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A library for fast import of Windows Master File Table($MFT) into Elasticsearch.",
"version": "1.4.0",
"project_urls": {
"Homepage": "https://github.com/sumeshi/mft2es",
"Repository": "https://github.com/sumeshi/mft2es"
},
"split_keywords": [
"mft",
"elasticsearch",
"json"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7d651c946584f056f3b1b2bd21f9a4a66d935021ff90a8c591501bda060530d2",
"md5": "21558928de3b23b8353c8fc086950784",
"sha256": "fbde8982d38259b02d8383470fd53368ceeac7bb3520b5c9c53950d9277fec24"
},
"downloads": -1,
"filename": "mft2es-1.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "21558928de3b23b8353c8fc086950784",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11,<4.0",
"size": 11746,
"upload_time": "2023-11-26T13:42:17",
"upload_time_iso_8601": "2023-11-26T13:42:17.293182Z",
"url": "https://files.pythonhosted.org/packages/7d/65/1c946584f056f3b1b2bd21f9a4a66d935021ff90a8c591501bda060530d2/mft2es-1.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cdfefc12f4bb23ce0c0ef3de39e0a0a60e06a8f4cdbe38162f997e6e33bd26e0",
"md5": "3a7cb322bf7ad2c82832771537d8a460",
"sha256": "6ee746f634c0757136817d8649b2800ddec3dbe9fabe872984659fc189e6474a"
},
"downloads": -1,
"filename": "mft2es-1.4.0.tar.gz",
"has_sig": false,
"md5_digest": "3a7cb322bf7ad2c82832771537d8a460",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11,<4.0",
"size": 10384,
"upload_time": "2023-11-26T13:42:18",
"upload_time_iso_8601": "2023-11-26T13:42:18.993255Z",
"url": "https://files.pythonhosted.org/packages/cd/fe/fc12f4bb23ce0c0ef3de39e0a0a60e06a8f4cdbe38162f997e6e33bd26e0/mft2es-1.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-26 13:42:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sumeshi",
"github_project": "mft2es",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mft2es"
}
JSON
