| Name | getdents JSON |
| Version |
0.4.1
JSON |
| download |
| home_page | None |
| Summary | Python binding to linux syscall getdents64. |
| upload_time | 2024-08-15 15:49:00 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | BSD-2-Clause |
| keywords |
getdents
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
===============
Python getdents
===============
Iterate large directories efficiently with python.
About
=====
``python-getdents`` is a simple wrapper around Linux system call ``getdents64`` (see ``man getdents`` for details). `More details <http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html>`_ on approach.
TODO
====
* Verify that implementation works on platforms other than ``x86_64``.
Install
=======
.. code-block:: sh
pip install getdents
For development
---------------
.. code-block:: sh
python3 -m venv env
. env/bin/activate
pip install -e .[test]
Building Wheels
~~~~~~~~~~~~~~~
.. code-block:: sh
pip install cibuildwheel
cibuildwheel --platform linux --output-dir wheelhouse
Run tests
=========
.. code-block:: sh
ulimit -v 33554432 && py.test tests/
Or
.. code-block:: sh
ulimit -v 33554432 && ./setup.py test
Usage
=====
.. code-block:: python
from getdents import getdents
for inode, type, name in getdents('/tmp', 32768):
print(name)
Advanced
--------
.. code-block:: python
import os
from getdents import *
fd = os.open('/tmp', O_GETDENTS)
for inode, type, name in getdents_raw(fd, 2**20):
print({
DT_BLK: 'blockdev',
DT_CHR: 'chardev ',
DT_DIR: 'dir ',
DT_FIFO: 'pipe ',
DT_LNK: 'symlink ',
DT_REG: 'file ',
DT_SOCK: 'socket ',
DT_UNKNOWN: 'unknown ',
}[type], {
True: 'd',
False: ' ',
}[inode == 0],
name,
)
os.close(fd)
CLI
---
Usage
~~~~~
::
python-getdents [-h] [-b N] [-o NAME] PATH
Options
~~~~~~~
+--------------------------+-------------------------------------------------+
| Option | Description |
+==========================+=================================================+
| ``-b N`` | Buffer size (in bytes) to allocate when |
| | iterating over directory. Default is 32768, the |
| | same value used by glibc, you probably want to |
+--------------------------+ increase this value. Try starting with 16777216 |
| ``--buffer-size N`` | (16 MiB). Best performance is achieved when |
| | buffer size rounds to size of the file system |
| | block. |
+--------------------------+-------------------------------------------------+
| ``-o NAME`` | Output format: |
| | |
| | * ``plain`` (default) Print only names. |
| | * ``csv`` Print as comma-separated values in |
+--------------------------+ order: inode, type, name. |
| ``--output-format NAME`` | * ``csv-headers`` Same as ``csv``, but print |
| | headers on the first line also. |
| | * ``json`` output as JSON array. |
| | * ``json-stream`` output each directory entry |
| | as single json object separated by newline. |
+--------------------------+-------------------------------------------------+
Exit codes
~~~~~~~~~~
* 3 - Requested buffer is too large
* 4 - ``PATH`` not found.
* 5 - ``PATH`` is not a directory.
* 6 - Not enough permissions to read contents of the ``PATH``.
Examples
~~~~~~~~
.. code-block:: sh
python-getdents /path/to/large/dir
python -m getdents /path/to/large/dir
python-getdents /path/to/large/dir -o csv -b 16777216 > dir.csv
Raw data
{
"_id": null,
"home_page": null,
"name": "getdents",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "getdents",
"author": null,
"author_email": "ZipFile <zipfile.d@protonmail.com>",
"download_url": "https://files.pythonhosted.org/packages/26/ad/74c400ce490c34ac64a57708ca1e47f2d921be825fdbab40e9b0dce4a85d/getdents-0.4.1.tar.gz",
"platform": null,
"description": "===============\nPython getdents\n===============\n\nIterate large directories efficiently with python.\n\nAbout\n=====\n\n``python-getdents`` is a simple wrapper around Linux system call ``getdents64`` (see ``man getdents`` for details). `More details <http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html>`_ on approach.\n\nTODO\n====\n\n* Verify that implementation works on platforms other than ``x86_64``.\n\nInstall\n=======\n\n.. code-block:: sh\n\n pip install getdents\n\nFor development\n---------------\n\n.. code-block:: sh\n\n python3 -m venv env\n . env/bin/activate\n pip install -e .[test]\n\nBuilding Wheels\n~~~~~~~~~~~~~~~\n\n.. code-block:: sh\n\n pip install cibuildwheel\n cibuildwheel --platform linux --output-dir wheelhouse\n\nRun tests\n=========\n\n.. code-block:: sh\n\n ulimit -v 33554432 && py.test tests/\n\nOr\n\n.. code-block:: sh\n\n ulimit -v 33554432 && ./setup.py test\n\nUsage\n=====\n\n.. code-block:: python\n\n from getdents import getdents\n\n for inode, type, name in getdents('/tmp', 32768):\n print(name)\n\nAdvanced\n--------\n\n.. code-block:: python\n\n import os\n from getdents import *\n\n fd = os.open('/tmp', O_GETDENTS)\n\n for inode, type, name in getdents_raw(fd, 2**20):\n print({\n DT_BLK: 'blockdev',\n DT_CHR: 'chardev ',\n DT_DIR: 'dir ',\n DT_FIFO: 'pipe ',\n DT_LNK: 'symlink ',\n DT_REG: 'file ',\n DT_SOCK: 'socket ',\n DT_UNKNOWN: 'unknown ',\n }[type], {\n True: 'd',\n False: ' ',\n }[inode == 0],\n name,\n )\n\n os.close(fd)\n\nCLI\n---\n\nUsage\n~~~~~\n\n::\n\n python-getdents [-h] [-b N] [-o NAME] PATH\n\nOptions\n~~~~~~~\n\n+--------------------------+-------------------------------------------------+\n| Option | Description |\n+==========================+=================================================+\n| ``-b N`` | Buffer size (in bytes) to allocate when |\n| | iterating over directory. Default is 32768, the |\n| | same value used by glibc, you probably want to |\n+--------------------------+ increase this value. Try starting with 16777216 |\n| ``--buffer-size N`` | (16 MiB). Best performance is achieved when |\n| | buffer size rounds to size of the file system |\n| | block. |\n+--------------------------+-------------------------------------------------+\n| ``-o NAME`` | Output format: |\n| | |\n| | * ``plain`` (default) Print only names. |\n| | * ``csv`` Print as comma-separated values in |\n+--------------------------+ order: inode, type, name. |\n| ``--output-format NAME`` | * ``csv-headers`` Same as ``csv``, but print |\n| | headers on the first line also. |\n| | * ``json`` output as JSON array. |\n| | * ``json-stream`` output each directory entry |\n| | as single json object separated by newline. |\n+--------------------------+-------------------------------------------------+\n\nExit codes\n~~~~~~~~~~\n\n* 3 - Requested buffer is too large\n* 4 - ``PATH`` not found.\n* 5 - ``PATH`` is not a directory.\n* 6 - Not enough permissions to read contents of the ``PATH``.\n\nExamples\n~~~~~~~~\n\n.. code-block:: sh\n\n python-getdents /path/to/large/dir\n python -m getdents /path/to/large/dir\n python-getdents /path/to/large/dir -o csv -b 16777216 > dir.csv\n",
"bugtrack_url": null,
"license": "BSD-2-Clause",
"summary": "Python binding to linux syscall getdents64.",
"version": "0.4.1",
"project_urls": {
"Source": "https://github.com/ZipFile/python-getdents"
},
"split_keywords": [
"getdents"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ef0b618c2ca652418011f4b5665662d3c02dcbb8b737d44489e08dff4f34dff1",
"md5": "8bd2ab198c48781ac3a60269ccfde3f6",
"sha256": "f79dcd666c9fc17c09d8d7833a13f254b49fe82a65f2922dbb8d306d7e93a0d5"
},
"downloads": -1,
"filename": "getdents-0.4.1-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "8bd2ab198c48781ac3a60269ccfde3f6",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 16867,
"upload_time": "2024-08-15T15:48:54",
"upload_time_iso_8601": "2024-08-15T15:48:54.621186Z",
"url": "https://files.pythonhosted.org/packages/ef/0b/618c2ca652418011f4b5665662d3c02dcbb8b737d44489e08dff4f34dff1/getdents-0.4.1-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f5066eba5c00a3132d20f2e5f9b2ed43ca7fe791892c83b96605c7df056d45f0",
"md5": "71645e0c391bb7ac52db2e8cb259305e",
"sha256": "cd9681dc61f63b67b8127424aec26ebde216c9bcf92f8c8597fe1f64505a26fd"
},
"downloads": -1,
"filename": "getdents-0.4.1-cp38-abi3-musllinux_1_2_x86_64.whl",
"has_sig": false,
"md5_digest": "71645e0c391bb7ac52db2e8cb259305e",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 16768,
"upload_time": "2024-08-15T15:48:56",
"upload_time_iso_8601": "2024-08-15T15:48:56.370896Z",
"url": "https://files.pythonhosted.org/packages/f5/06/6eba5c00a3132d20f2e5f9b2ed43ca7fe791892c83b96605c7df056d45f0/getdents-0.4.1-cp38-abi3-musllinux_1_2_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bfd53c2f90c472ee79a06238c3706984e4819c96bdfe6732241b441dc7f06ca1",
"md5": "1f12201839cb4ff2c4c0e669a0e7885f",
"sha256": "4096ace1857970c70a6db3df01340cb0285dbaac01c4869fc70bb84a0dd07e87"
},
"downloads": -1,
"filename": "getdents-0.4.1-pp310-pypy310_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "1f12201839cb4ff2c4c0e669a0e7885f",
"packagetype": "bdist_wheel",
"python_version": "pp310",
"requires_python": ">=3.8",
"size": 13131,
"upload_time": "2024-08-15T15:48:57",
"upload_time_iso_8601": "2024-08-15T15:48:57.405960Z",
"url": "https://files.pythonhosted.org/packages/bf/d5/3c2f90c472ee79a06238c3706984e4819c96bdfe6732241b441dc7f06ca1/getdents-0.4.1-pp310-pypy310_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ef525890c3f4b353b182a8f45c5dcabfe0d0f40f0ee6ea82f9415ab6ae1262a7",
"md5": "5cf624871543c17507c504a738fe80a6",
"sha256": "4b521f830a803efdca5468576e24b6e4abbe0730dbcf9331a4590035e5e8a344"
},
"downloads": -1,
"filename": "getdents-0.4.1-pp38-pypy38_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "5cf624871543c17507c504a738fe80a6",
"packagetype": "bdist_wheel",
"python_version": "pp38",
"requires_python": ">=3.8",
"size": 13129,
"upload_time": "2024-08-15T15:48:58",
"upload_time_iso_8601": "2024-08-15T15:48:58.616419Z",
"url": "https://files.pythonhosted.org/packages/ef/52/5890c3f4b353b182a8f45c5dcabfe0d0f40f0ee6ea82f9415ab6ae1262a7/getdents-0.4.1-pp38-pypy38_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7ffd4ca64e6f4e25466bf81ec5209f6b6bcb4813dfb4196aafcb7d9fb94d22dc",
"md5": "d88480bae8bf347d1530c8323a60e117",
"sha256": "885c1f0915bf82651c657d93aa3d635b6e8eb22b9ab74c7ab0bb218a10d4d23a"
},
"downloads": -1,
"filename": "getdents-0.4.1-pp39-pypy39_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "d88480bae8bf347d1530c8323a60e117",
"packagetype": "bdist_wheel",
"python_version": "pp39",
"requires_python": ">=3.8",
"size": 13131,
"upload_time": "2024-08-15T15:48:59",
"upload_time_iso_8601": "2024-08-15T15:48:59.399472Z",
"url": "https://files.pythonhosted.org/packages/7f/fd/4ca64e6f4e25466bf81ec5209f6b6bcb4813dfb4196aafcb7d9fb94d22dc/getdents-0.4.1-pp39-pypy39_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "26ad74c400ce490c34ac64a57708ca1e47f2d921be825fdbab40e9b0dce4a85d",
"md5": "2acb00e4e78b226dae5af8fd62ec3188",
"sha256": "75332d2a20d4ac7b0f623b98d5aaabe71236ab346013edf913e795c2952522d9"
},
"downloads": -1,
"filename": "getdents-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "2acb00e4e78b226dae5af8fd62ec3188",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 11701,
"upload_time": "2024-08-15T15:49:00",
"upload_time_iso_8601": "2024-08-15T15:49:00.597782Z",
"url": "https://files.pythonhosted.org/packages/26/ad/74c400ce490c34ac64a57708ca1e47f2d921be825fdbab40e9b0dce4a85d/getdents-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-15 15:49:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ZipFile",
"github_project": "python-getdents",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "getdents"
}
