# Overview
Often there is a need to temporarily save a file to 'disk' for the consumption of external tools. Or maybe you can pipe some input info to an external tool
but has no way of forcing such external tool to pipe its output straight to your software: **it wants to write a file to disk**.
Disk operations are slow and if repeated too often can shorten the lifespan of underlying media.
On Linux, most distributions offer a /tmp directory BUT it is usually on physical media. However, modern distributions often offer at east two
places where one can safely create temporary files in RAM: /dev/run/<uid>, /run/shm and /dev/shm
/dev/run/<uid> is ideal for your temporary files. It is writable and readable only by your user.
/dev/shm is usually world-readable and world-writable (just like /tmp), it is often used for IPC (inter process communication) and can serve well as a temporary RAM-based tempdir.
This module is very simple and tries not to reinvent the wheel. It will check /tmp to see if it in a ramdisk or not. And it will also check
if you have other options where to place your temporary files/dirs on a memory-based file system like tmpfs or ramfs.
Once you know the suitable path on a memory-based file system where you can have your files, you are well served by python's builtin modules and external packages like pathlib
or pyfilesystem2 to move on to do your things.
**To know more, I recommend the following links:**
https://unix.stackexchange.com/questions/162900/what-is-this-folder-run-user-1000
https://superuser.com/questions/45342/when-should-i-use-dev-shm-and-when-should-i-use-tmp
# API
This module searches for paths hosted on filesystems of type belonging to MEM_BASED_FS=['tmpfs', 'ramfs']
Paths in SUITABLE_PATHS=['/tmp', '/run/user/{uid}', '/run/shm', '/dev/shm'] are searched and the first path found that exists and is stored on a filesystem whose type belongs to MEM_BASED_FS will be used as the tempdir.
If no suitable path is found, then if fallback = True, we will fallback to default tempdir (as determined by tempfile stdlib). If fallback is a path, then we will default to it.
If fallback is false, a RunTimeError exception is raised.
The MemoryTempfile constructor has arguments that let you change how the algorithm works.
You can change the order of paths (with 'preferred_paths'), add new paths to the search (with 'preferred_paths' and/or with 'additional_paths') and you can exclude certain paths from SUITABLE_PATHS (with removed_paths). All paths containing the string {uid} will have it replaced by the user id.
You can change the filesystem types you accept (with filesystem_types) and specify whether or not to fallback to a vanilla tempdir as a last resort.
Then, all methods available from tempfile stdlib are available through MemoryTempfile.
# The constructor:
**Here is the list of accepted parameters:**
- preferred_paths: list = None
- remove_paths: list or bool = None
- additional_paths: list = None
- filesystem_types: list = None
- fallback: str or bool = None
The path list that will be searched from first to last item will be constructed using the algorith:
paths = preferred_paths + (SUITABLE_PATHS - remove_paths) + additional_paths
If remove_paths is boolean 'true', SUITABLE_PATHS will be eliminated, this is a way for you to take complete control of the path list
that will be used without relying on this package's hardcoded constants.
The only other hardcoded constant MEM_BASED_FS=['tmpfs', 'ramfs'] will not be used at all if you pass your own 'filesystem_types' argument.
By the way, if you wish to add other file system types, you must match what Linux uses in /proc/self/mountinfo (at the 9th column).
# Requirements
- Python 3
- Works only on Linux
- Compatible with chroot and/or namespaces, needs access to /proc/self/mountinfo
# Usage
## Example 1:
from memory_tempfile import MemoryTempfile
tempfile = MemoryTempfile()
with tempfile.TemporaryFile() as tf:
# as usual...
## Example 2:
# We now do not want to use /dev/shm or /run/shm and no ramfs paths
# If /run/user/{uid} is available, we prefer it to /tmp
# And we want to try /var/run as a last resort
# If all fails, fallback to platform's tmp dir
from memory_tempfile import MemoryTempfile
import memory_tempfile
# By the way, all paths with string {uid} will have it replaced with the user id
tempfile = MemoryTempfile(preferred_paths=['/run/user/{uid}'], remove_paths=['/dev/shm', '/run/shm'],
additional_paths=['/var/run'], filesystem_types=['tmpfs'], fallback=True)
if tempfile.found_mem_tempdir():
print('We could use any of the followig paths: {}'.format(tempfile.get_usable_mem_tempdir_paths()))
print('And we are using now: {}'.format(tempfile.gettempdir()))
with tempfile.NamedTemporaryFile() as ntf:
# use it as usual...
pass
Raw data
{
"_id": null,
"home_page": "https://github.com/mbello/memory-tempfile",
"name": "memory-tempfile",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6,<4.0",
"maintainer_email": "",
"keywords": "tempfile in RAM,tempfile in memory,file in memory,tempfs,ramdisk,/dev/shm,/run/user",
"author": "mbello",
"author_email": "mbello@users.noreply.github.com",
"download_url": "https://files.pythonhosted.org/packages/7c/da/588403f523b1dfc9f70891b21d70f3d0f23b8c56985ca60af6b99c2c9dfc/memory-tempfile-2.2.3.tar.gz",
"platform": "",
"description": "# Overview\n\nOften there is a need to temporarily save a file to 'disk' for the consumption of external tools. Or maybe you can pipe some input info to an external tool\nbut has no way of forcing such external tool to pipe its output straight to your software: **it wants to write a file to disk**.\nDisk operations are slow and if repeated too often can shorten the lifespan of underlying media.\n\nOn Linux, most distributions offer a /tmp directory BUT it is usually on physical media. However, modern distributions often offer at east two\nplaces where one can safely create temporary files in RAM: /dev/run/<uid>, /run/shm and /dev/shm\n\n/dev/run/<uid> is ideal for your temporary files. It is writable and readable only by your user.\n/dev/shm is usually world-readable and world-writable (just like /tmp), it is often used for IPC (inter process communication) and can serve well as a temporary RAM-based tempdir.\n\nThis module is very simple and tries not to reinvent the wheel. It will check /tmp to see if it in a ramdisk or not. And it will also check\nif you have other options where to place your temporary files/dirs on a memory-based file system like tmpfs or ramfs.\n\nOnce you know the suitable path on a memory-based file system where you can have your files, you are well served by python's builtin modules and external packages like pathlib\nor pyfilesystem2 to move on to do your things.\n\n**To know more, I recommend the following links:**\nhttps://unix.stackexchange.com/questions/162900/what-is-this-folder-run-user-1000\nhttps://superuser.com/questions/45342/when-should-i-use-dev-shm-and-when-should-i-use-tmp\n\n\n# API\nThis module searches for paths hosted on filesystems of type belonging to MEM_BASED_FS=['tmpfs', 'ramfs']\nPaths in SUITABLE_PATHS=['/tmp', '/run/user/{uid}', '/run/shm', '/dev/shm'] are searched and the first path found that exists and is stored on a filesystem whose type belongs to MEM_BASED_FS will be used as the tempdir.\nIf no suitable path is found, then if fallback = True, we will fallback to default tempdir (as determined by tempfile stdlib). If fallback is a path, then we will default to it.\nIf fallback is false, a RunTimeError exception is raised.\n\nThe MemoryTempfile constructor has arguments that let you change how the algorithm works.\nYou can change the order of paths (with 'preferred_paths'), add new paths to the search (with 'preferred_paths' and/or with 'additional_paths') and you can exclude certain paths from SUITABLE_PATHS (with removed_paths). All paths containing the string {uid} will have it replaced by the user id.\nYou can change the filesystem types you accept (with filesystem_types) and specify whether or not to fallback to a vanilla tempdir as a last resort.\n\nThen, all methods available from tempfile stdlib are available through MemoryTempfile.\n\n# The constructor:\n\n**Here is the list of accepted parameters:**\n- preferred_paths: list = None\n- remove_paths: list or bool = None\n- additional_paths: list = None\n- filesystem_types: list = None\n- fallback: str or bool = None\n\nThe path list that will be searched from first to last item will be constructed using the algorith:\n\n paths = preferred_paths + (SUITABLE_PATHS - remove_paths) + additional_paths\n\nIf remove_paths is boolean 'true', SUITABLE_PATHS will be eliminated, this is a way for you to take complete control of the path list\nthat will be used without relying on this package's hardcoded constants.\n\nThe only other hardcoded constant MEM_BASED_FS=['tmpfs', 'ramfs'] will not be used at all if you pass your own 'filesystem_types' argument.\nBy the way, if you wish to add other file system types, you must match what Linux uses in /proc/self/mountinfo (at the 9th column).\n\n# Requirements\n\n- Python 3\n- Works only on Linux\n- Compatible with chroot and/or namespaces, needs access to /proc/self/mountinfo\n\n# Usage\n\n## Example 1:\n\n from memory_tempfile import MemoryTempfile\n \n tempfile = MemoryTempfile()\n \n with tempfile.TemporaryFile() as tf:\n # as usual...\n \n## Example 2:\n\n # We now do not want to use /dev/shm or /run/shm and no ramfs paths\n # If /run/user/{uid} is available, we prefer it to /tmp\n # And we want to try /var/run as a last resort\n # If all fails, fallback to platform's tmp dir\n \n from memory_tempfile import MemoryTempfile\n import memory_tempfile\n\n # By the way, all paths with string {uid} will have it replaced with the user id\n tempfile = MemoryTempfile(preferred_paths=['/run/user/{uid}'], remove_paths=['/dev/shm', '/run/shm'],\n additional_paths=['/var/run'], filesystem_types=['tmpfs'], fallback=True)\n \n if tempfile.found_mem_tempdir():\n print('We could use any of the followig paths: {}'.format(tempfile.get_usable_mem_tempdir_paths()))\n print('And we are using now: {}'.format(tempfile.gettempdir()))\n \n with tempfile.NamedTemporaryFile() as ntf:\n # use it as usual...\n pass\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Helper functions to identify and use paths on the OS (Linux-only for now) where RAM-based tempfiles can be created.",
"version": "2.2.3",
"split_keywords": [
"tempfile in ram",
"tempfile in memory",
"file in memory",
"tempfs",
"ramdisk",
"/dev/shm",
"/run/user"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "6e7b5bdf9b79dc02dee393214115d86a",
"sha256": "dcea50b967f75b494fae8e242dc095e97280cfe6a53631473887b05f943cafeb"
},
"downloads": -1,
"filename": "memory_tempfile-2.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6e7b5bdf9b79dc02dee393214115d86a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6,<4.0",
"size": 5719,
"upload_time": "2020-03-11T00:58:44",
"upload_time_iso_8601": "2020-03-11T00:58:44.648520Z",
"url": "https://files.pythonhosted.org/packages/4a/08/43af249eed4ffadc4df084549994ad606a62119fa37bf3a3857acd2d61f4/memory_tempfile-2.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "f6a373ab06257702a5df29748332ff85",
"sha256": "4f23842924359e0ef9ecf9194f9e01437e119a96e77c9ad26af0e706aca849d5"
},
"downloads": -1,
"filename": "memory-tempfile-2.2.3.tar.gz",
"has_sig": false,
"md5_digest": "f6a373ab06257702a5df29748332ff85",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6,<4.0",
"size": 5623,
"upload_time": "2020-03-11T00:58:42",
"upload_time_iso_8601": "2020-03-11T00:58:42.807898Z",
"url": "https://files.pythonhosted.org/packages/7c/da/588403f523b1dfc9f70891b21d70f3d0f23b8c56985ca60af6b99c2c9dfc/memory-tempfile-2.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2020-03-11 00:58:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "mbello",
"github_project": "memory-tempfile",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "memory-tempfile"
}
JSON
