| Name | slurm2sql JSON |
| Version |
0.9.2
JSON |
| download |
| home_page | None |
| Summary | Import Slurm accounting database from sacct to sqlite3 database |
| upload_time | 2024-10-23 12:33:10 |
| maintainer | None |
| docs_url | None |
| author | Richard Darst |
| requires_python | >=3.6 |
| license | None |
| keywords |
slurm
sqlite3
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Read a Slurm accounting database to a sqlite3 file
==================================================
This contains a utility, ``slurm2sql``, which uses the `Slurm
<https://slurm.schedmd.com/overview>`__ workload manager's ``sacct``,
to export statistics from jobs and load them to a well-formed SQLite3
file (the database is also made so that it can be quried with DuckDB).
This file can then be queried for analytics much more easily than the
raw database or your own exports. The main features are:
- Parse ``sacct`` output (this was made before it had JSON output,
which you might want to look at instead - it's hard to use though).
- Preprocess all the values in to basic units, including values like
GPU usage that currently have to be extracted from other fields.
Even if SQLite isn't what you need, it provides an easy intermediate
file on the way to convert to whatever format you want. In
particular, it defines the database so that it can be used with
DuckDB, which is a more efficient tool for analytics.
There are also some command line frontends, ``slurm2sql-sacct`` and
``slurm2sql-seff`` that use this parsing to print out data in better
forms than built-in Slurm commands. This is especially useful for
``sacct``. You can design your own tools like this.
Installation
------------
Normal ``pip`` installation, name ``slurm2sql`` for the command line
programs. This installs the library and command line programs.
::
pip install slurm2sql
There is only a single file with no depencecies for the core
``slurm2sql`` library (which could also be manually downloaded - HPC,
right?), though the command line programs require ``tabulate``. It's
made to support very old Python.
Usage
-----
``slurm2sql``
~~~~~~~~~~~~~
Sample usage::
slurm2sql.py OUTPUT_DB -- [SACCT_FILTER_OPTIONS]
For example, to get all data from July and August (``-S``) for all
users (``-a``)::
slurm2sql.py sincejuly.sqlite3 -- -S 2019-07-1 -a
To get the data from the last *N* days. This will, day by day, get
each of these history and cumulatively update the database. This
updates a database by default, so that it can be used every day in
order to efficiently keep a running database. The ``-u`` option means
"don't delete existing database" (jobs with the same JobID get
updated, not duplicated)::
slurm2sql.py --history-days=N -u sincejuly.sqlite3 -- -a
The ``--history-start=YYYY-MM-DD`` option can do a similar thing
starting from a certain day, and ``--history=DD-HH:MM:SS`` starts
collecting from a given interval of time ago (the time format is as in
Slurm).
To resume from where you left off, first run with one of the history
options. Then, you can do ``--history-resume`` (no ``-u`` needed) and
it will continue fetching day-by-day from the time you last fetched.
You can also run this every day, to first load old historykeep a database updated::
slurm2sql.py --history-days=N -u sincejuly.sqlite3 -- -a
slurm2sql.py --history-resume sincejuly.sqlite3 -- -a
``slurm2sql-sacct``
~~~~~~~~~~~~~~~~~~~
This probably isn't the most useful part. Look at command line options.
.. code-block:: console
$ slurm2sql-sacct SACCT_FILTER
``slurm2sql-seff``
~~~~~~~~~~~~~~~~~~
This is more useful: it prints ``seff`` like output in a tabular
format. MemReqGiB is per-node, to compare withMaxRSSGiB.
.. code-block:: console
$ slurm2sql-sacct SACCT_FILTER
.. code-block:: console
$ slurm2sql-seff -S now-3day
JobID User hours NCPUS CPUeff MemReqGiB MaxRSSGiB MemEff NGpus GPUeff read_MiBps write_MiBps
------- ------- ----- ----- ------ --------- --------- ------ ----- ------ ---------- -----------
1860854 darstr1 0.28 1 87% 50 9.76 20% 213.88 14.51
1877467 darstr1 0 0 0% 0 0%
1884493 darstr1 0 1 0% 0.49 0 0%
1884494 darstr1 0 1 0% 0.49 0 0%
From Python
~~~~~~~~~~~
It can also be used from Python as what is essentially a glorified
parser.
.. code-block:: python
db = sqlite3.connect(':memory:')
slurm2sql.slurm2sql(db, ['-S', '2019-08-26'])
# For example, you can then convert to a dataframe:
import pandas as pd
df = pd.read_sql('SELECT * FROM slurm', db)
From DuckDB
~~~~~~~~~~~
DuckDB is a lot like SQLite, but column-oriented and optimized for
fast processing of data. The main downsides are slow inserts and
columns must have consistent data types, but that's the tradeoff we
need. Slurm2sql's SQLite database is created with type definitions,
so that you can easily open it with DuckDB even without conversion:
.. code-block:: console
$ duckdb dump.sqlite3
Or for even more speed, make a temporary in-memory copy (or this could
also be made into a file):
.. code-block:: sql
-- command line: $ duckdb database.db
ATTACH ':memory:' AS tmp;
CREATE TABLE tmp.slurm AS (SELECT * FROM slurm);
USE tmp; -- optional but makes tmp the default
Converting to DuckDB:
.. code-block:: console
$ duckdb new.duckdb "CREATE TABLE slurm AS (SELECT * FROM sqlite_scan('original.sqlite3', 'slurm'))"
Using via DuckDB from Python (with the raw sqlite database):
.. code-block:: python
conn = duckdb.connect("database.sqlite3")
conn.execute("select avg(cputime) from slurm").df()
Database format
---------------
Tables and views:
* Table ``slurm``: the main table with all of the data. There is one
row for each item returned by ``sacct``.
* View ``allocations``: has only the jobs (not job steps) (``where
JobStep is null``).
* View ``eff``: Does a lot of processing of ``slurm`` to produce some
``CPUEff``, ``MemEff``, and ``GPUeff`` values (0.0-1.0 usage
fractions), in addition to a bit more.
In general, there is one column for each item returned by ``sacct``,
but some of them are converted into a more useful form. Some columns
are added by re-processing other columns. See ``COLUMNS`` in
``slurm2sql.py`` for details. Extra columns can easily be added.
Developer note: There are two types of converter functions to make the
columns: easy ones, which map one slurm column directly to a database
column via a function, and line functions, which take the whole row
and can do arbitrary remixing of the data (to compute things like CpuEff.
Columns
~~~~~~~
All column values are converted to standard units: *bytes* (not MB,
KB, etc), *seconds*, *fraction 0.0-1.0* for things like
percentages, and *unixtime*.
Columns which are the same in raw ``sacct`` output aren't documented
specifically here (but note the default units above).
Below are some notable columns which do not exist in sacct (for the
rest, check out the `sacct manual page <https://slurm.schedmd.com/sacct.html#lbAF>`_). It's good
to verify that any of our custom columns make sense before trusting
them. For other columns, check ``man sacct``.
* ``Time``: approximation of last active time of a job. The first of
these that exists: ``End``, ``Start``, ``Submitted``. This is
intended to be used when you need to classify a job by when it ran,
but you don't care to be that specific. (Only the Time column is
indexed by default, not the other times)
* ``Submit``, ``Start``, ``End``: like the sacct equivalents,
but unixtime. Assume that the sacct timestamps are in localtime of
the machine doing the conversion. (``slurm2sql.unixtime`` converts
slurm-format timestamp to unixtime)
* ``QueueTime`` is Start-Submit in seconds. Start/End do not include
timezones, so expect inaccuracies around summer time changes.
* Job IDs. Slurm Job ID is by default of format
``JobID.JobStep`` or ``ArrayJobID_ArrayTaskID.JobStep``.
Furthermore, each array job has a "Raw JobID" (different for each
job, and is an actual JobID) in addition to the "ArrayJobID" which
is the same for all jobs in an array. We split all of these
different IDs into the following fields:
* ``JobID``: The full raw value that Slurm gives. The same for each
job in an array.
Only the integer Job ID, without the trailing array
tasks or job IDs. For array jobs, this is the "Raw JobID" as
described above, use ``ArrayJobID`` to filter jobs that are the
same. Integer
* ``JobIDnostep``: The part of JobID without anything after the ``.``
(no steps)
* ``JobIDonly``: The integer part of the JobID.
* ``JobIDRawonly``: The integer part of the Raw JobID (so this is
different for each job in an aray).
* ``ArrayTaskID``: As used above. Integer on null.
* ``JobStep``: Job step - only. If you SQL filter for ``StepID is
null`` you get only the main allocations. String.
* Note: HetJob offsets are not currently handled and silently
stripped out and give invalid data. File an issue and this will
be added.
* ``ReqMem``: The raw slurm value in a format like "5Gn". Instead of
parsing this, you probably want to use one of the other values below.
* ``ReqMemNode``, ``ReqMemCPU``: Requested memory per node or CPU,
either taken from ReqMem (if it matches) or computed (you might want
to check our logic if you rely on this). In Slurm, you
can request memory either per-node or per-core, and this calculates
the other one for you.
* ``ReqMemType``: ``c`` if the user requested mem-per-core originally,
``n`` if mem-per-node. Extracted from ``ReqMem``. Modern Slurm has
nothing here, and the column value is null.
* ``ReqMemRaw``: The numeric value of the ``ReqMem``, whether it is
``c`` or ``n``.
* ``ReqGPU``: Number of GPUs requested. Extracted from ``ReqTRES``.
* GPU information. These use values from the ``TRESUsageInAve``
fields in modern Slurm
* ``GpuMem``: ``gres/gpumem``
* ``GpuUtil``: ``gres/gpuutil`` (fraction 0.0-1.0).
* ``NGpus``: Number of GPUs. Should be the same as ``ReqGPU``, but
who knows.
* ``GpuUtilTot``, ``GpuMemTot``: like above but using the
``TRESUsageInTot`` sacct field.
* ``MemEff``: This is null in the Slurm table now, since Slurm gives
ReqMem in allocations and memory used in steps. The ``eff`` table
calculates this now.
* ``CPUEff``: CPU efficiency (0.0-1.0). All the same caveats as above
apply: test before trusting.
Quick reference of the other most important columns from the
accounting database that are hardest to remember:
* ``Elapsed``: Wall clock time
* ``CPUTime``: Reserved CPU time (Elapsed * number of CPUs). CPUEff ≈
TotalCPU/CPUTime = TotalCPU/(NCPUs x Elapsed)
* ``TotalCPU``: SystemCPU + TotalCPU, seconds of productive work.
The ``eff`` table adds the following:
* ``CPUEff``: like CPUEff but for the whole job
* ``MemEff``: Memory efficiency for the whole job (max(MaxRSS) /
ReqMem)
* And more, see the code for now.
Changelog
---------
Next
* This is the biggest column clean-up in a while.
* Add slurm2sql-{seff,sacct} commands.
* JobID columns adjusted: ``JobID`` is the raw thing that slurm gives,
``*only`` integer IDs without any trailing things,
``JobIDrawonly`` is the RawJobID without any trailing things.
* ReqMem has been updated: it no longer parses ``n`` and ``c``
suffixes for mem-per-node/cpu, and that respective column has been
removed.
* MemEff has been removed from the ``slurm`` table, since it is always
empty. The ``eff`` view has been added instead.
0.9.1
* Slurm >= 20.11 deprecates the ``AllocGRES`` and ``ReqGRES`` columns
(using ``Alloc/ReqTRES`` instead).
* From this slurm2sql version, a ReqTRES column will be requested
and databases will need to be re-created (or manually added to the
databases).
* If run on Slurm > 20.11, it will not request ReqGRES and only use
ReqTRES.
Development and maintenance
---------------------------
This could be considered beta right now, but it works and is in use by
people. If this is important for you, comment about your use case
in the Github issue tracker. Watch the repo if you want to give
comments on future data schema updates and backwards compatibility
when Slurm changes.
There are many different variations of Slurm, if it doesn't
work for you, send an issue or pull request to help us make it more
general - development is only done in response to feedback.
Development principles:
- All values are the most basic (metric) units: bytes, seconds,
seconds-since-epoch, etc.
- Try to use existing Slurm column names as much as possible (even if
they are hard to remember).
- Try to support as many Slurm versions as possible, but if something
becomes hard to support, don't worry too much about breaking
compatibly. SchedMD support slurm for 18 months after release. Try
to support at least those versions. (Until someone asks for it,
don't assume we can import data from very old Slurm versions)
- Don't try to maintain database compatibility. It's expected that for
all schema changes, you have to delete and re-import. But try to
avoid this if not needed.
Release process::
python setup.py sdist bdist_wheel
twine upload [--repository-url https://test.pypi.org/legacy/] dist/*0.9.0*
Originally developed at Aalto University, Finland.
Raw data
{
"_id": null,
"home_page": null,
"name": "slurm2sql",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "slurm, sqlite3",
"author": "Richard Darst",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/3f/21/9b17fea0bf2a67092610d6b756bfeffb97a6e19bd7fe92f8b0d0a8bfd0de/slurm2sql-0.9.2.tar.gz",
"platform": null,
"description": "Read a Slurm accounting database to a sqlite3 file\n==================================================\n\nThis contains a utility, ``slurm2sql``, which uses the `Slurm\n<https://slurm.schedmd.com/overview>`__ workload manager's ``sacct``,\nto export statistics from jobs and load them to a well-formed SQLite3\nfile (the database is also made so that it can be quried with DuckDB).\nThis file can then be queried for analytics much more easily than the\nraw database or your own exports. The main features are:\n\n- Parse ``sacct`` output (this was made before it had JSON output,\n which you might want to look at instead - it's hard to use though).\n- Preprocess all the values in to basic units, including values like\n GPU usage that currently have to be extracted from other fields.\n\nEven if SQLite isn't what you need, it provides an easy intermediate\nfile on the way to convert to whatever format you want. In\nparticular, it defines the database so that it can be used with\nDuckDB, which is a more efficient tool for analytics.\n\nThere are also some command line frontends, ``slurm2sql-sacct`` and\n``slurm2sql-seff`` that use this parsing to print out data in better\nforms than built-in Slurm commands. This is especially useful for\n``sacct``. You can design your own tools like this.\n\n\n\nInstallation\n------------\n\nNormal ``pip`` installation, name ``slurm2sql`` for the command line\nprograms. This installs the library and command line programs.\n\n::\n\n pip install slurm2sql\n\nThere is only a single file with no depencecies for the core\n``slurm2sql`` library (which could also be manually downloaded - HPC,\nright?), though the command line programs require ``tabulate``. It's\nmade to support very old Python.\n\n\n\nUsage\n-----\n\n\n``slurm2sql``\n~~~~~~~~~~~~~\n\nSample usage::\n\n slurm2sql.py OUTPUT_DB -- [SACCT_FILTER_OPTIONS]\n\n\nFor example, to get all data from July and August (``-S``) for all\nusers (``-a``)::\n\n slurm2sql.py sincejuly.sqlite3 -- -S 2019-07-1 -a\n\n\nTo get the data from the last *N* days. This will, day by day, get\neach of these history and cumulatively update the database. This\nupdates a database by default, so that it can be used every day in\norder to efficiently keep a running database. The ``-u`` option means\n\"don't delete existing database\" (jobs with the same JobID get\nupdated, not duplicated)::\n\n slurm2sql.py --history-days=N -u sincejuly.sqlite3 -- -a\n\nThe ``--history-start=YYYY-MM-DD`` option can do a similar thing\nstarting from a certain day, and ``--history=DD-HH:MM:SS`` starts\ncollecting from a given interval of time ago (the time format is as in\nSlurm).\n\nTo resume from where you left off, first run with one of the history\noptions. Then, you can do ``--history-resume`` (no ``-u`` needed) and\nit will continue fetching day-by-day from the time you last fetched.\nYou can also run this every day, to first load old historykeep a database updated::\n\n slurm2sql.py --history-days=N -u sincejuly.sqlite3 -- -a\n slurm2sql.py --history-resume sincejuly.sqlite3 -- -a\n\n\n``slurm2sql-sacct``\n~~~~~~~~~~~~~~~~~~~\n\nThis probably isn't the most useful part. Look at command line options.\n\n.. code-block:: console\n\n $ slurm2sql-sacct SACCT_FILTER\n\n\n``slurm2sql-seff``\n~~~~~~~~~~~~~~~~~~\n\nThis is more useful: it prints ``seff`` like output in a tabular\nformat. MemReqGiB is per-node, to compare withMaxRSSGiB.\n\n.. code-block:: console\n\n $ slurm2sql-sacct SACCT_FILTER\n\n.. code-block:: console\n\n $ slurm2sql-seff -S now-3day\n JobID User hours NCPUS CPUeff MemReqGiB MaxRSSGiB MemEff NGpus GPUeff read_MiBps write_MiBps\n ------- ------- ----- ----- ------ --------- --------- ------ ----- ------ ---------- -----------\n 1860854 darstr1 0.28 1 87% 50 9.76 20% 213.88 14.51\n 1877467 darstr1 0 0 0% 0 0%\n 1884493 darstr1 0 1 0% 0.49 0 0%\n 1884494 darstr1 0 1 0% 0.49 0 0%\n\n\nFrom Python\n~~~~~~~~~~~\n\nIt can also be used from Python as what is essentially a glorified\nparser.\n\n.. code-block:: python\n\n db = sqlite3.connect(':memory:')\n slurm2sql.slurm2sql(db, ['-S', '2019-08-26'])\n\n # For example, you can then convert to a dataframe:\n import pandas as pd\n df = pd.read_sql('SELECT * FROM slurm', db)\n\n\nFrom DuckDB\n~~~~~~~~~~~\n\nDuckDB is a lot like SQLite, but column-oriented and optimized for\nfast processing of data. The main downsides are slow inserts and\ncolumns must have consistent data types, but that's the tradeoff we\nneed. Slurm2sql's SQLite database is created with type definitions,\nso that you can easily open it with DuckDB even without conversion:\n\n.. code-block:: console\n\n $ duckdb dump.sqlite3\n\nOr for even more speed, make a temporary in-memory copy (or this could\nalso be made into a file):\n\n.. code-block:: sql\n\n -- command line: $ duckdb database.db\n ATTACH ':memory:' AS tmp;\n CREATE TABLE tmp.slurm AS (SELECT * FROM slurm);\n USE tmp; -- optional but makes tmp the default\n\nConverting to DuckDB:\n\n.. code-block:: console\n\n $ duckdb new.duckdb \"CREATE TABLE slurm AS (SELECT * FROM sqlite_scan('original.sqlite3', 'slurm'))\"\n\nUsing via DuckDB from Python (with the raw sqlite database):\n\n.. code-block:: python\n\n conn = duckdb.connect(\"database.sqlite3\")\n conn.execute(\"select avg(cputime) from slurm\").df()\n\n\n\nDatabase format\n---------------\n\nTables and views:\n\n* Table ``slurm``: the main table with all of the data. There is one\n row for each item returned by ``sacct``.\n* View ``allocations``: has only the jobs (not job steps) (``where\n JobStep is null``).\n* View ``eff``: Does a lot of processing of ``slurm`` to produce some\n ``CPUEff``, ``MemEff``, and ``GPUeff`` values (0.0-1.0 usage\n fractions), in addition to a bit more.\n\nIn general, there is one column for each item returned by ``sacct``,\nbut some of them are converted into a more useful form. Some columns\nare added by re-processing other columns. See ``COLUMNS`` in\n``slurm2sql.py`` for details. Extra columns can easily be added.\n\nDeveloper note: There are two types of converter functions to make the\ncolumns: easy ones, which map one slurm column directly to a database\ncolumn via a function, and line functions, which take the whole row\nand can do arbitrary remixing of the data (to compute things like CpuEff.\n\nColumns\n~~~~~~~\n\nAll column values are converted to standard units: *bytes* (not MB,\nKB, etc), *seconds*, *fraction 0.0-1.0* for things like\npercentages, and *unixtime*.\n\nColumns which are the same in raw ``sacct`` output aren't documented\nspecifically here (but note the default units above).\n\nBelow are some notable columns which do not exist in sacct (for the\nrest, check out the `sacct manual page <https://slurm.schedmd.com/sacct.html#lbAF>`_). It's good\nto verify that any of our custom columns make sense before trusting\nthem. For other columns, check ``man sacct``.\n\n* ``Time``: approximation of last active time of a job. The first of\n these that exists: ``End``, ``Start``, ``Submitted``. This is\n intended to be used when you need to classify a job by when it ran,\n but you don't care to be that specific. (Only the Time column is\n indexed by default, not the other times)\n\n* ``Submit``, ``Start``, ``End``: like the sacct equivalents,\n but unixtime. Assume that the sacct timestamps are in localtime of\n the machine doing the conversion. (``slurm2sql.unixtime`` converts\n slurm-format timestamp to unixtime)\n\n* ``QueueTime`` is Start-Submit in seconds. Start/End do not include\n timezones, so expect inaccuracies around summer time changes.\n\n* Job IDs. Slurm Job ID is by default of format\n ``JobID.JobStep`` or ``ArrayJobID_ArrayTaskID.JobStep``.\n Furthermore, each array job has a \"Raw JobID\" (different for each\n job, and is an actual JobID) in addition to the \"ArrayJobID\" which\n is the same for all jobs in an array. We split all of these\n different IDs into the following fields:\n\n * ``JobID``: The full raw value that Slurm gives. The same for each\n job in an array.\n\n Only the integer Job ID, without the trailing array\n tasks or job IDs. For array jobs, this is the \"Raw JobID\" as\n described above, use ``ArrayJobID`` to filter jobs that are the\n same. Integer\n\n * ``JobIDnostep``: The part of JobID without anything after the ``.``\n (no steps)\n\n * ``JobIDonly``: The integer part of the JobID.\n\n * ``JobIDRawonly``: The integer part of the Raw JobID (so this is\n different for each job in an aray).\n\n * ``ArrayTaskID``: As used above. Integer on null.\n\n * ``JobStep``: Job step - only. If you SQL filter for ``StepID is\n null`` you get only the main allocations. String.\n\n * Note: HetJob offsets are not currently handled and silently\n stripped out and give invalid data. File an issue and this will\n be added.\n\n* ``ReqMem``: The raw slurm value in a format like \"5Gn\". Instead of\n parsing this, you probably want to use one of the other values below.\n\n* ``ReqMemNode``, ``ReqMemCPU``: Requested memory per node or CPU,\n either taken from ReqMem (if it matches) or computed (you might want\n to check our logic if you rely on this). In Slurm, you\n can request memory either per-node or per-core, and this calculates\n the other one for you.\n\n* ``ReqMemType``: ``c`` if the user requested mem-per-core originally,\n ``n`` if mem-per-node. Extracted from ``ReqMem``. Modern Slurm has\n nothing here, and the column value is null.\n\n* ``ReqMemRaw``: The numeric value of the ``ReqMem``, whether it is\n ``c`` or ``n``.\n\n* ``ReqGPU``: Number of GPUs requested. Extracted from ``ReqTRES``.\n\n* GPU information. These use values from the ``TRESUsageInAve``\n fields in modern Slurm\n\n * ``GpuMem``: ``gres/gpumem``\n\n * ``GpuUtil``: ``gres/gpuutil`` (fraction 0.0-1.0).\n\n * ``NGpus``: Number of GPUs. Should be the same as ``ReqGPU``, but\n who knows.\n\n * ``GpuUtilTot``, ``GpuMemTot``: like above but using the\n ``TRESUsageInTot`` sacct field.\n\n* ``MemEff``: This is null in the Slurm table now, since Slurm gives\n ReqMem in allocations and memory used in steps. The ``eff`` table\n calculates this now.\n\n* ``CPUEff``: CPU efficiency (0.0-1.0). All the same caveats as above\n apply: test before trusting.\n\nQuick reference of the other most important columns from the\naccounting database that are hardest to remember:\n\n* ``Elapsed``: Wall clock time\n\n* ``CPUTime``: Reserved CPU time (Elapsed * number of CPUs). CPUEff \u2248\n TotalCPU/CPUTime = TotalCPU/(NCPUs x Elapsed)\n\n* ``TotalCPU``: SystemCPU + TotalCPU, seconds of productive work.\n\nThe ``eff`` table adds the following:\n\n* ``CPUEff``: like CPUEff but for the whole job\n\n* ``MemEff``: Memory efficiency for the whole job (max(MaxRSS) /\n ReqMem)\n\n* And more, see the code for now.\n\n\n\nChangelog\n---------\n\nNext\n\n* This is the biggest column clean-up in a while.\n* Add slurm2sql-{seff,sacct} commands.\n* JobID columns adjusted: ``JobID`` is the raw thing that slurm gives,\n ``*only`` integer IDs without any trailing things,\n ``JobIDrawonly`` is the RawJobID without any trailing things.\n* ReqMem has been updated: it no longer parses ``n`` and ``c``\n suffixes for mem-per-node/cpu, and that respective column has been\n removed.\n* MemEff has been removed from the ``slurm`` table, since it is always\n empty. The ``eff`` view has been added instead.\n\n0.9.1\n\n* Slurm >= 20.11 deprecates the ``AllocGRES`` and ``ReqGRES`` columns\n (using ``Alloc/ReqTRES`` instead).\n\n * From this slurm2sql version, a ReqTRES column will be requested\n and databases will need to be re-created (or manually added to the\n databases).\n * If run on Slurm > 20.11, it will not request ReqGRES and only use\n ReqTRES.\n\n\n\nDevelopment and maintenance\n---------------------------\n\nThis could be considered beta right now, but it works and is in use by\npeople. If this is important for you, comment about your use case\nin the Github issue tracker. Watch the repo if you want to give\ncomments on future data schema updates and backwards compatibility\nwhen Slurm changes.\n\nThere are many different variations of Slurm, if it doesn't\nwork for you, send an issue or pull request to help us make it more\ngeneral - development is only done in response to feedback.\n\nDevelopment principles:\n\n- All values are the most basic (metric) units: bytes, seconds,\n seconds-since-epoch, etc.\n- Try to use existing Slurm column names as much as possible (even if\n they are hard to remember).\n- Try to support as many Slurm versions as possible, but if something\n becomes hard to support, don't worry too much about breaking\n compatibly. SchedMD support slurm for 18 months after release. Try\n to support at least those versions. (Until someone asks for it,\n don't assume we can import data from very old Slurm versions)\n- Don't try to maintain database compatibility. It's expected that for\n all schema changes, you have to delete and re-import. But try to\n avoid this if not needed.\n\nRelease process::\n\n python setup.py sdist bdist_wheel\n twine upload [--repository-url https://test.pypi.org/legacy/] dist/*0.9.0*\n\nOriginally developed at Aalto University, Finland.\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Import Slurm accounting database from sacct to sqlite3 database",
"version": "0.9.2",
"project_urls": {
"Repository": "https://github.com/NordicHPC/slurm2sql"
},
"split_keywords": [
"slurm",
" sqlite3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e3d65de45690b23b80320f5125aa7180dee3e30e07a63dd3eae85a4c2ffc4858",
"md5": "f62f46f653b0d8f664b7d78dc94e144e",
"sha256": "b8c55fffa39de11643c671481a41cc7270213eb2d4c7e3a9d1ce0e6f33140453"
},
"downloads": -1,
"filename": "slurm2sql-0.9.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f62f46f653b0d8f664b7d78dc94e144e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 19957,
"upload_time": "2024-10-23T12:33:09",
"upload_time_iso_8601": "2024-10-23T12:33:09.046023Z",
"url": "https://files.pythonhosted.org/packages/e3/d6/5de45690b23b80320f5125aa7180dee3e30e07a63dd3eae85a4c2ffc4858/slurm2sql-0.9.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3f219b17fea0bf2a67092610d6b756bfeffb97a6e19bd7fe92f8b0d0a8bfd0de",
"md5": "af4ee4d3547685f8da6759adec683c25",
"sha256": "c4733004f0f47087114a12fe951bb27de330cb34c2083c9917e5853c824eaba5"
},
"downloads": -1,
"filename": "slurm2sql-0.9.2.tar.gz",
"has_sig": false,
"md5_digest": "af4ee4d3547685f8da6759adec683c25",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 29867,
"upload_time": "2024-10-23T12:33:10",
"upload_time_iso_8601": "2024-10-23T12:33:10.663379Z",
"url": "https://files.pythonhosted.org/packages/3f/21/9b17fea0bf2a67092610d6b756bfeffb97a6e19bd7fe92f8b0d0a8bfd0de/slurm2sql-0.9.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-23 12:33:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "NordicHPC",
"github_project": "slurm2sql",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "slurm2sql"
}
