| Name | crashstats-tools JSON |
| Version |
2.1.0
JSON |
| download |
| home_page | None |
| Summary | Tools for working with Crash Stats (https://crash-stats.mozilla.org/) |
| upload_time | 2024-10-30 23:48:02 |
| maintainer | None |
| docs_url | None |
| author | Will Kahn-Greene |
| requires_python | >=3.9 |
| license | MPLv2 |
| keywords |
socorro
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
================
crashstats-tools
================
Command line tools and library for interacting with Crash Stats
(`<https://crash-stats.mozilla.org/>`_).
:Code: https://github.com/mozilla-services/crashstats-tools
:Documentation: Check the ``README.rst`` file
:Changelog: Check the ``HISTORY.rst`` file
:Issue tracker: https://github.com/mozilla-services/crashstats-tools/issues
:License: MPLv2
:Chat: `#crashreporting matrix channel <https://chat.mozilla.org/#/room/#crashreporting:mozilla.org>`__
:Community Participation Guidelines: `<https://github.com/mozilla-services/crashstats-tools/blob/main/CODE_OF_CONDUCT.md>`_
Installing
==========
crashstats-tools is available on `PyPI <https://pypi.org>`_. You can install it
with `pipx <https://pipxproject.github.io/pipx/>`_::
$ pipx install crashstats-tools
For developing crashstats-tools, clone the Git repository, create a virtual
environment, and install crashstats-tools and dev dependencies with::
$ pip install -e '.[dev]'
Tools
=====
.. [[[cog
from click.testing import CliRunner
def execute_help(cmd):
result = CliRunner().invoke(cmd, ["--help"])
cog.out("\n::\n\n")
for line in result.output.splitlines():
if line.strip():
cog.out(f" {line}\n")
else:
cog.out("\n")
cog.out("\n")
]]]
.. [[[end]]]
supersearch
-----------
.. [[[cog
from crashstats_tools.cmd_supersearch import supersearch_cli
execute_help(supersearch_cli)
]]]
::
Usage: supersearch [OPTIONS]
Performs a basic search on Crash Stats using the Super Search API and outputs
the results.
A basic search uses filters and can span multiple pages of results.
If you want to use facets, aggregations, histograms, or cardinatlities, use
supersearchfacet.
There are two ways to supersearch:
First, you can specify Super Search API fields to generate the query.
For example:
$ supersearch --product=Firefox --num=10
Second, you can pass in a url from a Super Search on Crash Stats. This command
will then pull out the filter parameters. You can override those parameters
with command line arguments.
$ supersearch --supersearch-url='https://crash-stats.mozilla.org/search/...'
Make sure to use single quotes when specifying values so that your shell
doesn't expand variables or parse escape sequences.
By default, supersearch looks at crash report data for the last week. You can
specify start and end date filters to change the window of crash reports you
want to look at.
For example:
$ supersearch --date='>=2024-10-01' --date='<2024-10-15'
You can specify returned fields using the Super Search field "_columns".
For example:
$ supersearch --_columns=uuid --_columns=product --_columns=build_id
Results are tab-delimited by default. You can specify other output formats
using "--format". Tabs and newlines in output are escaped.
For list of available fields and Super Search API documentation, see:
https://crash-stats.mozilla.org/documentation/supersearch/
https://crash-stats.mozilla.org/documentation/supersearch/api/
This requires an API token in order to search and get results for protected
data fields. Using an API token also reduces rate-limiting. Set the
CRASHSTATS_API_TOKEN environment variable to your API token value:
CRASHSTATS_API_TOKEN=xyz supersearch ...
To create an API token for Crash Stats, visit:
https://crash-stats.mozilla.org/api/tokens/
Remember to abide by the data access policy when using data from Crash Stats!
The policy is specified here:
https://crash-stats.mozilla.org/documentation/protected_data_access/
Options:
--host TEXT host for system to fetch crashids from
[default: https://crash-stats.mozilla.org]
--supersearch-url TEXT Super Search url to base query on
--num TEXT number of crash ids you want or "all" for all
of them [default: 100]
--headers / --no-headers whether or not to show table headers
[default: no-headers]
--format [table|tab|csv|json|markdown]
format to print output [default: tab]
--verbose / --no-verbose whether to print debugging output [default:
no-verbose]
--color / --no-color whether or not to colorize output; note that
color is shut off when stdout is not an
interactive terminal automatically [default:
color]
--dotenv / --no-dotenv whether or not to load a .env file for
environment variables [default: no-dotenv]
--help Show this message and exit.
.. [[[end]]]
Fetch 10 crash ids for Firefox::
$ supersearch --num=10 --product=Firefox
Fetch all crash reports that have ``libc2.30.so/E22A1E7AEF7C58504AF2C60A5AD3A7AE0``
in the ``modules_in_stack`` field::
$ supersearch --modules_in_stack=libc2.30.so/E22A1E7AEF6C58504AF2C60A5AD3A7AE0
This is helpful when you need to reprocess crash reports after uploading symbols
for a module that we didn't have symbols for.
Fetch all crash reports that have ``libgallium_dri.so`` in the
``modules_in_stack`` field::
$ supersearch --modules_in_stack='^libgallium_dri.so'
Fetch 57 crash ids that match a Super Search query::
$ supersearch --num=57 \
--supersearch-url='https://crash-stats.mozilla.org/search/?release_channel=nightly&version=70.0a1&product=Firefox&_sort=-date'
Use single quotes for values so that your shell doesn't do any shell expansion.
Fetch uuid, product, version, and build_id for crash reports that have "OOM" in
the signature::
$ supersearch --_columns=uuid --_columns=product --_columns=version \
--_columns=build_id --signature='~OOM'
Note that this doesn't support Super Search aggregations--use
the ``supersearchfacet`` command for that.
See Super Search API documentation for details on notation and fields:
* https://crash-stats.mozilla.org/documentation/supersearch/
* https://crash-stats.mozilla.org/documentation/supersearch/api/
supersearchfacet
----------------
.. [[[cog
from crashstats_tools.cmd_supersearchfacet import supersearchfacet
execute_help(supersearchfacet)
]]]
::
Usage: supersearchfacet [OPTIONS]
Fetches facet data from Crash Stats using Super Search
There are two ways to run this:
First, you can specify Super Search API fields to generate the query.
For example:
$ supersearchfacet --product=Firefox --_facets=version
Second, you can pass in a url from a Super Search on Crash Stats. This command
will then pull out the parameters. You can override those parameters with
command line arguments.
$ supersearchfacet --_facets=version \
--supersearch-url='https://crash-stats.mozilla.org/search/...'
Make sure to use single quotes when specifying values so that your shell
doesn't expand variables.
You can get a facet of a field using ``_facets``.
For example, this filters on Firefox and returns a facet on version for the
last 7 days (the default time range).
$ supersearchfacet --product=Firefox --_facets=version
You can get cardinality (number of possible values), too. For example, this
shows the number of different versions for Firefox crash reports in the last 7
days.
$ supersearchfacet --product=Firefox --_facets=_cardinality.version
You can perform histograms. For example, this shows you counts for products
per day for the last week:
$ supersearchfacet --_histogram.date=product --relative-range=1w
You can get a cardinality for the data for a field. For example, this tells
you how many build ids there were for Firefox 124:
$ supersearchfacet --product=Firefox --version=124.0
--_facets=_cardinality.build_id
You can do nested aggregations. For example, this shows the count of crash
reports by product by release channel:
$ supersearchfacet --_aggs.product=release_channel
This shows count of crash reports by product, version, cardinality of
install_time:
$ supersearchfacet --_aggs.product.version=_cardinality.install_time
Make sure to specify at least one of ``_facets``, ``_aggs``, ``_histogram``,
or ``_cardinality``.
By default, returned data is in a table. Tabs and newlines in output is
escaped. Use ``--format`` to specify a different output format.
For list of available fields and Super Search API documentation, see:
https://crash-stats.mozilla.org/documentation/supersearch/
https://crash-stats.mozilla.org/documentation/supersearch/api/
This requires an API token in order to search and get results for protected
data fields. Using an API token also reduces rate-limiting. Set the
CRASHSTATS_API_TOKEN environment variable to your API token value:
CRASHSTATS_API_TOKEN=xyz supersearchfacet ...
To create an API token for Crash Stats, visit:
https://crash-stats.mozilla.org/api/tokens/
Remember to abide by the data access policy when using data from Crash Stats!
The policy is specified here:
https://crash-stats.mozilla.org/documentation/protected_data_access/
Options:
--host TEXT host for system to fetch facets from
[default: https://crash-stats.mozilla.org]
--supersearch-url TEXT Super Search url to base query on
--start-date TEXT start date for range; 'YYYY-MM-DD' and 'YYYY-
MM-DD HH:MM:SS' formats; defaults to 00:00:00
when no time specified
--end-date TEXT end date for range; 'YYYY-MM-DD' and 'YYYY-MM-
DD HH:MM:SS' formats; defaults to 23:59:59
when no time specified [default: today]
--relative-range TEXT relative range ending on end-date [default:
7d]
--format [table|tab|csv|markdown|json|raw]
format to print output [default: table]
--verbose / --no-verbose whether to print debugging output [default:
no-verbose]
--color / --no-color whether or not to colorize output; note that
color is shut off when stdout is not an
interactive terminal automatically [default:
color]
--denote-weekends / --no-denote-weekends
This will add a * for values that are
datestamps and on a Saturday or Sunday.
[default: no-denote-weekends]
--leftover-count / --no-leftover-count
Calculates the leftover that is the difference
between the total minus the sum of all term
counts [default: no-leftover-count]
--dotenv / --no-dotenv whether or not to load a .env file for
environment variables [default: no-dotenv]
--help Show this message and exit.
.. [[[end]]]
See the breakdown of crash reports by product for the last 7 days::
$ supersearchfacet --_facets=product
See crashes broken down by product and down by day for the last 7 days::
$ supersearchfacet --_histogram.date=product --relative=range=7d
Histograms, facets, aggs, and cardinality can be filtered using Super Search
filters. See crashes in Firefox by release channel broken down by day for the
last 7 days::
$ supersearchfacet \
--_histogram.date=release_channel \
--release_channel=nightly \
--release_channel=beta \
--release_channel=release \
--release_channel=esr \
--product=Firefox \
--relative-range=7d
Get the table in Markdown for easy cut-and-paste into Markdown things::
$ supersearchfacet --_histogram.date=product --relative=range=7d \
--format=markdown
See Super Search API documentation for details on notation and fields:
* https://crash-stats.mozilla.org/documentation/supersearch/
* https://crash-stats.mozilla.org/documentation/supersearch/api/
fetch-data
----------
.. [[[cog
from crashstats_tools.cmd_fetch_data import fetch_data
execute_help(fetch_data)
]]]
::
Usage: fetch-data [OPTIONS] OUTPUTDIR [CRASH_IDS]...
Fetches crash data from Crash Stats (https://crash-stats.mozilla.org/) system.
Given one or more crash ids via command line or stdin (one per line), fetches
crash data and puts it in specified directory.
Crash data is split up into directories: raw_crash/, dump_names/,
processed_crash/, and directories with the same name as the dump type.
https://antenna.readthedocs.io/en/latest/overview.html#aws-s3-file-hierarchy
This requires an API token in order to download dumps and protected data.
Using an API token also reduces rate-limiting. Set the CRASHSTATS_API_TOKEN
environment variable to your API token value:
CRASHSTATS_API_TOKEN=xyz fetch-data crashdata ...
To create an API token for Crash Stats, visit:
https://crash-stats.mozilla.org/api/tokens/
Remember to abide by the data access policy when using data from Crash Stats!
The policy is specified here:
https://crash-stats.mozilla.org/documentation/protected_data_access/
Options:
--host TEXT host to pull crash data from; this needs to
match CRASHSTATS_API_TOKEN value [default:
https://crash-stats.mozilla.org]
--overwrite / --no-overwrite whether or not to overwrite existing data
[default: overwrite]
--raw / --no-raw whether or not to save raw crash data [default:
raw]
--dumps / --no-dumps whether or not to save dumps [default: no-
dumps]
--processed / --no-processed whether or not to save processed crash data
[default: no-processed]
--workers INTEGER RANGE how many workers to use to download data;
requires CRASHSTATS_API_TOKEN [default: 1;
1<=x<=10]
--stats / --no-stats prints download stats for large fetch-data jobs;
if it's printing download stats, it's not
printing other things [default: no-stats]
--color / --no-color whether or not to colorize output; note that
color is shut off when stdout is not an
interactive terminal automatically [default:
color]
--dotenv / --no-dotenv whether or not to load a .env file for
environment variables [default: no-dotenv]
--help Show this message and exit.
.. [[[end]]]
This lets you download raw crash, dumps, and processed crash from Crash Stats.
Fetch processed crash data for specific crash id::
$ fetch-data --no-raw --no-dumps --processed 723cacd6-1684-420e-a1c7-f04240190731
Fetch raw crash data using supersearch command to generate crash ids::
$ supersearch --product=Firefox --num=10 | \
fetch-data --raw --no-dumps --no-processed crashdir
reprocess
---------
.. [[[cog
from crashstats_tools.cmd_reprocess import reprocess
execute_help(reprocess)
]]]
::
Usage: reprocess [OPTIONS] [CRASHIDS]...
Sends specified crashes for reprocessing
This requires CRASHSTATS_API_TOKEN to be set in the environment to a valid API
token.
To create an API token for Crash Stats, visit:
https://crash-stats.mozilla.org/api/tokens/
Note: If you're processing more than 10,000 crashes, you should use a sleep
value that balances the rate of crash ids being added to the queue and the
rate of crash ids being processed. For example, you could use "--sleep 10"
which will sleep for 10 seconds between submitting groups of crashes.
Also, if you're processing a lot of crashes, you should let us know before you
do it.
Options:
--host TEXT host for system to reprocess in [default:
https://crash-stats.mozilla.org]
--sleep INTEGER how long in seconds to sleep before submitting
the next group [default: 1]
--ruleset TEXT processor pipeline ruleset to use for
reprocessing these crash ids
--allow-many / --no-allow-many don't prompt user about letting us know about
reprocessing more than 10,000 crashes
[default: no-allow-many]
--color / --no-color whether or not to colorize output; note that
color is shut off when stdout is not an
interactive terminal automatically [default:
color]
--dotenv / --no-dotenv whether or not to load a .env file for
environment variables [default: no-dotenv]
--help Show this message and exit.
.. [[[end]]]
Reprocess an individual crash report::
$ reprocess 723cacd6-1684-420e-a1c7-f04240190731
Reprocess crash reports based on a supersearch::
$ supersearch --num=5 | reprocess
.. Note::
The ``reprocess`` command requires that you set ``CRASHSTATS_API_TOKEN`` in
your environment with an API token that has the "Reprocess Crashes"
permission.
.. Note::
If you intend to reprocess more than 10,000 crash reports, please tell
us first.
API token
=========
For ``supersearch`` and ``fetch-data``, you need to use a API token to:
* download data containing personally identifiable information
* download security sensitive data
* get out from the shadow of extreme API use rate limiting
You need an API token for ``reprocess``--it doesn't work without one.
If you have access, you can generate an API token here:
https://crash-stats.mozilla.org/api/tokens/
Once you have acquired one, set the ``CRASHSTATS_API_TOKEN`` environment
variable when using crashstats-tools commands.
Remember to abide by the data access policy when using data from Crash Stats!
The policy is specified here:
https://crash-stats.mozilla.org/documentation/memory_dump_access/
Use cases
=========
These tools are helpful when downloading data for analysis as well as
downloading data to test other tools with.
Example 1
---------
I want to collect a bunch of crash report data to look at possible values of an
annotation in Firefox crash reports that's not available in Super Search, yet.
Since I'm looking just at annotations, all I need is the raw crash.
I would do something like this::
$ mkdir crashdata
$ supersearch --product=Firefox --num=1000 | \
fetch-data --raw --no-dumps --no-processed crashdata
Then I can use ``jq`` or whatever to look at the crash report data in
``crashdata/raw_crash/``.
Example 2
---------
I want to test out a new JIT analysis tool that works on minidump files.
I would write a script like this::
#!/bin/bash
CRASHSTATS_API_TOKEN=foo
DATADIR=./crashdata
CRASHIDS=$(supersearch --product=Firefox --num=1000)
mkdir -p "${DATADIR}"
for crashid in ${CRASHIDS}
do
echo "crashid ${crashid}"
fetch-data --raw --dumps --no-processed "${DATADIR}" "${crashid}"
# Not all crash reports have dumps--we only want to run analysis
# on the ones that do.
if [[ -e "crashdata/dump/${crashid}" ]]
then
echo "analyze dump ${crashid}..."
# run my tool on the dump
fi
done
Example 3
---------
I want to get a list of crash ids for today (2019-07-30) where
``DOMFissionEnabled`` exists in the crash report.
I would do this::
$ supersearch --date=">=2019-07-30" --date='<2019-07-31' \
--dom_fission_enabled='!__null__'
Example 4
---------
I want to see number of crash reports for the last 14 days broken down by day
and by product where ``DOMFissionEnabled`` exists in the crash report.
I would do this::
$ supersearchfacet --period=daily --format=markdown --relative-range=14d \
--dom_fission_enabled='!__null__' --_facets=product
Library
=======
Further, this provides a library interface to some Crash Stats API endpoints:
``crashstats_tools.libcrashstats``
``get_crash_annotations(crash_id, api_token=None, host=DEFAULT_HOST)``
Fetches crash annotations for a given crash report.
If you don't provide an API token, then it only returns crash annotations
that are marked public.
``get_dump(crash_id, dump_name, api_token, host=DEFAULT_HOST)``
Fetches dumps, memory reports, and other crash report binaries for given
crash id.
This requires an api token.
``get_processed_crash(crash_id, api_token=None, host=DEFAULT_HOST)``
Fetches the processed crash for given crash id.
``supersearch(params, num_results, host=DEFAULT_HOST, api_token=None, logger=None)``
Performs a super search and returns generator of result hits.
This doesn't return facet, aggregation, cardinality, or histogram data.
If you want that, use ``supersearch_facet``.
``supersearch_facet(params, api_token=None, host=DEFAULT_HOST, logger=None)``
Performs a super search and returns facet data
Prior art and related projects
==============================
https://github.com/mozilla/libmozdata
Python library which has a ``Supersearch`` class for performing queries and
a ``ProcessedCrash`` class for fetching processed crash data.
https://github.com/mozilla-services/socorro
Socorro (the code base for Crash Stats) has a Docker-based local dev
environment which includes a series of commands for manipulating data.
https://socorro.readthedocs.io/en/latest/service/processor.html#processing-crashes
Raw data
{
"_id": null,
"home_page": null,
"name": "crashstats-tools",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "socorro",
"author": "Will Kahn-Greene",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/6c/fd/14a7cba76e9ea5e33265c24f01329c20fd9834ddef131521f7d531bc4ad8/crashstats_tools-2.1.0.tar.gz",
"platform": null,
"description": "================\ncrashstats-tools\n================\n\nCommand line tools and library for interacting with Crash Stats\n(`<https://crash-stats.mozilla.org/>`_).\n\n:Code: https://github.com/mozilla-services/crashstats-tools\n:Documentation: Check the ``README.rst`` file\n:Changelog: Check the ``HISTORY.rst`` file\n:Issue tracker: https://github.com/mozilla-services/crashstats-tools/issues\n:License: MPLv2\n:Chat: `#crashreporting matrix channel <https://chat.mozilla.org/#/room/#crashreporting:mozilla.org>`__\n:Community Participation Guidelines: `<https://github.com/mozilla-services/crashstats-tools/blob/main/CODE_OF_CONDUCT.md>`_\n\n\nInstalling\n==========\n\ncrashstats-tools is available on `PyPI <https://pypi.org>`_. You can install it\nwith `pipx <https://pipxproject.github.io/pipx/>`_::\n\n $ pipx install crashstats-tools\n\n\nFor developing crashstats-tools, clone the Git repository, create a virtual\nenvironment, and install crashstats-tools and dev dependencies with::\n\n $ pip install -e '.[dev]'\n\n\nTools\n=====\n\n.. [[[cog\n from click.testing import CliRunner\n\n def execute_help(cmd):\n result = CliRunner().invoke(cmd, [\"--help\"])\n cog.out(\"\\n::\\n\\n\")\n for line in result.output.splitlines():\n if line.strip():\n cog.out(f\" {line}\\n\")\n else:\n cog.out(\"\\n\")\n cog.out(\"\\n\")\n ]]]\n.. [[[end]]]\n\nsupersearch\n-----------\n\n.. [[[cog\n from crashstats_tools.cmd_supersearch import supersearch_cli\n execute_help(supersearch_cli)\n ]]]\n\n::\n\n Usage: supersearch [OPTIONS]\n\n Performs a basic search on Crash Stats using the Super Search API and outputs\n the results.\n\n A basic search uses filters and can span multiple pages of results.\n\n If you want to use facets, aggregations, histograms, or cardinatlities, use\n supersearchfacet.\n\n There are two ways to supersearch:\n\n First, you can specify Super Search API fields to generate the query.\n\n For example:\n\n $ supersearch --product=Firefox --num=10\n\n Second, you can pass in a url from a Super Search on Crash Stats. This command\n will then pull out the filter parameters. You can override those parameters\n with command line arguments.\n\n $ supersearch --supersearch-url='https://crash-stats.mozilla.org/search/...'\n\n Make sure to use single quotes when specifying values so that your shell\n doesn't expand variables or parse escape sequences.\n\n By default, supersearch looks at crash report data for the last week. You can\n specify start and end date filters to change the window of crash reports you\n want to look at.\n\n For example:\n\n $ supersearch --date='>=2024-10-01' --date='<2024-10-15'\n\n You can specify returned fields using the Super Search field \"_columns\".\n\n For example:\n\n $ supersearch --_columns=uuid --_columns=product --_columns=build_id\n\n Results are tab-delimited by default. You can specify other output formats\n using \"--format\". Tabs and newlines in output are escaped.\n\n For list of available fields and Super Search API documentation, see:\n\n https://crash-stats.mozilla.org/documentation/supersearch/\n\n https://crash-stats.mozilla.org/documentation/supersearch/api/\n\n This requires an API token in order to search and get results for protected\n data fields. Using an API token also reduces rate-limiting. Set the\n CRASHSTATS_API_TOKEN environment variable to your API token value:\n\n CRASHSTATS_API_TOKEN=xyz supersearch ...\n\n To create an API token for Crash Stats, visit:\n\n https://crash-stats.mozilla.org/api/tokens/\n\n Remember to abide by the data access policy when using data from Crash Stats!\n The policy is specified here:\n\n https://crash-stats.mozilla.org/documentation/protected_data_access/\n\n Options:\n --host TEXT host for system to fetch crashids from\n [default: https://crash-stats.mozilla.org]\n --supersearch-url TEXT Super Search url to base query on\n --num TEXT number of crash ids you want or \"all\" for all\n of them [default: 100]\n --headers / --no-headers whether or not to show table headers\n [default: no-headers]\n --format [table|tab|csv|json|markdown]\n format to print output [default: tab]\n --verbose / --no-verbose whether to print debugging output [default:\n no-verbose]\n --color / --no-color whether or not to colorize output; note that\n color is shut off when stdout is not an\n interactive terminal automatically [default:\n color]\n --dotenv / --no-dotenv whether or not to load a .env file for\n environment variables [default: no-dotenv]\n --help Show this message and exit.\n\n.. [[[end]]]\n\nFetch 10 crash ids for Firefox::\n\n $ supersearch --num=10 --product=Firefox\n\nFetch all crash reports that have ``libc2.30.so/E22A1E7AEF7C58504AF2C60A5AD3A7AE0``\nin the ``modules_in_stack`` field::\n\n $ supersearch --modules_in_stack=libc2.30.so/E22A1E7AEF6C58504AF2C60A5AD3A7AE0\n\nThis is helpful when you need to reprocess crash reports after uploading symbols\nfor a module that we didn't have symbols for.\n\nFetch all crash reports that have ``libgallium_dri.so`` in the\n``modules_in_stack`` field::\n\n $ supersearch --modules_in_stack='^libgallium_dri.so'\n\nFetch 57 crash ids that match a Super Search query::\n\n $ supersearch --num=57 \\\n --supersearch-url='https://crash-stats.mozilla.org/search/?release_channel=nightly&version=70.0a1&product=Firefox&_sort=-date'\n\nUse single quotes for values so that your shell doesn't do any shell expansion.\n\nFetch uuid, product, version, and build_id for crash reports that have \"OOM\" in\nthe signature::\n\n $ supersearch --_columns=uuid --_columns=product --_columns=version \\\n --_columns=build_id --signature='~OOM'\n\nNote that this doesn't support Super Search aggregations--use\nthe ``supersearchfacet`` command for that.\n\nSee Super Search API documentation for details on notation and fields:\n\n* https://crash-stats.mozilla.org/documentation/supersearch/\n* https://crash-stats.mozilla.org/documentation/supersearch/api/\n\n\nsupersearchfacet\n----------------\n\n.. [[[cog\n from crashstats_tools.cmd_supersearchfacet import supersearchfacet\n execute_help(supersearchfacet)\n ]]]\n\n::\n\n Usage: supersearchfacet [OPTIONS]\n\n Fetches facet data from Crash Stats using Super Search\n\n There are two ways to run this:\n\n First, you can specify Super Search API fields to generate the query.\n\n For example:\n\n $ supersearchfacet --product=Firefox --_facets=version\n\n Second, you can pass in a url from a Super Search on Crash Stats. This command\n will then pull out the parameters. You can override those parameters with\n command line arguments.\n\n $ supersearchfacet --_facets=version \\\n --supersearch-url='https://crash-stats.mozilla.org/search/...'\n\n Make sure to use single quotes when specifying values so that your shell\n doesn't expand variables.\n\n You can get a facet of a field using ``_facets``.\n\n For example, this filters on Firefox and returns a facet on version for the\n last 7 days (the default time range).\n\n $ supersearchfacet --product=Firefox --_facets=version\n\n You can get cardinality (number of possible values), too. For example, this\n shows the number of different versions for Firefox crash reports in the last 7\n days.\n\n $ supersearchfacet --product=Firefox --_facets=_cardinality.version\n\n You can perform histograms. For example, this shows you counts for products\n per day for the last week:\n\n $ supersearchfacet --_histogram.date=product --relative-range=1w\n\n You can get a cardinality for the data for a field. For example, this tells\n you how many build ids there were for Firefox 124:\n\n $ supersearchfacet --product=Firefox --version=124.0\n --_facets=_cardinality.build_id\n\n You can do nested aggregations. For example, this shows the count of crash\n reports by product by release channel:\n\n $ supersearchfacet --_aggs.product=release_channel\n\n This shows count of crash reports by product, version, cardinality of\n install_time:\n\n $ supersearchfacet --_aggs.product.version=_cardinality.install_time\n\n Make sure to specify at least one of ``_facets``, ``_aggs``, ``_histogram``,\n or ``_cardinality``.\n\n By default, returned data is in a table. Tabs and newlines in output is\n escaped. Use ``--format`` to specify a different output format.\n\n For list of available fields and Super Search API documentation, see:\n\n https://crash-stats.mozilla.org/documentation/supersearch/\n\n https://crash-stats.mozilla.org/documentation/supersearch/api/\n\n This requires an API token in order to search and get results for protected\n data fields. Using an API token also reduces rate-limiting. Set the\n CRASHSTATS_API_TOKEN environment variable to your API token value:\n\n CRASHSTATS_API_TOKEN=xyz supersearchfacet ...\n\n To create an API token for Crash Stats, visit:\n\n https://crash-stats.mozilla.org/api/tokens/\n\n Remember to abide by the data access policy when using data from Crash Stats!\n The policy is specified here:\n\n https://crash-stats.mozilla.org/documentation/protected_data_access/\n\n Options:\n --host TEXT host for system to fetch facets from\n [default: https://crash-stats.mozilla.org]\n --supersearch-url TEXT Super Search url to base query on\n --start-date TEXT start date for range; 'YYYY-MM-DD' and 'YYYY-\n MM-DD HH:MM:SS' formats; defaults to 00:00:00\n when no time specified\n --end-date TEXT end date for range; 'YYYY-MM-DD' and 'YYYY-MM-\n DD HH:MM:SS' formats; defaults to 23:59:59\n when no time specified [default: today]\n --relative-range TEXT relative range ending on end-date [default:\n 7d]\n --format [table|tab|csv|markdown|json|raw]\n format to print output [default: table]\n --verbose / --no-verbose whether to print debugging output [default:\n no-verbose]\n --color / --no-color whether or not to colorize output; note that\n color is shut off when stdout is not an\n interactive terminal automatically [default:\n color]\n --denote-weekends / --no-denote-weekends\n This will add a * for values that are\n datestamps and on a Saturday or Sunday.\n [default: no-denote-weekends]\n --leftover-count / --no-leftover-count\n Calculates the leftover that is the difference\n between the total minus the sum of all term\n counts [default: no-leftover-count]\n --dotenv / --no-dotenv whether or not to load a .env file for\n environment variables [default: no-dotenv]\n --help Show this message and exit.\n\n.. [[[end]]]\n\nSee the breakdown of crash reports by product for the last 7 days::\n\n $ supersearchfacet --_facets=product\n\nSee crashes broken down by product and down by day for the last 7 days::\n\n $ supersearchfacet --_histogram.date=product --relative=range=7d\n\nHistograms, facets, aggs, and cardinality can be filtered using Super Search\nfilters. See crashes in Firefox by release channel broken down by day for the\nlast 7 days::\n\n $ supersearchfacet \\\n --_histogram.date=release_channel \\\n --release_channel=nightly \\\n --release_channel=beta \\\n --release_channel=release \\\n --release_channel=esr \\\n --product=Firefox \\\n --relative-range=7d\n\nGet the table in Markdown for easy cut-and-paste into Markdown things::\n\n $ supersearchfacet --_histogram.date=product --relative=range=7d \\\n --format=markdown\n\nSee Super Search API documentation for details on notation and fields:\n\n* https://crash-stats.mozilla.org/documentation/supersearch/\n* https://crash-stats.mozilla.org/documentation/supersearch/api/\n\n\nfetch-data\n----------\n\n.. [[[cog\n from crashstats_tools.cmd_fetch_data import fetch_data\n execute_help(fetch_data)\n ]]]\n\n::\n\n Usage: fetch-data [OPTIONS] OUTPUTDIR [CRASH_IDS]...\n\n Fetches crash data from Crash Stats (https://crash-stats.mozilla.org/) system.\n\n Given one or more crash ids via command line or stdin (one per line), fetches\n crash data and puts it in specified directory.\n\n Crash data is split up into directories: raw_crash/, dump_names/,\n processed_crash/, and directories with the same name as the dump type.\n\n https://antenna.readthedocs.io/en/latest/overview.html#aws-s3-file-hierarchy\n\n This requires an API token in order to download dumps and protected data.\n Using an API token also reduces rate-limiting. Set the CRASHSTATS_API_TOKEN\n environment variable to your API token value:\n\n CRASHSTATS_API_TOKEN=xyz fetch-data crashdata ...\n\n To create an API token for Crash Stats, visit:\n\n https://crash-stats.mozilla.org/api/tokens/\n\n Remember to abide by the data access policy when using data from Crash Stats!\n The policy is specified here:\n\n https://crash-stats.mozilla.org/documentation/protected_data_access/\n\n Options:\n --host TEXT host to pull crash data from; this needs to\n match CRASHSTATS_API_TOKEN value [default:\n https://crash-stats.mozilla.org]\n --overwrite / --no-overwrite whether or not to overwrite existing data\n [default: overwrite]\n --raw / --no-raw whether or not to save raw crash data [default:\n raw]\n --dumps / --no-dumps whether or not to save dumps [default: no-\n dumps]\n --processed / --no-processed whether or not to save processed crash data\n [default: no-processed]\n --workers INTEGER RANGE how many workers to use to download data;\n requires CRASHSTATS_API_TOKEN [default: 1;\n 1<=x<=10]\n --stats / --no-stats prints download stats for large fetch-data jobs;\n if it's printing download stats, it's not\n printing other things [default: no-stats]\n --color / --no-color whether or not to colorize output; note that\n color is shut off when stdout is not an\n interactive terminal automatically [default:\n color]\n --dotenv / --no-dotenv whether or not to load a .env file for\n environment variables [default: no-dotenv]\n --help Show this message and exit.\n\n.. [[[end]]]\n\nThis lets you download raw crash, dumps, and processed crash from Crash Stats.\n\nFetch processed crash data for specific crash id::\n\n $ fetch-data --no-raw --no-dumps --processed 723cacd6-1684-420e-a1c7-f04240190731\n\nFetch raw crash data using supersearch command to generate crash ids::\n\n $ supersearch --product=Firefox --num=10 | \\\n fetch-data --raw --no-dumps --no-processed crashdir\n\n\nreprocess\n---------\n\n.. [[[cog\n from crashstats_tools.cmd_reprocess import reprocess\n execute_help(reprocess)\n ]]]\n\n::\n\n Usage: reprocess [OPTIONS] [CRASHIDS]...\n\n Sends specified crashes for reprocessing\n\n This requires CRASHSTATS_API_TOKEN to be set in the environment to a valid API\n token.\n\n To create an API token for Crash Stats, visit:\n\n https://crash-stats.mozilla.org/api/tokens/\n\n Note: If you're processing more than 10,000 crashes, you should use a sleep\n value that balances the rate of crash ids being added to the queue and the\n rate of crash ids being processed. For example, you could use \"--sleep 10\"\n which will sleep for 10 seconds between submitting groups of crashes.\n\n Also, if you're processing a lot of crashes, you should let us know before you\n do it.\n\n Options:\n --host TEXT host for system to reprocess in [default:\n https://crash-stats.mozilla.org]\n --sleep INTEGER how long in seconds to sleep before submitting\n the next group [default: 1]\n --ruleset TEXT processor pipeline ruleset to use for\n reprocessing these crash ids\n --allow-many / --no-allow-many don't prompt user about letting us know about\n reprocessing more than 10,000 crashes\n [default: no-allow-many]\n --color / --no-color whether or not to colorize output; note that\n color is shut off when stdout is not an\n interactive terminal automatically [default:\n color]\n --dotenv / --no-dotenv whether or not to load a .env file for\n environment variables [default: no-dotenv]\n --help Show this message and exit.\n\n.. [[[end]]]\n\nReprocess an individual crash report::\n\n $ reprocess 723cacd6-1684-420e-a1c7-f04240190731\n\nReprocess crash reports based on a supersearch::\n\n $ supersearch --num=5 | reprocess\n\n.. Note::\n\n The ``reprocess`` command requires that you set ``CRASHSTATS_API_TOKEN`` in\n your environment with an API token that has the \"Reprocess Crashes\"\n permission.\n\n\n.. Note::\n\n If you intend to reprocess more than 10,000 crash reports, please tell\n us first.\n\n\nAPI token\n=========\n\nFor ``supersearch`` and ``fetch-data``, you need to use a API token to:\n\n* download data containing personally identifiable information\n* download security sensitive data\n* get out from the shadow of extreme API use rate limiting\n\nYou need an API token for ``reprocess``--it doesn't work without one.\n\nIf you have access, you can generate an API token here:\n\nhttps://crash-stats.mozilla.org/api/tokens/\n\nOnce you have acquired one, set the ``CRASHSTATS_API_TOKEN`` environment\nvariable when using crashstats-tools commands.\n\nRemember to abide by the data access policy when using data from Crash Stats!\nThe policy is specified here:\n\nhttps://crash-stats.mozilla.org/documentation/memory_dump_access/\n\n\nUse cases\n=========\n\nThese tools are helpful when downloading data for analysis as well as\ndownloading data to test other tools with.\n\n\nExample 1\n---------\n\nI want to collect a bunch of crash report data to look at possible values of an\nannotation in Firefox crash reports that's not available in Super Search, yet.\n\nSince I'm looking just at annotations, all I need is the raw crash.\n\nI would do something like this::\n\n $ mkdir crashdata\n $ supersearch --product=Firefox --num=1000 | \\\n fetch-data --raw --no-dumps --no-processed crashdata\n\nThen I can use ``jq`` or whatever to look at the crash report data in\n``crashdata/raw_crash/``.\n\n\nExample 2\n---------\n\nI want to test out a new JIT analysis tool that works on minidump files.\n\nI would write a script like this::\n\n #!/bin/bash\n \n CRASHSTATS_API_TOKEN=foo\n DATADIR=./crashdata\n CRASHIDS=$(supersearch --product=Firefox --num=1000)\n \n mkdir -p \"${DATADIR}\"\n \n for crashid in ${CRASHIDS}\n do\n echo \"crashid ${crashid}\"\n fetch-data --raw --dumps --no-processed \"${DATADIR}\" \"${crashid}\"\n \n # Not all crash reports have dumps--we only want to run analysis\n # on the ones that do.\n if [[ -e \"crashdata/dump/${crashid}\" ]]\n then\n echo \"analyze dump ${crashid}...\"\n # run my tool on the dump\n fi\n done\n \n\nExample 3\n---------\n\nI want to get a list of crash ids for today (2019-07-30) where\n``DOMFissionEnabled`` exists in the crash report.\n\nI would do this::\n\n $ supersearch --date=\">=2019-07-30\" --date='<2019-07-31' \\\n --dom_fission_enabled='!__null__'\n\n\nExample 4\n---------\n\nI want to see number of crash reports for the last 14 days broken down by day\nand by product where ``DOMFissionEnabled`` exists in the crash report.\n\nI would do this::\n\n $ supersearchfacet --period=daily --format=markdown --relative-range=14d \\\n --dom_fission_enabled='!__null__' --_facets=product\n\n\nLibrary\n=======\n\nFurther, this provides a library interface to some Crash Stats API endpoints:\n\n``crashstats_tools.libcrashstats``\n\n``get_crash_annotations(crash_id, api_token=None, host=DEFAULT_HOST)``\n Fetches crash annotations for a given crash report.\n\n If you don't provide an API token, then it only returns crash annotations\n that are marked public.\n\n``get_dump(crash_id, dump_name, api_token, host=DEFAULT_HOST)``\n Fetches dumps, memory reports, and other crash report binaries for given\n crash id.\n\n This requires an api token.\n\n``get_processed_crash(crash_id, api_token=None, host=DEFAULT_HOST)``\n Fetches the processed crash for given crash id.\n\n``supersearch(params, num_results, host=DEFAULT_HOST, api_token=None, logger=None)``\n Performs a super search and returns generator of result hits.\n\n This doesn't return facet, aggregation, cardinality, or histogram data.\n If you want that, use ``supersearch_facet``.\n\n``supersearch_facet(params, api_token=None, host=DEFAULT_HOST, logger=None)``\n Performs a super search and returns facet data\n\n\nPrior art and related projects\n==============================\n\nhttps://github.com/mozilla/libmozdata\n Python library which has a ``Supersearch`` class for performing queries and\n a ``ProcessedCrash`` class for fetching processed crash data.\n\nhttps://github.com/mozilla-services/socorro\n Socorro (the code base for Crash Stats) has a Docker-based local dev\n environment which includes a series of commands for manipulating data.\n\n https://socorro.readthedocs.io/en/latest/service/processor.html#processing-crashes\n",
"bugtrack_url": null,
"license": "MPLv2",
"summary": "Tools for working with Crash Stats (https://crash-stats.mozilla.org/)",
"version": "2.1.0",
"project_urls": {
"Homepage": "https://github.com/mozilla-services/crashstats-tools",
"Issues": "https://github.com/mozilla-services/crashstats-tools",
"Source": "https://github.com/mozilla-services/crashstats-tools"
},
"split_keywords": [
"socorro"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dc20a36a56d6dd297255d3f46a82318ef539b6c6f03e1f674c2e72d7a421c1b7",
"md5": "f342699732a2ce35c333746ec0da1e88",
"sha256": "155065f86e5b026895775a3389d493feb800fe85e60b37b3dc556b481f459097"
},
"downloads": -1,
"filename": "crashstats_tools-2.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f342699732a2ce35c333746ec0da1e88",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 34939,
"upload_time": "2024-10-30T23:48:00",
"upload_time_iso_8601": "2024-10-30T23:48:00.596973Z",
"url": "https://files.pythonhosted.org/packages/dc/20/a36a56d6dd297255d3f46a82318ef539b6c6f03e1f674c2e72d7a421c1b7/crashstats_tools-2.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6cfd14a7cba76e9ea5e33265c24f01329c20fd9834ddef131521f7d531bc4ad8",
"md5": "356b4f31072a136224c629b7c857f40f",
"sha256": "18bd2f954beb126f3f42a9e7ffce228c53b52bae46488e5476c957a3870646a4"
},
"downloads": -1,
"filename": "crashstats_tools-2.1.0.tar.gz",
"has_sig": false,
"md5_digest": "356b4f31072a136224c629b7c857f40f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 45932,
"upload_time": "2024-10-30T23:48:02",
"upload_time_iso_8601": "2024-10-30T23:48:02.473690Z",
"url": "https://files.pythonhosted.org/packages/6c/fd/14a7cba76e9ea5e33265c24f01329c20fd9834ddef131521f7d531bc4ad8/crashstats_tools-2.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-30 23:48:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mozilla-services",
"github_project": "crashstats-tools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "crashstats-tools"
}
