# datagrepper
Datagrepper is a web application and JSON API to retrieve historical messages sent via Fedora Messaging. [Datanommer](https://github.com/fedora-infra/datanommer/) is a seperate project and service that consumes messages from the Fedora Messaging queue and puts them in a database. These messages is what datagrepper queries.
Datagrepper is curently running in production at https://apps.fedoraproject.org/datagrepper/
## Development Environment
Vagrant allows contributors to get quickly up and running with a datagrepper development environment by automatically configuring a virtual machine.
The datagrepper Vagrant environment configures configures and enables a datanommer service and database. The datanommer instance is configured to be empty when first provisioned, but to consume messages from the stage Fedora Messaging queue.
### Install vagrant
To get started, run the following commands to install the Vagrant and Virtualization packages needed, and start the libvirt service:
$ sudo dnf install ansible libvirt vagrant-libvirt vagrant-sshfs vagrant-hostmanager
$ sudo systemctl enable libvirtd
$ sudo systemctl start libvirtd
### Checkout and Provision
Next, check out the datagrepper code and run vagrant up:
$ git clone https://github.com/fedora-infra/datagrepper
$ cd datanommer
$ vagrant up
### Interacting with your development datagrepper
After successful provisioning of the Datagrepper vagrant setup, the datagrepper web application will be accessible from your host machine's web browser at
http://datagrepper.test:5000/
### Using the development environment
SSH into your newly provisioned development environment:
$ vagrant ssh
The vagrant setup also defines 4 handy commands to interact with the service that runs the datagrepper flask application:
$ datagrepper-start
$ datagrepper-stop
$ datagrepper-restart
$ dataprepper-logs
Additionally, the following commands are also available for interacting with the datanommer service:
$ datanommer-consumer-start
$ datanommer-consumer-stop
$ datanommer-consumer-restart
$ datanommer-consumer-logs
### Running the tests
Datanommer is comprised of 3 seperate modules in this single repository. There is a handy script in the top directory of this repo to run the tests on all 3 modules:
$ ./runtests.sh
However, tests can also be run on a single module by invotking tox in that modules' directory. For example:
$ cd datanommer.models/
$ tox
Note, that the tests use virtual environments that are not created from scratch with every subsequent run of the tests. Therefore, **when changes happen to dependencies, the tests may fail to run correctly**. To recreate the virtual envrionments, run the tests commands with the `-r` flag, for example:
$ ./runtests.sh -r
or
$ cd datanommer.models/
$ tox -r
Raw data
{
"_id": null,
"home_page": "https://github.com/fedora-infra/datagrepper",
"name": "datagrepper",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Fedora Infrastructure",
"author_email": "admin@fedoraproject.org",
"download_url": "https://files.pythonhosted.org/packages/d9/2b/6d36f46610750d75b755cee90e156eece31e13e474d4427943d8563de035/datagrepper-1.1.0.tar.gz",
"platform": null,
"description": "# datagrepper\n\nDatagrepper is a web application and JSON API to retrieve historical messages sent via Fedora Messaging. [Datanommer](https://github.com/fedora-infra/datanommer/) is a seperate project and service that consumes messages from the Fedora Messaging queue and puts them in a database. These messages is what datagrepper queries. \n\nDatagrepper is curently running in production at https://apps.fedoraproject.org/datagrepper/\n\n## Development Environment\n\nVagrant allows contributors to get quickly up and running with a datagrepper development environment by automatically configuring a virtual machine. \n\nThe datagrepper Vagrant environment configures configures and enables a datanommer service and database. The datanommer instance is configured to be empty when first provisioned, but to consume messages from the stage Fedora Messaging queue.\n\n### Install vagrant\nTo get started, run the following commands to install the Vagrant and Virtualization packages needed, and start the libvirt service:\n\n $ sudo dnf install ansible libvirt vagrant-libvirt vagrant-sshfs vagrant-hostmanager\n $ sudo systemctl enable libvirtd\n $ sudo systemctl start libvirtd\n\n### Checkout and Provision\nNext, check out the datagrepper code and run vagrant up:\n\n $ git clone https://github.com/fedora-infra/datagrepper\n $ cd datanommer\n $ vagrant up\n\n### Interacting with your development datagrepper\nAfter successful provisioning of the Datagrepper vagrant setup, the datagrepper web application will be accessible from your host machine's web browser at\n\nhttp://datagrepper.test:5000/\n\n\n\n\n### Using the development environment\nSSH into your newly provisioned development environment:\n\n $ vagrant ssh\n\nThe vagrant setup also defines 4 handy commands to interact with the service that runs the datagrepper flask application: \n\n $ datagrepper-start\n $ datagrepper-stop\n $ datagrepper-restart\n $ dataprepper-logs\n\nAdditionally, the following commands are also available for interacting with the datanommer service:\n\n $ datanommer-consumer-start\n $ datanommer-consumer-stop\n $ datanommer-consumer-restart\n $ datanommer-consumer-logs\n\n### Running the tests\nDatanommer is comprised of 3 seperate modules in this single repository. There is a handy script in the top directory of this repo to run the tests on all 3 modules:\n\n $ ./runtests.sh\n\nHowever, tests can also be run on a single module by invotking tox in that modules' directory. For example:\n\n $ cd datanommer.models/\n $ tox\n\nNote, that the tests use virtual environments that are not created from scratch with every subsequent run of the tests. Therefore, **when changes happen to dependencies, the tests may fail to run correctly**. To recreate the virtual envrionments, run the tests commands with the `-r` flag, for example:\n\n $ ./runtests.sh -r\n\nor\n\n $ cd datanommer.models/\n $ tox -r\n",
"bugtrack_url": null,
"license": "GPL-2.0-or-later",
"summary": "A webapp to query datanommer",
"version": "1.1.0",
"project_urls": {
"Homepage": "https://github.com/fedora-infra/datagrepper",
"Repository": "https://github.com/fedora-infra/datagrepper"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f6e600e2e1c58501b32ee93e3542866742ecc7299c2c754590e451709943c561",
"md5": "9289d60b8fc20670df1913eb061def18",
"sha256": "8c774363ca9094294ad02d6b49373e6feb5e8491b8536ddd607b48bc0ba7b832"
},
"downloads": -1,
"filename": "datagrepper-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9289d60b8fc20670df1913eb061def18",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 3131385,
"upload_time": "2023-09-26T09:49:05",
"upload_time_iso_8601": "2023-09-26T09:49:05.652645Z",
"url": "https://files.pythonhosted.org/packages/f6/e6/00e2e1c58501b32ee93e3542866742ecc7299c2c754590e451709943c561/datagrepper-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d92b6d36f46610750d75b755cee90e156eece31e13e474d4427943d8563de035",
"md5": "d90529efbe63919a721267ca047928a3",
"sha256": "3b461cbb5f54ef1b133bb462a8f2ed872c8832ae4bddce110c086d822b98ec35"
},
"downloads": -1,
"filename": "datagrepper-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "d90529efbe63919a721267ca047928a3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 3142088,
"upload_time": "2023-09-26T09:49:07",
"upload_time_iso_8601": "2023-09-26T09:49:07.924092Z",
"url": "https://files.pythonhosted.org/packages/d9/2b/6d36f46610750d75b755cee90e156eece31e13e474d4427943d8563de035/datagrepper-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-26 09:49:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fedora-infra",
"github_project": "datagrepper",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "datagrepper"
}
JSON
