| Name | madoop JSON |
| Version |
1.3.0
JSON |
| download |
| home_page | None |
| Summary | A light weight MapReduce framework for education. |
| upload_time | 2024-11-06 20:50:28 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.6 |
| license | MIT License Copyright (c) 2021 EECS 485 Staff Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
madoop
hadoop
mapreduce
michigan hadoop
hadoop streaming
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
Madoop: Michigan Hadoop
=======================
[](https://pypi.org/project/madoop/)
[](https://github.com/eecs485staff/madoop/actions?query=branch%3Adevelop)
[](https://codecov.io/gh/eecs485staff/madoop)
Michigan Hadoop (`madoop`) is a light weight MapReduce framework for education. Madoop implements the [Hadoop Streaming](https://hadoop.apache.org/docs/r1.2.1/streaming.html) interface. Madoop is implemented in Python and runs on a single machine.
For an in-depth explanation of how to write MapReduce programs in Python for Hadoop Streaming, see our [Hadoop Streaming tutorial](README_Hadoop_Streaming.md).
## Quick start
Install Madoop.
```console
$ pip install madoop
```
Create example MapReduce program with input files.
```console
$ madoop --example
$ tree example
example
├── input
│ ├── input01.txt
│ └── input02.txt
├── map.py
└── reduce.py
```
Run example word count MapReduce program.
```console
$ madoop \
-input example/input \
-output example/output \
-mapper example/map.py \
-reducer example/reduce.py
```
Concatenate and print the output.
```console
$ cat example/output/part-*
Goodbye 1
Bye 1
Hadoop 2
World 2
Hello 2
```
## Comparison with Apache Hadoop and CLI
Madoop implements a subset of the [Hadoop Streaming](https://hadoop.apache.org/docs/r1.2.1/streaming.html) interface. You can simulate the Hadoop Streaming interface at the command line with `cat` and `sort`.
Here's how to run our example MapReduce program on Apache Hadoop.
```console
$ hadoop \
jar path/to/hadoop-streaming-X.Y.Z.jar
-input example/input \
-output output \
-mapper example/map.py \
-reducer example/reduce.py
$ cat output/part-*
```
Here's how to run our example MapReduce program at the command line using `cat` and `sort`.
```console
$ cat input/* | ./map.py | sort | ./reduce.py
```
| Madoop | Hadoop | `cat`/`sort` |
|-|-|-|
| Implement some Hadoop options | All Hadoop options | No Hadoop options |
| Multiple mappers and reducers | Multiple mappers and reducers | One mapper, one reducer |
| Single machine | Many machines | Single Machine |
| `jar hadoop-streaming-X.Y.Z.jar` argument ignored | `jar hadoop-streaming-X.Y.Z.jar` argument required | No arguments |
| Lines within a group are sorted | Lines within a group are sorted | Lines within a group are sorted |
## Contributing
Contributions from the community are welcome! Check out the [guide for contributing](CONTRIBUTING.md).
## Acknowledgments
Michigan Hadoop is written by Andrew DeOrio <awdeorio@umich.edu>.
Raw data
{
"_id": null,
"home_page": null,
"name": "madoop",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "madoop, Hadoop, MapReduce, Michigan Hadoop, Hadoop Streaming",
"author": null,
"author_email": "Andrew DeOrio <awdeorio@umich.edu>",
"download_url": "https://files.pythonhosted.org/packages/2b/aa/50559a4248cc2eb15b0e1c173472be43824e41ea4e203fd48aaf987f04b2/madoop-1.3.0.tar.gz",
"platform": null,
"description": "Madoop: Michigan Hadoop\n=======================\n\n[](https://pypi.org/project/madoop/)\n[](https://github.com/eecs485staff/madoop/actions?query=branch%3Adevelop)\n[](https://codecov.io/gh/eecs485staff/madoop)\n\nMichigan Hadoop (`madoop`) is a light weight MapReduce framework for education. Madoop implements the [Hadoop Streaming](https://hadoop.apache.org/docs/r1.2.1/streaming.html) interface. Madoop is implemented in Python and runs on a single machine.\n\nFor an in-depth explanation of how to write MapReduce programs in Python for Hadoop Streaming, see our [Hadoop Streaming tutorial](README_Hadoop_Streaming.md).\n\n\n## Quick start\nInstall Madoop.\n```console\n$ pip install madoop\n```\n\nCreate example MapReduce program with input files.\n```console\n$ madoop --example\n$ tree example\nexample\n\u251c\u2500\u2500 input\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 input01.txt\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 input02.txt\n\u251c\u2500\u2500 map.py\n\u2514\u2500\u2500 reduce.py\n```\n\nRun example word count MapReduce program.\n```console\n$ madoop \\\n -input example/input \\\n -output example/output \\\n -mapper example/map.py \\\n -reducer example/reduce.py\n```\n\nConcatenate and print the output.\n```console\n$ cat example/output/part-*\nGoodbye 1\nBye 1\nHadoop 2\nWorld 2\nHello 2\n```\n\n## Comparison with Apache Hadoop and CLI\nMadoop implements a subset of the [Hadoop Streaming](https://hadoop.apache.org/docs/r1.2.1/streaming.html) interface. You can simulate the Hadoop Streaming interface at the command line with `cat` and `sort`.\n\nHere's how to run our example MapReduce program on Apache Hadoop.\n```console\n$ hadoop \\\n jar path/to/hadoop-streaming-X.Y.Z.jar\n -input example/input \\\n -output output \\\n -mapper example/map.py \\\n -reducer example/reduce.py\n$ cat output/part-*\n```\n\nHere's how to run our example MapReduce program at the command line using `cat` and `sort`.\n```console\n$ cat input/* | ./map.py | sort | ./reduce.py\n```\n\n| Madoop | Hadoop | `cat`/`sort` |\n|-|-|-|\n| Implement some Hadoop options | All Hadoop options | No Hadoop options |\n| Multiple mappers and reducers | Multiple mappers and reducers | One mapper, one reducer |\n| Single machine | Many machines | Single Machine |\n| `jar hadoop-streaming-X.Y.Z.jar` argument ignored | `jar hadoop-streaming-X.Y.Z.jar` argument required | No arguments |\n| Lines within a group are sorted | Lines within a group are sorted | Lines within a group are sorted |\n\n\n## Contributing\nContributions from the community are welcome! Check out the [guide for contributing](CONTRIBUTING.md).\n\n\n## Acknowledgments\nMichigan Hadoop is written by Andrew DeOrio <awdeorio@umich.edu>.\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2021 EECS 485 Staff Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "A light weight MapReduce framework for education.",
"version": "1.3.0",
"project_urls": {
"documentation": "https://github.com/eecs485staff/madoop/blob/develop/README_Hadoop_Streaming.md#hadoop-streaming-in-python",
"repository": "https://github.com/eecs485staff/madoop/"
},
"split_keywords": [
"madoop",
" hadoop",
" mapreduce",
" michigan hadoop",
" hadoop streaming"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "73c2fab53dc23d3fb17bb4b93303e2838ba9c299ae80ec283f0bd48d4a3396cd",
"md5": "110ee893cee68c6fd15f135739134536",
"sha256": "e8b709be5c65f10f1ff4091af36f80c4084028f48255507fa8ab644387efcfab"
},
"downloads": -1,
"filename": "madoop-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "110ee893cee68c6fd15f135739134536",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 11443,
"upload_time": "2024-11-06T20:50:27",
"upload_time_iso_8601": "2024-11-06T20:50:27.552334Z",
"url": "https://files.pythonhosted.org/packages/73/c2/fab53dc23d3fb17bb4b93303e2838ba9c299ae80ec283f0bd48d4a3396cd/madoop-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2baa50559a4248cc2eb15b0e1c173472be43824e41ea4e203fd48aaf987f04b2",
"md5": "9a987d3f97bdf4ff81fe3bf1054e2814",
"sha256": "b897588572384187e333a182beafaedde0aaa350a0e62b4b59ce6f74d45c4432"
},
"downloads": -1,
"filename": "madoop-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "9a987d3f97bdf4ff81fe3bf1054e2814",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 20924,
"upload_time": "2024-11-06T20:50:28",
"upload_time_iso_8601": "2024-11-06T20:50:28.399854Z",
"url": "https://files.pythonhosted.org/packages/2b/aa/50559a4248cc2eb15b0e1c173472be43824e41ea4e203fd48aaf987f04b2/madoop-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-06 20:50:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "eecs485staff",
"github_project": "madoop",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "madoop"
}
