spark-dataframe-tools


Namespark-dataframe-tools JSON
Version 0.6.14 PyPI version JSON
download
home_pagehttps://github.com/jonaqp/spark_dataframe_tools/
Summaryspark_dataframe_tools
upload_time2024-10-18 05:26:52
maintainerNone
docs_urlNone
authorJonathan Quiza
requires_pythonNone
licenseNone
keywords spark dataframe
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # spark_dataframe_tools

[![Github License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Updates](https://pyup.io/repos/github/woctezuma/google-colab-transfer/shield.svg)](pyup)
[![Python 3](https://pyup.io/repos/github/woctezuma/google-colab-transfer/python-3-shield.svg)](pyup)
[![Code coverage](https://codecov.io/gh/woctezuma/google-colab-transfer/branch/master/graph/badge.svg)](codecov)

spark_dataframe_tools is a Python library that implements styles in the Dataframe

## Installation

The code is packaged for PyPI, so that the installation consists in running:

```sh
pip install spark-dataframe-tools --user --upgrade
```

## Usage
```sh
import spark_dataframe_tools 
```

```sh
from pyspark.sql.types import StructType,StructField, StringType, IntegerType
data2 = [("James","","Smith","36636","M",3000),
    ("Michael","Rose","","40288","M",4000),
    ("Robert","","Williams","42114","M",4000),
    ("Maria","Anne","Jones","39192","F",4000),
    ("Jen","Mary","Brown","","F",-1)
  ]

schema = StructType([ \
    StructField("firstname",StringType(),True), \
    StructField("middlename",StringType(),True), \
    StructField("lastname",StringType(),True), \
    StructField("id", StringType(), True), \
    StructField("gender", StringType(), True), \
    StructField("salary", IntegerType(), True) \
  ])
 
df = spark.createDataFrame(data=data2, schema=schema)
```

## Pandas

```sh
df_pandas = df.toPandas()
df_pandas.show2()
```

## Spark

```sh
# Dataframe template table
df.show2()

# Dataframe memory usage
df.size()
```



## License

[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).

## New features v1.0

## BugFix

- choco install visualcpp-build-tools

## Reference

- Jonathan Quiza [github](https://github.com/jonaqp).
- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/jonaqp/spark_dataframe_tools/",
    "name": "spark-dataframe-tools",
    "maintainer": null,
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": "spark, dataframe",
    "author": "Jonathan Quiza",
    "author_email": "jony327@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/45/80/d687eacfd54aff08bfbc7a749413039a5389c9da69ae7be6a6b3e55302bf/spark_dataframe_tools-0.6.14.tar.gz",
    "platform": null,
    "description": "# spark_dataframe_tools\r\n\r\n[![Github License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\r\n[![Updates](https://pyup.io/repos/github/woctezuma/google-colab-transfer/shield.svg)](pyup)\r\n[![Python 3](https://pyup.io/repos/github/woctezuma/google-colab-transfer/python-3-shield.svg)](pyup)\r\n[![Code coverage](https://codecov.io/gh/woctezuma/google-colab-transfer/branch/master/graph/badge.svg)](codecov)\r\n\r\nspark_dataframe_tools is a Python library that implements styles in the Dataframe\r\n\r\n## Installation\r\n\r\nThe code is packaged for PyPI, so that the installation consists in running:\r\n\r\n```sh\r\npip install spark-dataframe-tools --user --upgrade\r\n```\r\n\r\n## Usage\r\n```sh\r\nimport spark_dataframe_tools \r\n```\r\n\r\n```sh\r\nfrom pyspark.sql.types import StructType,StructField, StringType, IntegerType\r\ndata2 = [(\"James\",\"\",\"Smith\",\"36636\",\"M\",3000),\r\n    (\"Michael\",\"Rose\",\"\",\"40288\",\"M\",4000),\r\n    (\"Robert\",\"\",\"Williams\",\"42114\",\"M\",4000),\r\n    (\"Maria\",\"Anne\",\"Jones\",\"39192\",\"F\",4000),\r\n    (\"Jen\",\"Mary\",\"Brown\",\"\",\"F\",-1)\r\n  ]\r\n\r\nschema = StructType([ \\\r\n    StructField(\"firstname\",StringType(),True), \\\r\n    StructField(\"middlename\",StringType(),True), \\\r\n    StructField(\"lastname\",StringType(),True), \\\r\n    StructField(\"id\", StringType(), True), \\\r\n    StructField(\"gender\", StringType(), True), \\\r\n    StructField(\"salary\", IntegerType(), True) \\\r\n  ])\r\n \r\ndf = spark.createDataFrame(data=data2, schema=schema)\r\n```\r\n\r\n## Pandas\r\n\r\n```sh\r\ndf_pandas = df.toPandas()\r\ndf_pandas.show2()\r\n```\r\n\r\n## Spark\r\n\r\n```sh\r\n# Dataframe template table\r\ndf.show2()\r\n\r\n# Dataframe memory usage\r\ndf.size()\r\n```\r\n\r\n\r\n\r\n## License\r\n\r\n[Apache License 2.0](https://www.dropbox.com/s/8t6xtgk06o3ij61/LICENSE?dl=0).\r\n\r\n## New features v1.0\r\n\r\n## BugFix\r\n\r\n- choco install visualcpp-build-tools\r\n\r\n## Reference\r\n\r\n- Jonathan Quiza [github](https://github.com/jonaqp).\r\n- Jonathan Quiza [RumiMLSpark](http://rumi-ml.herokuapp.com/).\r\n\r\n\r\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "spark_dataframe_tools",
    "version": "0.6.14",
    "project_urls": {
        "Download": "https://github.com/jonaqp/spark_dataframe_tools/archive/main.zip",
        "Homepage": "https://github.com/jonaqp/spark_dataframe_tools/"
    },
    "split_keywords": [
        "spark",
        " dataframe"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "39788a079059a7d4de277895ff76d7ac3ac3eccb339cf0079f30f54acac90aa3",
                "md5": "976d3df3fb3cf941aad596f78133cb75",
                "sha256": "4a8c12e55b2770df5a567e0696294131131db8f934ead8e33910bb782bb30611"
            },
            "downloads": -1,
            "filename": "spark_dataframe_tools-0.6.14-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "976d3df3fb3cf941aad596f78133cb75",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 12459,
            "upload_time": "2024-10-18T05:26:50",
            "upload_time_iso_8601": "2024-10-18T05:26:50.570822Z",
            "url": "https://files.pythonhosted.org/packages/39/78/8a079059a7d4de277895ff76d7ac3ac3eccb339cf0079f30f54acac90aa3/spark_dataframe_tools-0.6.14-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "4580d687eacfd54aff08bfbc7a749413039a5389c9da69ae7be6a6b3e55302bf",
                "md5": "b469ba7f34f229b097e0b01888d55f93",
                "sha256": "c9a080273102cd49f9f73a5d6c1d82feb2eb8b596de92d8c2c9edd2a1a491af8"
            },
            "downloads": -1,
            "filename": "spark_dataframe_tools-0.6.14.tar.gz",
            "has_sig": false,
            "md5_digest": "b469ba7f34f229b097e0b01888d55f93",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 11050,
            "upload_time": "2024-10-18T05:26:52",
            "upload_time_iso_8601": "2024-10-18T05:26:52.154099Z",
            "url": "https://files.pythonhosted.org/packages/45/80/d687eacfd54aff08bfbc7a749413039a5389c9da69ae7be6a6b3e55302bf/spark_dataframe_tools-0.6.14.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-10-18 05:26:52",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "jonaqp",
    "github_project": "spark_dataframe_tools",
    "github_not_found": true,
    "lcname": "spark-dataframe-tools"
}
Jonathan Quiza
Elapsed time: 2.20225s