# SIBR Module
[](https://badge.fury.io/py/sibr-module) A collection of helper modules for interacting with Google Cloud Platform services, designed to simplify common workflows. This package provides easy-to-use classes for BigQuery, Google Cloud Storage, Secret Manager, and Cloud Logging.
## Features
* **`BigQuery`**: Easily upload DataFrames to BigQuery tables with automatic schema detection and support for `append`, `replace`, and `merge` operations.
* **`CStorage`**: Upload and download files to and from Google Cloud Storage buckets.
* **`SecretsManager`**: Securely access secrets from Google Secret Manager.
* **`Logger`**: A flexible logger that supports both local file logging and integration with Google Cloud Logging.
## Installation
You can install the package from PyPI:
```pip install sibr-module ```
## Quickstart
Ensure you are authenticated with Google Cloud. You can do this by running:
```gcloud auth application-default login```
If in use locally, make sure to set your credentials available as environment variables using: `GOOGLE_APPLICATION_CREDENTIALS=path-to-your-credentials`.
```python
import pandas as pd
from sibr_module import BigQuery, Logger
# --- 1. Set up a logger ---
# This will log to a local file and, if enabled, to Google Cloud Logging
logger = Logger(log_name="my_app_logger", enable_cloud_logging=True)
logger.info("Application starting up.")
# --- 2. Prepare your data ---
data = {'name': ['Alice', 'Bob'], 'score': [85, 92]}
my_dataframe = pd.DataFrame(data)
# --- 3. Use the BigQuery helper ---
try:
# Initialize the client with your Google Cloud Project ID
bq_client = BigQuery(project_id="your-gcp-project-id", logger=logger)
# Upload the DataFrame to a BigQuery table
bq_client.to_bq(
df=my_dataframe,
dataset_name="my_dataset",
table_name="my_table",
if_exists="append" # Options: 'append', 'replace', or 'merge'
)
logger.info("Successfully uploaded data to BigQuery.")
except Exception as e:
logger.error(f"An error occurred: {e}")
```
## Usage Details
### BigQuery
The BigQuery class handles interactions with Google BigQuery.
* to_bq(df, dataset_name, table_name, if_exists='append', merge_on=None): Uploads a pandas DataFrame.
* if_exists='append': Adds data to an existing table.
* if_exists='replace': Deletes the existing table and creates a new one.
* if_exists='merge': Updates existing rows and inserts new ones. Requires merge_on to be set with a list of key columns.
* read_bq(query): Executes a query and returns the result as a pandas DataFrame.
* dtype_map: Optional argument if the user wishes to specify a certain mapping of dtypes to Big Query types
### CStorage
The CStorage class simplifies file operations with Google Cloud Storage.
* upload(local_file_path, destination_blob_name): Uploads a local file to the bucket.
* download(source_blob_name, destination_file_path): Downloads a file from the bucket.
### SecretsManager
Access your secrets easily.
* get_secret(secret_id): Retrieves the latest version of a secret by its ID.
## Contributing
Contributions are welcome! Please open an issue or submit a pull request if you have any improvements or bug fixes.
## License
This project is licensed under the MIT License. See the LICENSE file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "sibr-module",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "google cloud, bigquery, gcs, utilities",
"author": null,
"author_email": "Sigvard Bratlie <sigvard@sibr.no>",
"download_url": "https://files.pythonhosted.org/packages/63/ce/535994b5fbead1c556cefac8e6331dc9954a049784fcdcf6e3205aa74379/sibr_module-0.1.7.tar.gz",
"platform": null,
"description": "# SIBR Module\n\n[](https://badge.fury.io/py/sibr-module) A collection of helper modules for interacting with Google Cloud Platform services, designed to simplify common workflows. This package provides easy-to-use classes for BigQuery, Google Cloud Storage, Secret Manager, and Cloud Logging.\n\n## Features\n\n* **`BigQuery`**: Easily upload DataFrames to BigQuery tables with automatic schema detection and support for `append`, `replace`, and `merge` operations.\n* **`CStorage`**: Upload and download files to and from Google Cloud Storage buckets.\n* **`SecretsManager`**: Securely access secrets from Google Secret Manager.\n* **`Logger`**: A flexible logger that supports both local file logging and integration with Google Cloud Logging.\n\n## Installation\n\nYou can install the package from PyPI:\n\n```pip install sibr-module ```\n\n## Quickstart\nEnsure you are authenticated with Google Cloud. You can do this by running:\n\n```gcloud auth application-default login```\n\nIf in use locally, make sure to set your credentials available as environment variables using: `GOOGLE_APPLICATION_CREDENTIALS=path-to-your-credentials`.\n\n```python\nimport pandas as pd\nfrom sibr_module import BigQuery, Logger\n\n# --- 1. Set up a logger ---\n# This will log to a local file and, if enabled, to Google Cloud Logging\nlogger = Logger(log_name=\"my_app_logger\", enable_cloud_logging=True)\nlogger.info(\"Application starting up.\")\n\n# --- 2. Prepare your data ---\ndata = {'name': ['Alice', 'Bob'], 'score': [85, 92]}\nmy_dataframe = pd.DataFrame(data)\n\n# --- 3. Use the BigQuery helper ---\ntry:\n # Initialize the client with your Google Cloud Project ID\n bq_client = BigQuery(project_id=\"your-gcp-project-id\", logger=logger)\n\n # Upload the DataFrame to a BigQuery table\n bq_client.to_bq(\n df=my_dataframe,\n dataset_name=\"my_dataset\",\n table_name=\"my_table\",\n if_exists=\"append\" # Options: 'append', 'replace', or 'merge'\n )\n\n logger.info(\"Successfully uploaded data to BigQuery.\")\n\nexcept Exception as e:\n logger.error(f\"An error occurred: {e}\")\n```\n\n## Usage Details\n### BigQuery\nThe BigQuery class handles interactions with Google BigQuery. \n* to_bq(df, dataset_name, table_name, if_exists='append', merge_on=None): Uploads a pandas DataFrame.\n * if_exists='append': Adds data to an existing table.\n * if_exists='replace': Deletes the existing table and creates a new one.\n * if_exists='merge': Updates existing rows and inserts new ones. Requires merge_on to be set with a list of key columns.\n * read_bq(query): Executes a query and returns the result as a pandas DataFrame.\n * dtype_map: Optional argument if the user wishes to specify a certain mapping of dtypes to Big Query types\n\n### CStorage\nThe CStorage class simplifies file operations with Google Cloud Storage.\n\n* upload(local_file_path, destination_blob_name): Uploads a local file to the bucket.\n* download(source_blob_name, destination_file_path): Downloads a file from the bucket.\n\n### SecretsManager\nAccess your secrets easily.\n\n* get_secret(secret_id): Retrieves the latest version of a secret by its ID.\n\n## Contributing\nContributions are welcome! Please open an issue or submit a pull request if you have any improvements or bug fixes.\n\n## License\nThis project is licensed under the MIT License. See the LICENSE file for details.\n",
"bugtrack_url": null,
"license": "LICENSE",
"summary": "Help modules for different applications, especially for google cloud",
"version": "0.1.7",
"project_urls": {
"Bug Tracker": "https://github.com/sigvardbratlie/sibr-module/issues",
"Homepage": "https://github.com/sigvardbratlie/sibr-module"
},
"split_keywords": [
"google cloud",
" bigquery",
" gcs",
" utilities"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "d16c7a8c4566ff6f1e66ec98fd02f7b15366befca8349fa698aba58f5c7ce01e",
"md5": "8a04c71e1b8ddbef35af6d77dff76520",
"sha256": "ed15e5cd0c6ca42e7e6d8d5b9aed1980429e8289f11f5556053eefa9ae44aa0d"
},
"downloads": -1,
"filename": "sibr_module-0.1.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8a04c71e1b8ddbef35af6d77dff76520",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 11634,
"upload_time": "2025-08-14T17:39:57",
"upload_time_iso_8601": "2025-08-14T17:39:57.138226Z",
"url": "https://files.pythonhosted.org/packages/d1/6c/7a8c4566ff6f1e66ec98fd02f7b15366befca8349fa698aba58f5c7ce01e/sibr_module-0.1.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "63ce535994b5fbead1c556cefac8e6331dc9954a049784fcdcf6e3205aa74379",
"md5": "0857ac8e5ec139b4909379a9c187895a",
"sha256": "b192f7356cc3cf9b5468320c68e22d02374b1eadb29b79f89ae3faab2f273721"
},
"downloads": -1,
"filename": "sibr_module-0.1.7.tar.gz",
"has_sig": false,
"md5_digest": "0857ac8e5ec139b4909379a9c187895a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 13116,
"upload_time": "2025-08-14T17:39:58",
"upload_time_iso_8601": "2025-08-14T17:39:58.021975Z",
"url": "https://files.pythonhosted.org/packages/63/ce/535994b5fbead1c556cefac8e6331dc9954a049784fcdcf6e3205aa74379/sibr_module-0.1.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-14 17:39:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sigvardbratlie",
"github_project": "sibr-module",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "aiohappyeyeballs",
"specs": [
[
"==",
"2.6.1"
]
]
},
{
"name": "aiohttp",
"specs": [
[
"==",
"3.12.15"
]
]
},
{
"name": "aiosignal",
"specs": [
[
"==",
"1.4.0"
]
]
},
{
"name": "asttokens",
"specs": [
[
"==",
"3.0.0"
]
]
},
{
"name": "async-timeout",
"specs": [
[
"==",
"5.0.1"
]
]
},
{
"name": "atpublic",
"specs": [
[
"==",
"5.1"
]
]
},
{
"name": "attrs",
"specs": [
[
"==",
"25.3.0"
]
]
},
{
"name": "bigframes",
"specs": [
[
"==",
"2.15.0"
]
]
},
{
"name": "cachetools",
"specs": [
[
"==",
"5.5.2"
]
]
},
{
"name": "certifi",
"specs": [
[
"==",
"2025.8.3"
]
]
},
{
"name": "charset-normalizer",
"specs": [
[
"==",
"3.4.3"
]
]
},
{
"name": "cloudpickle",
"specs": [
[
"==",
"3.1.1"
]
]
},
{
"name": "comm",
"specs": [
[
"==",
"0.2.3"
]
]
},
{
"name": "contourpy",
"specs": [
[
"==",
"1.3.0"
]
]
},
{
"name": "cycler",
"specs": [
[
"==",
"0.12.1"
]
]
},
{
"name": "db-dtypes",
"specs": [
[
"==",
"1.4.3"
]
]
},
{
"name": "decorator",
"specs": [
[
"==",
"5.2.1"
]
]
},
{
"name": "exceptiongroup",
"specs": [
[
"==",
"1.3.0"
]
]
},
{
"name": "executing",
"specs": [
[
"==",
"2.2.0"
]
]
},
{
"name": "fonttools",
"specs": [
[
"==",
"4.59.0"
]
]
},
{
"name": "frozenlist",
"specs": [
[
"==",
"1.7.0"
]
]
},
{
"name": "fsspec",
"specs": [
[
"==",
"2025.7.0"
]
]
},
{
"name": "gcsfs",
"specs": [
[
"==",
"2025.7.0"
]
]
},
{
"name": "geopandas",
"specs": [
[
"==",
"1.0.1"
]
]
},
{
"name": "google-api-core",
"specs": [
[
"==",
"2.25.1"
]
]
},
{
"name": "google-auth",
"specs": [
[
"==",
"2.40.3"
]
]
},
{
"name": "google-auth-oauthlib",
"specs": [
[
"==",
"1.2.2"
]
]
},
{
"name": "google-cloud",
"specs": [
[
"==",
"0.34.0"
]
]
},
{
"name": "google-cloud-appengine-logging",
"specs": [
[
"==",
"1.6.2"
]
]
},
{
"name": "google-cloud-audit-log",
"specs": [
[
"==",
"0.3.2"
]
]
},
{
"name": "google-cloud-bigquery",
"specs": [
[
"==",
"3.35.1"
]
]
},
{
"name": "google-cloud-bigquery-connection",
"specs": [
[
"==",
"1.18.3"
]
]
},
{
"name": "google-cloud-bigquery-storage",
"specs": [
[
"==",
"2.32.0"
]
]
},
{
"name": "google-cloud-core",
"specs": [
[
"==",
"2.4.3"
]
]
},
{
"name": "google-cloud-functions",
"specs": [
[
"==",
"1.20.4"
]
]
},
{
"name": "google-cloud-iam",
"specs": [
[
"==",
"2.19.1"
]
]
},
{
"name": "google-cloud-logging",
"specs": [
[
"==",
"3.12.1"
]
]
},
{
"name": "google-cloud-resource-manager",
"specs": [
[
"==",
"1.14.2"
]
]
},
{
"name": "google-cloud-secret-manager",
"specs": [
[
"==",
"2.24.0"
]
]
},
{
"name": "google-cloud-storage",
"specs": [
[
"==",
"3.3.0"
]
]
},
{
"name": "google-crc32c",
"specs": [
[
"==",
"1.7.1"
]
]
},
{
"name": "google-resumable-media",
"specs": [
[
"==",
"2.7.2"
]
]
},
{
"name": "googleapis-common-protos",
"specs": [
[
"==",
"1.70.0"
]
]
},
{
"name": "grpc-google-iam-v1",
"specs": [
[
"==",
"0.14.2"
]
]
},
{
"name": "grpcio",
"specs": [
[
"==",
"1.74.0"
]
]
},
{
"name": "grpcio-status",
"specs": [
[
"==",
"1.74.0"
]
]
},
{
"name": "humanize",
"specs": [
[
"==",
"4.12.3"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.10"
]
]
},
{
"name": "importlib-metadata",
"specs": [
[
"==",
"8.7.0"
]
]
},
{
"name": "importlib-resources",
"specs": [
[
"==",
"6.5.2"
]
]
},
{
"name": "ipython",
"specs": [
[
"==",
"8.18.1"
]
]
},
{
"name": "ipywidgets",
"specs": [
[
"==",
"8.1.7"
]
]
},
{
"name": "jedi",
"specs": [
[
"==",
"0.19.2"
]
]
},
{
"name": "jupyterlab-widgets",
"specs": [
[
"==",
"3.0.15"
]
]
},
{
"name": "kiwisolver",
"specs": [
[
"==",
"1.4.7"
]
]
},
{
"name": "markdown-it-py",
"specs": [
[
"==",
"3.0.0"
]
]
},
{
"name": "matplotlib",
"specs": [
[
"==",
"3.9.4"
]
]
},
{
"name": "matplotlib-inline",
"specs": [
[
"==",
"0.1.7"
]
]
},
{
"name": "mdurl",
"specs": [
[
"==",
"0.1.2"
]
]
},
{
"name": "multidict",
"specs": [
[
"==",
"6.6.4"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"2.0.2"
]
]
},
{
"name": "oauthlib",
"specs": [
[
"==",
"3.3.1"
]
]
},
{
"name": "opentelemetry-api",
"specs": [
[
"==",
"1.36.0"
]
]
},
{
"name": "packaging",
"specs": [
[
"==",
"25.0"
]
]
},
{
"name": "pandas",
"specs": [
[
"==",
"2.3.1"
]
]
},
{
"name": "pandas-gbq",
"specs": [
[
"==",
"0.29.2"
]
]
},
{
"name": "parso",
"specs": [
[
"==",
"0.8.4"
]
]
},
{
"name": "pexpect",
"specs": [
[
"==",
"4.9.0"
]
]
},
{
"name": "pillow",
"specs": [
[
"==",
"11.3.0"
]
]
},
{
"name": "prompt-toolkit",
"specs": [
[
"==",
"3.0.51"
]
]
},
{
"name": "propcache",
"specs": [
[
"==",
"0.3.2"
]
]
},
{
"name": "proto-plus",
"specs": [
[
"==",
"1.26.1"
]
]
},
{
"name": "protobuf",
"specs": [
[
"==",
"6.31.1"
]
]
},
{
"name": "ptyprocess",
"specs": [
[
"==",
"0.7.0"
]
]
},
{
"name": "pure-eval",
"specs": [
[
"==",
"0.2.3"
]
]
},
{
"name": "pyarrow",
"specs": [
[
"==",
"21.0.0"
]
]
},
{
"name": "pyasn1",
"specs": [
[
"==",
"0.6.1"
]
]
},
{
"name": "pyasn1-modules",
"specs": [
[
"==",
"0.4.2"
]
]
},
{
"name": "pydata-google-auth",
"specs": [
[
"==",
"1.9.1"
]
]
},
{
"name": "pygments",
"specs": [
[
"==",
"2.19.2"
]
]
},
{
"name": "pyogrio",
"specs": [
[
"==",
"0.11.1"
]
]
},
{
"name": "pyparsing",
"specs": [
[
"==",
"3.2.3"
]
]
},
{
"name": "pyproj",
"specs": [
[
"==",
"3.6.1"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.9.0.post0"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2025.2"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.32.4"
]
]
},
{
"name": "requests-oauthlib",
"specs": [
[
"==",
"2.0.0"
]
]
},
{
"name": "rich",
"specs": [
[
"==",
"13.9.4"
]
]
},
{
"name": "rsa",
"specs": [
[
"==",
"4.9.1"
]
]
},
{
"name": "shapely",
"specs": [
[
"==",
"2.0.7"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.17.0"
]
]
},
{
"name": "sqlglot",
"specs": [
[
"==",
"27.6.0"
]
]
},
{
"name": "stack-data",
"specs": [
[
"==",
"0.6.3"
]
]
},
{
"name": "tabulate",
"specs": [
[
"==",
"0.9.0"
]
]
},
{
"name": "toolz",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "traitlets",
"specs": [
[
"==",
"5.14.3"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"==",
"4.14.1"
]
]
},
{
"name": "tzdata",
"specs": [
[
"==",
"2025.2"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"2.5.0"
]
]
},
{
"name": "wcwidth",
"specs": [
[
"==",
"0.2.13"
]
]
},
{
"name": "widgetsnbextension",
"specs": [
[
"==",
"4.0.14"
]
]
},
{
"name": "yarl",
"specs": [
[
"==",
"1.20.1"
]
]
},
{
"name": "zipp",
"specs": [
[
"==",
"3.23.0"
]
]
}
],
"lcname": "sibr-module"
}
