Mintospy
====
Mintos web scraper implemented in Python. **(For research purposes only)**
**Warning:** Mintos prohibit commercial use of web scrapers in their Terms & Conditions,
so make sure you only utilise this project for personal use.
Mintos platform: https://www.mintos.com/en/
Features
----
- Get portfolio data (Active funds, late funds, etc).
- Get Notes and Claims that you've invested in, with support for pagination and Mintos' filters.
- Get Notes that are listed on the Mintos primary and secondary marketplace, with support for pagination and Mintos' filters.
- Get details provided by Mintos on Notes and Claims.
- Returns Notes/Claims data in an easy-to-use Pandas DataFrame.
- Allows option to get raw data in JSON, instead of the parsed data in the Pandas DataFrame.
Installation
----
.. code-block:: bash
$ python -m pip install mintospy
This scraper uses audio transcription to automatically solve ReCAPTCHA challenges,
so you need to have FFmpeg installed on your machine and in your PATH (If using windows)
or just installed on your machine (For Linux or MacOS).
- Guide for installing FFmpeg on Windows: https://phoenixnap.com/kb/ffmpeg-windows
- Guide for installing FFmpeg on Linux: https://www.tecmint.com/install-ffmpeg-in-linux/
- Guide for installing FFmpeg on MacOS: https://phoenixnap.com/kb/ffmpeg-mac
Link to project used to solve ReCAPTCHA challenges: https://github.com/thicccat688/selenium-recaptcha-solver
Usage
----
.. code-block:: python
from mintospy import API
mintos_api = API(
email='YOUR EMAIL HERE',
password='YOUR PASSWORD HERE',
tfa_secret='YOUR BASE32 TFA SECRET HERE', # This is only required if you have TFA enabled on your account (It should look something like this: PJJORHUYVGZVPQSF)
)
# Gets data for EUR (€) portfolio
print(mintos_api.get_portfolio_data(currency='EUR'))
# Gets 200 of the EUR (€) denominated notes from your "Current investments" section
print(mintos_api.get_investments(currency='EUR', quantity=200))
# Gets 300 of the EUR (€) denominated notes from your "Finished investments" section
print(mintos_api.get_investments(currency='EUR', quantity=300, claims=True))
# Gets 400 KZT (₸) denominated notes available in the primary marketplace for investment
print(mintos_api.get_loans(currency='KZT', quantity=400))
# Gets 400 KZT (₸) denominated notes available in the secondary marketplace for investment
print(mintos_api.get_loans(currency='KZT', quantity=400, secondary_market=True))
How it works
----
You already have everything you need above, but if you're curious about how I've made this work, I've put the automation process below!
|
**Authentication process:**
- This part uses a headless browser to fill out the login form, resolve all the ReCAPTCHA challenges that appear, and, if applicable, generate the current TOTP token using the base32 secret provided by the user and fill out the TFA section.
- After a successful login, the driver pickles and saves the cookies, then load those cookies to avoid logging in again the next time the scraper is used (If the cookies haven't expired).
- To solve the ReCAPTCHA challenges, I'm using a package I made which works with Selenium. It solves the ReCAPTCHA challenges by using Google's speech recognition API to transcribe the audio and fill out the form as needed.
- If you're interested, here is the repository's URL: https://github.com/thicccat688/selenium-recaptcha-solver
Demonstration of the authentication process:
.. raw:: html
<a href="https://gyazo.com/920db679a5af97ba8726ea7124a81cf8"><img src="https://i.gyazo.com/920db679a5af97ba8726ea7124a81cf8.gif" alt="Image from Gyazo" width="1280"/></a>
|
**API request process and getting around Cloudflare:**
- This part took a great deal of work to figure out and implement. On top of using ReCAPTCHA, Mintos uses Cloudflare to detect bots and secure their API.
- Cloudflare makes it so requests made by Python to Mintos' API endpoints, even given the correct headers, are rejected with a 403 HTTP response.
- Cloudflare runs a series of checks to guarantee the requestor is a legitimate browser, making it virtually impossible to make requests without a web driver to emulate a browser's properties.
- I went around this by constructing the request payloads in Python and using said payloads to execute the desired API calls using the Fetch API in the web driver's console.
- Due to performance constraints, I also made a function that can do this concurrently, which I use for mass retrieval of investments or loans.
- This workaround means there's no need to scrape Mintos' UI to get the data we need so that we can perform data extraction more efficiently and in a less error-prone way.
|
**Final message:**
If you've reached this far, thank you! If you have any criticism or ideas about what can be improved, please get in touch with me through discord (ThiccCat#3210). Thanks again, and I hope this package can be of use to you!
Raw data
{
"_id": null,
"home_page": "https://github.com/thicccat688/mintospy",
"name": "mintospy",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "python,api,api-wrapper,mintos",
"author": "Tom\u00e1s Perestrelo",
"author_email": "tomasperestrelo21@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/76/73/dccfb73423e9661498157615747a193dbc46f2206b00b322d6610f41fbed/mintospy-0.6.3.tar.gz",
"platform": null,
"description": "Mintospy\n====\n\nMintos web scraper implemented in Python. **(For research purposes only)**\n\n**Warning:** Mintos prohibit commercial use of web scrapers in their Terms & Conditions,\nso make sure you only utilise this project for personal use.\n\nMintos platform: https://www.mintos.com/en/\n\nFeatures\n----\n\n- Get portfolio data (Active funds, late funds, etc).\n- Get Notes and Claims that you've invested in, with support for pagination and Mintos' filters.\n- Get Notes that are listed on the Mintos primary and secondary marketplace, with support for pagination and Mintos' filters.\n- Get details provided by Mintos on Notes and Claims.\n- Returns Notes/Claims data in an easy-to-use Pandas DataFrame.\n- Allows option to get raw data in JSON, instead of the parsed data in the Pandas DataFrame.\n\nInstallation\n----\n.. code-block:: bash\n\n $ python -m pip install mintospy\n\nThis scraper uses audio transcription to automatically solve ReCAPTCHA challenges,\nso you need to have FFmpeg installed on your machine and in your PATH (If using windows) \nor just installed on your machine (For Linux or MacOS).\n\n- Guide for installing FFmpeg on Windows: https://phoenixnap.com/kb/ffmpeg-windows\n- Guide for installing FFmpeg on Linux: https://www.tecmint.com/install-ffmpeg-in-linux/\n- Guide for installing FFmpeg on MacOS: https://phoenixnap.com/kb/ffmpeg-mac\n\nLink to project used to solve ReCAPTCHA challenges: https://github.com/thicccat688/selenium-recaptcha-solver\n\nUsage\n----\n.. code-block:: python\n\n from mintospy import API\n\n mintos_api = API(\n email='YOUR EMAIL HERE',\n password='YOUR PASSWORD HERE',\n tfa_secret='YOUR BASE32 TFA SECRET HERE', # This is only required if you have TFA enabled on your account (It should look something like this: PJJORHUYVGZVPQSF)\n )\n \n # Gets data for EUR (\u20ac) portfolio\n print(mintos_api.get_portfolio_data(currency='EUR'))\n\n # Gets 200 of the EUR (\u20ac) denominated notes from your \"Current investments\" section\n print(mintos_api.get_investments(currency='EUR', quantity=200))\n\n # Gets 300 of the EUR (\u20ac) denominated notes from your \"Finished investments\" section\n print(mintos_api.get_investments(currency='EUR', quantity=300, claims=True))\n\n # Gets 400 KZT (\u20b8) denominated notes available in the primary marketplace for investment\n print(mintos_api.get_loans(currency='KZT', quantity=400))\n \n # Gets 400 KZT (\u20b8) denominated notes available in the secondary marketplace for investment\n print(mintos_api.get_loans(currency='KZT', quantity=400, secondary_market=True))\n\nHow it works\n----\nYou already have everything you need above, but if you're curious about how I've made this work, I've put the automation process below!\n\n| \n\n**Authentication process:**\n\n- This part uses a headless browser to fill out the login form, resolve all the ReCAPTCHA challenges that appear, and, if applicable, generate the current TOTP token using the base32 secret provided by the user and fill out the TFA section.\n- After a successful login, the driver pickles and saves the cookies, then load those cookies to avoid logging in again the next time the scraper is used (If the cookies haven't expired).\n- To solve the ReCAPTCHA challenges, I'm using a package I made which works with Selenium. It solves the ReCAPTCHA challenges by using Google's speech recognition API to transcribe the audio and fill out the form as needed.\n- If you're interested, here is the repository's URL: https://github.com/thicccat688/selenium-recaptcha-solver\n\nDemonstration of the authentication process:\n\n.. raw:: html\n\n <a href=\"https://gyazo.com/920db679a5af97ba8726ea7124a81cf8\"><img src=\"https://i.gyazo.com/920db679a5af97ba8726ea7124a81cf8.gif\" alt=\"Image from Gyazo\" width=\"1280\"/></a>\n\n|\n\n**API request process and getting around Cloudflare:**\n\n- This part took a great deal of work to figure out and implement. On top of using ReCAPTCHA, Mintos uses Cloudflare to detect bots and secure their API. \n- Cloudflare makes it so requests made by Python to Mintos' API endpoints, even given the correct headers, are rejected with a 403 HTTP response. \n- Cloudflare runs a series of checks to guarantee the requestor is a legitimate browser, making it virtually impossible to make requests without a web driver to emulate a browser's properties.\n- I went around this by constructing the request payloads in Python and using said payloads to execute the desired API calls using the Fetch API in the web driver's console. \n- Due to performance constraints, I also made a function that can do this concurrently, which I use for mass retrieval of investments or loans.\n- This workaround means there's no need to scrape Mintos' UI to get the data we need so that we can perform data extraction more efficiently and in a less error-prone way. \n\n|\n\n**Final message:**\n\nIf you've reached this far, thank you! If you have any criticism or ideas about what can be improved, please get in touch with me through discord (ThiccCat#3210). Thanks again, and I hope this package can be of use to you!\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "",
"version": "0.6.3",
"split_keywords": [
"python",
"api",
"api-wrapper",
"mintos"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1e6e89392df25af0a4a05bce227616ec6867609b18e1ac93f05ae528e4d730e0",
"md5": "09c180a4bff72c9c790e0b52a6d99baf",
"sha256": "f7dd9156cca2613ad9c31402ca7f7cd824144184f8a80a0ba8f16329fa0a83cf"
},
"downloads": -1,
"filename": "mintospy-0.6.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "09c180a4bff72c9c790e0b52a6d99baf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 16709,
"upload_time": "2023-02-07T16:20:45",
"upload_time_iso_8601": "2023-02-07T16:20:45.916801Z",
"url": "https://files.pythonhosted.org/packages/1e/6e/89392df25af0a4a05bce227616ec6867609b18e1ac93f05ae528e4d730e0/mintospy-0.6.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7673dccfb73423e9661498157615747a193dbc46f2206b00b322d6610f41fbed",
"md5": "5748c7e8d63395419a022d07ba2af0b8",
"sha256": "73b70c7204166bcb6a854c22b3ccb6e8c414105f8b76dd4b7629fb04182312a5"
},
"downloads": -1,
"filename": "mintospy-0.6.3.tar.gz",
"has_sig": false,
"md5_digest": "5748c7e8d63395419a022d07ba2af0b8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 18624,
"upload_time": "2023-02-07T16:20:48",
"upload_time_iso_8601": "2023-02-07T16:20:48.679461Z",
"url": "https://files.pythonhosted.org/packages/76/73/dccfb73423e9661498157615747a193dbc46f2206b00b322d6610f41fbed/mintospy-0.6.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-07 16:20:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "thicccat688",
"github_project": "mintospy",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "anyio",
"specs": [
[
"==",
"3.6.2"
]
]
},
{
"name": "async-generator",
"specs": [
[
"==",
"1.10"
]
]
},
{
"name": "attrs",
"specs": [
[
"==",
"22.1.0"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
"==",
"4.11.1"
]
]
},
{
"name": "bleach",
"specs": [
[
"==",
"5.0.1"
]
]
},
{
"name": "build",
"specs": [
[
"==",
"0.9.0"
]
]
},
{
"name": "certifi",
"specs": [
[
"==",
"2022.12.7"
]
]
},
{
"name": "cffi",
"specs": [
[
"==",
"1.15.1"
]
]
},
{
"name": "charset-normalizer",
"specs": [
[
"==",
"2.1.1"
]
]
},
{
"name": "click",
"specs": [
[
"==",
"8.1.3"
]
]
},
{
"name": "colorama",
"specs": [
[
"==",
"0.4.6"
]
]
},
{
"name": "commonmark",
"specs": [
[
"==",
"0.9.1"
]
]
},
{
"name": "cryptography",
"specs": [
[
"==",
"39.0.0"
]
]
},
{
"name": "docutils",
"specs": [
[
"==",
"0.19"
]
]
},
{
"name": "exceptiongroup",
"specs": [
[
"==",
"1.0.4"
]
]
},
{
"name": "filelock",
"specs": [
[
"==",
"3.8.0"
]
]
},
{
"name": "h11",
"specs": [
[
"==",
"0.14.0"
]
]
},
{
"name": "httpcore",
"specs": [
[
"==",
"0.16.2"
]
]
},
{
"name": "httpx",
"specs": [
[
"==",
"0.23.1"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.4"
]
]
},
{
"name": "importlib-metadata",
"specs": [
[
"==",
"5.2.0"
]
]
},
{
"name": "iniconfig",
"specs": [
[
"==",
"1.1.1"
]
]
},
{
"name": "jaraco.classes",
"specs": [
[
"==",
"3.2.3"
]
]
},
{
"name": "jeepney",
"specs": [
[
"==",
"0.8.0"
]
]
},
{
"name": "keyring",
"specs": [
[
"==",
"23.13.1"
]
]
},
{
"name": "more-itertools",
"specs": [
[
"==",
"9.0.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"1.23.5"
]
]
},
{
"name": "outcome",
"specs": [
[
"==",
"1.2.0"
]
]
},
{
"name": "packaging",
"specs": [
[
"==",
"22.0"
]
]
},
{
"name": "pandas",
"specs": [
[
"==",
"1.5.2"
]
]
},
{
"name": "pep517",
"specs": [
[
"==",
"0.13.0"
]
]
},
{
"name": "pip-tools",
"specs": [
[
"==",
"6.12.1"
]
]
},
{
"name": "pkginfo",
"specs": [
[
"==",
"1.9.2"
]
]
},
{
"name": "pluggy",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "pycparser",
"specs": [
[
"==",
"2.21"
]
]
},
{
"name": "pydub",
"specs": [
[
"==",
"0.25.1"
]
]
},
{
"name": "Pygments",
"specs": [
[
"==",
"2.13.0"
]
]
},
{
"name": "pyotp",
"specs": [
[
"==",
"2.8.0"
]
]
},
{
"name": "pyparsing",
"specs": [
[
"==",
"3.0.9"
]
]
},
{
"name": "PySocks",
"specs": [
[
"==",
"1.7.1"
]
]
},
{
"name": "pytest",
"specs": [
[
"==",
"7.2.1"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.8.2"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"==",
"0.19.2"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2022.6"
]
]
},
{
"name": "pywin32-ctypes",
"specs": [
[
"==",
"0.2.0"
]
]
},
{
"name": "readme-renderer",
"specs": [
[
"==",
"37.3"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.28.1"
]
]
},
{
"name": "requests-file",
"specs": [
[
"==",
"1.5.1"
]
]
},
{
"name": "requests-toolbelt",
"specs": [
[
"==",
"0.10.1"
]
]
},
{
"name": "rfc3986",
"specs": [
[
"==",
"1.5.0"
]
]
},
{
"name": "rich",
"specs": [
[
"==",
"12.6.0"
]
]
},
{
"name": "SecretStorage",
"specs": [
[
"==",
"3.3.3"
]
]
},
{
"name": "selenium",
"specs": [
[
"==",
"4.7.2"
]
]
},
{
"name": "selenium-recaptcha-solver",
"specs": [
[
"==",
"1.0.8"
]
]
},
{
"name": "selenium-stealth",
"specs": [
[
"==",
"1.0.6"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "sniffio",
"specs": [
[
"==",
"1.3.0"
]
]
},
{
"name": "sortedcontainers",
"specs": [
[
"==",
"2.4.0"
]
]
},
{
"name": "soupsieve",
"specs": [
[
"==",
"2.3.2.post1"
]
]
},
{
"name": "SpeechRecognition",
"specs": [
[
"==",
"3.8.1"
]
]
},
{
"name": "tldextract",
"specs": [
[
"==",
"3.4.0"
]
]
},
{
"name": "tomli",
"specs": [
[
"==",
"2.0.1"
]
]
},
{
"name": "tqdm",
"specs": [
[
"==",
"4.64.1"
]
]
},
{
"name": "trio",
"specs": [
[
"==",
"0.22.0"
]
]
},
{
"name": "trio-websocket",
"specs": [
[
"==",
"0.9.2"
]
]
},
{
"name": "twine",
"specs": [
[
"==",
"4.0.2"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"1.26.13"
]
]
},
{
"name": "webdriver-manager",
"specs": [
[
"==",
"3.8.5"
]
]
},
{
"name": "webencodings",
"specs": [
[
"==",
"0.5.1"
]
]
},
{
"name": "wsproto",
"specs": [
[
"==",
"1.2.0"
]
]
},
{
"name": "zipp",
"specs": [
[
"==",
"3.11.0"
]
]
},
{
"name": "cloudscraper",
"specs": [
[
"~=",
"1.2.68"
]
]
},
{
"name": "setuptools",
"specs": [
[
"~=",
"65.5.1"
]
]
}
],
"lcname": "mintospy"
}
