# Beancount Red's Importers
Simple importers and tools for [Beancount](https://beancount.github.io/), software for
[plain text](https://plaintextaccounting.org/), double entry bookkeeping. _More
importantly, a framework to allow you to easily write your own importers._
### Introduction
This is a reference implementation of the principles expressed in
**[The Five Minute Ledger Update](https://reds-rants.netlify.app/personal-finance/the-five-minute-ledger-update/).**
Importers can be ugly and painful to write, yet are important in automating the grunt
work out of maintaining personal finance software. The philosophy is to make writing
sustainable, dependable importers easy. To achieve this, the design separates importers
in to three parts:
1. file format reader (reusable)
2. transaction builder (reusable)
3. institution-specific declarations and code (minimal, institution specific) <- _The
only one you have to write_
This design helps move most of the heavy-lifting common code into (1) and (2) above.
Writing new importers is made easier since one only has to write code to address the
institution-specific formatting and quirks for each bank/brokerage. See working examples
of an [ofx based](https://github.com/redstreet/beancount_reds_importers/blob/main/beancount_reds_importers/importers/citi/__init__.py)
and [csv](https://github.com/redstreet/beancount_reds_importers/blob/main/beancount_reds_importers/importers/schwab/schwab_csv_brokerage.py)
based importers.
### Importers
File format readers included are:
- `.ofx`
- `.csv` (single and multitable support)
- `.xlsx` (single and multitable support) (`pip3 install xlrd` if you plan to use this)
Transaction builders included are:
- Banking (for banks and credit cards, which benefit from a postings predictor like
[smart_importer](https://github.com/beancount/smart_importer)).
- Investments/brokerages (to handle the very many distinct cases of investment related
transactions).
- Paychecks (to handle paychecks, which typically contain very many mostly
pre-determined postings in a single entry).
[Input in `.ofx` format (over `.csv`) is preferred](https://reds-rants.netlify.app/personal-finance/a-word-about-input-formats-use-ofx-when-you-can/),
when provided by the institution, as it minimizes data and coding errors, eliminates
format breaking changes in .csv files, and typically includes balances that are used to
generate balance assertions, and commodity prices.
See [here](https://github.com/redstreet/beancount_reds_importers/tree/main/beancount_reds_importers)
for a list of institutions built-in. More investment, credit card, and banking
institutions will be added in the future. Contributions welcome.
### Tools and Utilities
These commands are installed as a part of the pip installation:
- `ofx-summarize`: Quick and dirty way to summarize a .ofx file, and peek inside it
- `bean-download`: [Download account statements automatically](https://reds-rants.netlify.app/personal-finance/direct-downloads/)
(for supporting institutions), from your configuration of accounts. Multi-threaded.
- `bean-download needs-update` is a configurable utility that shows you the last time
each account was updated, based on the latest balance assertion in your journal. See
[this article](https://reds-rants.netlify.app/personal-finance/how-up-to-date-are-my-accounts/)
for more.
The commands include shell auto-completion (tab-to-complete) via
[click](https://click.palletsprojects.com/en/8.1.x/shell-completion/). `bean-download`, in
particular, can complete the account or account groups you want to download, which can
be handy. To enable it in zsh, do
([see here for other shells](https://click.palletsprojects.com/en/8.1.x/shell-completion/)):
```
mkdir -p ~/.zcomplete
_OFX_SUMMARIZE_COMPLETE=zsh_source ofx-summarize > ~/.zcomplete/ofx-summarize-complete.zsh
_BEAN_DOWNLOAD_COMPLETE=zsh_source bean-download > ~/.zcomplete/bean-download-complete.zsh
# Place this in your shell's rc file (.zshrc or .bashrc or .fishrc):
for f in ~/.zcomplete/*; do source $f; done
```
## Features
- supports [Beancount](https://github.com/beancount/beancount) output via `bean-extract`
- should be easy to extend to ledger/hledger, etc. (contributions welcome)
- automatically generates [balance assertions](https://reds-rants.netlify.app/personal-finance/automating-balance-assertions/)
- support for:
- investment accounts (brokerages including retirement accounts)
- handles sweep funds, money market funds, and all standard brokerage transactions
- banking and credit card
- paychecks
- file format independent (ofx, csv, xlsx supported out of the box; single and
multitable for csv and xlsx; write your own reusable handler if needed)
- supports commodity-leaf accounts
- see [The Five Minute Ledger Update](https://reds-rants.netlify.app/personal-finance/the-five-minute-ledger-update/)
for automating downloads via `ofxclient`, connecting to `smart_importer` to
auto-classify transactions, and more
## Installation
```
pip3 install beancount-reds-importers
```
Or to install the bleeding edge version from git:
```
pip3 install git+https://github.com/redstreet/beancount_reds_importers
```
## Running
### Running the included examples:
1. `cd <your pip installed dir>/example #eg: cd ~/.local/lib/python3.8/site-packages/beancount_reds_importers/example`
2. `./import.sh OfxDownload.qfx` # Imports investments
3. `./import.sh transactions.qfx` # Import bank transactions; uses smart_importer to classify transactions
### Creating and running your own config:
1. Create your own my.import. An example my.import is provided. At the least, include your account numbers
2. Include fund information. Copy the included `fund_info.py` to start with.
3. You can now run `bean-identify`, `bean-extract`, etc. See the included script: Run `./import.sh <your_input_ofx>`
4. If identifier/cusip/isin info is missing, the importer will let you know. Add it to your
`fund_info.py` See
[this article](https://reds-rants.netlify.app/personal-finance/tickers-and-identifiers/)
for automating and managing identifier info
### Configuring balance assertion dates
Choices for the date of the generated balance assertion can be specified as a key in
the importer config, `balance_assertion_date_type`, which can be set to:
- `smart`: smart date (default; see below)
- `ofx_date`: date specified in ofx file
- `last_transaction`: max transaction date
- `today`: today's date
If you want something else, simply override this method in individual importer
`smart` dates: Banks and credit cards typically have pending transactions that are not
included in downloads. When we download the next statement, new transactions may appear
prior to the balance assertion date that we generate for this statement, which renders
this balance assertion invalid. This problem manifests occasionally as an existing
balance statement breaking when a new statement is downloaded and is an annoyance as it
needs manual fixing.
To minimize this, we set the balance assertion date to either two days (fudge factor to
account for pending transactions) before the statement's end date or the last
transaction's date, whichever is later. To choose a different fudge factor, simply set
`balance_assertion_date_fudge` in your config.
### Note
Depending on the institution, the `payee` and `narration` fields in generated
transactions may appear to be switched. This is described by
[libtransactionbuilder/banking.py](https://github.com/redstreet/beancount_reds_importers/blob/main/beancount_reds_importers/libtransactionbuilder/banking.py),
and the fields can be swapped in a `custom_init`.
## Testing
First:
```
pip3 install xlrd
```
Some importers are tested with
[regression_pytest.py](https://github.com/beancount/beancount/blob/v2/beancount/ingest/regression_pytest.py).
Run `pytest --generate` then `pytest`.
Anonymized data to increase regression test coverage is most welcome. Please submit a
PR if you can. See [here](beancount_reds_importers/importers/schwab/tests/schwab_csv_checking)
for an example to follow.
More broadly I run tests across hundreds of actual ofx and csv files, against reference
outputs that I know to be correct from my personal file. However, I'm unable to share
them since these are personal. Testing against real world files is best, so I recommend
you do this with your own input files.
## Contact
Feel free to post questions/concerns in the [Beancount groups](https://groups.google.com/forum/#!forum/beancount)
or on [The Five Minute Ledger Update](https://reds-rants.netlify.app/personal-finance/the-five-minute-ledger-update/)
site. For bugs, open an issue here on Github.
## Contributions
Features, fixes, and improvements welcome. New importers for institutions with test
input files appreciated. Sharing importers helps the community! Remember:
- Feel free to send send pull requests. Please include unit tests
- For larger changes or changes that might need discussion, please reach out and discuss
first to save time (open an issue)
- Please squash your commits (reasonably)
- Use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) for commit messages
Thank you for contributing!
Raw data
{
"_id": null,
"home_page": "https://github.com/redstreet/beancount_reds_importers",
"name": "beancount-reds-importers",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "importer ingestor beancount accounting",
"author": "Red Street",
"author_email": "redstreet@users.noreply.github.com",
"download_url": "https://files.pythonhosted.org/packages/eb/14/8e1e4ab820ef58d0e019546a70f37df75e94389e6d4007a2ce63c0d25c2e/beancount_reds_importers-0.9.0.tar.gz",
"platform": null,
"description": "# Beancount Red's Importers\n\nSimple importers and tools for [Beancount](https://beancount.github.io/), software for\n[plain text](https://plaintextaccounting.org/), double entry bookkeeping. _More\nimportantly, a framework to allow you to easily write your own importers._\n\n### Introduction\n\nThis is a reference implementation of the principles expressed in\n**[The Five Minute Ledger Update](https://reds-rants.netlify.app/personal-finance/the-five-minute-ledger-update/).**\n\nImporters can be ugly and painful to write, yet are important in automating the grunt\nwork out of maintaining personal finance software. The philosophy is to make writing\nsustainable, dependable importers easy. To achieve this, the design separates importers\nin to three parts:\n\n1. file format reader (reusable)\n2. transaction builder (reusable)\n3. institution-specific declarations and code (minimal, institution specific) <- _The\n only one you have to write_\n\nThis design helps move most of the heavy-lifting common code into (1) and (2) above.\nWriting new importers is made easier since one only has to write code to address the\ninstitution-specific formatting and quirks for each bank/brokerage. See working examples\nof an [ofx based](https://github.com/redstreet/beancount_reds_importers/blob/main/beancount_reds_importers/importers/citi/__init__.py)\nand [csv](https://github.com/redstreet/beancount_reds_importers/blob/main/beancount_reds_importers/importers/schwab/schwab_csv_brokerage.py)\nbased importers.\n\n### Importers\n\nFile format readers included are:\n- `.ofx`\n- `.csv` (single and multitable support)\n- `.xlsx` (single and multitable support) (`pip3 install xlrd` if you plan to use this)\n\nTransaction builders included are:\n- Banking (for banks and credit cards, which benefit from a postings predictor like\n [smart_importer](https://github.com/beancount/smart_importer)).\n- Investments/brokerages (to handle the very many distinct cases of investment related\n transactions).\n- Paychecks (to handle paychecks, which typically contain very many mostly\n pre-determined postings in a single entry).\n\n[Input in `.ofx` format (over `.csv`) is preferred](https://reds-rants.netlify.app/personal-finance/a-word-about-input-formats-use-ofx-when-you-can/),\nwhen provided by the institution, as it minimizes data and coding errors, eliminates\nformat breaking changes in .csv files, and typically includes balances that are used to\ngenerate balance assertions, and commodity prices.\n\nSee [here](https://github.com/redstreet/beancount_reds_importers/tree/main/beancount_reds_importers)\nfor a list of institutions built-in. More investment, credit card, and banking\ninstitutions will be added in the future. Contributions welcome.\n\n### Tools and Utilities\nThese commands are installed as a part of the pip installation:\n\n- `ofx-summarize`: Quick and dirty way to summarize a .ofx file, and peek inside it\n- `bean-download`: [Download account statements automatically](https://reds-rants.netlify.app/personal-finance/direct-downloads/)\n (for supporting institutions), from your configuration of accounts. Multi-threaded.\n - `bean-download needs-update` is a configurable utility that shows you the last time\n each account was updated, based on the latest balance assertion in your journal. See\n [this article](https://reds-rants.netlify.app/personal-finance/how-up-to-date-are-my-accounts/)\n for more.\n\nThe commands include shell auto-completion (tab-to-complete) via\n[click](https://click.palletsprojects.com/en/8.1.x/shell-completion/). `bean-download`, in\nparticular, can complete the account or account groups you want to download, which can\nbe handy. To enable it in zsh, do\n([see here for other shells](https://click.palletsprojects.com/en/8.1.x/shell-completion/)):\n\n```\nmkdir -p ~/.zcomplete\n_OFX_SUMMARIZE_COMPLETE=zsh_source ofx-summarize > ~/.zcomplete/ofx-summarize-complete.zsh\n_BEAN_DOWNLOAD_COMPLETE=zsh_source bean-download > ~/.zcomplete/bean-download-complete.zsh\n\n# Place this in your shell's rc file (.zshrc or .bashrc or .fishrc):\nfor f in ~/.zcomplete/*; do source $f; done\n```\n\n## Features\n- supports [Beancount](https://github.com/beancount/beancount) output via `bean-extract`\n - should be easy to extend to ledger/hledger, etc. (contributions welcome)\n- automatically generates [balance assertions](https://reds-rants.netlify.app/personal-finance/automating-balance-assertions/)\n- support for:\n - investment accounts (brokerages including retirement accounts)\n - handles sweep funds, money market funds, and all standard brokerage transactions\n - banking and credit card\n - paychecks\n- file format independent (ofx, csv, xlsx supported out of the box; single and\n multitable for csv and xlsx; write your own reusable handler if needed)\n- supports commodity-leaf accounts\n- see [The Five Minute Ledger Update](https://reds-rants.netlify.app/personal-finance/the-five-minute-ledger-update/)\n for automating downloads via `ofxclient`, connecting to `smart_importer` to\n auto-classify transactions, and more\n\n\n## Installation\n```\npip3 install beancount-reds-importers\n```\n\nOr to install the bleeding edge version from git:\n```\npip3 install git+https://github.com/redstreet/beancount_reds_importers\n```\n\n\n## Running\n\n### Running the included examples:\n1. `cd <your pip installed dir>/example #eg: cd ~/.local/lib/python3.8/site-packages/beancount_reds_importers/example`\n2. `./import.sh OfxDownload.qfx` # Imports investments\n3. `./import.sh transactions.qfx` # Import bank transactions; uses smart_importer to classify transactions\n\n\n### Creating and running your own config:\n1. Create your own my.import. An example my.import is provided. At the least, include your account numbers\n2. Include fund information. Copy the included `fund_info.py` to start with.\n3. You can now run `bean-identify`, `bean-extract`, etc. See the included script: Run `./import.sh <your_input_ofx>`\n4. If identifier/cusip/isin info is missing, the importer will let you know. Add it to your\n `fund_info.py` See\n [this article](https://reds-rants.netlify.app/personal-finance/tickers-and-identifiers/)\n for automating and managing identifier info\n\n### Configuring balance assertion dates\nChoices for the date of the generated balance assertion can be specified as a key in\nthe importer config, `balance_assertion_date_type`, which can be set to:\n- `smart`: smart date (default; see below)\n- `ofx_date`: date specified in ofx file\n- `last_transaction`: max transaction date\n- `today`: today's date\n\nIf you want something else, simply override this method in individual importer\n\n`smart` dates: Banks and credit cards typically have pending transactions that are not\nincluded in downloads. When we download the next statement, new transactions may appear\nprior to the balance assertion date that we generate for this statement, which renders\nthis balance assertion invalid. This problem manifests occasionally as an existing\nbalance statement breaking when a new statement is downloaded and is an annoyance as it\nneeds manual fixing.\n\nTo minimize this, we set the balance assertion date to either two days (fudge factor to\naccount for pending transactions) before the statement's end date or the last\ntransaction's date, whichever is later. To choose a different fudge factor, simply set\n`balance_assertion_date_fudge` in your config.\n\n\n### Note\n\nDepending on the institution, the `payee` and `narration` fields in generated\ntransactions may appear to be switched. This is described by\n[libtransactionbuilder/banking.py](https://github.com/redstreet/beancount_reds_importers/blob/main/beancount_reds_importers/libtransactionbuilder/banking.py),\nand the fields can be swapped in a `custom_init`.\n\n## Testing\nFirst:\n```\npip3 install xlrd\n```\n\nSome importers are tested with\n[regression_pytest.py](https://github.com/beancount/beancount/blob/v2/beancount/ingest/regression_pytest.py).\nRun `pytest --generate` then `pytest`.\n\nAnonymized data to increase regression test coverage is most welcome. Please submit a\nPR if you can. See [here](beancount_reds_importers/importers/schwab/tests/schwab_csv_checking)\nfor an example to follow.\n\n\nMore broadly I run tests across hundreds of actual ofx and csv files, against reference\noutputs that I know to be correct from my personal file. However, I'm unable to share\nthem since these are personal. Testing against real world files is best, so I recommend\nyou do this with your own input files.\n\n## Contact\nFeel free to post questions/concerns in the [Beancount groups](https://groups.google.com/forum/#!forum/beancount)\nor on [The Five Minute Ledger Update](https://reds-rants.netlify.app/personal-finance/the-five-minute-ledger-update/)\nsite. For bugs, open an issue here on Github.\n\n## Contributions\n\nFeatures, fixes, and improvements welcome. New importers for institutions with test\ninput files appreciated. Sharing importers helps the community! Remember:\n- Feel free to send send pull requests. Please include unit tests\n- For larger changes or changes that might need discussion, please reach out and discuss\n first to save time (open an issue) \n- Please squash your commits (reasonably)\n- Use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) for commit messages\n\nThank you for contributing!\n",
"bugtrack_url": null,
"license": "GPL-3.0",
"summary": "Importers for various institutions for Beancount",
"version": "0.9.0",
"project_urls": {
"Homepage": "https://github.com/redstreet/beancount_reds_importers"
},
"split_keywords": [
"importer",
"ingestor",
"beancount",
"accounting"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "53cec119d4ec7f62779e3901c3604f083d8b192b799fc9b4134120bfeba1e09c",
"md5": "17bffeae96b8635dbd58602b97d5084d",
"sha256": "84ad1549e8aa2930a66193b5f869bab9b1195361c15f65d816ce257f76767c59"
},
"downloads": -1,
"filename": "beancount_reds_importers-0.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "17bffeae96b8635dbd58602b97d5084d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 183170,
"upload_time": "2024-05-14T08:22:27",
"upload_time_iso_8601": "2024-05-14T08:22:27.693778Z",
"url": "https://files.pythonhosted.org/packages/53/ce/c119d4ec7f62779e3901c3604f083d8b192b799fc9b4134120bfeba1e09c/beancount_reds_importers-0.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "eb148e1e4ab820ef58d0e019546a70f37df75e94389e6d4007a2ce63c0d25c2e",
"md5": "d45383d59822b5e5b4705fad36f09656",
"sha256": "07a2d8db843308dad4aacc2e5a44e996b25b2a360ad850286524e538cfeae332"
},
"downloads": -1,
"filename": "beancount_reds_importers-0.9.0.tar.gz",
"has_sig": false,
"md5_digest": "d45383d59822b5e5b4705fad36f09656",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 151246,
"upload_time": "2024-05-14T08:22:29",
"upload_time_iso_8601": "2024-05-14T08:22:29.361152Z",
"url": "https://files.pythonhosted.org/packages/eb/14/8e1e4ab820ef58d0e019546a70f37df75e94389e6d4007a2ce63c0d25c2e/beancount_reds_importers-0.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-14 08:22:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "redstreet",
"github_project": "beancount_reds_importers",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "beancount",
"specs": [
[
">=",
"2.3.5"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
">=",
"4.12.3"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"8.1.7"
]
]
},
{
"name": "click-aliases",
"specs": [
[
">=",
"1.0.4"
]
]
},
{
"name": "dateparser",
"specs": [
[
">=",
"1.2.0"
]
]
},
{
"name": "importlib-metadata",
"specs": [
[
">=",
"6.8.0"
]
]
},
{
"name": "ofxparse",
"specs": [
[
">=",
"0.21"
]
]
},
{
"name": "openpyxl",
"specs": [
[
">=",
"3.1.2"
]
]
},
{
"name": "packaging",
"specs": [
[
">=",
"23.1"
]
]
},
{
"name": "pdfplumber",
"specs": [
[
">=",
"0.11.0"
]
]
},
{
"name": "petl",
"specs": [
[
">=",
"1.7.15"
]
]
},
{
"name": "setuptools",
"specs": [
[
">=",
"69.0.2"
]
]
},
{
"name": "setuptools-scm",
"specs": [
[
">=",
"8.0.4"
]
]
},
{
"name": "tabulate",
"specs": [
[
">=",
"0.9.0"
]
]
},
{
"name": "tomli",
"specs": [
[
">=",
"2.0.1"
]
]
},
{
"name": "tqdm",
"specs": [
[
">=",
"4.66.2"
]
]
},
{
"name": "typing_extensions",
"specs": [
[
">=",
"4.7.1"
]
]
},
{
"name": "xlrd",
"specs": [
[
">=",
"2.0.1"
]
]
}
],
"lcname": "beancount-reds-importers"
}
JSON
