# Fast Partitioned Learned Bloom Filter
This repository contains python implementation for our paper "[Fast Partitioned Learned Bloom Filter](https://arxiv.org/abs/2306.02846)" (NeurIPS 2023).
Three methods (PLBF, fast PLBF, and fast PLBF++) are implemented in two different frameworks: one was designed in the PLBF paper, and the other is our modified version.
## Abstract
A Bloom filter is a memory-efficient data structure for approximate membership queries used in numerous fields of computer science. Recently, learned Bloom filters that achieve better memory efficiency using machine learning models have attracted attention. One such filter, the partitioned learned Bloom filter (PLBF), achieves excellent memory efficiency. However, PLBF requires a $O(N^3 k)$ time complexity to construct the data structure, where $N$ and $k$ are the hyperparameters of PLBF. One can improve memory efficiency by increasing $N$, but the construction time becomes extremely long. Thus, we propose two methods that can reduce the construction time while maintaining the memory efficiency of PLBF. First, we propose fast PLBF, which can construct the same data structure as PLBF with a smaller time complexity $O(N^2 k)$. Second, we propose fast PLBF++, which can construct the data structure with even smaller time complexity $O(N k \log N + Nk^2)$. Fast PLBF++ does not necessarily construct the same data structure as PLBF. Still, it is almost as memory efficient as PLBF, and it is proved that fast PLBF++ has the same data structure as PLBF when the distribution satisfies a certain constraint. Our experimental results from real-world datasets show that (i) fast PLBF and fast PLBF++ can construct the data structure up to 233 and 761 times faster than PLBF, (ii) fast PLBF can achieve the same memory efficiency as PLBF, and (iii) fast PLBF++ can achieve almost the same memory efficiency as PLBF.
## Framework in the PLBF paper
This framework is designed to minimize the memory usage under the condition of expected false positive rate.
**Input arguments**
- `--data_path`: Csv file contains the items, scores, and labels
- `--N`: Number of *segments*
- `--k`: Number of *regions*
- `--F`: Target overall false positive rate
**Commands**
- run the PLBF: `python src/PLBFs/PLBF.py --data_path data/example.csv --N 50 --k 5 --F 0.01`
- run the FastPLBF: `python src/PLBFs/FastPLBF.py --data_path data/example.csv --N 50 --k 5 --F 0.01`
- run the FastPLBF++: `python src/PLBFs/FastPLBFpp.py --data_path data/example.csv --N 50 --k 5 --F 0.01`
## Framework in our paper
This framework is designed to minimize the expected false positive rate under the condition of memory usage.
**Input arguments**
- `--data_path`: Csv file contains the items, scores, and labels
- `--N`: Number of *segments*
- `--k`: Number of *regions*
- `--M`: Target memory usage for backup Bloom filters
**Commands**
- run the PLBF_M: `python src/PLBFs/PLBF_M.py --data_path data/example.csv --N 50 --k 5 --M 1000`
- run the FastPLBF_M: `python src/PLBFs/FastPLBF_M.py --data_path data/example.csv --N 50 --k 5 --M 1000`
- run the FastPLBF++_M: `python src/PLBFs/FastPLBFpp_M.py --data_path data/example.csv --N 50 --k 5 --M 1000`
Raw data
{
"_id": null,
"home_page": "https://github.com/atsukisato/FastPLBF",
"name": "fastplbftest",
"maintainer": "Atsuki Sato",
"docs_url": null,
"requires_python": "",
"maintainer_email": "sato.a.20010310@gmail.com",
"keywords": "bloom filter,learned index",
"author": "Atsuki Sato",
"author_email": "sato.a.20010310@gmail.com",
"download_url": "",
"platform": null,
"description": "# Fast Partitioned Learned Bloom Filter\n\nThis repository contains python implementation for our paper \"[Fast Partitioned Learned Bloom Filter](https://arxiv.org/abs/2306.02846)\" (NeurIPS 2023).\nThree methods (PLBF, fast PLBF, and fast PLBF++) are implemented in two different frameworks: one was designed in the PLBF paper, and the other is our modified version.\n\n## Abstract\n\nA Bloom filter is a memory-efficient data structure for approximate membership queries used in numerous fields of computer science. Recently, learned Bloom filters that achieve better memory efficiency using machine learning models have attracted attention. One such filter, the partitioned learned Bloom filter (PLBF), achieves excellent memory efficiency. However, PLBF requires a $O(N^3 k)$ time complexity to construct the data structure, where $N$ and $k$ are the hyperparameters of PLBF. One can improve memory efficiency by increasing $N$, but the construction time becomes extremely long. Thus, we propose two methods that can reduce the construction time while maintaining the memory efficiency of PLBF. First, we propose fast PLBF, which can construct the same data structure as PLBF with a smaller time complexity $O(N^2 k)$. Second, we propose fast PLBF++, which can construct the data structure with even smaller time complexity $O(N k \\log N + Nk^2)$. Fast PLBF++ does not necessarily construct the same data structure as PLBF. Still, it is almost as memory efficient as PLBF, and it is proved that fast PLBF++ has the same data structure as PLBF when the distribution satisfies a certain constraint. Our experimental results from real-world datasets show that (i) fast PLBF and fast PLBF++ can construct the data structure up to 233 and 761 times faster than PLBF, (ii) fast PLBF can achieve the same memory efficiency as PLBF, and (iii) fast PLBF++ can achieve almost the same memory efficiency as PLBF.\n\n## Framework in the PLBF paper\n\nThis framework is designed to minimize the memory usage under the condition of expected false positive rate.\n\n**Input arguments**\n- `--data_path`: Csv file contains the items, scores, and labels\n- `--N`: Number of *segments*\n- `--k`: Number of *regions*\n- `--F`: Target overall false positive rate\n\n**Commands**\n- run the PLBF: `python src/PLBFs/PLBF.py --data_path data/example.csv --N 50 --k 5 --F 0.01`\n- run the FastPLBF: `python src/PLBFs/FastPLBF.py --data_path data/example.csv --N 50 --k 5 --F 0.01`\n- run the FastPLBF++: `python src/PLBFs/FastPLBFpp.py --data_path data/example.csv --N 50 --k 5 --F 0.01`\n\n## Framework in our paper\n\nThis framework is designed to minimize the expected false positive rate under the condition of memory usage.\n\n**Input arguments**\n- `--data_path`: Csv file contains the items, scores, and labels\n- `--N`: Number of *segments*\n- `--k`: Number of *regions*\n- `--M`: Target memory usage for backup Bloom filters\n\n**Commands**\n- run the PLBF_M: `python src/PLBFs/PLBF_M.py --data_path data/example.csv --N 50 --k 5 --M 1000`\n- run the FastPLBF_M: `python src/PLBFs/FastPLBF_M.py --data_path data/example.csv --N 50 --k 5 --M 1000`\n- run the FastPLBF++_M: `python src/PLBFs/FastPLBFpp_M.py --data_path data/example.csv --N 50 --k 5 --M 1000`\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Fast PLBF: Partitioned Learned Bloom Filter with fast construction.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/atsukisato/FastPLBF"
},
"split_keywords": [
"bloom filter",
"learned index"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4159aba80197da13b5e76cd13e93401f8f2240d4dd51c4e4a8b53fa20ede963a",
"md5": "4317d48d606aa24bb6e5075a4f382fa8",
"sha256": "56312b60194138cbd0266e622c6749c20083c5b13c0715a66ceadc979f63ab78"
},
"downloads": -1,
"filename": "fastplbftest-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4317d48d606aa24bb6e5075a4f382fa8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17715,
"upload_time": "2023-11-05T19:38:34",
"upload_time_iso_8601": "2023-11-05T19:38:34.822207Z",
"url": "https://files.pythonhosted.org/packages/41/59/aba80197da13b5e76cd13e93401f8f2240d4dd51c4e4a8b53fa20ede963a/fastplbftest-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-05 19:38:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "atsukisato",
"github_project": "FastPLBF",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "fastplbftest"
}
JSON
