# the Caching, Highly-Available Postfix Policy Service
| **requires Python 3.8.10+**
| **makes use of Redis (or Sentinel) and a relational database (MariaDB)**
## Introduction
There is a need for a highly-available, high-performance, concurrent,
clusterable solution for handling various aspects of email policy.
Postfix farms out the job of policy decisions to a delegate over a
socket, so we can provide a framework for receiving that data, making
a decision about it, and then sending a response back to Postfix.
There are some projects which have provided smaller-scale solutions to
this issue. We handle rather a large volume of email, so we need
something more performant than a script which makes a database access
on every email. In order to simplify the language, the term "customer"
is used herein to mean a person who logs into an SMTP server in order
to send email.
My decision was to use Redis, since with Redis Sentinel it should be
possible to achieve a degree of high-availability using Redis as a
common datastore between the various email servers in the farm(s)
which will run local instances of the policy server, which will itself
use Redis to cache data and keep track of email quotas, etc.
In the first iteration, we propose to provide functionality for:
- outbound quota tracking on a continuous, rolling, per-interval basis;
- outbound sender domain authorization
- inbound email greylisting;
- inbound SPF checking
The framework is meant to be extensible, so that any conceivable email
policy might be implemented in the future.
## Configuration
The library will create a config file for itself if it does not find one
at its default config path, `/etc/chapps/chapps.ini`, or the value of
the environment variable `CHAPPS_CONFIG` if it is set. Note that
default settings for all available submodules will be produced. At
the time of writing, each script runs its own type of policy handler,
so only the settings for the policies of that handler will be needed,
plus the general CHAPPS settings and the Redis settings.
It is possible to adjust the number of connections allowed to be
waiting on the CHAPPS server to answer them. The default used by
`asyncio` when none is provided is 100, so that is also the default
value used by CHAPPS. It may be adjusted in the config file under
`[CHAPPS]`, and is called `listener_backlog`. In some cases it may be
desirable to increase that number, so that exceptionally busy mailers
do not run into the problem of having their connection attempts
rejected.
Policies may each specify separate listening addresses and ports, so
that they may run simultaneously on the same server. For multi-policy
handlers, the first handler specified will be the one whose network
settings are used. It is recommended to configure those elements only
on that policy, or to keep them in sync on all policies which are
handled together.
Example Postfix configs are included in the `postfix` directory,
classified by which service they are for. Most access control policy
services will be implemented in a very similar way in `main.cf`,
probably in combination with other policies. The examples provided
are the same configs used for testing, and are necessarily stripped
down to focus just on that particular service.
An example **rsyslog** config is also included; modification is
encouraged. If you wish to keep the debug logs in their special
destination, ensure that you create a log-rotation profile for it.
## Installation Overview
The recommended Debian packages are:
- `mysqlclient`
- `redis`
- `python3-pip`
- `python3-venv`
It is highly recommended to install CHAPPS into a venv. You may need
to install the system package `python3-venv` in order for this to
work:
```
python3 -m venv chapps-venv
. chapps-venv/bin/activate
```
The package may be installed via PyPI, using the following command:
```
python3 -m pip install chapps
```
### DB Initialization
As of this writing, it should be possible to run `apply-migrations` or
`chapps-cli admin db-setup` once the venv is activated, and that should
apply all of the necessary Alembic migrations to bring the database up
to date from zero, based on the database access configuration in the
CHAPPS config file.
If the config file has not yet been populated with database
credentials, do that first, and ensure that the named database exists
(has been created) on the database server before attempting to install
the schema into it.
**Please Note:**
Databases created with earlier versions of CHAPPS (v<=0.4.12) need to
have their data dumped to SQL, and the database dropped and the schema
re-created via the `apply-migrations` mechanism in order for it to exactly
match Alembic's notion of how it is built. Once Alembic has built the
schema, the data may be read back into the database. We will be using
Alembic going forward so this should be a one-time annoyance.
For convenience, here is the `mysqldump` commandline recommended for
dumping the data:
.. code:
mysqldump --skip-add-drop-table -tc chapps > chapps-data-only.sql
These options tell `mysqldump` not to drop the tables, not to try to create
the tables, and to use "complete" INSERT statements, which ensures that the
target columns are listed in the INSERT statement, in case the native column
order changes between dump and restore.
#### Database Adapters
In order to obtain control data from the database, CHAPPS policy
objects use a database adapter object tailored for their specific
needs. This allows all database logic to be factored completely out
of the policy layer.
As of CHAPPS v0.5.5, the database adapter layer is now based on
SQLAlchemy throughout. However, it is still possible to switch to
using the MariaDB adapter instead, for the actual services. The API
is built on SQLAlchemy and requires it.
**To set CHAPPS to use MySQL/MariaDB** directly, instead of
SQLAlchemy, set the environment variable `CHAPPS_DB_MODULE` to
`mysql`. Doing so will cause the policy layer to use the adapter
module based on `mysqlclient`, which also works just fine with
MariaDB. If it is not set or is set to something else, SQLAlchemy
will be used. This will need to be specified in the service
description file to take effect, unless some other method is being
used to launch CHAPPS.
The ability to switch database adapter modules may be eliminated in a
future release. However, since the application has not been tested in
production using SQLAlchemy, it seems prudent to provide a mechanism
for switching between them.
### Starting and Auto-launching Services
With a venv, the SystemD service files are installed to a folder
called `chapps/install` inside the venv directory, and Postfix
example/testing configs are located in the `chapps/postfix` folder. Scripts
and package go to `bin` and `lib/.../chapps` as expected. Use of a
venv is recommended, as the SystemD service description files provided
are formatted during the install process to launch the services correctly
within their venv.
Without a venv, they go to various system locations,
with the ancillary `chapps` directory usually showing up at
`/usr/local/chapps`. YMMV. A venv will keep things organized.
A Python script called `chapps_database_init.py` is included
to create the database schema required by the library. It
does not create the database itself. Before running this script,
ensure that the CHAPPS configuration file contains the correct
credentials and other control data to be able to connect to the
database server, and also ensure that the database named in that
config has been created on the server. The script will connect to the
database and create the tables. It uses `IF EXISTS` and does not
contain any kind of data deletion, so it should be safe to use at any
time.
For more information about installing, see the
[INSTALLATION](INSTALLATION.md) file.
### Redis configuration
Redis is used to store the real-time state of every active user's
outbound quota, sender-domain authorization status cache, and also to
keep track of greylisting status for greylisted emails. An active
user is one who has sent email in the last _interval_, that interval
defaulting to a day, since most quotas are expressed as
messages-per-day.
If your Redis deployment is on a different server and/or if CHAPPS is
sharing a Redis instance with some other services it may be necessary
to adjust the Redis-related settings in the config file, to adjust the
address and/or port to connect to, or what database to use. By
default, CHAPPS tries to connect to Redis on localhost, using the
standard port assignment and db 0.
If Sentinel is in use, populate the Sentinel-oriented configuration
elements `sentinel_servers` and `sentinel_dataset`. The servers list
should be a space-separated list of each Sentinel server half-socket;
for example, "10.1.9.10:26379 10.1.9.12:26379". The dataset name is
the one you specified to Sentinel when setting up the Sentinel
cluster. Sentinel's default dataset name is `mymaster`. We, of
course, recommend `chapps`, or perhaps `chapps-outbound` at a site
with a large volume of email. Since SPF doesn't make much use of
Redis, the inbound load may be lighter than the outbound load,
depending on which things happen more at a particular site.
### Logging
At this time, CHAPPS uses **syslog**, and transmits logs on the
`local0` facility. CHAPPS sends a fair amount of debug information at
the DEBUG level. Right now, the application's facility and level may
not be adjusted via the config file; later this may be implemented.
For the time being, it seems sufficient to control logging via the
**rsyslog** configuration used to control log entries on `local0`.
The example provided (in the `install` directory) sends all logs to a
special log (the path needs to exist and belong to the syslog user,
whether that be `syslog` (as on Ubuntu) or `root` under Debian). As
long as the path exists and is writable by **rsyslog**, it will create
the log. The example also sends logs at INFO level or above to
`/var/log/mail.log`, which generally is the destination that
**rsyslog** uses for mail-related logs.
Of course, site operators are encouraged to alter this example config
to their needs. For those who wish to monkeypatch facility and level,
it is set in one place, at the top of `chapps.logging`.
## REST API Service
Starting with version 0.4.0, a REST API service is included, based on
FastAPI, and using SQLAlchemy with the MySQLclient backend. A service
template for the API is provided, as well as a socket unit and an
**nginx** example config, for using a UDS to proxy between the web and
the application. By default, Gunicorn is used to launch Uvicorn
workers which serve the API directly on port 8080. The precise
details may be adjusted in the `chapps-api-gunicorn.service` unit
file.
It is also possible that other approaches are preferred at other sites.
The extra files, provided in the `install` directory, are provided in
the hope that they may be useful.
The API is self-documenting. Once it is running, visit it at
`<server>:8080/redoc/` or `<server>:8080/docs/` to browse the
documentation.
The API service needs to have the same configuration as the other
CHAPPS services it is meant to manage. Since it instantiates policy
objects, it is best to simply provide the same copy of the config to
all related nodes. Please note that the API service is completely
separate from the policy service(s), and need not run on the same
server -- and in fact probably _should_ not run on the same server --
with the policy service.
## Outbound Services
Policy services can be divided into those which work on outbound mail,
and those which work on inbound mail. Some, possibly, might be
applied to either flow, but none such are part of this project yet.
Outbound items share some characteristics.
Outbound mail, for our purposes, is assumed to originate with an
authenticated user. That user may authenticate with Postfix using a
username/password or a client-side SSL cert, in which case the
username or subject name (of the cert) will be passed along by Postfix
to the policy service.
In order to allow sites to specify exactly what field of the Postfix
policy data they would like to use to identify users, the
configuration allows the user to specify the first field to check.
### Setting the user key
Postfix submits a fairly large packet of data on each policy
delegation request. One prominent element of this data is the MAIL
FROM address, which is labelled as `sender`. This is perhaps the
obvious element to use to count quotas, but some other fields are more
interesting.
Current versions of the software allow the config file to specify what
element of that delegation request payload to use, defaulting to
`sasl_username`. This is because when customers use a password auth
process, the `sasl_username` corresponds to the customer whose
email quota is being checked. In certain circumstances (when
authentication fails), the `sasl_username` field is blank. Since
v0.3.11, when we find it blank we attribute that to authentication
failure, and we provide some extra config elements to control this
behavior.
If the config key `require_user_key` is set to **True**, then only the
key specified in `user_key` will be checked for contents to identify
the customer, and if it is empty, an `AuthenticationFailedException`
will be raised, which will cause the `no_user_key_response` to be sent
back via Postfix. If `require_user_key` is **False**, then a series
of fields will be searched as outlined below.
At present, there is little sanitation on the `user_key` field. It is
never evaluated as code, but it is used directly as the attribute name
for the value dereference. If that yields no value, or if it is not
specified, CHAPPS looks for `sasl_username` first, then
`ccert_subject`, and if there is none, it falls back to `sender`,
which can also be blank. In that extreme case, CHAPPS uses
`client_address`. This will not work very well long-term if a lot of
real customers share a mail gateway, so it is recommended to make sure
that the field specified is being populated.
Incidentally, this may be a reason for permitting customers which
don't appear in the user-list, since system-generated messages which
don't have a `sender` listed will end up quota'd on their client
address, and probably most of them will be denied by quota,
potentially generating a large number of confusing secondary error
messages. CHAPPS currently expects any permitted sender to appear in
the `users` table. Note that the name which appears in this table
needs to match what will be discovered in the specified key field.
For sites which use the customer's email address as their login name
for email access, this is easy. For cert issuers, it may simplify
things to use the email address as the subject of the cert, but any
unique string will work.
## Outbound Quota Policy Service
The service is designed to run locally side-by-side with the Postfix
server, and connect to a Redis instance, optionally via Sentinel. As
such it listens on 127.0.0.1, and on port 10225 by default, though
both may be adjusted in the config file. It obtains quota policy data
on a per-customer basis, from a relational database, and caches that
data in Redis for operational use. Once a user's quota data has been
stored, it will be cached for a day, so that database accesses may be
avoided.
Current quota usage is **not** kept in a relational database.
There is CLI access to data about current quota usage, providing
facilities for updating quota policy information immediately: clearing
quotas, upgrading them, adding new users, adding new quotas, etc.,
along with a number of other useful functions. The CLI, `chapps-cli`,
is self-documenting.
There is also a REST API service which can perform any of these tasks,
using cURL or similar.
In order to set up Postfix for policy delegation, consult [Postfix
documentation](http://www.postfix.org/SMTPD_POLICY_README.html) to
gain a complete understanding of how policy delegation works. In
short, the `smtpd_recipient_restrictions` block should contain the
setting `check_policy_service inet:127.0.0.1:10225`. In addition, it
is necessary to ensure that the service itself, the script
`chapps_outbound_quota.py` or `chapps_outbound_multi.py` is running.
This should be accomplished
using SystemD or similar; scripts/unit file assets to assist with that
are to be found in the `install` directory. (For now, according to
current wisdom, Postfix's own `spawn` functionality from `master.cf`
should be avoided.)
### Outbound Quota Policy Configuration: Database Setup
At present, the service expects to obtain quota policy enforcement
parameters from a relational database (MySQL or MariaDB). The
framework has been designed to make it easy to write adapters to any
particular backend datasource regarding quota information.
As of CHAPPS v0.4.13, Alembic is used to maintain the database schema
as it mutates across versions. [See above](#db-initialization) for
advice related to software upgrades and database migrations.
The database schema used has been kept as simple as possible: (please
note that this schema may differ slightly in terms of index and key
names from the one installed by Alembic)
```
CREATE TABLE `users` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`name` varchar(128) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE `quotas` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`name` varchar(32) NOT NULL,
`quota` bigint(20) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`),
UNIQUE KEY `quota` (`quota`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE `quota_user` (
`quota_id` bigint(20) NOT NULL,
`user_id` bigint(20) NOT NULL,
PRIMARY KEY (`user_id`)
KEY `fk_quota` (`quota_id`),
CONSTRAINT `fk_quota_user` FOREIGN KEY (`quota_id`) REFERENCES `quotas` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_user_quota` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
The `users` table contains a record for each authorized customer who
is allowed to send email. Customers without entries will not be able
to send email, despite authenticating with Postfix.
The `quotas` table contains quota definitions, the `name` is meant to
hold a user-readable tag for the quota and max outbound email count
(`quota`) of that quota.
The `quota_user` table joins the `users` table with the `quotas`
table. The `quota_user.user_id` column joins with `users.id` to map
usernames onto IDs. Usernames may be email addresses, but they also
may not. How they are obtained is configurable as `user_key` -- the
specified field will be extracted from the policy request payload
presented by Postfix.
Once the `quotas` table has been populated with the desired quota
policies, the `quota_user` table may then be populated to reflect each
user's quota.
The application sets cached quota limit data to expire after 24 hours,
so it will occasionally refresh quota policy settings, in case they
get changed. In order to flush the quota information, all that is
required is to delete that user's policy tracking data from Redis.
Routes for doing so are provided by the REST API.
**Please note:** Customers with no `users` entry will not be able to send
outbound email.
### Quota policy settings (non-database)
#### Counting all outbound messages against the quota
Some quota systems count any email as a single email regardless of the
number of recipients included in the envelope recipients list
(RCPT TO). This software can operate that way, but it can also
count an email for each recipient in the list. Whether it does so is
governed by the boolean setting `counting_recipients`: setting this to
True will cause CHAPPS OutboundQuotaPolicy to count a sent email for
each recipient.
#### Outbound quota grace margins
There is a `margin` setting which will allow for some fuzziness over
the established quota for multi-recipient emails, allowing a customer
to go over their quota on a single (multi-recipient) email as long as
the total number of mails sent fits within the margin. This obviously
has no meaning if recipients aren't being counted, since no email will
ever represent more than a single outbound message.
Margins specified in **integers** are absolute message counts.
Those specified as **floats** represent a proportion of the total
margin. If a float value is less than 1 it is assumed to be the
ratio. If it is larger than 1 and less than 100, it is assumed to be
a percentage, and it divided by 100.0 is used as the ratio.
## Sender Domain Authorization (Outbound multi-policy service)
As of CHAPPS v0.4.12 , sender-domain authorization (SDA) is only
available as part of the outbound multi-policy service, consisting of
SDA followed by outbound quota. There is a plan (TODO:) to produce a
standalone SDA service script.
The SDA policy allows an email service provider to specify on a
per-customer basis exactly which domains may appear after the @ in the
MAIL FROM address, the `sender` field in the Postfix policy delegation
data packet. Customer identification for outbound emails is covered
in a previous section of this document (see: ['Setting the user
key'](#setting-the-user-key)).
It is generally possible to configure vanilla Postfix to limit
outbound domains for users, but we encountered some difficulty getting
it to work reliably, and this method opens the door to a great deal of
additional nuance which would not otherwise be available to us.
CHAPPS stores SDA policy control data in its database, in a fairly
simple, normalized scheme. This feature uses a table each to store
source domains and email addresses, and a new join table for each to
link customers with domains and whole-email addresses they are allowed to
use for outbound mail.
The domain matching is intentionally inflexible -- the
entire string after the @ sign must match a domain in the table. That
is to say: in order to allow users to send from subdomains, those
subdomains must have entries in the domains table, and those entries
must be linked to the logged-in (email-sending) customer via the
`domain_user` join table.
Here is the schema, for reference: (please note that this schema may
differ slightly in terms of index and key names from the one installed
by Alembic)
```
CREATE TABLE `domains` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(64) NOT NULL,
`greylist` tinyint(1) NOT NULL,
`check_spf` tinyint(1) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ix_domains_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE `domain_user` (
`domain_id` bigint(20) NOT NULL,
`user_id` bigint(20) NOT NULL,
PRIMARY KEY (`domain_id`,`user_id`),
KEY `fk_user_domain` (`user_id`),
CONSTRAINT `fk_domain_user`
FOREIGN KEY (`domain_id`) REFERENCES `domains` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_user_domain`
FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE `emails` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`name` varchar(128) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE `email_user` (
`email_id` bigint(20) NOT NULL,
`user_id` bigint(20) NOT NULL,
PRIMARY KEY (`email_id`,`user_id`),
KEY `fk_user_email` (`user_id`),
CONSTRAINT `fk_email` FOREIGN KEY (`email_id`) REFERENCES `emails` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `fk_user_email` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
As with the quota policy, the logic used is inherently conservative.
If a customer has no entry in the `users` table, that customer will
not be able to send mail (even though they have authenticated). If a
customer is trying to send an email from an address (the `sender`
address) which has a domain string (everything after the @ sign) which
does not appear in the `domains` table, or for which that user lacks a
join record in `domain_user`, the email will be denied.
In practice, this will mean that when a new customer signs up,
the domain(s) included in that service agreement should be added to
the `domains` table. Any users which are authorized to send email
appearing to originate from that domain should be added to the `users`
table, with join records linking their IDs to the IDs of the domain(s)
they can send for, in `domain_user`.
### Whole-Email Matching
As a fallback to domain authorization, the SDA module also compares
the entire `sender` field with the whole-email entries associated to
the user. This allows an email provider to specify specific email
addresses which may be used for outbound masquerading. This helps to
prevent customers from pretending to be other customers, and helps to
create a specification for possible scanning tools which might
otherwise react negatively to the logs of such activity.
TODO: Currently, CHAPPS causes cached policy data to have an expiry
timer of a day. For outbound quota, this makes a great deal of sense
because the quotas are expressed in emails per day. However, a day's
worth of authorized email senders' Redis cache keys may actually cause
quite a bit of memory usage for no particular reason. Users don't
send an evenly-spaced stream of email throughout the day; they send
some emails, often in clusters, separated by long pauses. As such,
the expiry time of SDA Redis caches should probably be a tunable
parameter, in order to allow operators to tune how much RAM on their
Redis servers ends up devoted to SDA caching. 6 or 8 hours seems like
a reasonable trade-off between Redis-RAM bloat and RDBMS latency, but
different sites are different.
## Inbound Services
For inbound services, it is possible to enable either policy on a
per-domain basis. This may be accomplished via the CLI or API.
### HELO Whitelisting
Sometimes CHAPPS may not be the first line of defense for all email;
it is possible that most email needs policy enforcement, but that some
particular relay(s) already do the same job as CHAPPS, and so whatever
they relay should be whitelisted. In such a case, the relay will
probably always fail SPF anyway, and result in a header to that
effect.
In order to whitelist by **HELO**, specify the `helo_whitelist` option in
the `[CHAPPS]` section of the config file, with data about the server
to whitelist. Due to some limitations of ConfigParser, the data
needs to be packed into a list on a single line. The format is as
follows:
```
helo_whitelist=mx.example.com[:1.2.3.4][;mx2.example.com[:5.6.7.8][;...]]
```
To break this down in English, provide at a minimum the HELO name used
by the server to whitelist. In such a case, the IP will be obtained
from DNS and used as if it had also been supplied. In order to supply
the IP address of the server, use a colon (`:`) after the name. In
order to list more than one name (optionally with IP address), use
semi-colons (`;`) to separate entries.
At configuration time, the A record will be evaluated to see if it
matches the IP provided. The PTR of the IP, whether provided or
obtained by a lookup, will be obtained as well, and checked to make
sure the provided name matches. If all the elements do not match,
CHAPPS will log an error and will not perform any whitelisting.
Currently, if the address provided is the loopback address
"127.0.0.1", then no DNS lookups will be performed, and both name and
address values will be used without any checking or cross-checking.
In future versions this logic could be expanded to include all
"private" IPv4 address ranges, v6 ranges, etc.
## Greylisting Policy Service
Greylisting is an [approach to spam
prevention](https://en.wikipedia.org/wiki/Greylisting_(email)) based
on the tendency of spammers to emit emails without using gateways.
Spam is typically sent _to_ one or more gateways by malware programs
which amount to viral MUAs, able to connect to SMTP servers to send
mail, but not capable of noting response codes and retrying deferred
deliveries. Because a large proportion of spam is (or was) sent this
way, the simple act of deferring emails from unknown (untrusted)
sources eliminates a large amount of spam.
CHAPPS can, on a per-domain, opt-in basis, perform [RCPT or DATA
greylisting](https://datatracker.ietf.org/doc/html/rfc6647#section-2.4)
at present, since it wants to use the sender's email address and IP
address as well as the recipient list. CHAPPS is written to expect to
be invoked during the RCPT phase, but should work in the DATA phase.
That has not been tested.
Emails addressed to enforcing domains will be greylisted--that is,
deferred--when they are associated with source tuples which are not
recognized. Tracking data regarding recognized tuples and domain
option setting is stored in Redis. Config data regarding option
status is obtained from the database and cached in Redis. Domains
which do not set the `greylist` bit on their records will receive all
mail addressed to them immediately, without any greylisting.
The Greylisting module performs whitelisting at its own level, based
on a tally of successfully delivered deferred emails. That is to say,
emails which have been deferred and then redelivered on the required
schedule are counted in a tally per-client (by source IP address).
When that tally reaches 10, further emails with matching source IPs
are whitelisted. This threshhold may be adjusted in the
`GreylistingPolicy` config, as `whitelist_threshold`.
Please note that in the context of comprehensive inbound email
filtering, SPF and greylisting have an interesting relationship which
is not entirely straightforward, and so a special combined, inbound
multi-policy service has been provided which combines the features of
greylisting and SPF checking in a sane fashion, and provides a
framework for adding further policies.
## SPF Policy Enforcement
The [Sender Policy
Framework](https://en.wikipedia.org/wiki/Sender_Policy_Framework) is a
complicated and intricate beast, and so I will not try to describe it
in great detail, but instead link to relevant documentation about what
SPF is. Important to note is the fact that SPF is a framework for
using the DNS as the policy configuration source.
There is no provision in the RFC for the caching of SPF results in
order to apply them to other circumstances, such as another email with
the same inputs. It is possible that the policy itself, i.e. the TXT
record containing the SPF policy string, could change between emails.
As such, this module does not use Redis to cache operational data.
Redis is used to cache the per-domain SPF-enablement option.
There is a very widely-used and well-supported implementation of the
SPF check itself in the Python community called
[pyspf](https://pypi.org/project/pyspf/), by Stuart Gathman and
Terence Way. CHAPPS uses this library to get SPF check results.
The SPF policy enforcement framework included in CHAPPS makes it
possible for an operator to specify clearly and flexibly what they
would like to have happen in response to any of the different SPF
check results. The [SPF specification in RFC
7208](https://datatracker.ietf.org/doc/html/rfc7208) does not address
exactly what response to take in each case, saying that it is a site's
prerogative to decide the fates of those emails.
Domains must opt-in to SPF checking, just as with Greylisting.
Domains which opt out of both will simply receive all email addressed
to them without any SPF checking or Greylisting.
There is no standalone SPF service; it is part of the multipolicy
inbound service. It would be trivial to create a standalone SPF
service. Since we intend to provide both options to our customers,
creating a standalone service is a low priority. **Please note** that
as with Greylisting, the domain-level option will be honored even by
standalone handlers, meaning that in order for a domain's incoming
email to have SPF enforced, that option will still need to be set on
its record.
## Inbound Multi-policy Service (SPF + Greylisting)
**Please note that blanket use of Greylisting is not recommended.**
What does it mean to use both greylisting and SPF? The trivial answer
is to pass one filter, and then pass the next filter. But which comes
first?
If one greylists first, a legitimate email may be deferred for ten
minutes, then pass SPF checking; should emails which pass SPF be
subject to greylisting? Conversely, a greylisted email may also come
from a server which is not allowed by its SPF record, and then be
deferred only to be denied for an unrelated reason after ten minutes
of taking up disk space and using up cycles needlessly.
On the other hand, if one uses an SPF filter first, in a trivial
fashion, then emails must pass muster on the SPF check first, which
seems right and proper to me, certainly. And if greylisting is to be
used also, then it makes sense for emails which get `pass` from SPF to
be deferred. When they are sent again they will of course incur
another SPF check, and then they will pass greylisting, provided that
the SPF record they depend on has not changed in the meantime.
In the realm of SPF, there are a couple of grey areas, no pun
intended. SPF can return `softfail` if it isn't sure enough about the
check failing to indicate a hard fail. It can also return `none` or
`neutral` which are required to be treated the same way. In such
cases, the SPF checker is saying that the SPF record either doesn't
exist or might as well not exist for all the good it does in this
case.
Generally, sites are left to determine whether to accept these emails,
or possibly tag them and/or quarantine them. So far, this software
does not address any of those possible outcomes. But we can provide
the interesting option of using greylisting for grey areas.
By default, CHAPPS SPF policy enforcement service uses greylisting for
emails which receive `softfail` and `none`/`neutral` responses on
their SPF checks.
If a domain is set to enforce SPF **and** Greylisting, that will cause
CHAPPS to greylist even emails which receive `pass` from SPF, meaning
that any "deliverable" email will be deferred unless it is already
coming from a recognized source (tuple), when both are enabled.
(Non-deliverable categories are: `fail`, `temperror`, `permerror`.)
The messages which accompany the various rejections and deferrals
indicate what the reason was. In some cases, those messages indicate
that a message has been greylisted due to the SPF enforcement policy.
## Upcoming features
A mini-roadmap of upcoming changes:
minor:
- SDA (and other) Redis keys will have tunable expiry times
- Look into specifying log facility and level in the config file
major:
- Possibly support multiple enforcement intervals
- It seems inevitable that other features will also be added. There
is some skeletal code in the repo for building email content
filters, which are not the same as policy delegates.
- Using Redis makes it possible to send pub/sub messages when
certain sorts of conditions occur, such as a user making a large
number of attempts to send mail in a short time while overquota,
or when a user (repeatedly?) attempts to send email as being from
a domain that user lacks authorization for.
Raw data
{
"_id": null,
"home_page": "https://github.com/easydns/chapps",
"name": "chapps",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "Postfix,Policy,Daemon",
"author": "Caleb S. Cullen",
"author_email": "ccullen@easydns.com",
"download_url": "https://files.pythonhosted.org/packages/71/01/7e6cffa8057002ecd1cb50f40a6dd35697cbae75791340442121d2827112/chapps-0.5.18.tar.gz",
"platform": null,
"description": "# the Caching, Highly-Available Postfix Policy Service\n\n| **requires Python 3.8.10+**\n| **makes use of Redis (or Sentinel) and a relational database (MariaDB)**\n\n## Introduction\n\nThere is a need for a highly-available, high-performance, concurrent,\nclusterable solution for handling various aspects of email policy.\nPostfix farms out the job of policy decisions to a delegate over a\nsocket, so we can provide a framework for receiving that data, making\na decision about it, and then sending a response back to Postfix.\nThere are some projects which have provided smaller-scale solutions to\nthis issue. We handle rather a large volume of email, so we need\nsomething more performant than a script which makes a database access\non every email. In order to simplify the language, the term \"customer\"\nis used herein to mean a person who logs into an SMTP server in order\nto send email.\n\nMy decision was to use Redis, since with Redis Sentinel it should be\npossible to achieve a degree of high-availability using Redis as a\ncommon datastore between the various email servers in the farm(s)\nwhich will run local instances of the policy server, which will itself\nuse Redis to cache data and keep track of email quotas, etc.\n\nIn the first iteration, we propose to provide functionality for:\n - outbound quota tracking on a continuous, rolling, per-interval basis;\n - outbound sender domain authorization\n - inbound email greylisting;\n - inbound SPF checking\n\nThe framework is meant to be extensible, so that any conceivable email\npolicy might be implemented in the future.\n\n## Configuration\n\nThe library will create a config file for itself if it does not find one\nat its default config path, `/etc/chapps/chapps.ini`, or the value of\nthe environment variable `CHAPPS_CONFIG` if it is set. Note that\ndefault settings for all available submodules will be produced. At\nthe time of writing, each script runs its own type of policy handler,\nso only the settings for the policies of that handler will be needed,\nplus the general CHAPPS settings and the Redis settings.\n\nIt is possible to adjust the number of connections allowed to be\nwaiting on the CHAPPS server to answer them. The default used by\n`asyncio` when none is provided is 100, so that is also the default\nvalue used by CHAPPS. It may be adjusted in the config file under\n`[CHAPPS]`, and is called `listener_backlog`. In some cases it may be\ndesirable to increase that number, so that exceptionally busy mailers\ndo not run into the problem of having their connection attempts\nrejected.\n\nPolicies may each specify separate listening addresses and ports, so\nthat they may run simultaneously on the same server. For multi-policy\nhandlers, the first handler specified will be the one whose network\nsettings are used. It is recommended to configure those elements only\non that policy, or to keep them in sync on all policies which are\nhandled together.\n\nExample Postfix configs are included in the `postfix` directory,\nclassified by which service they are for. Most access control policy\nservices will be implemented in a very similar way in `main.cf`,\nprobably in combination with other policies. The examples provided\nare the same configs used for testing, and are necessarily stripped\ndown to focus just on that particular service.\n\nAn example **rsyslog** config is also included; modification is\nencouraged. If you wish to keep the debug logs in their special\ndestination, ensure that you create a log-rotation profile for it.\n\n## Installation Overview\n\nThe recommended Debian packages are:\n - `mysqlclient`\n - `redis`\n - `python3-pip`\n - `python3-venv`\n\nIt is highly recommended to install CHAPPS into a venv. You may need\nto install the system package `python3-venv` in order for this to\nwork:\n```\npython3 -m venv chapps-venv\n. chapps-venv/bin/activate\n```\n\nThe package may be installed via PyPI, using the following command:\n\n```\npython3 -m pip install chapps\n```\n\n### DB Initialization\n\nAs of this writing, it should be possible to run `apply-migrations` or\n`chapps-cli admin db-setup` once the venv is activated, and that should\napply all of the necessary Alembic migrations to bring the database up\nto date from zero, based on the database access configuration in the\nCHAPPS config file.\n\nIf the config file has not yet been populated with database\ncredentials, do that first, and ensure that the named database exists\n(has been created) on the database server before attempting to install\nthe schema into it.\n\n**Please Note:**\n\n\tDatabases created with earlier versions of CHAPPS (v<=0.4.12) need to\n\thave their data dumped to SQL, and the database dropped and the schema\n\tre-created via the `apply-migrations` mechanism in order for it to exactly\n\tmatch Alembic's notion of how it is built. Once Alembic has built the\n\tschema, the data may be read back into the database. We will be using\n\tAlembic going forward so this should be a one-time annoyance.\n\n\tFor convenience, here is the `mysqldump` commandline recommended for\n\tdumping the data:\n\n\t.. code:\n\n\t mysqldump --skip-add-drop-table -tc chapps > chapps-data-only.sql\n\n\tThese options tell `mysqldump` not to drop the tables, not to try to create\n\tthe tables, and to use \"complete\" INSERT statements, which ensures that the\n\ttarget columns are listed in the INSERT statement, in case the native column\n\torder changes between dump and restore.\n\n#### Database Adapters\n\nIn order to obtain control data from the database, CHAPPS policy\nobjects use a database adapter object tailored for their specific\nneeds. This allows all database logic to be factored completely out\nof the policy layer.\n\nAs of CHAPPS v0.5.5, the database adapter layer is now based on\nSQLAlchemy throughout. However, it is still possible to switch to\nusing the MariaDB adapter instead, for the actual services. The API\nis built on SQLAlchemy and requires it.\n\n**To set CHAPPS to use MySQL/MariaDB** directly, instead of\nSQLAlchemy, set the environment variable `CHAPPS_DB_MODULE` to\n`mysql`. Doing so will cause the policy layer to use the adapter\nmodule based on `mysqlclient`, which also works just fine with\nMariaDB. If it is not set or is set to something else, SQLAlchemy\nwill be used. This will need to be specified in the service\ndescription file to take effect, unless some other method is being\nused to launch CHAPPS.\n\nThe ability to switch database adapter modules may be eliminated in a\nfuture release. However, since the application has not been tested in\nproduction using SQLAlchemy, it seems prudent to provide a mechanism\nfor switching between them.\n\n### Starting and Auto-launching Services\n\nWith a venv, the SystemD service files are installed to a folder\ncalled `chapps/install` inside the venv directory, and Postfix\nexample/testing configs are located in the `chapps/postfix` folder. Scripts\nand package go to `bin` and `lib/.../chapps` as expected. Use of a\nvenv is recommended, as the SystemD service description files provided\nare formatted during the install process to launch the services correctly\nwithin their venv.\n\nWithout a venv, they go to various system locations,\nwith the ancillary `chapps` directory usually showing up at\n`/usr/local/chapps`. YMMV. A venv will keep things organized.\n\nA Python script called `chapps_database_init.py` is included\nto create the database schema required by the library. It\ndoes not create the database itself. Before running this script,\nensure that the CHAPPS configuration file contains the correct\ncredentials and other control data to be able to connect to the\ndatabase server, and also ensure that the database named in that\nconfig has been created on the server. The script will connect to the\ndatabase and create the tables. It uses `IF EXISTS` and does not\ncontain any kind of data deletion, so it should be safe to use at any\ntime.\n\nFor more information about installing, see the\n[INSTALLATION](INSTALLATION.md) file.\n\n### Redis configuration\n\nRedis is used to store the real-time state of every active user's\noutbound quota, sender-domain authorization status cache, and also to\nkeep track of greylisting status for greylisted emails. An active\nuser is one who has sent email in the last _interval_, that interval\ndefaulting to a day, since most quotas are expressed as\nmessages-per-day.\n\nIf your Redis deployment is on a different server and/or if CHAPPS is\nsharing a Redis instance with some other services it may be necessary\nto adjust the Redis-related settings in the config file, to adjust the\naddress and/or port to connect to, or what database to use. By\ndefault, CHAPPS tries to connect to Redis on localhost, using the\nstandard port assignment and db 0.\n\nIf Sentinel is in use, populate the Sentinel-oriented configuration\nelements `sentinel_servers` and `sentinel_dataset`. The servers list\nshould be a space-separated list of each Sentinel server half-socket;\nfor example, \"10.1.9.10:26379 10.1.9.12:26379\". The dataset name is\nthe one you specified to Sentinel when setting up the Sentinel\ncluster. Sentinel's default dataset name is `mymaster`. We, of\ncourse, recommend `chapps`, or perhaps `chapps-outbound` at a site\nwith a large volume of email. Since SPF doesn't make much use of\nRedis, the inbound load may be lighter than the outbound load,\ndepending on which things happen more at a particular site.\n\n### Logging\n\nAt this time, CHAPPS uses **syslog**, and transmits logs on the\n`local0` facility. CHAPPS sends a fair amount of debug information at\nthe DEBUG level. Right now, the application's facility and level may\nnot be adjusted via the config file; later this may be implemented.\nFor the time being, it seems sufficient to control logging via the\n**rsyslog** configuration used to control log entries on `local0`.\n\nThe example provided (in the `install` directory) sends all logs to a\nspecial log (the path needs to exist and belong to the syslog user,\nwhether that be `syslog` (as on Ubuntu) or `root` under Debian). As\nlong as the path exists and is writable by **rsyslog**, it will create\nthe log. The example also sends logs at INFO level or above to\n`/var/log/mail.log`, which generally is the destination that\n**rsyslog** uses for mail-related logs.\n\nOf course, site operators are encouraged to alter this example config\nto their needs. For those who wish to monkeypatch facility and level,\nit is set in one place, at the top of `chapps.logging`.\n\n## REST API Service\n\nStarting with version 0.4.0, a REST API service is included, based on\nFastAPI, and using SQLAlchemy with the MySQLclient backend. A service\ntemplate for the API is provided, as well as a socket unit and an\n**nginx** example config, for using a UDS to proxy between the web and\nthe application. By default, Gunicorn is used to launch Uvicorn\nworkers which serve the API directly on port 8080. The precise\ndetails may be adjusted in the `chapps-api-gunicorn.service` unit\nfile.\n\nIt is also possible that other approaches are preferred at other sites.\nThe extra files, provided in the `install` directory, are provided in\nthe hope that they may be useful.\n\nThe API is self-documenting. Once it is running, visit it at\n`<server>:8080/redoc/` or `<server>:8080/docs/` to browse the\ndocumentation.\n\nThe API service needs to have the same configuration as the other\nCHAPPS services it is meant to manage. Since it instantiates policy\nobjects, it is best to simply provide the same copy of the config to\nall related nodes. Please note that the API service is completely\nseparate from the policy service(s), and need not run on the same\nserver -- and in fact probably _should_ not run on the same server --\nwith the policy service.\n\n## Outbound Services\n\nPolicy services can be divided into those which work on outbound mail,\nand those which work on inbound mail. Some, possibly, might be\napplied to either flow, but none such are part of this project yet.\nOutbound items share some characteristics.\n\nOutbound mail, for our purposes, is assumed to originate with an\nauthenticated user. That user may authenticate with Postfix using a\nusername/password or a client-side SSL cert, in which case the\nusername or subject name (of the cert) will be passed along by Postfix\nto the policy service.\n\nIn order to allow sites to specify exactly what field of the Postfix\npolicy data they would like to use to identify users, the\nconfiguration allows the user to specify the first field to check.\n\n### Setting the user key\n\nPostfix submits a fairly large packet of data on each policy\ndelegation request. One prominent element of this data is the MAIL\nFROM address, which is labelled as `sender`. This is perhaps the\nobvious element to use to count quotas, but some other fields are more\ninteresting.\n\nCurrent versions of the software allow the config file to specify what\nelement of that delegation request payload to use, defaulting to\n`sasl_username`. This is because when customers use a password auth\nprocess, the `sasl_username` corresponds to the customer whose\nemail quota is being checked. In certain circumstances (when\nauthentication fails), the `sasl_username` field is blank. Since\nv0.3.11, when we find it blank we attribute that to authentication\nfailure, and we provide some extra config elements to control this\nbehavior.\n\nIf the config key `require_user_key` is set to **True**, then only the\nkey specified in `user_key` will be checked for contents to identify\nthe customer, and if it is empty, an `AuthenticationFailedException`\nwill be raised, which will cause the `no_user_key_response` to be sent\nback via Postfix. If `require_user_key` is **False**, then a series\nof fields will be searched as outlined below.\n\nAt present, there is little sanitation on the `user_key` field. It is\nnever evaluated as code, but it is used directly as the attribute name\nfor the value dereference. If that yields no value, or if it is not\nspecified, CHAPPS looks for `sasl_username` first, then\n`ccert_subject`, and if there is none, it falls back to `sender`,\nwhich can also be blank. In that extreme case, CHAPPS uses\n`client_address`. This will not work very well long-term if a lot of\nreal customers share a mail gateway, so it is recommended to make sure\nthat the field specified is being populated.\n\nIncidentally, this may be a reason for permitting customers which\ndon't appear in the user-list, since system-generated messages which\ndon't have a `sender` listed will end up quota'd on their client\naddress, and probably most of them will be denied by quota,\npotentially generating a large number of confusing secondary error\nmessages. CHAPPS currently expects any permitted sender to appear in\nthe `users` table. Note that the name which appears in this table\nneeds to match what will be discovered in the specified key field.\nFor sites which use the customer's email address as their login name\nfor email access, this is easy. For cert issuers, it may simplify\nthings to use the email address as the subject of the cert, but any\nunique string will work.\n\n## Outbound Quota Policy Service\n\nThe service is designed to run locally side-by-side with the Postfix\nserver, and connect to a Redis instance, optionally via Sentinel. As\nsuch it listens on 127.0.0.1, and on port 10225 by default, though\nboth may be adjusted in the config file. It obtains quota policy data\non a per-customer basis, from a relational database, and caches that\ndata in Redis for operational use. Once a user's quota data has been\nstored, it will be cached for a day, so that database accesses may be\navoided.\n\nCurrent quota usage is **not** kept in a relational database.\n\nThere is CLI access to data about current quota usage, providing\nfacilities for updating quota policy information immediately: clearing\nquotas, upgrading them, adding new users, adding new quotas, etc.,\nalong with a number of other useful functions. The CLI, `chapps-cli`,\nis self-documenting.\n\nThere is also a REST API service which can perform any of these tasks,\nusing cURL or similar.\n\nIn order to set up Postfix for policy delegation, consult [Postfix\ndocumentation](http://www.postfix.org/SMTPD_POLICY_README.html) to\ngain a complete understanding of how policy delegation works. In\nshort, the `smtpd_recipient_restrictions` block should contain the\nsetting `check_policy_service inet:127.0.0.1:10225`. In addition, it\nis necessary to ensure that the service itself, the script\n`chapps_outbound_quota.py` or `chapps_outbound_multi.py` is running.\nThis should be accomplished\nusing SystemD or similar; scripts/unit file assets to assist with that\nare to be found in the `install` directory. (For now, according to\ncurrent wisdom, Postfix's own `spawn` functionality from `master.cf`\nshould be avoided.)\n\n### Outbound Quota Policy Configuration: Database Setup\n\nAt present, the service expects to obtain quota policy enforcement\nparameters from a relational database (MySQL or MariaDB). The\nframework has been designed to make it easy to write adapters to any\nparticular backend datasource regarding quota information.\n\nAs of CHAPPS v0.4.13, Alembic is used to maintain the database schema\nas it mutates across versions. [See above](#db-initialization) for\nadvice related to software upgrades and database migrations.\n\nThe database schema used has been kept as simple as possible: (please\nnote that this schema may differ slightly in terms of index and key\nnames from the one installed by Alembic)\n\n```\nCREATE TABLE `users` (\n `id` bigint(20) NOT NULL AUTO_INCREMENT,\n `name` varchar(128) NOT NULL,\n PRIMARY KEY (`id`),\n UNIQUE KEY `name` (`name`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n\nCREATE TABLE `quotas` (\n `id` bigint(20) NOT NULL AUTO_INCREMENT,\n `name` varchar(32) NOT NULL,\n `quota` bigint(20) NOT NULL,\n PRIMARY KEY (`id`),\n UNIQUE KEY `name` (`name`),\n UNIQUE KEY `quota` (`quota`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n\nCREATE TABLE `quota_user` (\n `quota_id` bigint(20) NOT NULL,\n `user_id` bigint(20) NOT NULL,\n PRIMARY KEY (`user_id`)\n KEY `fk_quota` (`quota_id`),\n CONSTRAINT `fk_quota_user` FOREIGN KEY (`quota_id`) REFERENCES `quotas` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,\n CONSTRAINT `fk_user_quota` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n```\n\nThe `users` table contains a record for each authorized customer who\nis allowed to send email. Customers without entries will not be able\nto send email, despite authenticating with Postfix.\n\nThe `quotas` table contains quota definitions, the `name` is meant to\nhold a user-readable tag for the quota and max outbound email count\n(`quota`) of that quota.\n\nThe `quota_user` table joins the `users` table with the `quotas`\ntable. The `quota_user.user_id` column joins with `users.id` to map\nusernames onto IDs. Usernames may be email addresses, but they also\nmay not. How they are obtained is configurable as `user_key` -- the\nspecified field will be extracted from the policy request payload\npresented by Postfix.\n\nOnce the `quotas` table has been populated with the desired quota\npolicies, the `quota_user` table may then be populated to reflect each\nuser's quota.\n\nThe application sets cached quota limit data to expire after 24 hours,\nso it will occasionally refresh quota policy settings, in case they\nget changed. In order to flush the quota information, all that is\nrequired is to delete that user's policy tracking data from Redis.\nRoutes for doing so are provided by the REST API.\n\n**Please note:** Customers with no `users` entry will not be able to send\noutbound email.\n\n### Quota policy settings (non-database)\n\n#### Counting all outbound messages against the quota\n\nSome quota systems count any email as a single email regardless of the\nnumber of recipients included in the envelope recipients list\n(RCPT TO). This software can operate that way, but it can also\ncount an email for each recipient in the list. Whether it does so is\ngoverned by the boolean setting `counting_recipients`: setting this to\nTrue will cause CHAPPS OutboundQuotaPolicy to count a sent email for\neach recipient.\n\n#### Outbound quota grace margins\n\nThere is a `margin` setting which will allow for some fuzziness over\nthe established quota for multi-recipient emails, allowing a customer\nto go over their quota on a single (multi-recipient) email as long as\nthe total number of mails sent fits within the margin. This obviously\nhas no meaning if recipients aren't being counted, since no email will\never represent more than a single outbound message.\n\nMargins specified in **integers** are absolute message counts.\n\nThose specified as **floats** represent a proportion of the total\nmargin. If a float value is less than 1 it is assumed to be the\nratio. If it is larger than 1 and less than 100, it is assumed to be\na percentage, and it divided by 100.0 is used as the ratio.\n\n## Sender Domain Authorization (Outbound multi-policy service)\n\nAs of CHAPPS v0.4.12 , sender-domain authorization (SDA) is only\navailable as part of the outbound multi-policy service, consisting of\nSDA followed by outbound quota. There is a plan (TODO:) to produce a\nstandalone SDA service script.\n\nThe SDA policy allows an email service provider to specify on a\nper-customer basis exactly which domains may appear after the @ in the\nMAIL FROM address, the `sender` field in the Postfix policy delegation\ndata packet. Customer identification for outbound emails is covered\nin a previous section of this document (see: ['Setting the user\nkey'](#setting-the-user-key)).\n\nIt is generally possible to configure vanilla Postfix to limit\noutbound domains for users, but we encountered some difficulty getting\nit to work reliably, and this method opens the door to a great deal of\nadditional nuance which would not otherwise be available to us.\n\nCHAPPS stores SDA policy control data in its database, in a fairly\nsimple, normalized scheme. This feature uses a table each to store\nsource domains and email addresses, and a new join table for each to\nlink customers with domains and whole-email addresses they are allowed to\nuse for outbound mail.\n\nThe domain matching is intentionally inflexible -- the\nentire string after the @ sign must match a domain in the table. That\nis to say: in order to allow users to send from subdomains, those\nsubdomains must have entries in the domains table, and those entries\nmust be linked to the logged-in (email-sending) customer via the\n`domain_user` join table.\n\nHere is the schema, for reference: (please note that this schema may\ndiffer slightly in terms of index and key names from the one installed\nby Alembic)\n\n```\nCREATE TABLE `domains` (\n `id` int(11) NOT NULL AUTO_INCREMENT,\n `name` varchar(64) NOT NULL,\n `greylist` tinyint(1) NOT NULL,\n `check_spf` tinyint(1) NOT NULL,\n PRIMARY KEY (`id`),\n UNIQUE KEY `ix_domains_name` (`name`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n\nCREATE TABLE `domain_user` (\n `domain_id` bigint(20) NOT NULL,\n `user_id` bigint(20) NOT NULL,\n PRIMARY KEY (`domain_id`,`user_id`),\n KEY `fk_user_domain` (`user_id`),\n CONSTRAINT `fk_domain_user`\n FOREIGN KEY (`domain_id`) REFERENCES `domains` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,\n CONSTRAINT `fk_user_domain`\n FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n\nCREATE TABLE `emails` (\n `id` bigint(20) NOT NULL AUTO_INCREMENT,\n `name` varchar(128) NOT NULL,\n PRIMARY KEY (`id`),\n UNIQUE KEY `name` (`name`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n\nCREATE TABLE `email_user` (\n `email_id` bigint(20) NOT NULL,\n `user_id` bigint(20) NOT NULL,\n PRIMARY KEY (`email_id`,`user_id`),\n KEY `fk_user_email` (`user_id`),\n CONSTRAINT `fk_email` FOREIGN KEY (`email_id`) REFERENCES `emails` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,\n CONSTRAINT `fk_user_email` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;\n```\n\nAs with the quota policy, the logic used is inherently conservative.\nIf a customer has no entry in the `users` table, that customer will\nnot be able to send mail (even though they have authenticated). If a\ncustomer is trying to send an email from an address (the `sender`\naddress) which has a domain string (everything after the @ sign) which\ndoes not appear in the `domains` table, or for which that user lacks a\njoin record in `domain_user`, the email will be denied.\n\nIn practice, this will mean that when a new customer signs up,\nthe domain(s) included in that service agreement should be added to\nthe `domains` table. Any users which are authorized to send email\nappearing to originate from that domain should be added to the `users`\ntable, with join records linking their IDs to the IDs of the domain(s)\nthey can send for, in `domain_user`.\n\n### Whole-Email Matching\n\nAs a fallback to domain authorization, the SDA module also compares\nthe entire `sender` field with the whole-email entries associated to\nthe user. This allows an email provider to specify specific email\naddresses which may be used for outbound masquerading. This helps to\nprevent customers from pretending to be other customers, and helps to\ncreate a specification for possible scanning tools which might\notherwise react negatively to the logs of such activity.\n\nTODO: Currently, CHAPPS causes cached policy data to have an expiry\ntimer of a day. For outbound quota, this makes a great deal of sense\nbecause the quotas are expressed in emails per day. However, a day's\nworth of authorized email senders' Redis cache keys may actually cause\nquite a bit of memory usage for no particular reason. Users don't\nsend an evenly-spaced stream of email throughout the day; they send\nsome emails, often in clusters, separated by long pauses. As such,\nthe expiry time of SDA Redis caches should probably be a tunable\nparameter, in order to allow operators to tune how much RAM on their\nRedis servers ends up devoted to SDA caching. 6 or 8 hours seems like\na reasonable trade-off between Redis-RAM bloat and RDBMS latency, but\ndifferent sites are different.\n\n## Inbound Services\n\nFor inbound services, it is possible to enable either policy on a\nper-domain basis. This may be accomplished via the CLI or API.\n\n### HELO Whitelisting\n\nSometimes CHAPPS may not be the first line of defense for all email;\nit is possible that most email needs policy enforcement, but that some\nparticular relay(s) already do the same job as CHAPPS, and so whatever\nthey relay should be whitelisted. In such a case, the relay will\nprobably always fail SPF anyway, and result in a header to that\neffect.\n\nIn order to whitelist by **HELO**, specify the `helo_whitelist` option in\nthe `[CHAPPS]` section of the config file, with data about the server\nto whitelist. Due to some limitations of ConfigParser, the data\nneeds to be packed into a list on a single line. The format is as\nfollows:\n```\nhelo_whitelist=mx.example.com[:1.2.3.4][;mx2.example.com[:5.6.7.8][;...]]\n```\nTo break this down in English, provide at a minimum the HELO name used\nby the server to whitelist. In such a case, the IP will be obtained\nfrom DNS and used as if it had also been supplied. In order to supply\nthe IP address of the server, use a colon (`:`) after the name. In\norder to list more than one name (optionally with IP address), use\nsemi-colons (`;`) to separate entries.\n\nAt configuration time, the A record will be evaluated to see if it\nmatches the IP provided. The PTR of the IP, whether provided or\nobtained by a lookup, will be obtained as well, and checked to make\nsure the provided name matches. If all the elements do not match,\nCHAPPS will log an error and will not perform any whitelisting.\n\nCurrently, if the address provided is the loopback address\n\"127.0.0.1\", then no DNS lookups will be performed, and both name and\naddress values will be used without any checking or cross-checking.\nIn future versions this logic could be expanded to include all\n\"private\" IPv4 address ranges, v6 ranges, etc.\n\n## Greylisting Policy Service\n\nGreylisting is an [approach to spam\nprevention](https://en.wikipedia.org/wiki/Greylisting_(email)) based\non the tendency of spammers to emit emails without using gateways.\nSpam is typically sent _to_ one or more gateways by malware programs\nwhich amount to viral MUAs, able to connect to SMTP servers to send\nmail, but not capable of noting response codes and retrying deferred\ndeliveries. Because a large proportion of spam is (or was) sent this\nway, the simple act of deferring emails from unknown (untrusted)\nsources eliminates a large amount of spam.\n\nCHAPPS can, on a per-domain, opt-in basis, perform [RCPT or DATA\ngreylisting](https://datatracker.ietf.org/doc/html/rfc6647#section-2.4)\nat present, since it wants to use the sender's email address and IP\naddress as well as the recipient list. CHAPPS is written to expect to\nbe invoked during the RCPT phase, but should work in the DATA phase.\nThat has not been tested.\n\nEmails addressed to enforcing domains will be greylisted--that is,\ndeferred--when they are associated with source tuples which are not\nrecognized. Tracking data regarding recognized tuples and domain\noption setting is stored in Redis. Config data regarding option\nstatus is obtained from the database and cached in Redis. Domains\nwhich do not set the `greylist` bit on their records will receive all\nmail addressed to them immediately, without any greylisting.\n\nThe Greylisting module performs whitelisting at its own level, based\non a tally of successfully delivered deferred emails. That is to say,\nemails which have been deferred and then redelivered on the required\nschedule are counted in a tally per-client (by source IP address).\nWhen that tally reaches 10, further emails with matching source IPs\nare whitelisted. This threshhold may be adjusted in the\n`GreylistingPolicy` config, as `whitelist_threshold`.\n\nPlease note that in the context of comprehensive inbound email\nfiltering, SPF and greylisting have an interesting relationship which\nis not entirely straightforward, and so a special combined, inbound\nmulti-policy service has been provided which combines the features of\ngreylisting and SPF checking in a sane fashion, and provides a\nframework for adding further policies.\n\n## SPF Policy Enforcement\n\nThe [Sender Policy\nFramework](https://en.wikipedia.org/wiki/Sender_Policy_Framework) is a\ncomplicated and intricate beast, and so I will not try to describe it\nin great detail, but instead link to relevant documentation about what\nSPF is. Important to note is the fact that SPF is a framework for\nusing the DNS as the policy configuration source.\n\nThere is no provision in the RFC for the caching of SPF results in\norder to apply them to other circumstances, such as another email with\nthe same inputs. It is possible that the policy itself, i.e. the TXT\nrecord containing the SPF policy string, could change between emails.\nAs such, this module does not use Redis to cache operational data.\nRedis is used to cache the per-domain SPF-enablement option.\n\nThere is a very widely-used and well-supported implementation of the\nSPF check itself in the Python community called\n[pyspf](https://pypi.org/project/pyspf/), by Stuart Gathman and\nTerence Way. CHAPPS uses this library to get SPF check results.\n\nThe SPF policy enforcement framework included in CHAPPS makes it\npossible for an operator to specify clearly and flexibly what they\nwould like to have happen in response to any of the different SPF\ncheck results. The [SPF specification in RFC\n7208](https://datatracker.ietf.org/doc/html/rfc7208) does not address\nexactly what response to take in each case, saying that it is a site's\nprerogative to decide the fates of those emails.\n\nDomains must opt-in to SPF checking, just as with Greylisting.\nDomains which opt out of both will simply receive all email addressed\nto them without any SPF checking or Greylisting.\n\nThere is no standalone SPF service; it is part of the multipolicy\ninbound service. It would be trivial to create a standalone SPF\nservice. Since we intend to provide both options to our customers,\ncreating a standalone service is a low priority. **Please note** that\nas with Greylisting, the domain-level option will be honored even by\nstandalone handlers, meaning that in order for a domain's incoming\nemail to have SPF enforced, that option will still need to be set on\nits record.\n\n## Inbound Multi-policy Service (SPF + Greylisting)\n\n**Please note that blanket use of Greylisting is not recommended.**\n\nWhat does it mean to use both greylisting and SPF? The trivial answer\nis to pass one filter, and then pass the next filter. But which comes\nfirst?\n\nIf one greylists first, a legitimate email may be deferred for ten\nminutes, then pass SPF checking; should emails which pass SPF be\nsubject to greylisting? Conversely, a greylisted email may also come\nfrom a server which is not allowed by its SPF record, and then be\ndeferred only to be denied for an unrelated reason after ten minutes\nof taking up disk space and using up cycles needlessly.\n\nOn the other hand, if one uses an SPF filter first, in a trivial\nfashion, then emails must pass muster on the SPF check first, which\nseems right and proper to me, certainly. And if greylisting is to be\nused also, then it makes sense for emails which get `pass` from SPF to\nbe deferred. When they are sent again they will of course incur\nanother SPF check, and then they will pass greylisting, provided that\nthe SPF record they depend on has not changed in the meantime.\n\nIn the realm of SPF, there are a couple of grey areas, no pun\nintended. SPF can return `softfail` if it isn't sure enough about the\ncheck failing to indicate a hard fail. It can also return `none` or\n`neutral` which are required to be treated the same way. In such\ncases, the SPF checker is saying that the SPF record either doesn't\nexist or might as well not exist for all the good it does in this\ncase.\n\nGenerally, sites are left to determine whether to accept these emails,\nor possibly tag them and/or quarantine them. So far, this software\ndoes not address any of those possible outcomes. But we can provide\nthe interesting option of using greylisting for grey areas.\n\nBy default, CHAPPS SPF policy enforcement service uses greylisting for\nemails which receive `softfail` and `none`/`neutral` responses on\ntheir SPF checks.\n\nIf a domain is set to enforce SPF **and** Greylisting, that will cause\nCHAPPS to greylist even emails which receive `pass` from SPF, meaning\nthat any \"deliverable\" email will be deferred unless it is already\ncoming from a recognized source (tuple), when both are enabled.\n(Non-deliverable categories are: `fail`, `temperror`, `permerror`.)\n\nThe messages which accompany the various rejections and deferrals\nindicate what the reason was. In some cases, those messages indicate\nthat a message has been greylisted due to the SPF enforcement policy.\n\n## Upcoming features\n\nA mini-roadmap of upcoming changes:\n\nminor:\n\n - SDA (and other) Redis keys will have tunable expiry times\n - Look into specifying log facility and level in the config file\n\nmajor:\n\n - Possibly support multiple enforcement intervals\n - It seems inevitable that other features will also be added. There\n is some skeletal code in the repo for building email content\n filters, which are not the same as policy delegates.\n - Using Redis makes it possible to send pub/sub messages when\n certain sorts of conditions occur, such as a user making a large\n number of attempts to send mail in a short time while overquota,\n or when a user (repeatedly?) attempts to send email as being from\n a domain that user lacks authorization for.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Caching, Highly-Available Postfix Policy Service",
"version": "0.5.18",
"split_keywords": [
"postfix",
"policy",
"daemon"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "71017e6cffa8057002ecd1cb50f40a6dd35697cbae75791340442121d2827112",
"md5": "c1ea74cd45b9cf1e14ecf53744bb20e3",
"sha256": "94b0a8d4323daad1ff0d40982114a7898b11cc6ff461e468bfbcc23f6b466a6d"
},
"downloads": -1,
"filename": "chapps-0.5.18.tar.gz",
"has_sig": false,
"md5_digest": "c1ea74cd45b9cf1e14ecf53744bb20e3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 150924,
"upload_time": "2023-04-05T20:33:38",
"upload_time_iso_8601": "2023-04-05T20:33:38.974242Z",
"url": "https://files.pythonhosted.org/packages/71/01/7e6cffa8057002ecd1cb50f40a6dd35697cbae75791340442121d2827112/chapps-0.5.18.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-05 20:33:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "easydns",
"github_project": "chapps",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "chapps"
}
JSON
