mtpolicyd - Getting started

Installation

GET STARTED WITH BASIC MTPOLICYD INSTALLATION

INSTALL MEMCACHED

memcached is required for mtpolicyd. A package of memcached should come with your os distribution. On Debian based distributions it can be installed by:

  apt-get install memcached

Check /etc/default/memcached if the service is enabled:

  ENABLE_MEMCACHED=yes

Start the memcached service

  /etc/init.d/memcached start

INSTALL MTPOLICYD

FROM PACKAGE

Packages for mtpolicyd are included in the Debian package repositories.

FROM SOURCE/CPAN

Since mtpolicyd source is shipped as a perl/CPAN package it could be installed from CPAN. To install the Mail::Mtpolicyd package with all dependencies required make sure you have installed cpanminus:

  apt-get install cpanminus

Then install the Mail::Mtpolicyd distribution with:

  cpanm Mail::MtPolicyd

It is recommended to create an system user and group for the daemon.

You can get a default configuration file etc/mtpolicyd.conf from the tarball.

The init scripts for the debian packages are located at debian/mtpolicyd.init and for redhat systems at rpm/mtpolicyd.init within the tarball.

TEST MTPOLICYD

Now the daemon should be up:

  $ ps -u mtpolicyd f
    PID TTY      STAT   TIME COMMAND
   2566 ?        Ss     0:12 /usr/bin/mtpolicyd (master)
   2731 ?        S      0:28  \_ /usr/bin/mtpolicyd (idle)
  19464 ?        S      0:26  \_ /usr/bin/mtpolicyd (idle)
  28858 ?        S      0:26  \_ /usr/bin/mtpolicyd (idle)
  32372 ?        S      0:24  \_ /usr/bin/mtpolicyd (idle)

And it should be listening on localhost:12345:

  $ netstat -aenpt | grep :12345
  tcp        0      0 127.0.0.1:12345         0.0.0.0:*               LISTEN      0          17333578    -

Now test it with a simple query:

  $ policyd-client -h localhost:12345

Paste the following request to the command:

  reverse_client_name=smtp.google.com
  sender=bob@gmail.com
  client_address=192.168.1.1
  recipient=ich@markusbenning.de
  helo_name=smtp.google.com

Terminate the request by a blank line. Just press enter.

The mtpolicyd should respond with a action like:

  PREPEND X-MtScore: NO score

ADD A MTPOLICYD QUERY TO YOUR POSTFIX SMTPD

Open you postfix main.cf configuration file in a text editor. It should be located below /etc/postfix.

Add a 'check_policyd_service inet:127.0.0.1:12345' check to your smtpd_recipient_restrictions. It should look like this one:

  smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, check_policy_service inet:127.0.0.1:12345

Now restart postfix. Now follow your maillog as new mails arrive. There should be a mtpolicyd line for every query.

CONGRATULATIONS

Your mtpolicyd is now configured and running with the default configuration.

You may now want to continue with reading Mail::MtPolicyd::Cookbook::DefaultConfig which explains what the default configuration does.

BasicModule

How to write your own mtpolicyd plugin

mtpolicyd makes use of Moose. If you're not yet familiar with Moose you should start reading the Moose::Intro first.

Basic skeleton of a mtpolicyd plugin

A plugin in mtpolicyd is basically a class which inherits from Mail::MtPolicyd::Plugin and is located below the Mail::MtPolicyd::Plugin:: namespace:

  package Mail::MtPolicyd::Plugin::HelloWorld;

  use Moose;
  use namespace::autoclean;

  # VERSION
  # ABSTRACT: a mtpolicyd plugin which just returns a hello world reject

  extends 'Mail::MtPolicyd::Plugin';

  use Mail::MtPolicyd::Plugin::Result;

  sub run {
    my ( $self, $r ) = @_;

    return Mail::MtPolicyd::Plugin::Result->new(
      action => 'reject Hello World!',
      abort => 1,
    );
  }

  __PACKAGE__->meta->make_immutable;

  1;

Every plugin must implement a run() method. mtpolicyd will call run() every time your module is called from the configuration to process a request. A Mail::MtPolicyd::Request object containing the current request is passed to the method. The run() method must return undef or a <Mail::MtPolicyd::Plugin::Result> object. If undef is return mtpolicyd will continue with the next plugin. If a result is returned mtpolicyd will push the result to the list of results and abort processing the request if abort is set.

After you placed the module with your lib search path you should be able to use the plugin within mtpolicyd.conf:

  <Plugin hello-world>
    module = "HelloWorld"
  </Plugin>

For now our plugin will just return an "reject Hello World!" action to the MTA.

Adding configuration options

All options defined in the configuration file will be passed to the object constructor new() when creating an object of your plugin class.

The parameter "module" is not passed to the object constructor because it contains the name of your class.

You can defined configuration parameters by adding attributes to your class.

You're class already inherits 3 attributes from the Plugin base class:

  • name (required)

    Which contains the name of your <Plugin> section.

  • log_level (default: 4)

    Which contains the level used when your plugin calls $self->log( $r, '...');.

  • on_error (default: undef)

    Tells mtpolicyd what to do when the plugin dies.

    If set to "continue" mtpolicyd will continue processing and just leaves a line in the log.

Add a new attribute to your plugin class:

  has 'text' => ( is => 'rw', isa => 'Str', default => 'Hello World!');

Return this string instead of the hard coded string:

      action => 'reject '.$self->text,

The string is now configurable from the configuration:

  <Plugin hello-world>
    module = "HelloWorld"
    text = "Hello Universe!"
  </Plugin>

ExtendedModule

Extending your mtpolicyd plugin

How to achieve common task within your plugin.

Logging

You can output log messages to the mail.log from within your plugin by calling:

  $self->log( $r, '<log message>' );

The log() method is inherited from the Plugin base class.

The default log_level used is 4.

To debug your plugin you can overwrite the log_level in your plugins configuration:

  <Plugin hello-world>
    module = "HelloWorld"
    log_level = 2
  </Plugin>

This will cause your plugin to log with an higher log_level of 2.

Caching lookup results

If your plugin is called from the smtpd_recipient_restrictions if will be called once for every recipient. If your plugin does an lookup (dns, database, ...) should cache the result.

The Mail::MtPolicyd::Request object implements the method do_cached() to achieve this:

  my ( $ip_result, $info ) = $r->do_cached('rbl-'.$self->name.'-result',
    sub { $self->_rbl->check( $ip ) } );

The first parameter is the key in the session to store the cached result. The second parameter is a function reference.

It will check if there's already an result stored in the given key within the session. In this case it will return the cached result as an array. If there is no result it will execute the code reference, store the result within the session and will also return an array containing the return values of the result.

Doing things only once per mail

If your plugin is called from the smtpd_recipient_restrictions if will be called once for every recipient but some tasks should only be performed once per mail.

The Mail::MtPolicyd::Request object implements the method is_already_done() to achieve this:

  if( defined $self->score && ! $r->is_already_done( $self->name.'-score' ) ) {
    $self->add_score($r, $self->name => $self->score);
  }

The method takes the key in the session in which the flag is stored.

The example above will add a new score to the scoring, but only once per mail since the session is persisted across different checks.

Use scoring

To add scoring to your plugin your plugin needs to consume the role Mail::MtPolicyd::Plugin::Role::Scoring.

This will add the method add_score( $r, $key, $value ) to your plugin class.

The $key is a name for the score you'll see when you display the detailed scores. eg. with the AddScoreHeader or ScoreAction plugin.

The $value is positive or negative number. In most cases you want to make this value configurable.

It is also recommended that you check that you add an score only once. See is_already_done() method above.

Here is an example:

  with 'Mail::MtPolicyd::Plugin::Role::Scoring';

  has 'score' => ( is => 'rw', isa => 'Maybe[Num]' );

And somewhere in your run() method:

  if( defined $self->score && ! $r->is_already_done( $self->name.'-score' ) ) {
    $self->add_score( $r, $self->name => $self->score );
  }

Make a configuration value user-configurable

To add user configurable parameters to your plugin the class must consume the Mail::MtPolicyd::Plugin::Role::UserConfig role.

  with 'Mail::MtPolicyd::Plugin::Role::UserConfig' => {
    'uc_attributes' => [ 'enabled', 'mode' ],
  };

The regular attributes:

  has 'enabled' => ( is => 'rw', isa => 'Str', default => 'on' );
  has 'mode' => ( is => 'rw', isa => 'Str', default => 'reject' );

The UserConfig role adds the get_uc( $session, $param ) method to your class. To retrieve the user-configurable values for this attributes use:

  my $session = $r->session;
  my $mode = $self->get_uc( $session, 'mode' );
  my $enabled = $self->get_uc( $session, 'enabled' );

Per user configuration in mtpolicyd works like this:

  • Retrieve configuration values and store them in the session

    A plugin like SqlUserConfig retrieves configuration values and stores them in the current session.

    For example it may set the following key value:

      hello_world_enabled = off
  • A Plugin with user configurable parameters

    Our HelloWorld plugin may be configured like this:

      <Plugin hello-world>
      	module = "HelloWorld"
    	enabled = on
    	uc_enabled = "hello_world_enabled"
      </Plugin>

    If the key "hello_world_enabled" is defined in the session it will use its value for $mode. If it is not defined it will fall back to value of the "enabled" attribute.

Set a mail header

The Mail::MtPolicyd::Plugin::Result object has an extra constructor for returning a PREPEND action for setting a header:

  Mail::MtPolicyd::Plugin::Result->new_header_once( $is_already_done, $header_name, $value );

It could be used like this:

  return Mail::MtPolicyd::Plugin::Result->new_header_once(
    $r->is_already_done( $self->name.'-tag' ),
    $header_name, $value );

Adding periodically scheduled tasks

When mtpolicyd is called with the option --cron <task1,task2,...> it will execute all plugins that implement a cron() function.

The function is expected to take the following parameters:

  $plugin->cron( $server, @tasks );

By default mtpolicyd ships with a crontab that will execute the tasks hourly,daily,weekly and monthly.

A plugin that implements a weekly task may look like this:

  sub cron {
    my $self = shift;
    my $server = shift;
    
    if( grep { $_ eq 'weekly' } @_ ) {
      # do some weekly tasks
      $server->log(3, 'i am a weekly task');
    }
  }

The $server object could be used for logging.

To see the output on the command line you may call mtpolicyd like this:

  mtpolicyd -f -l 4 --cron=weekly

HowtoAccountingQuota

SMTP level accounting and quotas with mtpolicyd

The mtpolicyd could be used to implement a smtp level accounting and quota system.

This guide explains how to setup accounting and quotas based on the sender ip on a monthly base and configurable quota limits.

The how to expects that mtpolicyd is already installed, working and assumes a MySQL database is used to hold accounting data and quota configuration.

Set up Accounting

The accounting and quota checks should be implemented in postfix smtpd_end_of_data_restrictions. If you're already using mtpolicyd for other check it may be necessary to setup a second virtual host for the accounting/quota configuration. Otherwise you can use the default port 12345 virual host.

Setup a second virtual host

First tell mtpolicyd to also listen on an addition port. In the global configuration add the new port to the port option:

  port="127.0.0.1:12345,127.0.0.1:12346"

Then add a new virtual host at the end of the configuration file:

  <VirtualHost 12346>
    name="accounting"
    # TODO: add plugins...
  </VirtualHost>
Configure the Accounting plugin

Now add the Accounting plugin to your virtual host:

  <Plugin AcctIP>
    module = "Accounting"
    fields = "client_address"
    # time_pattern = "%Y-%m"
    # table_prefix = "acct_"
  </Plugin>

And the restart mtpolicyd to reload the configuration.

The plugin will create a table for every field listed in "fields". By default the table prefix is acct_ so the table name will be acct_client_address in our example. The plugin will create a row within this table for every client_address and expanded time_pattern:

  mysql> select * from acct_client_address;
  +----+-------------------+---------+-------+------------+---------+-----------+
  | id | key               | time    | count | count_rcpt | size    | size_rcpt |
  +----+-------------------+---------+-------+------------+---------+-----------+
  |  1 | 2604:8d00:0:1::3  | 2015-01 |    18 |         18 |   95559 |     95559 |
  |  2 | 2604:8d00:0:1::4  | 2015-01 |    21 |         21 |   99818 |     99818 |
  ...
  +----+-------------------+---------+-------+------------+---------+-----------+
Activate the check in postfix

To active the check add the policyd to your smtpd_end_of_data_restrictions in main.cf:

  smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:12346

If you have multiple smtpd process configured in a smtp-filter setup make sure only one smtpd is doing accounting/quota checks. Deactivate the restrictions by adding the following option the the re-inject smtpd processes in master.cf:

  -o smtpd_end_of_data_restrictions=

Setup quota limits

To limit the number of messages a client_address is allowed to send add the following Quota plugin to your virtual host configuration before the Accounting plugin:

  <Plugin QuotaIP>
    module = "Quota"
    field = "client_address"
    metric = "count"
    threshold = 1000
    action = "defer you exceeded your monthly limit, please insert coin"
    # time_pattern = "%Y-%m"
    # table_prefix = "acct_"
  </Plugin>

Using per client_address quota limits

Create the following table structure in your MySQL database:

  CREATE TABLE `relay_policies` (
    `id` int(11) NOT NULL auto_increment,
    `desc` VARCHAR(64) NOT NULL,
    `config` TEXT NOT NULL,
    PRIMARY KEY  (`id`)
  ) ENGINE=InnoDB;

  INSERT INTO relay_policies VALUES(1, 'standard relay host', '{"quota_count":"10000"}');
  INSERT INTO relay_policies VALUES(2, 'premium relay host', '{"quota_count":"100000"}');

  CREATE TABLE `relay_hosts` (
    `id` int(11) NOT NULL auto_increment,
    `client_address` VARCHAR(64) NOT NULL,
    `relay_policy` int(11) NOT NULL,
    PRIMARY KEY  (`id`),
    KEY `relay_policy` (`relay_policy`),
    CONSTRAINT `relay_hosts_ibfk_1` FOREIGN KEY (`relay_policy`) REFERENCES `relay_policies` (`id`)
  ) ENGINE=InnoDB;

  INSERT INTO relay_hosts VALUES(NULL, '2604:8d00:0:1::3', 1);
  INSERT INTO relay_hosts VALUES(NULL, '2604:8d00:0:1::4', 2);

You can use the following SELECT statement to retrieve the configuration for a relay_host:

  mysql> SELECT p.config FROM relay_policies p JOIN relay_hosts h ON (h.relay_policy = p.id) WHERE h.client_address = '2604:8d00:0:1::4';
  +--------------------------+
  | config                   |
  +--------------------------+
  | {"quota_count":"100000"} |
  +--------------------------+
  1 row in set (0.00 sec)

To load the (JSON) configuration into the mtpolicyd session variables use the SqlUserConfig plugin and this SQL statement:

  <Plugin QuotaPolicyConfig>
    module = "SqlUserConfig"
    sql_query = "SELECT p.config FROM relay_policies p JOIN relay_hosts h ON (h.relay_policy = p.id) WHERE h.client_address=?"
    field = "client_address"
  </Plugin>

This plugin must be added before your Accounting and Quota plugins.

To use the quota_count value instead of the default threshold adjust your Quota plugin configuration:

  <Plugin QuotaIP>
    module = "Quota"
    field = "client_address"
    metric = "count"
    threshold = 1000
    uc_threshold = "quota_count"
    action = "defer you exceeded your monthly limit, please insert coin"
    # time_pattern = "%Y-%m"
    # table_prefix = "acct_"
  </Plugin>

If the session variable quota_count is defined it will be used as threshold instead of the value configured in mtpolicyd.conf.