postfix, integrating memcache as a lookup table using tcp_table

I have not had time to do the test “postfix memcached patch” because there are no idle servers that can be used for the experiment. instead, I’ve made a tutorial how to integrate memcached as a “postfix lookup table” with the help of tcp_table and a simple perl script.

Indeed, tcp_table “table lookup protocol” is one of the most powerful tools as well as the regexp and pcre, in my opinion. although client-server connection is not protected and and the server is not authenticated.

yes, I did a lot of experiments using tcp_table and perl scripts. it made me realize that I can do almost everything I need and make postfix as my favorite MTA.

Things required:

OK, first we create a simple perl script that allows you to handle the protocols of tcp_table. let’s call it memc.pl

#!/usr/bin/perl
use strict;
use warnings;
use Sys::Syslog qw(:DEFAULT setlogsock);
use Cache::Memcached;

# Configure the memcached server
my $memd = new Cache::Memcached {
            'servers' => [ '127.0.0.1:11211' ],
};

#
# Initalize and open syslog.
#
openlog('postfix/memcached','pid','mail');

sub qrymemc {
        return unless /^get\s+(.+)/i;
        my $kmemc = lc($1);
        chomp($kmemc);
        trim($kmemc);
        my $vmemc = $memd->get($kmemc);
        if (defined $vmemc) {
                return ($kmemc,$vmemc);
        }
        return;
}

sub trim{
        $_[0]=~s/^\s+//;
        $_[0]=~s/\s+$//;
        return;
}

#
# Autoflush standard output.
#
select STDOUT; $|++;

while (<>) {
        chomp;
        if (/^get\s+(.+)/i) {
                my $data = lc($1);
                my @res = qrymemc($data);
                syslog("info","data: %s", $data);
                if (@res) {
                        chomp(@res);
                        print "200 $res[1]\n";
                        syslog("info","Found: key = %s, value = %s", $res[0], $res[1]);
                        next;
                }
        }
        print "200 DUNNO\n";
}


Create an entry like this in master.cf

127.0.0.1:2552 inet  n       n       n       -       0      spawn
    user=nobody argv=/etc/postfix/memc.pl

Make memc.pl executable and don’t forget to reload postfix

# chmod 755 memc.pl
# postfix reload

To perform the entry/deletion “key, value” to the memcache I wrote a perl script called postmemc.pl

#!/usr/bin/perl
use strict;
use warnings;
use Cache::Memcached;
use Getopt::Long;

my $memd = new Cache::Memcached {
    'servers' => [ "localhost:11211" ],
  };

my ($k, $v, $help, $delk);

usage() if ( @ARGV < 1 or
           ! GetOptions ('help|?' => \$help, "k=s" => \$k, "v=s" => \$v, "del=s" => \$delk)
           or defined $help );

sub usage
{
        print "Unknown option: @_\n" if ( @_ );
        print "Usage: postmemc.pl --k=key --v='value'\n";
        print "       postmemc.pl --k key --v 'value'\n";
        print "       postmemc.pl --del=key\n";
        print "       postmemc.pl --del key\n";
        exit;
}

if (defined($k) && defined($v) && !defined($delk)) {
        my $val = $memd->get($k);

        if ($val)
        {
                print "Key: $k with Value: '$val' exist\n";
        } else {
                $memd->set($k, $v);
        }
} elsif (!defined($k) && !defined($v) && defined($delk)) {
        my $val = $memd->get($delk);

        if ($val)
        {
                $memd->delete($delk);
        } else {
                print "Key: $delk not' exist\n";
        }
} else {
        usage();
}

$memd->disconnect_all();

Now, let’s try to enter some data using postmemc.pl

$ ./postmemc.pl --k='spam@example.com' --v='REJECT not allowed'

And, for example we want to use it in smtpd_recipient_restrictions as check_recipient_access in main.cf

smtpd_recipient_restrictions =
   ...
   check_recipient_access tcp:[127.0.0.1]:2552,
   ...

Don’t forget to reload postfix. let’s try using postmap to query entries that we have input into memcached.

$ postmap -q spam@example.com tcp:[127.0.0.1]:2552
REJECT not allowed

yeah, another tcp_table’s greatness has been created.  :mrgreen:

7 Comments

  1. Anand

    i am getting the below error when i followed your steps

    /var/postfix/sbin/postmap -q vickykopu@shah-anand.com tcp:[127.0.0.1]:2552
    postmap: warning: read TCP map reply from [127.0.0.1]:2552: unexpected EOF (Connection reset by peer)
    postmap: warning: read TCP map reply from [127.0.0.1]:2552: unexpected EOF (Connection reset by peer)
    postmap: warning: read TCP map reply from [127.0.0.1]:2552: unexpected EOF (Connection reset by peer)
    postmap: warning: read TCP map reply from [127.0.0.1]:2552: unexpected EOF (Connection reset by peer)
    postmap: warning: read TCP map reply from [127.0.0.1]:2552: unexpected EOF (Connection reset by peer)
    postmap: warning: read TCP map reply from [127.0.0.1]:2552: unexpected EOF (Connection reset by peer)

    • looks like your perl (memc.pl) script replying something that caused postmap spits warning.
      what happen if you try telnet 127.0.0.1 2552 manually.

      $ telnet 127.0.0.1 2552
      $ get vickykopu@shah-anand.com
      
  2. Anand

    Thanks

    Now i am getting the reply; but it does not work when the message is injected to postfix. I am using qmail and postfix ; mails are injected by qmail and then forwarded to postfix to verify the sender and check if the sender is allowed to send emails.

    For this i have used tcp_table whcih querries memcache and finds out if the sender is allowed or not. This does not work when the mails are re-queued or flushed.

    I want it when the outgoing smtp daemon is invoked; please let me now how can i do this.

    Below is the output of
    /var/postfix/sbin/postmap -v -q qwnjwqjwqqwqwwqf@shah-anand.com tcp:[127.0.0.1]:2552

    Main.cf:-
    smtpd_sender_restrictions = check_sender_access tcp:[127.0.0.1]:2552

    postmap: inet_addr_local: configured 3 IPv4 addresses
    postmap: dict_open: tcp:[127.0.0.1]:2552
    postmap: dict_tcp_lookup: key qwnjwqjwqqwqwwqf@shah-anand.com
    postmap: trying… [127.0.0.1]
    postmap: dict_tcp_lookup: send: get qwnjwqjwqqwqwwqf@shah-anand.com
    postmap: dict_tcp_lookup: recv: 200 KEY NOT FOUND
    postmap: dict_tcp_lookup: found: KEY NOT FOUND
    KEY NOT FOUND

    • you should see the log what envelope sender sent by qmail to postfix. when you run postmap and replying with unexpected answer then there still a problem with perl script or the data itself on memcached server.

      the easiest way is test perl script in a first place.

      $./perlscript.pl
      get qwnjwqjwqqwqwwqf@shah-anand.com
      

      if there’s no reply then there might be problem with perl script itself, memcached data, or perlscript cannot communicate to memcached server.

      those things need to be checked berfore integrating the script to postfix spawn daemon

  3. gerald

    Thank you for this great topic.
    I learnt much from this.

    There is one more question for postfix.

    Could I change above checking from “check_recipient_access” to “sender email ip” instead?

    • sure why not?table(and the code) can be modified, either contained email address or ip address 🙂

  4. gerald

    I have tested successfully on all incoming mails using tcp_table.

    But, I fail to perform it on the outgoing mail of local machine. In fact, the mail was sent out directly without leave any warning message on mail log.

    Is there any way to achieve these kind of checking on all mail relay to this server (including same machine)?

Leave a Reply

Your email address will not be published. Required fields are marked *