Postfix, Rate Limiting Inbound Emails Using SenderScore And Memcache

I received email from someone fiew days ago, he directed me to an article about senderscore and and asked if I could make it usable. Actually, I’m not very familiar with how senderscore work. I’ve read the article and see the FAQ at https://senderscore.org/. I have found that senderscore can be queried with a format like this:

reversed.ip.address.score.senderscore.com

Ie, I want to know the score value of ip address 202.127.97.97, the format of the query would be like this:

$ dig a 97.97.127.202.score.senderscore.com +short
127.0.4.75

Look at the answers given by senderscore’s NS. last octet is the score of the ip address 202.127.97.97, which scored 75.

Excerpts from senderscore faq:

All scores are based on a scale of 0 to 100, where 0 is the worst, and 100 is the best possible score. A score represents that IP address’s rank as measured against other IP addresses, much like a percentile ranking.

Now back to the article, The authors make a perl module that can perform queries to senderscore ns, put a “reputation score” into memcache, at the same time, calculating how many times an ip address connected to our smtp.

Let’s begin, first of all download Policy::Memcache from this git repository 
Create a working directory, and extract the tarball.

$ mkdir pol-mem && cd pol-mem
$ tar --extract --file=petermblair-libemail-f73612c.tar.gz petermblair-libemail-f73612c/perl/senderscore/memcache/
$ mv petermblair-libemail-f73612c/perl/senderscore/memcache/* .

Postfix, Omar Kilani’s Memcache Patch Try-Out

I was rewrote Omar Kilani’s memcache patch couple of weeks ago. But that was not tested due to lack of time and unavailability of servers that can be used.

Now, i got chance to implement simple test. This is my configuration:
main.cf

smtpd_recipient_restrictions =
   ...
   ...
   check_recipient_access memcache:/etc/postfix/memcache.cf,
   ...
   ...

memcache.cf

servers = localhost:11211
key_format = %s

Entry on memcache

spam@example.com	REJECT	not allowed

Query using postmap

$ postmap -q "spam@example.com" memcache:/etc/postfix/memcache.cf
postmap: dict_memcache_lookup: using key_format '%s'
postmap: plmemcache_get: fetching key spam@example.com from memcache
postmap: plmemcache_get: key spam@example.com =>; REJECT not allowed
postmap: dict_memcache_lookup: spam@example.com returned REJECT not allowed
REJECT not allowed

A little bit too verbose i guess, but it can be adjusted by modifying source code.

Geo Location DNSBL Using Perl, Memcached And GeoIP

In my last article about the DNSBL and memcached, I wrote how to use memcached to store the data for the DNSBL. It led me to other new ideas to make the Geo Location DNSBL. In previous article I put data into memcached with the following format:

_prefix_ip 127.0.0.2-10
_prefix_ = _a_ or _txt_
ip = ip address to blacklist
Example:
_a_192.168.1.1	127.0.0.3
_txt_192.168.1.1	Blacklisted

Each ip address has a pair of _a_ and _txt_ record.
For This experiment records will be store in following format:

_prefix_country_code 127.0.0.2-10
_prefix_ = _a_ or _txt_
country_code = country code to blacklist
Example:
_a_ID	127.0.0.3
_txt_ID	Blacklisted

dnsbl-geo.pl Perl script

#!/usr/bin/perl
use Net::DNS::Nameserver;
use Cache::Memcached;
use Geo::IP;
use strict;
use warnings;

our $our_dnsbl = ".dnsbl.example.com";

# Configure the memcached server
my $memd = new Cache::Memcached {
            'servers' => [ '127.0.0.1:11211' ],
};

my $gi = Geo::IP->new(GEOIP_STANDARD);

#sub reverse_ipv4 {
#        my $ip = $_[0];
#        my ($a1, $a2, $a3, $a4) = split /\./, $ip;
#        my $reversed_ipv4 = join('.', $a4,$a3,$a2,$a1);
#        return $reversed_ipv4;
#}

sub reverse_ipv4 {
        my @ips = split /\./, $_[0];
        my @r;
        push @r, pop @ips while @ips;
        return join('.', @r);
}

sub strip_domain_part {
        my $strip_domain = $_[0];
        $strip_domain =~ s/$our_dnsbl//ig;
        return $strip_domain;
}

sub reply_handler {
        my ($qname, $qclass, $qtype, $peerhost,$query,$conn) = @_;
        my ($rcode, @ans, @auth, @add);
        my ($memc_a_val, $memc_txt_val);

        #print "Received query from $peerhost to ". $conn->{"sockhost"}. "\n";
        #$query->print;

        my $striped_domain = strip_domain_part($qname);
        my $reverse_striped_domain = reverse_ipv4($striped_domain);

        my $country_code = $gi->country_code_by_addr($reverse_striped_domain);

        if ($qtype eq "A" && $qname eq $striped_domain . $our_dnsbl) {
                my $vmemc_a_val = sprintf("_a_%s", lc($country_code));
                $memc_a_val = $memd->get($vmemc_a_val);
                if (defined($memc_a_val)) {
                        my ($ttl, $rdata) = (86400, $memc_a_val);
                        push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                        $rcode = "NOERROR";
                }
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }

        if ($qtype eq "TXT" && $qname eq $striped_domain . $our_dnsbl) {
                my $vmemc_txt_val = sprintf("_txt_%s", lc($country_code));
                $memc_txt_val = $memd->get($vmemc_txt_val);
                if (defined($memc_txt_val)) {
                        my ($ttl, $rdata) = (86400, $memc_txt_val);
                        push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                        $rcode = "NOERROR";
                }
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }

        # mark the answer as authoritive (by setting the 'aa' flag
        return ($rcode, \@ans, \@auth, \@add, { aa => 1 });
        $memd->disconnect_all();
}

my $ns = Net::DNS::Nameserver->new(
     LocalAddr    => "192.168.200.18",
     LocalPort    => 5353,
     ReplyHandler => \&reply_handler,
     Verbose      => 0,
) || die "couldn't create nameserver object\n";

$ns->main_loop;

DNSBL Using Perl And Memcached

DNSBL is a DNS based blackhole list, which can be used as countermeasure against unsolicited mail spam. One of the most efficient ways to block mail spam is to do it on SMTP conversation stage by denying incoming connects from spam sources, where the source machine is identified by its IP address which is checked against one or more DNSBLs on the fly.

I wrote a very simple perl script, which mimicked the way the original DNSBL works. However, I’ve created this script just respond to A and TXT records. But that’s the basic principle of how the DNSBL works.

Note: This is not recommended for real use. it’s volatile, the records will be vaporized upon server reboot. Think of this experiment just for learning purposes and fun only :mrgreen: .

DNSBL server perl scripts we called it dnsbl.pl (This script is lousy, fix it if necessary):

#!/usr/bin/perl
use Net::DNS::Nameserver;
use Net::CIDR 'addr2cidr';
use Cache::Memcached;
use strict;
use warnings;

our $our_dnsbl = ".dnsbl.example.com";

# Configure the memcached server
my $memd = new Cache::Memcached {
            'servers' => [ '127.0.0.1:11211' ],
};

sub reverse_ipv4 {
        my $ip = $_[0];
        my ($a1, $a2, $a3, $a4) = split /\./, $ip;
        my $reversed_ipv4 = join('.', $a4,$a3,$a2,$a1);
        return $reversed_ipv4;
}

#sub reverse_ipv4 {
#        my @ips = split /\./, $_[0];
#        my @r;
#        push @r, pop @ips while @ips;
#        return join('.', @r);
#}

sub strip_domain_part {
        my $strip_domain = $_[0];
        $strip_domain =~ s/$our_dnsbl//ig;
        return $strip_domain;
}

sub truncating_ipv4 {
        my $ip_addr = $_[0];
        my ($a1, $a2, $a3, $a4) = split /\./, $ip_addr;

        my $net_work_addr_ess = $ip_addr;
        my $net_work_addr = join('.', $a1,$a2,$a3);
        my $net_work = join('.', $a1,$a2);
        my $net = $a1;

        my @truncated_ipv4_lists = ($net_work_addr_ess,$net_work_addr,$net_work,$net);
        return @truncated_ipv4_lists;
}

sub test_cidr {
        my @cidr_list = Net::CIDR::addr2cidr($_[0]);
        return @cidr_list;
}

sub reply_handler {
        my ($qname, $qclass, $qtype, $peerhost,$query,$conn) = @_;
        my ($rcode, @ans, @auth, @add);
        my ($memc_a_val, $memc_txt_val, $memc_cidr_val, $memc_match_val);

        #print "Received query from $peerhost to ". $conn->{"sockhost"}. "\n";
        #$query->print;

        my $striped_domain = strip_domain_part($qname);
        my $reverse_striped_domain = reverse_ipv4($striped_domain);
        my @truncated_ipv4_lists = truncating_ipv4($reverse_striped_domain);
        my @cidr_lists = test_cidr($reverse_striped_domain);

        if ($qtype eq "A" && $qname eq $striped_domain . $our_dnsbl) {
                foreach my $truncated_ipv4_list (@truncated_ipv4_lists) {
                        $memc_a_val = $memd->get("_a_" . $truncated_ipv4_list);
                        last if(defined($memc_a_val));
                }

                foreach my $cidr_list (@cidr_lists) {
                        $memc_cidr_val = $memd->get("_a_" . $cidr_list);
                        last if(defined($memc_cidr_val));
                }

                for(;;) {
                        if (defined($memc_a_val)) {
                                $memc_match_val = $memc_a_val;
                                last;
                        }
                        if (defined($memc_cidr_val)) {
                                $memc_match_val = $memc_cidr_val;
                                last;
                        }
                }

                my ($ttl, $rdata) = (86400, $memc_match_val);
                push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                $rcode = "NOERROR";
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }

        if ($qtype eq "TXT" && $qname eq $striped_domain . $our_dnsbl) {
                foreach my $truncated_ipv4_list (@truncated_ipv4_lists) {
                        $memc_txt_val = $memd->get("_txt_" . $truncated_ipv4_list);
                        last if(defined($memc_txt_val));
                }

                foreach my $cidr_list (@cidr_lists) {
                        $memc_cidr_val = $memd->get("_txt_" . $cidr_list);
                        last if(defined($memc_cidr_val));
                }

                for(;;) {
                        if (defined($memc_txt_val)) {
                                $memc_match_val = $memc_txt_val;
                                last;
                        }
                        if (defined($memc_cidr_val)) {
                                $memc_match_val = $memc_cidr_val;
                                last;
                        }
                }

                my ($ttl, $rdata) = (86400, $memc_match_val);
                push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                $rcode = "NOERROR";
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }
        # mark the answer as authoritive (by setting the 'aa' flag
        return ($rcode, \@ans, \@auth, \@add, { aa => 1 });
        $memd->disconnect_all();
}

my $ns = Net::DNS::Nameserver->new(
     LocalAddr    => "192.168.200.18",
     LocalPort    => 5353,
     ReplyHandler => \&reply_handler,
     Verbose      => 0,
) || die "couldn't create nameserver object\n";

$ns->main_loop;

postfix, integrating memcache as a lookup table using tcp_table

I have not had time to do the test “postfix memcached patch” because there are no idle servers that can be used for the experiment. instead, I’ve made a tutorial how to integrate memcached as a “postfix lookup table” with the help of tcp_table and a simple perl script.

Indeed, tcp_table “table lookup protocol” is one of the most powerful tools as well as the regexp and pcre, in my opinion. although client-server connection is not protected and and the server is not authenticated.

yes, I did a lot of experiments using tcp_table and perl scripts. it made me realize that I can do almost everything I need and make postfix as my favorite MTA.

Things required:

OK, first we create a simple perl script that allows you to handle the protocols of tcp_table. let’s call it memc.pl

#!/usr/bin/perl
use strict;
use warnings;
use Sys::Syslog qw(:DEFAULT setlogsock);
use Cache::Memcached;

# Configure the memcached server
my $memd = new Cache::Memcached {
            'servers' => [ '127.0.0.1:11211' ],
};

#
# Initalize and open syslog.
#
openlog('postfix/memcached','pid','mail');

sub qrymemc {
        return unless /^get\s+(.+)/i;
        my $kmemc = lc($1);
        chomp($kmemc);
        trim($kmemc);
        my $vmemc = $memd->get($kmemc);
        if (defined $vmemc) {
                return ($kmemc,$vmemc);
        }
        return;
}

sub trim{
        $_[0]=~s/^\s+//;
        $_[0]=~s/\s+$//;
        return;
}

#
# Autoflush standard output.
#
select STDOUT; $|++;

while (<>) {
        chomp;
        if (/^get\s+(.+)/i) {
                my $data = lc($1);
                my @res = qrymemc($data);
                syslog("info","data: %s", $data);
                if (@res) {
                        chomp(@res);
                        print "200 $res[1]\n";
                        syslog("info","Found: key = %s, value = %s", $res[0], $res[1]);
                        next;
                }
        }
        print "200 DUNNO\n";
}

Postfix, old memcache lookup table patch

Yesterday, I was idly fiddling with the old patch postfix “memcached lookup table” created by Omar Kilani . unfortunately, patches can only be used for old postfix distributions (2.1.x – Released 2005-04-01, 2.2.x – Released 2005-04-01).

I rewrote the patch (code was not modified) so it can be applied against last postfix-2.9-20110706 snapshot.
This patches required memcached and libmemcache .

I was successfully compiled it, but not test it yet whether it will work or not. so it’s not recommended for use on production servers.

$ wget ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/postfix-2.9-20110706.tar.gz
$ tar xzf postfix-2.9-20110706.tar.gz
$ cd postfix-2.9-20110706

Postfix memcache patch can be download here:
[download#41]

Patch postfix source distribution

$ patch -p1 < ../postfix-2.9-20110706-memcache.patch
patching file html/DATABASE_README.html
patching file html/Makefile.in
patching file html/MEMCACHE_README.html
patching file html/memcache_table.5.html
patching file man/Makefile.in
patching file man/man5/memcache_table.5
patching file proto/DATABASE_README.html
patching file proto/Makefile.in
patching file proto/MEMCACHE_README.html
patching file proto/memcache_table
patching file README_FILES/AAAREADME
patching file README_FILES/DATABASE_README
patching file README_FILES/MEMCACHE_README
patching file src/global/dict_memcache.c
patching file src/global/dict_memcache.h
patching file src/global/mail_dict.c
patching file src/global/Makefile.in

Memcached Replication On Server Clusters

When I was looking for ways to replicate the contents of memcached for high-availability performance, I found this memcached-repcached application. that has the ability to replicate the contents of one memcached to another.

Repcached key features

  • Multi master replication.
  • Asynchronous data repliacation.
  • Support all memcached command (set, add, delete, incr/decr, flush_all, cas)

People probably already know about memcached . It’s a robust, high performance key-value based memory object cache interface. but unfortunately, lack the ability to create redundancy and replication in memcached server clusters. although replication could be done at the application level. However, it all depends on each individual’s taste.

This is a quick and dirty experiment I have tried using memcached-repcached application on 2 servers.

Download memcached-repcached from repcached.lab.klab.org
Or you can download source rpm version from here
[download#33]

On the first server and second server extract memcached-1.2.8-repcached-2.2.tar.gz, compile with –enable-replication option when configure.

$ tar xvzf memcached-1.2.8-repcached-2.2.tar.gz
$ cd memcached-1.2.8-repcached-2.2
$ ./configure --enable-replication
$ make

Monitoring And Debugging Memcached Server Using phpMemCachedAdmin

I have found a very cool program for monitoring and debugging Memcached server, it’s called  phpMemcachedAdmin. This program allows to see in real-time (top-like) or from the start of the server, stats for get, set, delete, increment, decrement, evictions, reclaimed, cas command, as well as server stats (network, items, server version) with googlecharts and server internal configuration You can go further to see each server slabs, occupation, memory wasted and items (key & value).

Another part can execute commands to any memcached server : get, set, delete, flush_all, as well as execute any commands (like stats) with telnet To extract these informations, phpMemCacheAdmin uses, as you wish, direct communication with server, PECL Memcache or PECL Memcached API.

Explanation above is an excerpt from the website. This is a simple guide how do I install phpMemCachedAdmin.

Download phpMemCachedAdmin tarball

# wget http://phpmemcacheadmin.googlecode.com/files/phpMemcachedAdmin-1.2.1-r233.tar.gz

Since we will install in /var/www/html/memcached/ directory, run this command:

# mkdir -p /var/www/html/memcached
# tar -xvzf phpMemcachedAdmin-1.2.1-r233.tar.gz -C /var/www/html/memcached/
# chmod 0777 /var/www/html/memcached/Config/Memcache.php

Apache Configuration

( i’m using apache as reverse proxy backend )

# cd /etc/httpd/conf.d
# vi memcached.conf

Sharing PHP Session In web Clusters Using Memcache

If you have multiple load balanced webservers all serving the same site, sessions should be shared among thosememcache logo servers, and not reside on each server individually. Because we don’t know where user gets load-balanced to which backend server. A common way solving this problems are by using custom class that overrides default behavior and stores sessions in a MySQL database. All webservers in clusters connect to the same database. However, the main problem that must be taken into consideration when using a database is the bottleneck.

Example loadbalancing backend using nginx

http {
  upstream backend {
    server 192.168.1.1:8000 weight=3;
    server 192.168.1.2:8001;
    server 192.168.1.3:8002;
    server 192.168.1.4:8003;
  }

  server {
    listen 80;
    server_name www.example.com;
    location / {
      proxy_pass http://backend;
    }
  }
}

How To Graph Nginx Statistics

Sometimes, we prefer to view statistics in graphical form rather than numerical values​​, which is not too attractive to be displayed. Nginx supports stub_status directive module, which we can use to print:

  • Active connections
  • Server accepts handled requests
  • Reading
  • Writing
  • Waiting

For example:

Active connections: 6
server accepts handled requests
 15561 15561 26602
Reading: 4 Writing: 2 Waiting: 0

However, This module is not compiled by default and must be specified using this argument when compiling nginx.

--with-http_stub_status_module

First, to get statistics like the above example, you should modify nginx config file and add location directive this

server {
....
....
	location /nginx_status {
		stub_status on;
		access_log   off;
		allow 1.2.3.4;
		allow 5.6.7.8;
		deny all;
	}
}

perl scripts used to generate statistical images can be downloaded here:

http://kovyrin.net/files/mrtg/rrd_nginx.pl.txt

Rename rrd_nginx.pl.txt to rrd_nginx.pl and make it executable