Postfix Changing Outgoing IP By Time Interval Using TCP_TABLE And Perl

Someone asked me if i can make a perl scripts that can change the ip address based on time interval, say he want ip address 1.2.3.4 used within one hour, if done next ip address will be used within next one hour..and so on. when it came to highest number of ip address in array, they will be reset back to the start. first i suggest him to look at the articles i wrote. But then i decide to write Perl script which was made for the purposes mention above.

here we are..

Postfix section:

master.cf
127.0.0.1:2527 inet  n       n       n       -       0      spawn
          user=nobody argv=/etc/postfix/ip_by_time.pl

ip1  unix -       -       n       -       -       smtp
          -o syslog_name=postfix-ip1
          -o smtp_helo_name=smtp1.example.com
          -o smtp_bind_address=1.2.3.1

ip2  unix -       -       n       -       -       smtp
          -o syslog_name=postfix-ip2
          -o smtp_helo_name=smtp2.example.com
          -o smtp_bind_address=1.2.3.2

ip3  unix -       -       n       -       -       smtp
          -o syslog_name=postfix-ip3
          -o smtp_helo_name=smtp3.example.com
          -o smtp_bind_address=1.2.3.3

ip4  unix -       -       n       -       -       smtp
          -o syslog_name=postfix-ip4
          -o smtp_helo_name=smtp4.example.com
          -o smtp_bind_address=1.2.3.4
....
....

main.cf

transport_maps = tcp:[127.0.0.1]:2527
127.0.0.1:2527_time_limit = 3600s

Geo Location DNSBL Using Perl, Memcached And GeoIP

In my last article about the DNSBL and memcached, I wrote how to use memcached to store the data for the DNSBL. It led me to other new ideas to make the Geo Location DNSBL. In previous article I put data into memcached with the following format:

_prefix_ip 127.0.0.2-10
_prefix_ = _a_ or _txt_
ip = ip address to blacklist
Example:
_a_192.168.1.1	127.0.0.3
_txt_192.168.1.1	Blacklisted

Each ip address has a pair of _a_ and _txt_ record.
For This experiment records will be store in following format:

_prefix_country_code 127.0.0.2-10
_prefix_ = _a_ or _txt_
country_code = country code to blacklist
Example:
_a_ID	127.0.0.3
_txt_ID	Blacklisted

dnsbl-geo.pl Perl script

#!/usr/bin/perl
use Net::DNS::Nameserver;
use Cache::Memcached;
use Geo::IP;
use strict;
use warnings;

our $our_dnsbl = ".dnsbl.example.com";

# Configure the memcached server
my $memd = new Cache::Memcached {
            'servers' => [ '127.0.0.1:11211' ],
};

my $gi = Geo::IP->new(GEOIP_STANDARD);

#sub reverse_ipv4 {
#        my $ip = $_[0];
#        my ($a1, $a2, $a3, $a4) = split /\./, $ip;
#        my $reversed_ipv4 = join('.', $a4,$a3,$a2,$a1);
#        return $reversed_ipv4;
#}

sub reverse_ipv4 {
        my @ips = split /\./, $_[0];
        my @r;
        push @r, pop @ips while @ips;
        return join('.', @r);
}

sub strip_domain_part {
        my $strip_domain = $_[0];
        $strip_domain =~ s/$our_dnsbl//ig;
        return $strip_domain;
}

sub reply_handler {
        my ($qname, $qclass, $qtype, $peerhost,$query,$conn) = @_;
        my ($rcode, @ans, @auth, @add);
        my ($memc_a_val, $memc_txt_val);

        #print "Received query from $peerhost to ". $conn->{"sockhost"}. "\n";
        #$query->print;

        my $striped_domain = strip_domain_part($qname);
        my $reverse_striped_domain = reverse_ipv4($striped_domain);

        my $country_code = $gi->country_code_by_addr($reverse_striped_domain);

        if ($qtype eq "A" && $qname eq $striped_domain . $our_dnsbl) {
                my $vmemc_a_val = sprintf("_a_%s", lc($country_code));
                $memc_a_val = $memd->get($vmemc_a_val);
                if (defined($memc_a_val)) {
                        my ($ttl, $rdata) = (86400, $memc_a_val);
                        push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                        $rcode = "NOERROR";
                }
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }

        if ($qtype eq "TXT" && $qname eq $striped_domain . $our_dnsbl) {
                my $vmemc_txt_val = sprintf("_txt_%s", lc($country_code));
                $memc_txt_val = $memd->get($vmemc_txt_val);
                if (defined($memc_txt_val)) {
                        my ($ttl, $rdata) = (86400, $memc_txt_val);
                        push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                        $rcode = "NOERROR";
                }
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }

        # mark the answer as authoritive (by setting the 'aa' flag
        return ($rcode, \@ans, \@auth, \@add, { aa => 1 });
        $memd->disconnect_all();
}

my $ns = Net::DNS::Nameserver->new(
     LocalAddr    => "192.168.200.18",
     LocalPort    => 5353,
     ReplyHandler => \&reply_handler,
     Verbose      => 0,
) || die "couldn't create nameserver object\n";

$ns->main_loop;

DNSBL Using Perl And Memcached

DNSBL is a DNS based blackhole list, which can be used as countermeasure against unsolicited mail spam. One of the most efficient ways to block mail spam is to do it on SMTP conversation stage by denying incoming connects from spam sources, where the source machine is identified by its IP address which is checked against one or more DNSBLs on the fly.

I wrote a very simple perl script, which mimicked the way the original DNSBL works. However, I’ve created this script just respond to A and TXT records. But that’s the basic principle of how the DNSBL works.

Note: This is not recommended for real use. it’s volatile, the records will be vaporized upon server reboot. Think of this experiment just for learning purposes and fun only :mrgreen: .

DNSBL server perl scripts we called it dnsbl.pl (This script is lousy, fix it if necessary):

#!/usr/bin/perl
use Net::DNS::Nameserver;
use Net::CIDR 'addr2cidr';
use Cache::Memcached;
use strict;
use warnings;

our $our_dnsbl = ".dnsbl.example.com";

# Configure the memcached server
my $memd = new Cache::Memcached {
            'servers' => [ '127.0.0.1:11211' ],
};

sub reverse_ipv4 {
        my $ip = $_[0];
        my ($a1, $a2, $a3, $a4) = split /\./, $ip;
        my $reversed_ipv4 = join('.', $a4,$a3,$a2,$a1);
        return $reversed_ipv4;
}

#sub reverse_ipv4 {
#        my @ips = split /\./, $_[0];
#        my @r;
#        push @r, pop @ips while @ips;
#        return join('.', @r);
#}

sub strip_domain_part {
        my $strip_domain = $_[0];
        $strip_domain =~ s/$our_dnsbl//ig;
        return $strip_domain;
}

sub truncating_ipv4 {
        my $ip_addr = $_[0];
        my ($a1, $a2, $a3, $a4) = split /\./, $ip_addr;

        my $net_work_addr_ess = $ip_addr;
        my $net_work_addr = join('.', $a1,$a2,$a3);
        my $net_work = join('.', $a1,$a2);
        my $net = $a1;

        my @truncated_ipv4_lists = ($net_work_addr_ess,$net_work_addr,$net_work,$net);
        return @truncated_ipv4_lists;
}

sub test_cidr {
        my @cidr_list = Net::CIDR::addr2cidr($_[0]);
        return @cidr_list;
}

sub reply_handler {
        my ($qname, $qclass, $qtype, $peerhost,$query,$conn) = @_;
        my ($rcode, @ans, @auth, @add);
        my ($memc_a_val, $memc_txt_val, $memc_cidr_val, $memc_match_val);

        #print "Received query from $peerhost to ". $conn->{"sockhost"}. "\n";
        #$query->print;

        my $striped_domain = strip_domain_part($qname);
        my $reverse_striped_domain = reverse_ipv4($striped_domain);
        my @truncated_ipv4_lists = truncating_ipv4($reverse_striped_domain);
        my @cidr_lists = test_cidr($reverse_striped_domain);

        if ($qtype eq "A" && $qname eq $striped_domain . $our_dnsbl) {
                foreach my $truncated_ipv4_list (@truncated_ipv4_lists) {
                        $memc_a_val = $memd->get("_a_" . $truncated_ipv4_list);
                        last if(defined($memc_a_val));
                }

                foreach my $cidr_list (@cidr_lists) {
                        $memc_cidr_val = $memd->get("_a_" . $cidr_list);
                        last if(defined($memc_cidr_val));
                }

                for(;;) {
                        if (defined($memc_a_val)) {
                                $memc_match_val = $memc_a_val;
                                last;
                        }
                        if (defined($memc_cidr_val)) {
                                $memc_match_val = $memc_cidr_val;
                                last;
                        }
                }

                my ($ttl, $rdata) = (86400, $memc_match_val);
                push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                $rcode = "NOERROR";
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }

        if ($qtype eq "TXT" && $qname eq $striped_domain . $our_dnsbl) {
                foreach my $truncated_ipv4_list (@truncated_ipv4_lists) {
                        $memc_txt_val = $memd->get("_txt_" . $truncated_ipv4_list);
                        last if(defined($memc_txt_val));
                }

                foreach my $cidr_list (@cidr_lists) {
                        $memc_cidr_val = $memd->get("_txt_" . $cidr_list);
                        last if(defined($memc_cidr_val));
                }

                for(;;) {
                        if (defined($memc_txt_val)) {
                                $memc_match_val = $memc_txt_val;
                                last;
                        }
                        if (defined($memc_cidr_val)) {
                                $memc_match_val = $memc_cidr_val;
                                last;
                        }
                }

                my ($ttl, $rdata) = (86400, $memc_match_val);
                push @ans, Net::DNS::RR->new("$qname $ttl $qclass $qtype $rdata");
                $rcode = "NOERROR";
        } elsif ( $qname eq "dnsbl.example.com" ) {
                $rcode = "NOERROR";

        } else {
                $rcode = "NXDOMAIN";
        }
        # mark the answer as authoritive (by setting the 'aa' flag
        return ($rcode, \@ans, \@auth, \@add, { aa => 1 });
        $memd->disconnect_all();
}

my $ns = Net::DNS::Nameserver->new(
     LocalAddr    => "192.168.200.18",
     LocalPort    => 5353,
     ReplyHandler => \&reply_handler,
     Verbose      => 0,
) || die "couldn't create nameserver object\n";

$ns->main_loop;

Measure Response Time Of SMTP Connections Using Perl

Normally, to check if my server smtp connection alive, I just do telnet to port 25 from my workstation. if the smtp banner displayed, it means that the connection to the smtp server is good. I have done this for years.  😆

$ telnet smtp.example.com 25
Trying xxx.xxx.xx.xxx...
Connected to xxx.xxx.xx.x.
Escape character is '^]'.
220 smtp.example.com ESMTP Postfix
ehlo host.example.com
250-smtp.example.com
250-PIPELINING
250-SIZE 52428800
250-ETRN
250-STARTTLS
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
quit
221 2.0.0 Bye
Connection closed by foreign host.

Although many tools that are deliberately created for this purpose, still, I prefer just using telnet to port 25. I have made a simple perl script, with the intention that the things I do for years manually can be done automatically.

Modules required:

How To Graph Nginx Statistics

Sometimes, we prefer to view statistics in graphical form rather than numerical values​​, which is not too attractive to be displayed. Nginx supports stub_status directive module, which we can use to print:

  • Active connections
  • Server accepts handled requests
  • Reading
  • Writing
  • Waiting

For example:

Active connections: 6
server accepts handled requests
 15561 15561 26602
Reading: 4 Writing: 2 Waiting: 0

However, This module is not compiled by default and must be specified using this argument when compiling nginx.

--with-http_stub_status_module

First, to get statistics like the above example, you should modify nginx config file and add location directive this

server {
....
....
	location /nginx_status {
		stub_status on;
		access_log   off;
		allow 1.2.3.4;
		allow 5.6.7.8;
		deny all;
	}
}

perl scripts used to generate statistical images can be downloaded here:

http://kovyrin.net/files/mrtg/rrd_nginx.pl.txt

Rename rrd_nginx.pl.txt to rrd_nginx.pl and make it executable

Postfix Rotating Outgoing IP Using TCP_TABLE And Perl

In The last article “Postfix Randomizing Outgoing IP Using TCP_TABLE And Perl“, i was writing about my experiment randomizing outbound ip. someone asked if the script can be mimicked as in the iptables statistic module. indeed, when using the previous script, the results obtained will really random. although that is what the script was created.

I found this wiki explaining about Weighted Round-Robin Scheduling.

Supposing that there is a server set S = {S0, S1, …, Sn-1};
W(Si) indicates the weight of Si;
i indicates the server selected last time, and i is initialized with -1;
cw is the current weight in scheduling, and cw is initialized with zero;
max(S) is the maximum weight of all the servers in S;
gcd(S) is the greatest common divisor of all server weights in S;

while (true) {
    i = (i + 1) mod n;
    if (i == 0) {
        cw = cw - gcd(S);
        if (cw <= 0) {
            cw = max(S);
            if (cw == 0)
            return NULL;
        }
    }
    if (W(Si) >= cw)
        return Si;
}

anyway, I do not have much time to implement that algorithm into the script, so I use an existing perl module List:: util:: WeightedRoundRobin

Check Disk Quota Usage By Parsing Maildirsize

In the previous article, I have explained how to check the quota usage by using perl scripts. maildir quota checking is done by using File::Find function which seems a bit slow and too much consuming I/O process. same as the find function on bash script. find function in perl perform disk usage by determining the size of all files and directories recursively. all sizes of files and directories then accumulated.

find(sub{ -f and ( $size += -s ) }, $diruser );

actually, examination of maildir quota usage is also done by imap server which is written into a file called maildirsize.  So, rather than wasting resources on the server. we can directly read the specified quota and usage on maildirsize file.

Maildir quota usage using File::Find

sub checksize {
        my $diruser = $_[0];
        trim($diruser);
        my $size;
        find(sub{ -f and ( $size += -s ) }, $diruser );
        if (defined $size) {
                $size = sprintf("%u",$size);
                return $size;
        }
        return undef;
}

Postfix Checking Maildir Disk Usage / Quota On The Fly

Postfix does not have built-in quota inspection feature. i’ve seen people on mailing list asking how to check maildir usage on the fly. Add-ons such as VDA , vda inspect quota after email was accepted. When someone over quota, they will bounce mail, Just imagine, when bad guys sent lots of email to overdrawn account with fake return address/sender envelope, on purpose. Our mail server will be backscatter source. spamming undelivered message to innocent people.

Maildrop and Dovecot can handle quota better, but still they’re all inspect maildir usage after accepting the mail. And likely they will bounce email after the first inspect overquota maildir. Ideally, sender should be rejected at smtp conversation time. RCPT TO stage will perfect place for inspecting recipient maildir usage. Before postfix introducing tcp_table, the best solutions was creating map for overquota user. this can be done by using script by querying user quota constant specified in database, then compared to usage in maildirsize file or maildir disk usage.

I wrote this simple perl script, has functions to inspect user quota specified in database, and maildir disk usage. it runs as daemon. it’s not perfect. the script lack of ability when dealing with email address alias or email address extension. Just keep in mind tcp_table connection is not protected and the server is not authenticated.

There are two main functions which are used in this script. checksqlsize and checksize.

checksqlsize is used to check user quota specified in the database. you can adjust parameters in the script as needed,

sub checksqlsize {
        my $user = $_[0];
        my $sqlresult;
        trim($user);
        my $dbh = DBI->connect('DBI:mysql:postfixdb:localhost', 'user', 'password', { RaiseError => 1 });
        my $sth = $dbh->prepare(qq{SELECT quota FROM mailbox WHERE username='$user'});
        $sth->execute();
        while (my @row = $sth->fetchrow_array) {
                $sqlresult = $row[0];
        }
        $sth->finish();
        $dbh->disconnect;
        if ($sqlresult >= 0 ) {
                return $sqlresult;
        } else {
                return undef;
        }
}

Postfix header_checks using tcp_table and checkdbl.pl script

Postfix implements the header_checks as built-in content inspection classes while receiving mail. Usually the best performance is obtained with pcre (Perl Compatible Regular Expression) tables or slower regexp (POSIX regular expressions). Googling on the net, i’ve found tiny perl script that can queries to dbl.spamhaus.org, multi.surbl.org, black.uribl.com. ( Sahil Tandon wrote it, based on  João Gouveia perl script, i think..)

first download the script

# cd /etc/postfix
# wget http://people.freebsd.org/~sahil/scripts/checkdbl.pl.txt

Rename and make it executable

# mv checkdbl.pl.txt checkdbl.pl
# chmod 755 checkdbl.pl

Edit master.cf add this two lines

127.0.0.1:2526 inet  n       n       n       -       0      spawn
user=nobody argv=/etc/postfix/checkdbl.pl

Make preliminary test, to ensure checkdb.pl sih really spawned and answering our queries

# postfix reload
# telnet 127.0.0.1 2526