May 2011 – KutuKupret colorful website

How To Graph Nginx Statistics

Sometimes, we prefer to view statistics in graphical form rather than numerical values, which is not too attractive to be displayed. Nginx supports stub_status directive module, which we can use to print:

Active connections
Server accepts handled requests
Reading
Writing
Waiting

For example:

Active connections: 6
server accepts handled requests
 15561 15561 26602
Reading: 4 Writing: 2 Waiting: 0

However, This module is not compiled by default and must be specified using this argument when compiling nginx.

--with-http_stub_status_module

First, to get statistics like the above example, you should modify nginx config file and add location directive this

server {
....
....
	location /nginx_status {
		stub_status on;
		access_log   off;
		allow 1.2.3.4;
		allow 5.6.7.8;
		deny all;
	}
}

perl scripts used to generate statistical images can be downloaded here:

http://kovyrin.net/files/mrtg/rrd_nginx.pl.txt

Rename rrd_nginx.pl.txt to rrd_nginx.pl and make it executable

admin May 31, 2011memcache, nginx statistic, perlNo Comments

geoip postfix

Postfix GeoIP Based Rejections

If you intend to be able to reject connections from remote IP addresses if they’re from certain countries. This is how you do it. This method will reject ip address that has been mapped in GeoIP at smtp conversation stage. However, This tutorial has never been tested. nothing more than a prototype that i created in leisure time.

Software required:

postfix (tcp_table)
Perl
Perl Geo::IP module

main.cf:

127.0.0.1:2528_time_limit = 3600s
smtpd_client_restrictions =
	check_client_access tcp:[127.0.0.1]:2528

master.cf

127.0.0.1:2528 inet  n       n       n       -       0      spawn
	user=nobody argv=/etc/postfix/geo-reject.pl

admin May 29, 2011geoip, postfix7

geoip postfix

Postfix Bind Sender Outgoing IP, Based On GeoIP Location

This morning, when I took my daughter to school, I got the idea to experiment with postfix and GeoIP location. the idea is, if mx emails are in a geo targeted a specific location, mail delivery will be done with a certain ip address.

Ie:

Every emails with the mx hosts that have IP addresses/host mapped to the US country code, will be bind to ip 1.2.3.4.
Every emails with the mx hosts that have IP addresses/host mapped to the HK country code, will be bind to ip 5.6.7.8.

Every emails with the mx hosts that have IP addresses/host mapped to the CN country code, will be relay to our smtp nexthop in china.

And so on..

what is geolocation?

Geolocation is used to deduce the geolocation (geographic location) of another party. For example, on the Internet, one geolocation approach is to identify the subject party’s IP address, then determine what country (including down to the city and post/ZIP code level), organization, or user the IP address has been assigned to, and finally, determine that party’s location. Other methods include examination of a MAC address, image metadata, or credit card information.

But, in this experiment we just need ip/host to country code map and perl script.

Perl module required:

Net::DNS
Geo::IP
Sys::Syslog

Basic Usage perl geoip

#!/usr/bin/perl
use Geo::IP;
my $gi = Geo::IP->new(GEOIP_STANDARD);
print $gi->country_name_by_name("amazon.com");

I would still be using transport_maps and tcp_table to interact with Perl scripts. so here’s the prototype.

In Postfix part, we have custom transport like this in master.cf:

smtp-JP  unix -       -       n       -       -       smtp
	-o syslog_name=postfix-smtp-JP
	-o smtp_helo_name=smtp-JP.example.com
	-o smtp_bind_address=1.2.3.1
smtp-US  unix -       -       n       -       -       smtp
	-o syslog_name=postfix-smtp-US
	-o smtp_helo_name=smtp-US.example.com
	-o smtp_bind_address=1.2.3.2
smtp-ID  unix -       -       n       -       -       smtp
	-o syslog_name=postfix-smtp-ID
	-o smtp_helo_name=smtp-ID.example.com
	-o smtp_bind_address=1.2.3.3
smtp-CN  unix -       -       n       -       -       smtp
	-o syslog_name=postfix-smtp-CN
	-o smtp_helo_name=smtp-CN.example.com
	-o smtp_bind_address=1.2.3.4
smtp-HK  unix -       -       n       -       -       smtp
	-o syslog_name=postfix-smtp-HK
	-o smtp_helo_name=smtp-HK.example.com
	-o smtp_bind_address=1.2.3.5

admin May 28, 2011geoip, postfix3

Uncategorized

Kutukupret New Layout / Themes

Today, I will not talk about the technical, tutorial or howto. yesterday I was speaking with friends about the layout of my website. she said my website is too "messy", hard to find the main content. indeed, from the first time since making these blogs, I only use themes that…

admin May 27, 2011No Comments

nginx

Nginx And Simple Permalink

What is permalink?

permalink definition on a wiki website:

A permalink, or permanent link, is a URL that points to a specific blog or forum entry after it has passed from the front page to the archives. Because a permalink remains unchanged indefinitely, it is less susceptible to link rot. Most modern weblogging and content-syndication software systems support such links. Other types of websites use the term permanent links, but the term permalink is most common within the blogosphere. Permalink is a portmanteau word made from permanent link. Permalinks are often simply stated so as to be human-readable.

How’s the shape of permalink structure?

Generally, dynamic page’s url will look like this:

http://www.example.com/index.php?page=10

become:

http://www.example.com/what-is-permalink,10.html

admin May 26, 2011nginx, permalinkNo Comments

array programing

Sort an integer array, What language do you use?

Array data type, used in a programming language to specify a variable that can be indexed

During my experiment with array / hash variable, i’ve found this interesting website. The wiki explains how to sort the integers in the array by using a variety of languages. even some programming languages that I do not know at all.

I’ll pick a few examples of sorting functions described in the wiki

Sort an integer array using C language:

#include <stdlib.h>  /* qsort() */
#include <stdio.h>   /* printf() */

int intcmp(const void *aa, const void *bb)
{
	const int *a = aa; *b = bb;
	return (*a < *b) ? -1 : (*a > *b);
}

int main()
{
	int nums[5] = {2,4,3,1,2};
	qsort(nums, 5, sizeof(int), intcmp);
	printf("result: %d %d %d %d %d\n",
		nums[0], nums[1], nums[2], nums[3], nums[4]);
	return 0;
}

admin May 25, 2011array, c, programingNo Comments

perl postfix

Postfix Rotating Outgoing IP Using TCP_TABLE And Perl

In The last article “Postfix Randomizing Outgoing IP Using TCP_TABLE And Perl“, i was writing about my experiment randomizing outbound ip. someone asked if the script can be mimicked as in the iptables statistic module. indeed, when using the previous script, the results obtained will really random. although that is what the script was created.

I found this wiki explaining about Weighted Round-Robin Scheduling.

Supposing that there is a server set S = {S0, S1, …, Sn-1};
W(Si) indicates the weight of Si;
i indicates the server selected last time, and i is initialized with -1;
cw is the current weight in scheduling, and cw is initialized with zero;
max(S) is the maximum weight of all the servers in S;
gcd(S) is the greatest common divisor of all server weights in S;

while (true) {
    i = (i + 1) mod n;
    if (i == 0) {
        cw = cw - gcd(S);
        if (cw <= 0) {
            cw = max(S);
            if (cw == 0)
            return NULL;
        }
    }
    if (W(Si) >= cw)
        return Si;
}

anyway, I do not have much time to implement that algorithm into the script, so I use an existing perl module List:: util:: WeightedRoundRobin

admin May 22, 2011perl, postfix, tcp_table5

perl postfix

Check Disk Quota Usage By Parsing Maildirsize

In the previous article, I have explained how to check the quota usage by using perl scripts. maildir quota checking is done by using File::Find function which seems a bit slow and too much consuming I/O process. same as the find function on bash script. find function in perl perform disk usage by determining the size of all files and directories recursively. all sizes of files and directories then accumulated.

find(sub{ -f and ( $size += -s ) }, $diruser );

actually, examination of maildir quota usage is also done by imap server which is written into a file called maildirsize. So, rather than wasting resources on the server. we can directly read the specified quota and usage on maildirsize file.

Maildir quota usage using File::Find

sub checksize {
        my $diruser = $_[0];
        trim($diruser);
        my $size;
        find(sub{ -f and ( $size += -s ) }, $diruser );
        if (defined $size) {
                $size = sprintf("%u",$size);
                return $size;
        }
        return undef;
}

admin May 18, 2011maildirsize, perl, postfix, quota2

perl postfix

Postfix Checking Maildir Disk Usage / Quota On The Fly

Postfix does not have built-in quota inspection feature. i’ve seen people on mailing list asking how to check maildir usage on the fly. Add-ons such as VDA , vda inspect quota after email was accepted. When someone over quota, they will bounce mail, Just imagine, when bad guys sent lots of email to overdrawn account with fake return address/sender envelope, on purpose. Our mail server will be backscatter source. spamming undelivered message to innocent people.

Maildrop and Dovecot can handle quota better, but still they’re all inspect maildir usage after accepting the mail. And likely they will bounce email after the first inspect overquota maildir. Ideally, sender should be rejected at smtp conversation time. RCPT TO stage will perfect place for inspecting recipient maildir usage. Before postfix introducing tcp_table, the best solutions was creating map for overquota user. this can be done by using script by querying user quota constant specified in database, then compared to usage in maildirsize file or maildir disk usage.

I wrote this simple perl script, has functions to inspect user quota specified in database, and maildir disk usage. it runs as daemon. it’s not perfect. the script lack of ability when dealing with email address alias or email address extension. Just keep in mind tcp_table connection is not protected and the server is not authenticated.

There are two main functions which are used in this script. checksqlsize and checksize.

checksqlsize is used to check user quota specified in the database. you can adjust parameters in the script as needed,

sub checksqlsize {
        my $user = $_[0];
        my $sqlresult;
        trim($user);
        my $dbh = DBI->connect('DBI:mysql:postfixdb:localhost', 'user', 'password', { RaiseError => 1 });
        my $sth = $dbh->prepare(qq{SELECT quota FROM mailbox WHERE username='$user'});
        $sth->execute();
        while (my @row = $sth->fetchrow_array) {
                $sqlresult = $row[0];
        }
        $sth->finish();
        $dbh->disconnect;
        if ($sqlresult >= 0 ) {
                return $sqlresult;
        } else {
                return undef;
        }
}

admin May 16, 2011maildir, perl, postfix, quota4

memcache nginx

Nginx Memcached Module, Caching Webiste Image Using Memcached

During my exploration of optimzing website using memcached, i’ve found this tutorial. It’s quite interesting using key value application (memcached) for serving website’s static content, Such as image, jss or js file. As we know, Reading/writing Hardrive is slower than accessing memory. although compared with SSD or 7,200 RPM hard drives. Even when an application is loaded, any files that are opened for use in that application are loaded into memory. As well as how web servers work.

First time following the tutorial, it isn’t work. image files not loaded into memcached, no warn, now error. after slightly modifying the script, image files can be stored into the memcached perfectly.

Here’s the php script after modification, i saved it as mem.php

<?php

function rscandir($base='', &$data=array()) {
        $array = array_diff(scandir($base), array('.', '..'));

        foreach($array as $value) {
                if (is_dir($base."/".$value)) {
                        $data = rscandir($base."/".$value,$data);

                }
                elseif (is_file($base."/".$value)) {
                $rest = substr($value, -4);
                        if ((!strcmp($rest,'.jpg')) || (!strcmp($rest,'.png')) || (!strcmp($rest,'.gif'))) {
                                $data[] = $base."/".$value;
                        }
                }

        }
return $data;

}

$mylist=rscandir("/path/to/public_html");

$srch = array('/path/to/public_html');
$newval = array('');

$memcache_obj = memcache_connect("127.0.0.1", 11211);

while (list($key, $val) = each($mylist)) {
        $url=str_replace($srch,$newval,$val);
        echo "$key => $val -> ".filesize($val)."\n";
        $value = file_get_contents($val);
        memcache_add($memcache_obj, $url, $value, false, 0);
}

?>

admin May 13, 2011memcache, nginx3