admin – Page 6 – KutuKupret colorful website

Check Disk Quota Usage By Parsing Maildirsize

In the previous article, I have explained how to check the quota usage by using perl scripts. maildir quota checking is done by using File::Find function which seems a bit slow and too much consuming I/O process. same as the find function on bash script. find function in perl perform disk usage by determining the size of all files and directories recursively. all sizes of files and directories then accumulated.

find(sub{ -f and ( $size += -s ) }, $diruser );

actually, examination of maildir quota usage is also done by imap server which is written into a file called maildirsize. So, rather than wasting resources on the server. we can directly read the specified quota and usage on maildirsize file.

Maildir quota usage using File::Find

sub checksize {
        my $diruser = $_[0];
        trim($diruser);
        my $size;
        find(sub{ -f and ( $size += -s ) }, $diruser );
        if (defined $size) {
                $size = sprintf("%u",$size);
                return $size;
        }
        return undef;
}

admin May 18, 2011maildirsize, perl, postfix, quota2

perl postfix

Postfix Checking Maildir Disk Usage / Quota On The Fly

Postfix does not have built-in quota inspection feature. i’ve seen people on mailing list asking how to check maildir usage on the fly. Add-ons such as VDA , vda inspect quota after email was accepted. When someone over quota, they will bounce mail, Just imagine, when bad guys sent lots of email to overdrawn account with fake return address/sender envelope, on purpose. Our mail server will be backscatter source. spamming undelivered message to innocent people.

Maildrop and Dovecot can handle quota better, but still they’re all inspect maildir usage after accepting the mail. And likely they will bounce email after the first inspect overquota maildir. Ideally, sender should be rejected at smtp conversation time. RCPT TO stage will perfect place for inspecting recipient maildir usage. Before postfix introducing tcp_table, the best solutions was creating map for overquota user. this can be done by using script by querying user quota constant specified in database, then compared to usage in maildirsize file or maildir disk usage.

I wrote this simple perl script, has functions to inspect user quota specified in database, and maildir disk usage. it runs as daemon. it’s not perfect. the script lack of ability when dealing with email address alias or email address extension. Just keep in mind tcp_table connection is not protected and the server is not authenticated.

There are two main functions which are used in this script. checksqlsize and checksize.

checksqlsize is used to check user quota specified in the database. you can adjust parameters in the script as needed,

sub checksqlsize {
        my $user = $_[0];
        my $sqlresult;
        trim($user);
        my $dbh = DBI->connect('DBI:mysql:postfixdb:localhost', 'user', 'password', { RaiseError => 1 });
        my $sth = $dbh->prepare(qq{SELECT quota FROM mailbox WHERE username='$user'});
        $sth->execute();
        while (my @row = $sth->fetchrow_array) {
                $sqlresult = $row[0];
        }
        $sth->finish();
        $dbh->disconnect;
        if ($sqlresult >= 0 ) {
                return $sqlresult;
        } else {
                return undef;
        }
}

admin May 16, 2011maildir, perl, postfix, quota4

memcache nginx

Nginx Memcached Module, Caching Webiste Image Using Memcached

During my exploration of optimzing website using memcached, i’ve found this tutorial. It’s quite interesting using key value application (memcached) for serving website’s static content, Such as image, jss or js file. As we know, Reading/writing Hardrive is slower than accessing memory. although compared with SSD or 7,200 RPM hard drives. Even when an application is loaded, any files that are opened for use in that application are loaded into memory. As well as how web servers work.

First time following the tutorial, it isn’t work. image files not loaded into memcached, no warn, now error. after slightly modifying the script, image files can be stored into the memcached perfectly.

Here’s the php script after modification, i saved it as mem.php

<?php

function rscandir($base='', &$data=array()) {
        $array = array_diff(scandir($base), array('.', '..'));

        foreach($array as $value) {
                if (is_dir($base."/".$value)) {
                        $data = rscandir($base."/".$value,$data);

                }
                elseif (is_file($base."/".$value)) {
                $rest = substr($value, -4);
                        if ((!strcmp($rest,'.jpg')) || (!strcmp($rest,'.png')) || (!strcmp($rest,'.gif'))) {
                                $data[] = $base."/".$value;
                        }
                }

        }
return $data;

}

$mylist=rscandir("/path/to/public_html");

$srch = array('/path/to/public_html');
$newval = array('');

$memcache_obj = memcache_connect("127.0.0.1", 11211);

while (list($key, $val) = each($mylist)) {
        $url=str_replace($srch,$newval,$val);
        echo "$key => $val -> ".filesize($val)."\n";
        $value = file_get_contents($val);
        memcache_add($memcache_obj, $url, $value, false, 0);
}

?>

admin May 13, 2011memcache, nginx3

memcache nginx php

Nginx Memcached module And PHP

I’ve seen lots of tutorials on internet explaining how set nginx to use memcached module. But, i’ve rarely seen them explaining, even the easiest one on how to populate memcached with data generated by php. When i’m getting in touch with memcached for the first time, i thought default configuration like this will be working out of the box. without touching php script. 😀

server {

	listen 80;
	server_name  example.com www.example.com;

	# ...

	location ~ \.php$ {
		set $memcached_key "kutu:$request_uri";
		memcached_pass 127.0.0.1:11211;

		default_type       text/html;
		error_page 404 405 502 = @cache_miss;
	}

	location @cache_miss {
		fastcgi_pass 127.0.0.1:9000;
		fastcgi_index index.php;
		fastcgi_param SCRIPT_FILENAME /path/to/yoursite/htdocs$fastcgi_script_name;
		include /etc/nginx/fastcgi_params;
	}

	# ...
}

Well, i was totally confused :D. so, to prevent other people misinterpret on how nginx memcached module works, i wrote simple examples, and simple explainations on how to configure nginx and populate data generated from php.

admin May 12, 2011memcache, nginx, php9

nginx

Nginx And Unix User Directories

As mentioned in the nginx wiki webiste, Nginx does not natively support user dirs, but as of 0.7.42 it can be done by using regex captures. i've tested it using this simple configuration. [text highlight="6,7,8"] server { listen 192.168.200.18:80; server_name _; access_log /var/log/nginx/nginx-userdirs-access.log main; location ~ ^/~(.+?)(/.*)?$ { alias /home/$1/public_html$2; }…

admin May 11, 2011nginx, user directoriesNo Comments

nginx

Nginx Reverse Proxying Multiple Domains Using map Module

Instead of writing multiple server blocks repeatedly in configuration, we can simplified configuration using nginx map module ie: [bash] map_hash_bucket_size 128; map $http_host $backend_servers { hostnames; default www.example.com; frontend.example2.com backend.example2.com frontend.example3.com backend.example3.com www.example.org backend.example.org } proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; server { location / { proxy_pass…

admin May 11, 2011map Module, nginx, reverse proxyNo Comments

apache

Graphing ApacheBench Results Using GnuPlot

Ab / ApacheBench is a tool commonly used to perform benchmark by many people. unfortunately, the results are displayed somewhat difficult to read by most people. but it can be tricked by turning it into a plot image. this time, I will make examples how to change the results of apachebench into a form of image plots using gnuplot. although apachebench’s report is very helpful, we need to output the data in a parse-able format. To do this we use the -g switch which will output a gnuplot-friendly tab delimited table of data.

First we need to installed gnuplot if we don’t have it yet.

$sudo yum -y install gnuplot

In this example i’ll benchmark request on static png file.

Run first webserver benchmark

$ ab -k -n 50000 -c 100 -g server1.txt http://server1/server1.png

Run second webserver benchmark

$ ab -k -n 50000 -c 100 -g server2.txt http://server2/server2.png

Example result server1.txt/server2.txt

starttime       seconds ctime   dtime   ttime   wait
Tue May 10 16:42:28 2011        1305020548      0       2       2       2
Tue May 10 16:42:28 2011        1305020548      0       2       2       2
Tue May 10 16:42:28 2011        1305020548      0       3       3       2
Tue May 10 16:42:28 2011        1305020548      0       3       3       3
Tue May 10 16:42:28 2011        1305020548      0       3       3       3
Tue May 10 16:42:28 2011        1305020548      0       3       3       3
Tue May 10 16:42:28 2011        1305020548      0       3       3       3
Tue May 10 16:42:28 2011        1305020548      0       3       3       3

admin May 10, 2011apache, ApacheBench, GnuPlot3

postfix

postfix-2.8.2 postfix-2.8.3 src rpm (sqlite3 support)

postfix-2.8.2.src.rpm postfix-2.8.3.src.rpm This is how to build it: https://kutukupret.com/2011/05/02/postfix-2-9-20110501-src-rpm-with-sqlite3-support/

admin May 9, 2011postfix, sqlite3 supportNo Comments

memcache

Memcached In A shell Using nc And echo

Some example accessing memcached from a shell using nc and echo

Storage Commands

set
add
get
gets
replace
append
prepend
cas
delete
command [key] [flags] [exptime] [bytes] [noreply]\r\n[value]\r

Parameters :

[key]			: the key of the data stored
[flags]			: 32-bit unsigned integer that the server store with the data (provided by the user), and return along the data when the item is retrieved
[exptime]		: expiration time in seconds, 0 mean no delay, if exptime is superior to 30 day, Memcached will use it as a UNIX timestamps for expiration
[bytes]			: number of bytes in the data block
[cas unique]		: unique 64-bit value of an existing entry (retrieved with gets command) to use with cas command
[noreply]		: optional parameter that inform the server to not send the reply

admin May 5, 2011echo, linux, memcache, nc, netcat, shell1

memcache mysql php

Memcached And PHP, Caching Mysql Query Result

I’ve been messing around with memcached and php-pecl-memcache to cache sql query result. Many web sites & applications such as Facebook, LiveJournal, Flickr, Slashdot, WikiPedia/MediaWiki, SourceForge, Digg and Twitter use memcached to enhance their performance.

Memcached (Memory Cache Daemon) was developed by the team at LiveJournal to improve performance of their social blogging site by minimizing the impact of the bottleneck caused by reading data directly from the database. Memcached is a server that caches Name Value Pairs in memory. The “Name”, or key, is limited to 250 characters, and the “Value” is limited to 1MB in size. Values can consist of data, HTML Fragments, or binary objects; almost any type of data that can be serialized and fits in memcached can be stored.

here is simple example/demonstration how to cache regular sql query

First of all, we need memcached daemon run on system

$ ps ax | grep memcached
 8955 ?        Ssl    0:00 memcached -d -p 11211 -u memcached -m 256 -c 1024 -P /var/run/memcached/memcached.pid -l 127.0.0.1

Setup simple mysql database/tables as shown bellow:

mysql-shell> CREATE DATABASE memcache;

Copy/Paste this tables schema to your mysql shell/console

CREATE TABLE memc
(
 personID int NOT NULL AUTO_INCREMENT,
 PRIMARY KEY(personID),
 FirstName varchar(15),
 LastName varchar(15),
 Age int
);
hit enter/return key

Insert some data

mysql-shell> INSERT INTO memc (FirstName, LastName, Age) VALUES('Memory', 'Cache', '100');

admin May 4, 2011memcache, mysql, php7