Spoofed Google Bot

I’ve found something interesting in the nginx mailing list today http://forum.nginx.org/read.php?2,202715,202715#msg-202715 . someone asked whether it is possible to block fake user agent such as google bot. sometimes, a lot of bots flood our servers disguise as google bot or other legal bot. Most likely, to scrape our website’s contents. Original google bot always uses the ip address which is owned by Google Inc. Many website owners complained, bad bots only drain their bandwidth usage. 🙂

The first option to use “if” directive in nginx.

if ($http_user_agent ~* "Google Bot") {
	allow 66.x;
	allow 70.x;
	deny all;
}

However, “if” directive considered to be a bad practice when use for anything rather than “return” or “rewrite”. Here’s an example from Igor Sysoev: