Illuminati Karate

At Illuminati Karate we love a challenge. So when our client called needing two lists of 1500 IP addresses geocoded as quickly as possible, we rose to the occasion. With the help of PHP and a freely available web service we were able to complete the task in just under an hour. Here’s how we did it.

The Challenge

Given 2 text files, each containing approximately 1500 anonymous IP addresses, we were to determine how many resolved to each US state. Separate counts were to be maintained for each list. The data was needed for a time sensitive report, and the clock was ticking.

The Tools

There are several freely available SQL databases which correlate IP addresses to their geographic location, such as this one maintained by iplocationtools.com. If we’d been building a web app setting up a local instance would have made sense, but at 4.1 million rows it was going to take too long to get a local instance running.

What we needed was a web service to do the lookups for us. In the past, we’d been unable to find a free web service to fulfill this role, as most will not provide data beyond a country level. But today we were lucky, and Google quickly turned up the perfect solution: a free REST web service running the very same database from iplocationtools.com. Its exactly what it says on the tin: a free IP geolocation webservice. It gave us even more accuracy than we needed (cities and zipcodes!), and even better it was running on Google’s App Engine, so we knew it could handle a heavy load.

With the primary problem solved, the only thing needed was a simple PHP script to read in the input files, query the web service for each IP, parse the JSON results, and maintain a total for each state.

The Solution

Thanks to PHP5’s excellent JSON support, the script to interact with the web service was trivial. I won’t bore you with the file I/O and tallying, but here’s the function which converts the IP address to its state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<?php
	function getStateFromIp($ip)
	{
		$resp = json_decode(file_get_contents('http://freegeoip.appspot.com/json/' . urlencode($ip)));
		if (!$resp->status)
		{
			// lookup failed
			return false;
		}
		else
		{
			return $resp->regioncode;		
		}
	}      
?>

Take note of the the json_decode function. If you’re not already familiar with it I’d highly recommend taking a few minutes to familiarize yourself with PHP5’s JSON functions. JSON is becoming a de facto standard for web services, so being able to painlessly convert to and from JSON in PHP is critical.

Finally, we simply used print_r to dump the array containing the per state totals, and used Monarch to pull the data into an Excel file which we then delivered to the customer. (Note: under normal circumstances we would have reached a more elegant solution, but given the time constraints brute force was the name of the game.)

Conclusion

On-demand web services are fundamentally changing the way we interact with data. Projects which were formerly large, slow, and expensive can now be completed in a matter of hours or days for a fraction of the cost. Perhaps most importantly, anyone can get involved.

Its an exciting time, and we’re happy to be a part of it all. Thanks for reading!

If your company needs help with a custom dataset contact us today. Even if you’re not sure exactly what to ask for, we can help fill in the blanks.