Last weekend I finished implementing a system that I'd been procrastinating on for months.

When Klaki moved to my dad's basement and onto his DSL connection, bandwidth became more of a bottleneck than it had been in the past. There are quite a few websites of family, friends and friends-of-friends hosted on Klaki.

Some of them seem to have been abandoned by their owners, but the sites are still up, and still getting traffic - still slowing down my parents' connection to the Internet.

Funnily enough, the main "culprits" are things which may just have been forgotten. Individual mp3 files in "tmp" folders, large images which were put online to share with friends... blog entries from years ago with cute pictures stored on Klaki. Most people don't realize that things they put online years ago are still costing bandwidth and resources today - mostly thanks to search engines helping strangers find things years later.

But to be clear; this post isn't supposed to make the users of Klaki feel guilty about that stuff. This is just how the nature of the web and the web is great.

I'm too lazy to contact people and ask them to clean up their stuff - and I know that even if I did, the problem would just come back in a few months anyway. I see it more as a technical problem that just needs fixing. So this weekend, that's what I did. :-)

I implemented the Godaddy Cache!

For a few years I've had a very, very cheap web-hosting subscription with Godaddy. I think it was on the order of $3 per month or something silly like that, and the bandwidth allotment for it is hundreds of gigabytes per month, and a few gigs of storage as well. And I've hardly used it at all...

So last weekend I wrote a program which will analyze the last two weeks of web-traffic on Klaki, identifying the busiest files (images, music, data). Anything causing more than 16MB of traffic over the time period is copied to gd.klaki.net, and the Klaki webserver configured to redirect requests for those files to there. The main exception is I never redirect HTML files, as that would almost certainly break things.

The system updates itself every night, keeping track of what is popular and what isn't, automatically deleting things from the "Godaddy cache" when they aren't needed anymore. I have yet to teach it to recognize when files change on Klaki and update the cache, but for the types of data it keeps track of that shouldn't be much of a problem. Maybe I'll fix that this weekend.

Yesterday gd.klaki.net served up about 500MB of data, so on a weekly basis I'm probably saving at least 3GB, or about 15GB per month. That's quite a bit for a lowly DSL connection.

This should be a win/win/win: my parent's internet connection will be less overloaded, the big files will be served from Godaddy's more powerful (faster) servers and the rest of the content on Klaki should become more responsive as a result. What could possibly go wrong?

This is mostly invisible, but every once in a while you may end up clicking a link on some Klaki-hosted web-site and get redirected to a strange-looking URL starting with gd.klaki.net. Now you know why.

So don't panic! :-)

But please let me know if this breaks anything...