ThoughtFactory.CreateThought() A random collection of thoughts from an easily bored developer

Twittering Heights

Until recently I honestly didn't "get" the Twitter phenomenon. I figured that it would be chock full of people posting about what they had for lunch, or other such inane comments.

There are quite a few excellent articles that go some way to refute this by pointing out the power of choosing who to follow, but that still wasn't enough for me. Finding people to follow seemed like a lot of hard work, and constantly maintaining this list was not of any particular appeal to me. I'd given up on trying to maintain a LiveJournal friends list, which also had the tendency to suffer from the same sort of noise problems, so why would Twitter be any different?

Thinking back, the reason I lost interest in LiveJournal had nothing to do with the work involved in tending the garden that was my friends list. I simply never felt compelled to post anything on the site, so I had no buy in. I was merely an observer, and I was never going to become interested in a service that I had no urge to contribute to.

Twitter is a little different. The short, 140 character messages and public API seemed like the perfect way of complimenting my blog, especially if I could put a Twitter feed on my site.  I could augment the larger posts with short, train of thought ideas, and generally ignore the horrors of trying to accumulate another Facebook-like friends list and just concentrate on producing content. At last, a social networking site with a chance of holding my interest!

There are a few ways of getting at a particular Twitter feed. Aside from the obvious "code your own" approach, Twitter themselves publish a series of "badges" that provide a variety of Flash of Javascript driven ways of getting your Twitter info onto your site. This blog runs on WordPress, so I could also the variety of Twitter based tools for WordPress that let you create and display tweets on your blog. I ended up using the main Javascript driven "badge" provided by Twitter. All it took was some minimal styling and I had a working feed of all my tweets.

It only took a few hours for me to figure out that the side bar's performance was greatly affected depending on Twitter's now legendary availability problems, and the Twitter results wouldn't even show if you were behind a proxy server that didn't allow social networking sites. The feed implementation I chose worked by including two Javascript blocks sourced directly from Twitter, one being a file that defines two functions used to display tweets and the other being a JSON file representing a call to one of these functions with data from my feed. Obviously if a user was trying to view my site from a location that does not allow Twitter, e.g. most work places, then they would not see the feed on my site. Even if they could view Twitter, there was a good chance that the service itself would be down, causing an HTTP timeout to occur on the client.

One option would have been to move the Twitter call to my server, making a PHP call from the WordPress site to populate the list of tweets, or even pointing the existing Javascript at a script on my site. Contacting Twitter on the server would allow me to bypass the "social network" proxy tax, and it would also give me a point of contact to introduce a cache for the Twitter feed.

By caching the data on my server, and only fetching it from Twitter if the cache was older than a certain period of time, I could at least make sure that the feed would always display. If Twitter was down then I'd just have to wait for the request to time out and return the cache results regardless of age. Unfortunately any potential HTTP timeout could still affect the timeliness of the response from my blog, meaning that a user still had the potential to feel the wrath of Twitter's downtime on my site.

It seemed that, no matter what I did, contacting Twitter to retrieve my tweets was going to be an expensive and unreliable operation, and not one that I wanted to impose on the user when they request pages from my site. After all, I'm trying to augment my blog, not cripple it.

In situations such as this I'd normally install a service of some sort to do the request outside of the request/response life cycle experienced by users of the site. However, this site runs on shared hosting Linux environment with no SSH/shell access, and my only way to configure it is through FTP or a web interface, so I assumed that my options here would be limited.

Fortunately, after a bit of poking around in the web configuration tool, I discovered that my hosting provider lets me schedule cron jobs. This was the missing link that would allow me to use a cached approach and keep the same Javascript display code, while still enabling the cache to be updated outside of the normal request/response lifecycle. Effectively I could create a poor man's Twitter update service just by executing a Python script every 15 minutes.

In the interests of spending as little time as possible on the problem, I decided that I was going to keep all the existing Javascript for displaying the tweets. All I needed to do was point both the function definition and the JSON file links at copies on my site, and update the JSON file on a regular basis using a script executed via a cron job. A few lines of Python and some site configuration later and I had a Twitter feed that updates every 15 minutes.

One of the most interesting aspects of the exercise was deciding how to handle failures in the script. Twitter is down so often that I really don't care if the web request fails. So much so that I didn't even bother logging it. I've effectively assumed that the service will fail a good proportion of the time, and if it fails because Twitter couldn't be contacted then so be it. I get told about everything else, such as being unable to write to the cache, but I could really do without 50 email a day reminding me about the precarious nature of a service I have no control over.