Yesterday the Dutch television channel RTL organized a political twitter debate. Most political party leaders participated but politicians, Dutch tweeps, as well as RTL’s servers became overloaded (and again #rtldebat became a world-wide trending topic). Instead of only splitting the debating screen between politicians and other participants, it might have been wise of RTL to have politicians only answer the most retweeted tweets (the way we analyzed the #iranelection tweets). By having people retweet what they deem important or good questions, the intelligibility of the debate might have been greatly enhanced.

It’s about time I release some of the code I’ve been writing. Today, a simple twitter scraper in PHP, a simplified version of the one we (DMI) used for our #iranelection project. Just copy-paste the code below into a file called twitterScraper.php and follow the directions in the comments.

Of course, this is not the official way to scrape Twitter. The nice thing about this script however, is that you can retrieve all public tweets up till two weeks ago; by continuing to pass the latest id of a result page as the max_id parameter. This script circumvents Twitter’s API limits on maximum number of requests as well as the maximum number of results returned. Of course, usage of this script and the possible consequences are your own responsibility.

(more…)

In response, and as an addition, to Wilbert Baans’ storytelling with public databases, I made a little twitter scraper. It indexed all twits for the query http://search.twitter.com/search?q=%22i+voted%22+%2B+McCain+OR+Obama+-twitvote, up till a month ago. From November the 4th till November the 5th the script scrapes new twits for this query every minute.

A point goes to Obama if the regular expression /vote.*?obama/i succeeds, it goes to McCain if the regular expression /vote.*mccain/i succeeds, else it is undecided / unrecognized.

The result can be found at Twitter Poll. So far Obama is winning.

Update: Wilbert Baan has made theses stats into a beautiful visualization:

There is an other twitter poll on Mashable.

Update 2: I tried making a similar exit poll based on MySpace, but that query does not yield results sorted on time. Also, just like Google, MySpace only returns a maximum of 1000 results per query. Unfortunately I’m thus stuck with the same numbers, no matter how many times I will scrape it.