I thought it would be fun to track the tweets during the Royal Wedding along with providing a test for some backend changes to Politwitter i've made to better handle high volume of tweets on election day.

I ran into a problem though! The Twitter Streaming API seemed to cap-out at 51 tweets/sec (3060 tweets/min). I know my database can handle inserts per/sec MUCH higher than that from my tests, so it seems to be a ceiling on the Twitter API. If you know more about this please let me know! I doubt Canadian political Twitter will ever reach that volume though. So the stats are not totally accurate because for 2 hours the tweets/sec were greater than this limit as you can see in the chart below. Lets get to the stats I was able to collect!

From 3:00am - 6:00pm (London time) there were 1,242,321 tweets!

Hashtags Usage

• #royalwedding 1,121,184
• #rw2011 88,011
• #katemiddleton 5,774
• #princewilliam 3,071

Top Retweeted Tweets

• @MileyCyrus: wow wow wow #royalwedding soooo romantic. FLAWLESS. =] How lucky is Kate? Its every girls dream to be a princess. Harry? ...
  • 4,750 retweets
• @BBCBreaking: Picture: Thousands cheer as the royal couple kiss on the Buckingham Palace balcony http://twitpic.com/4qybp0 #BBCWeddin ...
  • 1,329 retweets
• @KennyHamilton: Maybe @JustinBieber will go and sing "UP" #RoyalWedding
  • 1,250 retweets
• @Queen_UK: Packed one's handbag for tomorrow's wedding (miniature gin, remote weapons button, reading glasses, iPhone, bag of peanuts ...
  • 1,219 retweets
• @BigStaxxBoss: Imagine if Kanye interrupted the RoyalWedding, "Excuse me, Imma let ya'll finish but Jay-Z & Beyonce had the best wedd ...
  • 1,099 retweets

The Royal Wedding dominated the Twitter top trends

Just a quick post with a chart showing the number of tweets by candidates broken down by province.

The NDP had an online town hall today with questions from callers and people on Twitter using the #asklayton hashtag. The actual live video stream had many problems in the first half, low audio volume then suddenly spiking high. The video would often freeze or cut out completely, but these issues seemed to fade. It appeared that most questions were from callers on the phone and not twitter though.

So how popular was this on Twitter? There was 752 tweets during the townhall. So nothing mind blowing, not breaking into the Twitter.com top trends, but a clever use of Twitter none the less. Also interesting to note that they were moderating tweets that were displayed on the NDP website using the hashtag, probably a good idea.

3 MPs and 7 candidates tweeted with the hashtag.

Here are some of the most tweeted topics: promises, jobs, childcare, reform, postsecondary, internet

• most @replied tweets
• most retweeted tweets

Here is a short video clip of a few of the always running processes on the Politwitter server during the french language leaders debate.

Along with a few processes that are running 24/7 there are 40 or so tasks that run in the background on a periodic basis to keep the site updated, refresh stats along with other things. Doing all this processing in the background allows the website to load fast, which has always been a priority from day one.

Politwiter also caches the output of a lot of content, like the sidebar blocks, many are cached anywhere from 1 minute to 30 minutes. There are over 11,000 files in the cached content folder right now. Gzip is also used to speed up delivery.

Politwitter is run on a Xeon Quad core Dell Poweredge server running Windows Web Server 2008. A second Dell poweredge server is used as a dedicated database server. Bother servers have 4GB of ram, which is used to give MySQL plenty of room for in-memory caching. Politwitter is run on IIS7 using PHP. The website is custom coded by Trevor May and doens't use any 3rd party frameword or CMS. The uses jquery along with other 3rd party platfrms like Goole Visualization API for charts.

That's all for now, I might write a post detailing some of the ways Politwitter deals with the large amounts of data and how the Twitter API's are used.

I am excited to announce that Politwitter is teaming up with Global News to help provide them with insights into the election campaign on Twitter & other social media. Politwitter is providing content 'widgets' that will soon showup on www.globalnews.ca along with data used for articles like this. In return Politwitter is getting some great National exposure and we have some additional things planned for debate and election nights.

I also wanted to mention that Politwitter and myself remain completely indipendant from Global and they have no control over the content or direction of what I do on politwiter.ca. Additionaly while some of the widgets and data I am providing Global will be exclusive, anything on Politwitter.ca is open for media to source (credit and link to politwitter.ca is greatly appreciated). I also remain open to talk with other news media, if you have questions or are interested in an interview.

Thanks to Global for helping promote Politwitter, stay tuned for more!

As the first week of the election campaign wraps up, Twitter has shown that it has become a powerful tool during political campaigns. There have already been some high profile uses of Twitter, such as being able to see clear evidence of public support for the Green Party's Elizabeth May in her debate exclusion, Harper and Ignatieff trading challenges for a one-on-one debate and @senatorjake taking a shot at reporters.

A Look at the Numbers

Twitter has seen a surge of activity in the Canadian political discussion, there have already been over 130,000 tweets in the first week of the campaign. The media has been highlighting the impressive numbers of Twitter volume - 30,000 tweets in first 4-5 days of the campaign. This is not the whole picture though, these numbers only reflect tweets using the election hashtags like #elxn41. Politwitter.ca has a much larger data pool to sample from, including a large directory of MPs, candidates, riding associations and more. Over 600 accounts are tracked in the federal list. Politwitter also has numerous hashtags that are indexed along with per-riding hashtags and a list of keywords that are tracked. This enables Politwitter to provide a much broader picture of the Canadian political twitterverse.

With over 130,000 tweets in the first week (18,000 tweets a day) compared to around 44,000 tweets in the week before the election campaign began. However, only around 43% of these tweets are original content, the rest being retweets. So far, Liberal MPs and candidates have been out-tweeting their competitors by almost 2:1, but the Conservatives have more MPs tweeting each day. I suspect these numbers will even out as the parties get more of their candidates on Twitter. There are currently 170 MPs on Twitter, despite my assumptions I was surprised to find that nearly 90% of the their accounts are currently active! This number is much higher than during the pre-election period. MPs and candidates have only contributed 4,000 tweets (3% of the total amount). Liberals are retweeted the most, and Liberals and Conservatives are about even for replies to their tweets.

The now widely known #elxn41 hashtag has accounted for more than half the tweets during the first week. The French language version #fed2011 has only around 10% of #elxn41's volume.

Sentiment Analysis

One of the intriguing things that Politwitter has been doing with all of this data is sentiment tracking. Tweets are marked as positive, negative or neutral using a method developed by Stanford University called "Sentiment Classification using Distant Supervision." Only around 20% of Tweets can be classified, the remainder don't have any or enough emotional context. But, even with that low percentage we can produce interesting results because of the volume of Tweets during this election. For example, you can see the Green party's positive to negative percentage shoot up on Wednesday when the Elizabeth May exclusion from the debates was the hot topic. Sentiment for the Liberals has stayed more positive versus the Conservatives and the NDP for most of the campaign so far; the Bloc Quebecois has enjoyed high sentiment throughout the whole period.

Leaders Sentiment

Sentiment is also tracked for each of the party leaders, these numbers have moved around a fair bit through the week but at the weeks end the ranking is Gilles Duceppe, Elizabeth May, Michael Ignatieff , Stephen Harper, Jack Layton. Harper has the greatest volume of positive tweets, but the previous ranking comparing the ratio between positive and negative is more insightful. Elizabeth May has been tweeting more than the other leaders which makes sense since she declared the Greens would be using Social Media heavily and she needs to get her message out more than the others.

Election Promises

Politwitter is also tracking popularity and sentiment of the parties election promises. In the first week the promises by the NDP have received the greatest positive response followed by the Liberals then Conservatives. The Liberal promises have been the most popular on Twitter having the most Tweets and retweets mentioning them.

The first week of the "Twitter Election" started strong and I prediect will continue to grow in numbers with huge spikes during the debate and election day, May 2nd. What further surprises and insights will Twitter provide during the rest of the campaign? If the first week is any indication there will be more and it seems like it's the thing to be watching. The great unknown is if any of this actually affects voters and election results, time may tell.

On march 15^th I started my push for riding level hashtags, with the promotion of a “riding hashtag id lookup tool”. A week later a blogger posted a blog with suggestions for alpha-acronym based riding hashtags. I have had a couple people suggest they would be better than the EDID numeric hashtags I am using, this is my response to those queries.

I created the riding edid hashtags before that blog post was made and the edid hashtags have since really started to gather attention and usage. The Hashtag lookup tool has been retweeted by over 100 people http://politwitter.ca/page/edid_lookup and the riding hashtags have been used 220 times already.

I explained most my reasoning on the lookup page, but here is some more detail.

Why do we use hashtags? To categorize tweets so others can find tweets along the same topic, so uniqueness of a hashtag is paramount. This is one of the main reasons I chose not to pursue acronyms, so many acronyms or short-forms of riding names would result in hashtags already in wide usage on Twitter for things not related to Canadian Politics. If they are not in-use now, there is also a chance they could be taken over by some obscure foreign topic in the future. The author of the hashtag blog list didn’t take much care in choosing hashtags, he just used obvious shortforms or acronyms, SOOO many of those result in hashtags that would be flooded with non-relevant tweets. Making following those hashtags rather useless for those ridings.

I decided several months ago I wanted to capture riding level tweets, so I considered several options, and consulted with several others about it. Having unique hashtags is important for the user experience but also for Politwitter since I want to index all these riding level tweets so I can do riding by riding analysis of things like sentiment and volume.

Since riding hashtags are meant to be for tweets about a specific riding, it’s not as important that I know what the riding hashtag for Oakville is, All I really need to know is my own riding. For people who are interested in other ridings, I have made it easier to see what the riding hashtag represents on Politwitter by creating a hover box with details about the hashtag when you place your mouse over it. I am also working on a couple new features on Politwitter to make remembering riding hashtags easier on Politwitter, for people that use the site to tweet.

People can also use other hashtags in their tweets that are more human readable outside the Politwitter site, but it just makes it so much easier to index riding related tweets using the edid tag. With the tweets that contain hashtags, many often contain multiple hashtags anyway, It’s very. So I encourage people to sue both.

I am also indexing tweets that contain the riding name in the tweet, so riding tweets can also be indexed that way.

I cannot endorse or promote that list of hashtags from the blog, not enough thought was given in their uniqueness. I started down the acronym road over a month ago but quickly found out it was not practical, trying to find acronyms that weren’t already used, resulted in hashtags that were no more recognizable that the edid number. However if a particular riding acronym hashtag is found to be in wide use I will of course add it to the Politwitter database like I do for any Canadian political hashtag. As long as it doesn’t have a large percentage of non-relevant tweets also using the same hashtag. And I encourage anyone to contact me via email or a tweet with any suggestions of riding hashtags they think are in wide use. But I will continue to promote the riding edid hashtags as the defacto standard, its usage is growing each day.

But Like I said if there is a another hashtag growing in unique usage in a riding, I will add it to the Politwitter database and riding hashtag lookup tool, which will suggest both.

The mainstream media in the last few days have been citing polls that show Canadians don't want an election or are unhappy about having one, but what does the Twittersphere think?

Politwitter indexed 13,389 tweets from thursday to saturday that included the popular #elxn41 hashtag. Here is some analysis of all those tweets.

• Positive: 2,271
• Negative: 1,014
• Neutral: 10,410
• Percent postive: 69%

These results seem to indicate the overal tone of Twitter over the news of a federal election was positive, the opposite of the mainstream media polls. Interesting stuff! People tweeting with the #elxn41 are probably more engaged in politics than the random selection of people that would be polled traditonaly.

Here are how often the party leaders were mentioned in the #elxn41 tweets:

• Harper: 8,094
• Ignatieff: 4,154
• Layton: 1,712
• Duceppe: 968

These are the most tweeted words:

The most retweeted tweet during this period was (169 retweets):

view tweet details

The tweet with the most replies was (109 replies):

view tweet details

I’ve been working hard on Politwitter this past week, adding many new exciting features along with several bug fixes.

I just finished adding a new statistics page that tracks the sentiment of tweets mentioning MPs. The results are grouped by party and interesting charts are shown.
http://politwitter.ca/page/sentiment

I created a page listing all the Twitter hashtags for House of Commons committees and will now index all tweets using these hashtags.
http://politwitter.ca/page/committees

I created a list of all 308 federal ridings http://politwitter.ca/riding
You can click on any riding to view that ridings social media page, showing all related content for that riding.

I also started a federal riding hashtag format based on the riding EDID number for people to tag their tweets about local riding issues. To help people find your EDID hashtag I created this lookup toolhttp://politwitter.ca/page/edid_lookup

I remade the main menu using a multi-column dropdown format, allowing for much more links in a user friendly format.

I created a system to collect the total number of tweets each parties MPs make each week then to automatically tweet out those stats for each party using 4 Twitter accounts I have for each party. So people can follow which ever twitter account they are interested in. The stats can also be viewed on this page: http://politwitter.ca/page/statistics_weekly

I remade the “election fever” page updated for 2011 and switched the charts to use the new interactive Google visualization.
http://politwitter.ca/page/election_fever

I did some major backend work to allow the site to start tracking MPs & MLAs that are not on Twitter but do have other social networks. Previously everything was tied to a twitter username, so I wasn’t able to add MPs without Twitter.

Along with that I started creating new directory pages to list the politicians on other social networks than Twitter
Facebook: http://politwitter.ca/directory/facebook
Youtube: http://politwitter.ca/directory/youtube/mp/house

Also related are the new ‘profile overview’ pages for tracked users, which give a summary of all their social media content indexed by Politwitter. Example: http://politwitter.ca/profile/joycemurray
I’ve also added ‘Klout’ score to all MPs profiles.

I started populating the stats page on MPs profiles, here is an example: http://politwitter.ca/page/userstats/user/MinJK

There is now a “Bills & Votes” landing page http://politwitter.ca/page/bills and I’m working on indexing all tweets relating to any bill in parliament, then listing all the bills with related tweets.

The MP & Riding search was also improved. You can enter an MP or MLAs name, riding name or a postal code and get a list of MPs and ridings that match.
Example: http://politwitter.ca/mp_search.php?search_mp=Peter

————————————————————————–

And finally I added a new blog post asking for Donations to help fund the continuing development, upkeep and hosting of Politwitter.ca

Read the post here: http://politwitter.ca/blog/14/call-for-donations
Donate here: http://politwitter.ca/page/donate

This is a comment I made in response a blog post Does tweeting signal success?

I am very much looking forward to the next Canadian federal election. I will be using the wealth of Twitter and other social media data Politwitter has been gathering to try and predict the outcome of the election. Followers will be a big part of that but i'm going to use several other social media metrics.

I hope to not only have a prediction for the winning party nationally, but also a riding by riding prediction. And after the actual election results are known, it will be very interesting to see how accurate the predictions were and if Twitter and social media is a gauge.

If anyone wants to help with this effort, I could sure use it! I spend a lot of time doing the backend programming and database work, which leaves me with limited time to keep the directory of MPs and candidates updated, its a lot of work. Every once and awhile I go on a binge adding as many as I can find, but its time consuming finding each MP or candidates twitter, facebook, blog, youtube and flickr.

Anyone can help by submitting MP and candidate info using this simple form http://politwitter.ca/page/join

I also have a new page listing every riding and it shows which ridings are missing social media info for their local MP and candidates http://politwitter.ca/riding
which is A LOT right now. I've concentrated on MPs on Twitter and only recently added support to add MPs without Twitter but have other social media accounts such as Facebook.

This wasn't a great example but its noteworthy that the recent BC Liberal Leadership race was won by the candidate with the most Twitter followers and Facebook fans. I have the comparison herehttp://politwitter.ca/special/leadbcl (right side)