Summary: in 2015 I created a Twitter bot, @AnnArborVotes (code on GitHub). (2018 Sam says: after this project ceased I gave the Twitter handle to local civics hero Mary Morgan at A2CivCity). I searched Twitter for 52,000 unique voter names, matching names from the Ann Arbor, MI voter rolls to Twitter accounts based nearby. The bot then tweeted messages to a randomly-selected half of those 2,091 matched individuals, encouraging them to vote in a local primary election that is ordinarily very low-turnout.
I then examined who actually voted (a matter of public record). There was no overall difference between the treatment and control groups. I observed a promising difference in the voting rate when looking only at active Twitter users, i.e., those who had tweeted in the month before I visited their profile. These active users only comprised 7% of my matched voters, however, and the difference in this small subgroup was not statistically significant (n = 150, voting rates of 23% vs 15%, p = 0.28).
I gave a talk summarizing the experiment at Nerd Nite Ann Arbor that is accessible to laypeople (it was at a bar and meant to be entertainment):
This video is hosted by the amazing Ann Arbor District Library – here is their page with multiple formats of this video and a summary of the talk. Here are the slides from the talk (PDF), but they’ll make more sense with the video’s voiceover.
The full write-up:
I love the R programming language (#rstats) and wanted a side project. I’d been curious about Twitter bots. And I’m vexed by how low voter turnout is in local elections. Thus, this experiment.
The video covers the project at a high level; here are more details.
Everything was done in R, from data gathering to tweeting to analyzing the results.
Voter rolls information – names, addresses, etc. – is publicly available and was supplied by the Washtenaw County clerk. They also supplied the post-election list of who actually voted. I gathered the data from Twitter to match usernames using the rvest package for R, as I had too many names to search for the Twitter API to be an option. I struggled with rvest until I followed the recommendation to use it with SelectorGadget (I like the Chrome plug-in) – see the recording of my talk for a demo of this powerful combination.
My data gathering process involved a recursive search. I’d perform an accounts search on Twitter for a unique name in the voter rolls, then visit each of the account pages on the resulting list to see which, if any, were in Ann Arbor. There were high-confidence matches (though not 100% certainty) for about 4% of cases (2,091 out of 52,035 voter names).
I sent the tweets via the twitteR package. Getting a Twitter app set up and authenticated was by far the worst part of this project – installing Ruby just to run a short script that I didn’t understand was the pinnacle of command line bullshittery – but past that, actually tweeting was easy.
The bot cycled through a few canned tweets. The messages looked like this:
Researching bots before I started, I was initially concerned about getting banned. Twitter cautions against excessive unsolicited @-mentioning and that practice has gotten other bots shut down. To that end, I started sending my tweets very slowly (104 minutes apart) 6 days before the election, and when I did not get banned, I gradually ramped up to full speed (a tweet every minute, with some randomization) on the evening before the election.
I made it through the August primary without getting shut down! Then I started round 2 of the experiment in November, but resumed my tweeting at full speed (after not tweeting for weeks) and was promptly suspended by Twitter. Technically it was my application that was restricted (SocialGoodBot), not the @AnnArborVotes account.
Results – did it work?!
1. No overall difference between the treatment and control groups:
2. Statistically-significant difference between tweets with at least one engagement and the control group:
An “engagement” means that someone did something to the tweet besides viewing it: clicking the tweet to see it in full, clicking on the bot’s profile, replying, etc. Most engagements presumably came from the recipient, so we can use this as sort of a confirmation that the tweet was seen by the voter.
This difference was statistically significant at p < 0.05. This is not causative evidence of the reminders having an effect – but it is interesting, Do engagers vote because they engaged? Or do likely voters engage because they vote and are interested in the topic? I suspect it’s the latter.
3. Interesting difference when considering only active Twitter users:
This cut of the data is important because many tweets in the experiment were sent to lapsed Twitter users; that means, effectively, that the treatment likely did not reach them. The median number of lifetime tweets per user in our data set is 39 – that does not represent an engaged Twitter user!
So I tried looking only at active tweeters (at least 1 tweet sent in previous month). Unfortunately, this is a small subset of the data, and there wasn’t the statistical power needed to be confident that this difference isn’t due to chance, even though the difference observed would represent a giant effect size if reproducible.
4. Other outcomes.
I got a few replies, and a handful of retweets and favorites:
- 8,500 tweet views
- 267 engagements
- 11 favorites
- 9 replies
- 6 retweets
The low views and engagements aren’t surprising. Many targeted accounts were inactive, and even active users may be unlikely to engage with a bot.
Beyond the primary measurements of the experiment’s effects, hundreds of other people outside of the experiment were also subjected to reminders to vote, which may have had an unmeasured effect.
I’d love to see someone extend this research:
- It’s possible that reminder tweets could have a meaningful effect on the behavior of active Twitter users
- Most methods of increasing turnout are partisan and expensive; this is non-partisan and free!
- Voter turnout is good for democracy and society
Feel free to adapt or extend my code. If you have questions about it, Tweet me or open a GitHub issue. (2018 Sam says: gosh old self, that wasn’t very reproducible, was it?).
I hope to eventually explore another use of voter data to motivate participation: game-ifying voting history. Who has the best streak on your block, in your ward, in your city? Who is the best voter among your Facebook friends? badvoter.org is an inspirational starting point. Ted Cruz used mailers to this effect prior to the 2016 Iowa caucuses (though his campaign faked the data).
The handle @AnnArborVotes may have a future in the hands of another area resident fighting the battle for increased voter participation…
Thanks! to Jeff Gentry and Hadley Wickham for their R packages that made this possible, Andrew Cantino for SelectorGadget, Nerd Nite Ann Arbor for hosting and recording my talk, and the AADL for hosting the video.