Solving Twitter’s abuse problem

Twitter is awesome, but unless it gets its abuse problem under control, it’s going to struggle to attract new users.

Enter the Internet

Why is this such a big problem for social media?

I’m no expert, but that sounds a bit… slander-y to me.

And this…

Straight to the C word. Charming.

And this…

No caption required.

article continues below

Part 2 — Why creating a sensible policy on online abuse is complicated

Everybody agrees they want freedom, but they can’t agree what ‘freedom’ means.

Freedom to Speak

Freedom from Abuse

The friction is where the freedoms clash

Creating an abuse policy

The current section on abusive behaviour in ‘Twitter Rules’ is actually pretty good, but both under- and inconsistently enforced.
  • It’s crucial that any actions that are taken as a response to abuse are taken transparently.
  • There must be a way to correct mistakes made in the process of implementing the policy.
  • There needs to be a clear set of guidelines for what is OK and what isn’t.
  • Reported content must be dealt with as quickly as possible. In the case of Twitter, a news source that operates in the fractions-of-a-second timeframe, rather than the hours or days of most other websites, I’d suggest that ‘as quickly as possible’ should mean ‘less than an hour’.
  • The people dealing with abuse need a strong suite of tools to help draw contextual information from the posts they’ve had reported.

An aside on public figures


  • Creating an abuse policy is difficult, and impossible to make ‘correct’ for everyone involved: Emotions run high.
  • You cannot placate both the people who believe they deserve the freedom to say anything, and the people who believe they deserve a freedom from abuse. A good abuse policy is likely to impinge on what both of these groups perceive as their rights.
  • Not easy, but it’s important that any policy that is implemented is transparent, easy to understand, and hard to misinterpret.
  • People have different tolerances and expectations for abuse, and a policy should either reflect that, or pick an appropriate middle ground.

article continues below

Part 3 — Abusive or not — A challenge of sentiment analysis

“‘Fucking cocksucker scumbag asshole’, for example, is relatively unlikely to be meant as a phrase of endearment”

Abusive or not, here I come…

This isn’t even close to the worst of the worst: This is a perfectly normal day on Twitter.

(tweet recreated after I managed to misplace the screenshot. Apologies.)

Sometimes, spotting abuse is easy.

It’s pretty hard to see how this could be anything other than a threat.

… and …

This one is part of a huge, ongoing abuse campaign against a prominent feminist blogger. Definitely not a joke.

… and …

This one is taking the abuse to another level: Not just a threat and an instance of abuse, but also a promise of continued campaign of abuse from any number of Twitter accounts, against another prominent female blogger. Twitter has now suspended this account, but — as the tweeter points out — there’s more where that account came from.

… and …

There were dozens of this particular instance of abuse, from dozens of different accounts. It’d be hard to argue that this is a prank.

… and …

It is hard to imagine a scenario or context where this is a prank, a joke, or a misunderstanding. This tweet was at a blogger who has been the target of sustained abuse for years. I’d argue this particular example is probably serious enough to warrant a call to the police.

The ‘James is a cunt’ problem

Sometimes, it’s hard to determine abuse

Abusive? Probably not… But maybe?

Sometimes, it’s complicated…

All of these tweets happened in a span of a few hours. Shkreli isn’t the most-loved man in the world, for sure. Also shown: Some of the phrases that the algorithm considers abusive. No surprises there.
This tweet broadens the abuse from just Martin to further threatening his sisters, which is a further problem. While it is conceivable that Martin himself revels in the attention, I don’t know whether he has any siblings, far less whether they have given permission to be dragged into the limelight.

And sometimes, it’s impossible…

“If you get a blade sharp enough, it’ll cut through anything”

I can’t even imagine what goes through David’s head when he reads Roy’s tweet. No matter how much you disagree with David, first amendment be damned. That’s never an OK thing to say to another human being.

And then there’s images

Adding context to abuse analysis

Contextual indicators of abuse:

  1. How many followers does the abuser have? (Fewer than 50 is bad news, and fewer still increases the abuse risk)
  2. How many friends does the abuser have? (Fewer than 50 is bad news, and fewer still increases the abuse risk)
  3. How old is the account the abuser is using? (Less than a month is bad, less than a week is worse, less than a day is really bad news)
  4. What is the ratio of followers-to-friends? (If the abuser follows more people than what follows them, that’s an indicator of abuse. If their friend-to-follower ratio is 3:1 or above, it’s a strong indicator)
  5. What is the followers-to-followers ratio between the abuser and the target? (A ratio is 100:1 or higher is a strong indicator of abuse)
  6. How many times has the abuser tweeted? (Fewer than 100 tweets is bad, and it gets worse the fewer tweets there are)
  7. Is the abuser an ‘egg’? (i.e. did they set a picture? Yes is bad news)
  8. Has the bot caught the abuser been being abusive before? (Once is usually a strong indicator. Habitual abuse sends the risk of abuse off the charts)
  9. Has the target of abuse been abused before? (The more, the higher the risk of future abuse)
  10. Is the tweet part of a conversation? (If it is an at-mention out of the blue, that’s bad news.)
  11. Does the abuser follow the target? (If no, it’s a weak indicator of abuse, but there is some correlation)
  12. Is the target verified? (If yes, it can help inform some of the other indicators of the algorithm. Verified users also seem to get a disproportionate amount of abuse)

Contextual contra-indicators of abuse:

  1. Is the tweet part of a conversation? (The more tweets back-and-forth in that conversation, the less likely it is to be abusive)
  2. Has the abuser and the target had interactions in the past? (Yes is usually a contra-indication, but not always; it could mean two people constantly at each other’s throats, or two people who used to be friends but are no longer)
  3. Does the target follow the abuser? (If yes, it’s much less likely to be abusive)
  4. Do the target and abuser follow each other (If yes, it’s dramatically less likely to be abusive)
  5. Is the abuser verified? (If so: unlikely to be abusive. In fact, in the six weeks my bot was running, it only found a couple of instances of abuse by verified users, as I’m writing this, those users are no longer bearing the blue tickmark.)
  6. Did the target favourite the abusive tweet? (A favourite is usually a strong contra-indicator of abuse).
  7. Did the target retweet the abusive tweet? (A retweet can go either way; It is usually positive, but some that are in the crosshairs of abuse retweet their abusers, presumably to shame them).

And that’s just the beginning…

  • I wouldn’t be surprised if users with a verified phone number are much less likely to be abusive.
  • If the twitter account is tied to a mobile app, it may be possible to identify what other accounts are tied to that mobile device, and draw conclusions from that.
  • Does the target block the abuser? Blocking, muting, and reporting tweets as abusive would likely be strong indicators of abuse, and could be added to the algorithms.
  • It may be possible to do IP analysis on new accounts, and see whether they are likely to be abusive or not.



  • Analysing tweets for abuse is a surprisingly difficult problem
  • Context is everything, and the historical relationship between the users in question is often a better indicator for abuse than the tweets themselves
  • The Twitter API is too restrictive to really be able to analyse a big enough dataset to draw more conclusive, er, conclusions.

article continues below

Part 4 — Solving Twitter’s abuse problem

There is no doubt in my mind that on *my* Twitter, there is no space for threats, abuse, and harassment.

Step 1 — Outline the problem, create a good policy & set clear goals

Step 2 — Full buy-in or bust.

Step 3 — Run strong post-moderation

Step 4 — Implement better preventative moderation

Verified users are given a Quality Filter, which makes some types of content disappear.

Step 5 — Close the feedback loop

It won’t be easy, but it must be done.

Try the bot yourself



Writer, startup pitch coach, enthusiastic dabbler in photography.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store