Automated sentiment analysis is less accurate then flipping a coin

When talking to customers one recurring questions is about sentiment grading. Should it be automated?

After much research we couldn’t find or build something that came close to the accuracy we wanted for our clients to be able to make informed business decisions. We spoke to third party suppliers, universities and our smartest geeks internally - whilst there are some good attempts and some great research going on we decided that humans still have the edge.

So when we are asked the question our response is - it’s best if you grade it internally (you can do this manually in our product) as you understand your business and how people talk about it, failing that you can outsource it to us and we can do it for you.

It seems from the article below we might have been right in our thinking….

“Automated sentiment analysis is less accurate then flipping a coin when it comes to determining whether brand mentions in social media are positive or negative, according to a white paper from FreshMinds.

Tests of a range of different social media monitoring tools conducted by the research consultancy found that comments were, on average, correctly categorised only 30% of the time.

FreshMinds’ experiment involved tools from Alterian, Biz360, Brandwatch, Nielsen, Radian6, Scoutlabs and Sysomos. The products were tested on how well they assessed comments made about the coffee chain Starbucks, with the comments also having been manually coded.”