blog.brandprotect.com

Social Media Monitoring is complicated!

  

 Email Article  

Tweet  

  

  

  

Financial success on the internet is all about traffic. In the old days Meta Tags where being used to "prop up" the importance of web sites. I am sure you remember that in the early days of the web, key words were being repeated over and over again to fool traffic engines, all for the sake of SEO (search engine optimization). Well, the Googles of the world got smarter and reduced the influence of those types of old SEO techniques, but the opportunity was too big to dismiss. Smart humans always find a way! The new way to fool search engines, techniques like embedding key words, phrases and even paragraphs invisibly amongst text had successfully tricked most search engines. This text can't be seen when you read a web page, but is picked up by the search engine robots and thus make their way into search results. In most cases they have nothing to do with the context of the article, but they are highly charged with emotion to fool a search engine and they can fool us as well.

To put this all in context, let me give you some real data. We decided to use Manulife Financial as a test case to determine the accuracy of the data and our analysis. We compared our machine scored data against human validated scoring (yes, we had our people read EVERY post) and here are the results.

I won't go into specifics of this search other than reporting the numbers so you can get a feel for the outcome and the magnitude of the issues.

• Our system performed multi-source searches which resulted in 182,000 URLs.
• A number of relevancy filters were applied to only Blogs and Forums content which brought the number down to 4,239 URLs.
• Then the system performed sentiment analysis on the content, identifying 1,244 URLs that contained 5,675 mentions that carried positive, negative or neutral sentiment. Note that neutral sentiment is different from marketing messages. The latter is of no value to the client because most likely they are the client's own write up.
• To assess the quality of the machine discovered sentimental mentions, the same 1,244 URLs were routed to the human tagging pipeline. Without knowing the machine identified mentions, the human analyst examined the entire thread to identify sentimental mentions.
• The machine discovered sentimental mentions and human identified mentions where then correlated to determine the machine accuracy.
• The statistics are shown in the chart below:

So...

10% of the mentions Identified had strong sentiment and were completely relevant to the study. They are "the needles" that we are trying to identify. Human and machine were in total agreement on the sentiment polarities.

74% where correctly identified as sentimentally neutral and therefore not significant (This is the Hay). Human and machine were in total agreement on the lack of sentiment in the content.

1% where incorrectly scored. Machine assessed the sentiment as positive while human said it is negative and vice versa, or machine said there is no sentiment and human identified sentiment and vice versa. That means a polarity error of 1%.

15% where scored with correct sentiment, but human analyst considered them not relevant to the study. Something had fooled the system! On further analysis, we discovered that these posts had embedded text or were primarily designed for SEO purposes.

As far as the industry is concerned the above statistics would be incredible. Our automated systems scored the data for sentiment with unprecedented accuracy. If you assume all of the irrelevant comments were mistakes made by the system, our accuracy rate would be an industry leading 84%. If you assume that all of the irrelevant posts were scored accurately, the accuracy rating would move up to an incredible 99%.

So why are we not satisfied?

The data that we would be prioritizing for our clients would be those that had sentiment, or everything except posts without sentiment.

Out of those, the gems would be 585 out of 1,463 (585+817+61) or 40%

So we now have the real reason that customers are dissatisfied!

An industry leading 84%-99% accuracy rate means that more than every other post that a customer reviews is irrelevant.  Of course 1 in 2 is better than the 1 in over a thousand, which would be the case for unaided search.

We and the industry clearly have some work to do.  All I can tell you at this point is "We're on it!" As we are currently testing new technological advancements in this area I will keep you posted on our new test results :)