March 7, 2012

Stop Measuring Vanity Metrics, Measure What Matters Instead

Social Media is now becoming endemic in our PR, Marketing, and Customer Service plans.  But it is still a new area and presents several challenges when it comes to measuring your success.  Over the last few months as we have approached 4000 winery brands using our system, we have been continually asked “what should wineries be focused on measuring?” For starters, here is what you should NOT be focused on measuring.  Anything that is a “Vanity Metric.”  What do I mean?

Look, we all like to compete and the notion of achieving big numbers feels good.  But the reality is big numbers are meaningless without value. Some examples of Vanity Metrics include:

  • Number of Facebook Fans
  • Facebook reach
  • Number of Twitter Followers
  • Twitter Reach
  • Number of App downloads
  • Number of Registered users
  • Total email addresses
  • Klout Score

Why are these Vanity?


I am not saying you should completely ignore them as they are some metric of success, but they do not represent reality and mislead a brand in gauging meaningful performance in these channels. As an example, consider the post, The 7 Biggest Fan Page Marketing Mistakes, which dispels popular thinking when it comes to marketing and measuring the success of your Facebook fan page. In essence it says that although having more fans increases the probability of your posts being seen, it is hardly a meaningful insurance policy. Moreover the notion of “reach” for your posts that are shared or reposted is even more outrageous. It likens to the days of talking about a press placement and the circulation of the publication. The total circulation was the aspiration of the reach of the press but the reality was that only the people who read that piece within the publication mattered and no one knew how many actually read the article.

At VinTank we have assembled an amazing Board of Advisors. I am not bragging but stating a fact that these are some of the most talented visionaries in the digital arena with a keen focus on helping our industry succeed in digital and social media.  They constantly help us behind the scenes with a myriad of activities but when faced with this continual question, I went to three of them to help us bring clarity on what are the RIGHT metrics to measure in social media.  So here are three of the world’s foremost leaders in social media sharing what to measure:

Margot Savell – Senior Vice President, Measurement – Hill+Knowlton Strategies


First, in terms of strategy, all metrics should be connected to a winery’s business plan. Overall, the most important metric in social media is engagement. In terms of “owned” metrics (e.g., on a winery’s own Facebook page, Twitter account, blog, etc., where the winery activates its own messages), the most important metrics, in my opinion, are:

  • Facebook: People Talking About This (on both the page and post level)
  • # of Engaged Users (per post)
  • Twitter: # of Brand Mentions and # of Retweets (or Twitter Link Clicks)
  • Blog: # of On-topic comments by unique visitors

In terms of “public” channels (e.g., the universe of social media, where consumers activate the messages in blogs, forums, etc.), the most important metrics, in my opinion, are:

  • Share of Conversation
  • Share of Topic
  • # of Brand Mentions
  • # of Shares

Aaron Strout – Head of Location Based Marketing – WCG

For us, the biggest two are engagement (particularly on Facebook) and share of conversation (vs share of voice).

Michael Brito – SVP, Social Business Planning – Edelman Digital 

  1. Share of conversation around a given topic. For example, looking at the entire universe of conversations related to let’s say .. “basketball shoes” and the total is 1.8 million over the last 6 months, we would then extract let’s say “nike” mentions along with that term to determine the share of conversation. And then do the same for competitors. It shows relevance to a given topic.
  2. Sentiment – but not just sentiment. We are now looking to see if we can slightly alter sentiment based on messages that our clients are sharing in the social space. It’s real fuzzy and the sentiment is done by hand but the clients are asking and it’s important to them.
  3. We also categorize many of our metrics and roll them up to “Community Health” and measure engagement (RTs, mentions, likes, etc) and also watch for trending.

In sum, there are two major metrics for measuring success in social media: engagement and share of voice.  With this direction, these will be key metrics that we will be focusing on adding to our upcoming reporting features.  We will be developing reports like:

  • Group my customers by how much they have interacted with my fan page (likes, shares, comments).  How has it grown vs “x” time period?
  • Show me customers that have not interacted with my brand in more than x months.
  • Group my posts by how many interactions they have received.
  • Group my customers by how many times they have mentioned my brand.
  • Group my channels by interactions.
  • How many interactions am I getting from direct customers (people tagged as wine club members, commerce customers, etc) vs. other social customers.
  • How many trade accounts are talking about me and how often?
  • Show me my footprint vs. my competitors (enterprise version)
  • Show my my share of voice for “x” category (enterprise version).
  • http://www.blogyourwine.com/ Kris Chislett

    I’m glad someone finally addressed this issue. A perfect example of this is the new growth in “meme” Facebook Pages with huge amounts of followers. I’m sure they’re very proud of themselves, but what do these numbers REALLY mean from a business perspective…?

    Wineries suffer the same problem. I see the ones with 4,000+ fans acting high and mighty about the following they’ve built, but when you look at their engagement from fans, it’s close to zero.

  • Aaron Strout

    Thanks for including me in this. Great post Paul!

  • http://www.winetonite.com Ed Thralls

    Paul,

    From a small winery perspective how valuable can a metric of Share of Conversation, Share of Topic really be since the value will be so small?  Also, these types of metrics are tough to calculate accurately too…. unless that is you are planning to include these types of measures in the Social Connect dashboards by brand?

  • http://www.facebook.com/pmabray Paul Mabray

    We will be adding different elements of all the reporting into each of the versions.  Share of conversation/topic will probably only be for the enterprise package.

  • http://www.facebook.com/pmabray Paul Mabray

    Ed, sorry to have forgotten to answer the value for a small brand.  I think it is much like the buying adwords.  Owning a long tail adword is as valuable with more ROI than owning an expensive adword for a broad category.  So the answer is yes, depending on topic/conversation.

  • Craig Haserot

    So interaction and SOV cover payroll and pay bills?  Lol!  How about good old fashioned sales, as in Show me the Money. Still.

    I’m still waiting for anyone to show me Real ROI in social media related to wineries, as in this social media program/campaign generated this roi.  Many have promised though …

  • http://twitter.com/girlwithaglass Alana Gentry

    A short illustration…I’m at dinner in Raleigh NC, someone brings a bottle of wine from a small Napa producer who I have only met (and interacted with) online. I exclaim, “I know him! There are only 240 cases made of that wine.” I live in Napa/Sonoma. I facebooked the winemaker with a picture & story about the wine in real time. The wine enthusiast “friends” the winemaker. By the next day, the wine enthusiast had talked live to the winemaker and bought the last case available & signed up for his club. The price point is $75 a bottle. I watched the wine enthusiast share his glee with other serious wine buyers the entire week I was in NC, including a wine shop owner. Can’t win if you don’t play.

  • http://www.facebook.com/pmabray Paul Mabray

    Craig,
    I get the sneaking suspicion you are not only a disbeliever, but someone who is incorrectly equating ROI with sales.  ROI is a broad category and the goal is to get the highest Return on Investment of your time, your dollars, or your ideas.  But ROI is not solely a measure of sales for effort.  It is a measure of how much return you get vs. doing another activity.  As an example the ROI of using a spreadsheet vs. an abacus is pretty obvious.  The ROI of using a phone vs. Morse code is equally obvious.  The same notion goes for social media.  Its ROI as a communication channel is extremely positive vs. previous communication inventions.

    Moreover, since when is talking to customers, through whatever channel, not a good investment in ROI?  That almost always leads to sales (direct or indirect) when you invest the time and energy.  Why is talking to customers through social media any different?

  • http://twitter.com/RobertDwyer Robert Dwyer

    Hi Craig,

    I think social media may be spreading the word positively about your brand in indirect ways. I never would have tasted your wines in Cambridge without a heads up from @jhemming:twitter. And I never would have gotten 4 friends together to order a case without Twitter. And @adamjapko:twitter may not have placed an order with you if he hadn’t heard us all tweeting about your wines.

    I think at some point social media becomes like having a website or sending email. It’s just something that you do if you want to maintain a connection with customers.

    -Bob

    PS Love your wines!

  • Kjell Kallman

    Great story Alana. I’ve witness similar examples in all areas of marketing. The key here is to follow what Margot said and tie it back to the winery’s business plan and that would have to includes sales. But that’s really, really hard to do because there is not always direct correlation. Time passes, people start the conversation and others finish the sale and those two don’t do the “hand-off” well, so data gets lost.  

    The objective still remains – always try to tie the effort, campaign or program back to a sales result. Go look for the evidence. Tell everyone what the game plan is and ask them to go look for it and then share what they find with the entire team. And of course, use great tools like those offered by Vintank.

  • http://twitter.com/UCBeau Beau

    I first heard about Sojourn via Social Media (Facebook conversation with a wine-buddy on the East Coast) and that led to me signing up for the mailing list, buying wines, and telling my buddies here in Oregon that California still makes kick-ass pinot noirs… :) That said, I am still a bit skeptical of Social Media’s power..but it’s only because the vast majority of my winery’s current customers don’t use it to talk about wine..My job is to change that though.

  • Kjell Kallman

    Always enjoy reading your posts, Paul. I especially like the last list of bullets because they use real sentences like “Show me customers that have not interacted with my brand in x months”. That’s information someone can use to build a campaign and drive sales. You can do the ROI calculation pretty easily on that data. What are the chances we could adapt your tools to another industry ?

  • http://www.facebook.com/pmabray Paul Mabray

    Kjell, thank you so much for the kind words.  The tools work for all industries but especially any experiential ones (movies, books, software, travel, restaurants, etc).  We unfortunately just don’t have the bandwidth or resources to expand and we have a laser like focus on changing the wine industry through digital.

  • Moduswines

    I KNOW WHO YOU’RE TALKING ABOUT ALANA! HAHA!
    FYI you guys, it’s me. And she is right. SM isn’t always totally direct ie… Post a request for a sale on your FB… Orders pour in… Almost NEVER happens.
    SM, thru relationships (social), creates a path of shared conversation that can indirectly lead back to the shopping cart on your website. Or in my case, a phone call which solidifies the relationship.

  • http://www.facebook.com/AshleyPengilly Ashley Lynn

    Great article Paul. When starting for my wine company 6 months ago, I was left with the never ending task of creating an ‘online presence’ for all of our brands (30+ channels for all brands/accounts). I use the term ‘online presence’ (or OP for short) because social media isn’t only about how many fans or followers you have, but your virality online. VinTank has helped me gauge my OP by daily statistics provided by your firms highly specialized algorithm to see how much my brands have grown online over a given amount of time through mentions across hundreds of various social media portals. It also allows me to assess if I am reaching new consumers or reviving past consumers who have ‘forgotten’ about the brand, to drive these consumers to discuss and engage their family and friends with my brands… The ultimate goal of any marketing team.

    What makes being a wine social media (SM) head so special is that you understand the wine buyers mindset. A wine SM head knows it is much harder to have customers buy your products frequently and often, as most consumers tend to ‘try out’ other brands every trip to the store. My goal online is to engage those who try our wines, ask questions about our wines, visit our wineries, or are curious about the brand so that they see our brands as more than just a bottle on the shelf. I want customers to feel like a part of the brand, an exclusiveness… One of my OP missions. Although I can not measure my ROI in store sales, I can measure my viral growth through customer interactions, brand mentions online, length of conversations of our brands topics, new/past consumer capturing and thus make educated, modest estimates of our brands full exposure online over certain given amounts of time. The LITE software is the perfect set of virality ‘training wheels’ that VinTank offers. I hope in the near future we can upgrade to Enterprise so I can then measure our performance directly with my arch nemesis, other wine brands ;)  

    Cheers!

    Ashley Pengilly

  • http://www.winetonite.com Ed Thralls

    Paul does a good job explaining some obvious activities that will return higher return vs. others, a blanket response of show me the money or show me the ROI is simply a copout.  I bet most organizations couldn’t tell us right now the ROI of each of their sales channels, or the ads posted in Wine Spectator or for the 2 cases poured at the latest drunkfest at Fort Mason.  

    In my previous careers working with large global organizations with equally large investments, a thorough ROI analysis was always required.  And in that analysis there were always “intangible” benefits as well. A well-informed business decision should never be solely based on a single % or $ figure.  It doesn’t make business sense.  Also, when you are a smaller organization as are most wineries, you can’t afford to waste the time for analysis paralysis either.

    Having said that, while managing social media strategies for over 10 winery brands since 2010, I can sit here and tell you that we have had positive ROI activities and campaigns and business (as does my team) continues to grow.  Sorry, can’t share the detailed #’s due to company policy of course.  Positive results (sales included) have occurred from the following type of activities:

    1. Live Blogger tastings (both sales leading up selling tasting packs as well as downstream after bloggers content was released post-show) — also reaped 1.1M impressions reaching 55K+ potential and existing customers — the long tail of the actions from these impressions are impossible to track no matter what business you are in… just like how many people read that ad in Wine Spectator, but it must be greater than 0.  If you didn’t do it at all, you absolutely know that the return is 0.

    2. One FB channel sold several thousands of $ of wine directly from the tasting room during a CA Wine Month promotion last September ( this contradicts what ModusWines below stats would rarely, if at all, happen via FB)

    3. Increased eyeballs to brand websites.  By being in Social Media we have brought eyeballs and subsequent sales from blogs, FB and Twitter  – read that is new new clients you may not have had in your traditional sales channels

    4. Tasting room traffic for 4 brands are being tracked and reviewed for correlations to interaction and calls to action on many channels including, FB, Twitter, Foursquare, Yelp to name a few.  Currently, at most of those locations visitors are up vs. same period previous year as is engagement (via checkins, photos, posts, etc.) = increasing brand awareness and propensity to buy.

    I could go on as these are just a few of the winners we have experienced and readily report on this information monthly to our Executive team internally.

  • http://modeldeals.info/wine-pr-and-the-media-in-the-age-of-twitter/ Wine PR and the Media in the Age of Twitter « Gossip on the Web

    [...] Agile in the Age of HyperspecializationTruth in theText Part I – Is the Age of the Earth ImportantSusan’s Change The Age JourneyWine TodayAli Landry Opens Up About Cheating Ex Mario Lopez (VIDEO)Usher – Fed Up (featuring Usher, Young Jeezy, Drake, Rick Ross) :Celebrity News ScoopMy Love Affair With Sommelier Spokespeople + How To Make It A Happy Marriage – The Buzz Bin‘Cloud Atlas’ Felt ‘Impossible’ To Susan SarandonStop Measuring Vanity Metrics, Measure What Matters Instead – VinTank [...]

  • http://www.designbyhumphreys.com/vanity-metrics/ Vanity Metrics | Design by Humphreys

    [...] Vanity Metrics.  The term was first coined by Eric Ries in his book Lean Startup.  So what are Vanity Metrics?  Here are some examples from over at Vintank: [...]

  • http://www.scoop.it/t/running-lean/p/3994865609/stop-measuring-vanity-metrics-measure-what-matters-instead-vintank Stop Measuring Vanity Metrics, Measure What Matters Instead – VinTank | Running Lean | Scoop.it

    [...] Social Media is now becoming endemic in our PR, Marketing, and Customer Service plans. But it is still a new area and presents several challenges when it comes to measuring your success. Over the last few months as we have approached 4000 winery brands using our system, we have been continually asked “what should wineries be focused on measuring?” For starters, here is what you should NOT be focused on measuring. Anything that is a “Vanity Metric.” What do I mean?  [...]

For Your Continued Enjoyment...

January 3, 2013

The Atomic Weight of Social Media Interaction...

As we continue to create tools to better extract ROI or make social media easier,... more

March 19, 2012

The Profiles of Social Media People in the Wi...

We’ve had the privilege to see the wine industry evolve over three years in the usage of... more

March 17, 2012

Unraveling the Mystery of ROI in Social Media...

“Show me the ROI from social media.” I can’t tell you how often I hear... more

February 19, 2012

VinTank 2.0 Advanced Features: Innovation thr...

Yesterday, we quietly launched our new advanced features on VinTank Social Connect.  As promised, the... more