So it’s that time of year again when commercialism runs rampant, people spend with reckless abandon, and at any moment there could be fisticuffs at your local Wal-Mart. But alas, this is Holiday Season in America, so be joyous about it!
I’ve been watching online spending trends for the past decade and most recently tying to discern what impact mobile and social media plays in all that glitters online. All signs indicate that 2013 is door-busting records with all time highs for online sales, yet depending on which data you believe in, there’s different stories to be told.
Two analytics leaders, IBM and Adobe routinely benchmark holiday shopping. And while their methodologies differ, so too does their data. Here’s a snapshot of some of their published findings thus far:
Show me the Money
IBM’s Digital Analytics Benchmark reports a +18.9% increase from 2012 in Black Friday sales during this year’s holiday season. Average Order Value (AOV) was $135 with on average 3.8 items per order.
Adobe’s Digital Index reported slightly higher profits with a 39% increase from 2012 for a whopping $1.93 Billion in online sales. Adobe reported a similar AOV at $139 and also revealed that the peak shopping time on Black Friday was between 11AM and noon ET, when retailers accrued $150 Million during this single profitable hour.
While both companies reported lift on 2013 online sales during these two days of shopping, each indicates substantial lift in Thanksgiving Day sales, which may have cannibalized some of Friday’s profits. And while Cyber Monday numbers are still being tallied, all signs point to the biggest online shopping day yet, which likely has retailers grinning from ear to ear early on in this short 2013 holiday shopping season.
Both indices show mobile as a significant driver in online sales. Adobe reported that on Thanksgiving and Black Friday, nearly one out of every four sales was made via mobile device. IOS devices and in particular, iPads were the device of choice in both company’s findings. Adobe reported that a total of $417 Million was recognized in just two days (Thanksgiving and Black Friday) via iPad sales by businesses within their index.
This should come as no surprise to those of us following the data, but mobile now represents nearly 40% of all Black Friday traffic. That’s a trend that retailers just cannot ignore. And as a consumer, you probably can’t ignore it either. Tactics reported by IBM indicate that retailers sent 37% more push notifications via alerts and popup messages on installed apps during these two heavy online shopping days.
Where in the World?
The biggest discrepancy between the two online shopping benchmarks comes from the geographic perspective. Keep in mind here, that IBM’s Digital Analytics Benchmark is comprised of data from 800 US Retail websites; and the Adobe Digital Index data represents a wholly different set of US retailers that accrued 3 billion online visits during the Thanksgiving to Cyber Monday shopping spree. (Note that exact comparable data isn’t provided in publicly available information.)
Yet, Adobe’s data reflects the majority of online shopping on Black Friday coming from 1) Vermont, 2) Wyoming, 3) South Dakota, 4) North Dakota, and 5) Alaska. They cite weather and rural locations as rationale for these states topping the list. IBM on the other hand, indicates that on Black Friday 2013, the highest spending states from their benchmark include: 1) New York, California, Texas, Florida, and Georgia. It’s not atypical to see variances in data sets, yet keep in mind when interpreting results for yourself, it’s all about the data collection method. Results will vary based on who is in your benchmark and how you’re slicing the data.
While IBM’s early data cited in an article by All Things Digital made the outlook for social appear dreary,
Adobe weighed in with a contradictory and uplifting perspective on social. IBM did not report on social sales for Black Friday in 2013 apparently because the findings weren’t “interesting”, but their report from 2012 showed that directly attributable revenue from social media (last click) was a dismal .34% of Black Friday sales. By my math that equates to a paltry $3.5 Million total online dollars via social media sales for Black Friday. The AllThingsD reporter managed to eek out of Jay Henderson, IBM’s Strategy Director, that social sales were flat again this year. Moreover, the article quotes Henderson as saying “I don’t think the implication is that social isn’t important, but so far it hasn’t proven effective to driving traffic to the site or directly causing people to convert.” Hmm…
However, this year Adobe is telling a slightly different story. According to their Cyber Monday blog post, social media has referred a whopping $150 million in sales in just five days from Thanksgiving to Cyber Monday. While, it’s not clear if they’re tracking using a last- or first-click perspective, this data indicates that social is pulling its share of the holiday sled this 2013 season. Well, at least social is pulling about 2% of the sled based on a total of $7.4 billion in total online sales from Thanksgiving through Cyber Monday.
Whichever metrics you choose to believe, counting dollars in social media ROI is never an easy task and it usually doesn’t lead to riches. I’m about to publish a white paper on this very topic, so if you’d like to learn more about quantifying the impact of social, email me for more info.
The Bottom Line
This holiday season is shaping up to be the biggest yet for retailers of all sizes. Remember when just a few years ago people were afraid to buy ***anything*** online? Well, it certainly appears that those days are gone. So, as the days before Christmas (or whichever holiday you celebrate) wind down, and the free shipping deals get sweeter, and the door-busters swing closed until next year, take a close look at your data to see what the digital data trends leave for you.
Attributing credit across a multitude of marketing efforts is one of those sticky problems in digital analytics that seems to generate a whole lot of controversy. This is a topic that comes up with nearly all of my clients and is one that both Eric T. Peterson and I have been researching and writing about for some time now. My latest findings on attribution will be published in a whitepaper sponsored by Teradata Aster, titled, Attribution Methods and Models: A Marketer’s Framework, but you can tune in to our webcast on January 16th, to get the high notes.
While some pundits will argue that attribution is not worth the trouble and that all attribution models are flawed, others contend that attribution simply requires a healthy dose of marketing science, which will enable marketer’s to reap benefits tenfold. At the risk of opening up a whole can of Marketing Attribution worms, I’ll offer my Marketer’s Framework for Attribution, which is a pragmatic approach to organizing, analyzing, and optimizing your marketing mix using data. But first, let’s define marketing attribution:
Web Analytics Demystified defines Marketing Attribution as:
The process of quantifying the impact of multiple marketing exposures and touchpoints preceding a desired outcome.
The first question that you need to ask yourself is whether or not you really even need to include attribution in your analytical mix of tools, tricks, and technologies. I offer this as a starting point because attribution isn’t easy and if you don’t really need it, then you can save yourself a whole lot of headaches by short-cutting the process and offering a data-informed validation of why you don’t want to mess with attribution.
The approach I offer is shamelessly ripped-off from Derek Tangren of Adobe, who blogged; Do we really need an advanced attribution marketing model? Derek encourages his readers to answer this question by looking at their existing data to determine what percentage of orders occur on a user’s first visit to your website vs. those that occur on multiple visits. I bastardized Derek’s idea and applied it to help marketers understand how many visits typically precede a conversion event. While Derek offers a way to do this using Adobe Omniture, I’ve created a custom report within Google Analytics that does virtually the same thing. I call it the Attribution Litmus Test.
My version is a quick sanity check for those of you running Google Analytics to determine the number of conversions that occur on the first visit versus those that occur on subsequent visits. To use this, you must have your conversion events tagged as Goals within Google Analytics (which you should be doing anyway!). If you’d like to run the Attribution Litmus Test on your own data within Google Analytics, you can add the Custom Report to your GA account by following this link: http://bit.ly/Attribution_litmus_test. Remember that you must have goals set up in Google Analytics for this report to generate properly.
So now that you’ve determined that Attribution is a worthwhile endeavor to pursue for your organization, let’s dive into the Framework. According to a study conducted by eConsultancy, only 19% of Marketers have a framework for analyzing the customer journey across online and offline touch points. Yet, the reality of consumer behavior today illustrates that multi-channel marketing exposures and multiple digital touch points are commonplace. As such, Marketers need a method for understanding their cross-channel customers in a systematic and reproducible way.
Step 1: Identify Your Data Sources
The first step in utilizing an Attribution Framework is to identify and input your data sources. Because advanced attribution requires understanding marketing effectiveness across all channels, it means that you must acquire data from each channel that potentially impacts the customer path to purchase. Typical digital channels may include: display advertising, search, email, affiliates, social media, and website activity.
Step 2: Sequence Your Time Frame
All attribution models must consider time to understand which marketing exposures occurred first, and also to discern the latent impact of exposure across channels. This requires that organizations sequence their data. While numerous data formats will likely go into the model, we’ve seen the greatest success when attribution data is stored and aggregated within a relational database.
Step 3: Apply Attribution Models
The actual attribution models will determine how you look at your data and make determinations about which marketing channels, campaigns, and touch points are effective in the context of your entire marketing mix. There are five models that are commonly used in the attribution world: First Click, Last Click, Uniform, Weighted, Exponential. To learn more about these models, tune into the webcast where I explain each in more detail.
Step 4: Conduct Statistical Analysis
After the data has been prepped, sequenced, and cleansed; this is typically where Data Scientists conduct general queries, apply business logic, and run what-if analyses against the model. At agencies that specialize in attribution modeling like Razorfish, they have an advanced analytics team comprised of data scientists that attack the data. They’re looking for correlations to identify if users are exposed to marketing assets A>B>C, are they likely to take action D?
Step 5: Optimize Marketing Mix
Of course, the ultimate goal in utilizing an attribution framework is to make decisions that impact your marketing efforts. These decisions can be strategic such as: deciding to invest in a new social media channel; discontinuing use of a non-performing affiliate partner; or reallocating budget to highly successful channels. But an attribution model can also play a major role in making daily life marketing decisions such as: which keywords to bid on during a specific campaign; who should receive an email promotion; or where to place that out of home billboard to attract the most attention.
In conclusion, Marketing Attribution continues to be an Achilles’ heel to many marketers. But, the good news is that approaching attribution with the right toolset and a framework for solving the attribution riddle is definitely the way to go. Throughout my latest research, I talked with companies like Barnes & Noble, LinkedIn, and the Gilt Groupe to learn how they’re using and applying Marketing Attribution models. I’ve also had the good fortune to demo some of the latest attribution tools from industry leading vendors like Teradata Aster and Visual IQ. Through this research, I learned that there is some truly innovative work going on with regard to attribution, but there is no single best way to do it. I’d love to hear how you’re solving for attribution. Please shoot me a note, tune into our webcast, or comment on how you’re re-examining attribution.
Before too much time passes during these dog days of summer, I thought that I’d offer a recap of the eMetrics Marketing Optimization Summit that took place in Chicago recently. First of all, Chicago really digs analytics. Despite a smallish eMetrics crowd of around ~100 or so people, there was lots of energy, young talent and academic interest.
I had the privilege of sharing a few minutes of the opening keynote with Jim Sterne where I made a few announcements about the newly rebranded DAA (Digital Analytics Association). I proudly announced that we transitioned 25% of our Board of Directors by adding new members Eric Feinberg, Peter Fader and Terry Cohen to our diverse assembly of directors. I also took the stage in my new role as President of the DAA and shared my thoughts about the epic journey we’ve collectively embarked on in this industry that we call digital analytics. This is a theme that I reiterated during my closing presentation on The Evolution of Analytics, whereby I concluded, that the future state of evolution is up to each of us to determine.
But speaking of future success, I commend the local DAA Chicago Chapter for the great strides they’ve made in not only organizing our open industry meeting, but also in championing the cause for digital analytics in the windy city. The DAA has much better brand recognition and awareness in Chicago than I thought. But I suppose I shouldn’t be too surprised because after all, according to the DAA Compensation scan, Chicago is the second best place to live if your seeking a job in analytics.
Moving onto more details about the conference, Jim Sterne always encourages attendees to measure the value of eMetrics not just in the content, but also in the hallway conversations and the key tibits that you take back to your desk when all the sessions and lobby bar fun is over. In Chicago, for me the hallway conversations focused on several hot topics in analytics including: tag management, privacy and of course, the perennial analytics issues of people, process and technology.
I also learned (privately) that Amazon is doing some crazy brilliant stuff, but it’s so good that they can’t even talk about it. The senior brass at the really good companies are very protective, but web analysts can still be plied (at least a little) with alcohol at a Web Analytics Wednesday.
And finally, people who do know what we do are struggling to pull together the pieces for making an analytics program work…finding staff, selecting tools, building process. These are perennial issues in digital analytics and why we’ve built our consulting practice here at Web Analytics Demystified to help solve these problems.
But as always at eMetrics, I was invigorated to speak with new entrants to digital analytics and the usual suspects as well. For me, I’ll be taking from this eMetrics something back to my desk and to my clients…and that is a fresh perspective.
Anyone who has been in this game for any length of time should recognize that it’s easy to become steeped in your own myopic view of digital analytics and continue to rehash the same perennial issues with the same examples over and over again. Yet, any good analysis – or method of teaching – needs to evolve to remain relevant. And thus, for me this eMetrics taught me that experience needs to be tempered with the fresh eyes of unbridled passion and enthusiasm. While we may hold the frameworks and fundamentals, it is they who hold the spark. I for one appreciate what the next generation of digital analyst is bringing to this industry and hope to learn as much from them as I can offer.
The Web Analyst’s Code of Ethics is a reality! This Code represents an industry effort to promote ethical data practices and treat consumer data with the respect and attention it deserves.
I’m writing this on the eve before the official launch announcement of the Web Analyst’s Code of Ethics here at the WAA Symposium in Austin Texas. As you can see in the video above, this effort is the culmination of a ton of hard work by a community of contributors.
Yet, the conversation isn’t a new one. My partner Eric has been writing about the fact that We are our own worst enemy since August and our internal conversations about privacy regulation and public opinion of tracking practices have been going on long before that. The issue received mainstream attention from the Wall Street Journal in their What They Knowseries, which took a bias view in our opinion. Anything that starts out with the phrase; “Marketers are spying on Internet users…“ is FUD in my opinion.
So, in September of last year we decided to do something about it. I must say that Eric never fails to amaze me in his ability to make things happen, because not 24 hours after our conversation about launching a Code of Ethics, he had one drafted and in my inbox. We decided that the best avenue for getting this code out to the community was to work in conjunction with the WAA, where I am a member of the board. Thus, I shopped it around to my fellow board members and we all agreed that it was something that our industry needed. The issue was brought before the WAA Standards Committee and a sub-committee was formed to hash out the details. And the Code was offered to the community for public comment. After numerous iterations and literally dozens of comments and contributors, we arrived at the final Code you see here.
It’s important to recognize that this Code is a pledge for individuals and not organizations. We created it as such because we know that not every individual will be able to enforce policy within their company, but every individual can inform and educate their peers. Yet, as we state in the pledge itself, “I recognize that we are far stronger as a community…”. And this effort is about a community showing it’s commitment to ethical data collection and utilization practices.
Momentum for this project has been incredible thus far, but our work is far from over. It’s just beginning. Like any good analyst, I’ve created goals and success metrics for the code of ethics that I’ll be tracking and reporting on over time. The video above is the first effort to share a glimpse of the metrics, but ultimately I’m shooting for the following goals:
1) Gain 1,000 Pledges to the Code of Ethics in 2011
2) Attract mainstream media attention to this community effort within the first 90 days of launch (e.g., recognition by @WhatTheyKnow)
3) Ensure that our collective voice is heard by legislators and policy makers before regulation is forced upon us
Let us know what you think about the Code of Ethics here by leaving comments and joining the conversation. Or simply show your support by pledging to follow the Web Analyst’s Code of Ethics.
This weekend the Wall Street Journal produced a well researched article called The Web’s New Gold Mine: Your Secrets. Apparently, it’s the first in a series of articles about Internet tracking practices. It’s entirely informative and chock full of quotes, anecdotes, video and interesting visuals. I highly recommend giving this article a read if you subscribe to the WSJ, or encourage you to join the discussion on their blog. However, I take serious issue with the bias inherent within this first article. The author, Julia Angwin uses phraseology like “the business of spying on consumers”, and “…details about her, all to be put up for sale for a tenth of a penny”. Clearly, the conclusion drawn by the author and presented to readers is that tracking solutions are spawned from malice. I vehemently disagree.
While, it’s true that some tracking can be used for devious function, the majority of uses are fully anonymous and serve to benefit end users exponentially. The reality is that media fragmentation, facilitated by the Internet, has forced advertisers to compete for our attention. To do this, they’re hocking their wares in a significantly more relevant way. By serving up advertising content that’s based on activity, propensity and preference, they are saving us from the irrelevant fire hose of most advertising. Without being coarse, I find that the fact that some consumers are self-conscious and sensitive to advertising that’s targeted to their browsing activity as trivial. It’s trivial compared to the the benefits that targeting delivers to the rest of us.
I’ve got more to say on this topic, a lot more in fact, but I’ll stop short for now. My closing thought is that, while the author of the Web’s New Goldmine may see the art and science of tracking as a boon for advertisers… I see it as a significant win for consumers. A jackpot perhaps. I hope and expect that my online and offline interactions with brands will get increasingly better and more relevant as my interactions continue. Tracking will enable this to happen. But, that’s just me…I’d love to know what you think.
Okay…I’ve been quiet about the Coremetrics acquisition by IBM for long enough now. While the dust still won’t settle until sometime in Q3’10, when this deal passes FTC scrutiny, I’m compelled to weigh in and offer my $.02 USD mainly because there’s been some good dialog in the blogosphere from people I respect like: Eric, Joe Stanhope, Akin and more recently Brian Clifton.
I’ll take a slightly different approach and use the acquisition to talk about the state of the web analytics marketplace. For starters, let me just say that this acquisition was inevitable. So too will Webtrends be acquired by some player looking to incorporate metrics into their overarching set of technology capabilities. And as I blogged earlier this spring, yet another even bigger fish will eat the existing big fish and we’ll utter oooh’s and ahhh’s as the analytics technology market evolves into a vital organ for all businesses with a heartbeat. While not immune to arrhythmia, this course of events shouldn’t really take anyone by surprise. I’ve been saying this for a while now and even penned “Web Analytics is Destined to Become an Integrated Service” back in May 2009 when I wrote the Forrester US Web Analytics Forecast 2008-2014 (subscription required). I’ve been advocating web analytics as a function within the marketing organization, which seems to be a logical orientation. However, it’s interesting that the consumption of analytical technologies has come from a smattering of different perspectives.
Here’s how the post-acquisition landscape looks:
Adobe’s acquisition of Omniture undoubtedly took many by surprise (myself included – although you’re never allowed to admit surprise as an analyst). The promise Adobe made to investors was that they would incorporate the market leading web analytics technology into the creative life-cycle by enabling measurement at the point of content creation. Perhaps that’s not exactly how they positioned it, but that was my impression and they’re now executing on that promise. Say what you want about acquisitions and the slow moving integration process, but Creative Suite 5 debuted in April just six short months after the deal closed, with measurement hooks from FlashPro and Dreamweaver into both SiteCatalyst and Test & Target. They’ve also accomplished this remarkable feat using a visual interface allowing content editors and non power-users the ability to begin measuring their digital assets. This utilization of analytics places measurement at the operational level, yet by and large it’s still within the marketing group.
The Marketer’s Toolbox…
Enter Unica with their rebranded Marketing Innovation product suite where NetInsight (formerly Sane Solutions) web analytics sits at the core. While both Omniture and Coremetrics made pre-acquisition strides to amass a truly effective online marketing suite, they were merely playing second fiddle to Unica Campaign, Interact and Marketing Platform solutions. Unica is widely acclaimed as a leading Campaign Management tool and sits proudly in the marketing departments across many an enterprise business. They’ve worked web analytics into the DNA of their overall marketing perspective and use it to power the automation and decisioning that many organizations strive for with lust and admiration. Their utilization of analytics really does empower analytics as a lynchpin for integrated marketing.
With speculation still swirling about the how’s and why’s of IBM’s intended use of Coremetrics, it’s tough to ignore Coremetrics’ strength in the retail vertical. While Coremetrics has an impressive client based outside of retail, including publishers and financial institutions among others, they’ve clearly got some good mojo going with their triple-A retail clients. Just thinking of how Big Blue will assimilate the nimble teams of relentless Coremetrics marketers in San Mateo and Texas makes me slightly nervous. Not for any loss of focus by the Coremetrics team on their dedication to client support or from their delivery of leading analytical capabilities that they offer – rather – where will this newly acquired asset live within the IBM estate? The way I see it, two possible scenarios can play out here:
1. First is the scenario that Akin speculates upon whereby IBM is folded into the Websphere group and serves to illuminate the value of customer interactions within website platforms across IBM’s customer base. This would greatly benefit Websphere customers although it would narrowly define a finite application of a technology that is so much bigger than just online commerce.
2. The scenario that Eric envisions (and one that I believe would benefit our industry exponentially) is the one where IBM becomes the “business analytics” juggernaut in the enterprise. If this were to occur, IBM would need to integrate its SPSS and Cognos acquisitions to get really crafty about delivering extremely high value digital insights.
These are two very different outcomes and both speculatory, but I’m rooting for the latter simply because it has the potential to push analytics so much further along. My sources tell me that some long-time IBM’ers feel this way too. One confidant with access to IBM brass even shared with me that internally the acquisition will be deemed a failure by some at IBM if Coremetrics isn’t integrated with SPSS and Cognos. That’s great news, because wholesale failure of business analytics isn’t an option.
So here we have Webtrends as the only standalone web analytics player remaining from the set of truly original US-based technologies. They’re doing a good job of playing the part of Switzerland as they not-so-quietly establish a platform of Open Analytics whereby data flows in -and- out of the interface fueling other operations around the business. While this is not the same as an integrated approach, Webtrends is taking a strong stance on have-it-your-way analytics. Their open APIs and REST URLs make it easy to leverage their data collection and pump data to any application within the enterprise. Thus, they too offer an integrated approach yet do so by maintaining a position that supports rather than delivers the adjacent marketing functions.
The Low End Theory…
Any post about the state of the analytics marketplace would be remiss if Google Analytics wasn’t included in the conversation. I include the Big Googley in the Low End Theory – not because they’re trailing – but because they’re sneaky smart. Just in case you haven’t been watching, since Google acquired Urchin Software, GA has been quietly amassing millions of installations across businesses large and small adding to the democratization of web analytics. I’d argue that they’re not doing this in a concerted enterprise-wide way, but they are probably gaining the most ground across the enterprise by sheer adoption and hands-on utilization. What this means is that pockets of users are deploying Google Analytics for very focused use of the data and the organization is becoming more accustomed to seeing GA data and using it to make key decisions in their day-to-day operations.
Many other analytics programs are delivering similar value to business users, yet in an extremely isolated manner with tools like KissMetrics, Twitalyzer, Visible Measures and Radian6 just to name a few. This is truly the low end theory because the data is rarely seen by anyone outside the marketing group, but it’s driving key activity around specific marketing functions without the larger business really taking note. Think grassroots baby – under the radar – with potential super smartie effectiveness.
Can Marketing Come from the Heart?
By now you should be asking yourself; So where’s this all going? Despite how each of the companies I described above fit into the overall aspect of a company’s business, I think that we can all agree that analytics is about understanding business performance. Here is where Eric’s vision of the Coming Revolution in Web Analytics fits into the story and the quietly powerful behemoth that’s already penetrated the enterprise garden sits in wait down in Cary, North Carolina. Whether it’s SAS, another player, or an amalgamation of services from multiple players – analytics needs to be at the heart of the organization. Here’s where my analogy pays off…because if this is to happen, then data becomes the lifeblood of the enterprise and analytics allows companies to relate to their customers and offer more tuned in and relevant products and services. Marketing should control this blood flow but use it to power the brain and the working limbs of the organization. While this may start to look like Business Intelligence, I believe it’s different because it requires real-time information, automated decisioning and ultimately creativity. These are qualities that I have yet to see from a BI tool. But maybe I’m naive.
Before this diatribe gets any longer, and you dear reader need resuscitation I’ll call it quits. But I’ll offer fair warning that this is just the beginning of my thoughts on the matter and there’s more to follow. I’d also love to hear what you think.
I’ve been pondering this blog post for a couple of weeks now since I took the WAA Certification Exam along with eight others in the inaugural proctored exam at eMetrics in San Jose.
To be totally honest, I probably didn’t need to take this test. For starters, I’m not a traditional web analyst that’s down in the trenches doing the hard work of analysis, reporting and translating the massive amounts of data we’re all so fond of collecting into insights and recommendations. While these web analysts have something to prove to their organizations about the value of their jobs and the expertise they posses – frankly I have nothing to prove.
Additionally, I work for a well established consultancy with a great brand reputation and I’m not planning on looking for a new job anytime soon. Our clients are most likely going to work with us regardless of our certification status. Yet, I wanted to take this test because I do advise my clients on what they should be doing with web analytics from a strategic perspective. I speak frequently about analytics and how to interpret and deliver data in the most effective ways. So my vantage point cannot be void of practical knowledge that dictates what’s possible in a realistic world.
Thus, I took the test in part to illustrate to myself that I not only talk the talk, but am willing to put my practical skills to the test. And yes…I passed, so you’ll be seeing the CWA (Certified Web Analyst) designation show up on my credentials.
Further, many of you voted recently to elect me to the Web Analytics Association Board of Directors; and I thank you for that. I took the WAA Certification Exam, so that I could lead by example and educate others about what I genuinely believe to be a valuable test of digital measurement knowledge. I encouraged all of my fellow board members to take the test as well and several have done so and more are sure to follow.
But because I went through the experience of taking this exam, I am uniquely qualified to share my experiences that stretch way beyond the speculation of any detractors that criticize this exam. Thus, I give you the Good…the Bad…and the Ugly of the WAA Certification Exam.
This exam is a true test of analytical knowledge that requires both business acumen and a deep understanding of applied web analytics. Like all things analytics – it’s not easy. In fact, it’s downright hard. The guidance offered by the WAA regarding a recommended 3 years of practical experience is sound advice. And even then, this exam will require web analysts to dig deep into their skill set to come up with not just acceptable answers, but the best answer. Out of the initial nine exam-takers, seven passed the test, which is good. Yet, the minimum passing grade for the exam is 60% and the mean scores for our inaugural group was 61.7% (maybe I should have saved that for the ugly). The high score among all test takers thus far was 70%. While this may open questions about whether or not this test is too hard, to me it shows that there is plenty of runway for analysts to showcase their superstar skills with high scores. And if it was easy, where everyone could pass, then what validation of knowledge would that really be?
As my fellow WAA Board member Vicky Brock Tweeted: “As an employer I’d hire folk who ace this, as it tests analytical skills not recall”. Vicky also shared thoughts on her experience here. Much like Vicky, I believe this exam is a good test of knowledge that requires prospective certified analysts to know their stuff, which in turn demonstrates that the credential holds distinction.
The format is a familiar multiple choice answer system with four possible answers. Like most diligent test takers, I relied on the process of eliminating the ones that I knew were incorrect and then sorting through the remaining choices. This typically left me with two answer choices that could work, but knowing that one was better than the other, I was largely going on instinct to make the right choice. There is also a word question section that offered business scenarios and data sets leaving you to solve problems within the context of a specific business. These questions were the real gems of the exam and guaranteed to make your head spin. I love these types of questions, but perhaps I’m a glutton for punishment.
The big elephant in the room is the price. Without question, taking this exam is a financial commitment. I shelled out the bucks from my own pocket to do it because I believe in the value of certification. We as an industry are gaining momentum so quickly that analytics and data-driven cultures are all the rage today. The use of data is permeating organizations from the tactical to the strategic and ending up on the boardroom table, and in some cases, in financial analyst reports that end up on Wall Street. Yet, despite these significant gains, we have no designation to acknowledge that our Web Analysts are qualified for the job. This certification exam is that designation that will identify the truly proficient practitioners. In my opinion, this exam is worth every penny and I strongly believe that as more and more professionals acquire the CWA accreditation it will become the gold standard by which job candidates, consultants and trusted advisors are selected. When we reach this critical mass, those who aren’t Certified Web Analysts will be questioned with just cause…So why aren’t you certified?
I’ll be the first to admit that their are still some kinks in the system so it’s not perfect. Yet, nobody is so I’m willing to offer some leniency. For me, just downloading the application to sign up for the test was a chore. I offered feedback, so hopefully a fix is in the works now [there is], but when I registered the editable PDF application only worked if you had Acrobat writer on your machine, which I don’t. So after filling out the entire form, I couldn’t save it. I ended up printing out the pages and then scanning them back in to submit my application. Now, that’s more than I’d expect from your average exam taker, but I was on a mission. Also, be prepared to dig out your resume because the application requires listing all of your previous employers, their addresses, manager names and phone numbers. I was toggling between the application and my LinkedIn profile just to complete the darn thing.
**UPDATE** There is now a web based form that serves as the application, so no more downloading the PDF.
Next, it was very challenging for me to prepare for this exam. I did utilize the documents offered by the WAA including the Knowledge Required for Certification and the practice questions. The practice questions were actually great. They helped me to decide whether I was going to take the test and did closely resemble the actual questions on the test. I just wished there were more of them. The Knowledge Required document also contained a great deal of useful information, but after pouring through the 37 pages of material, I was still left feeling unprepared. The document mirrors the UBC course material, so it is thorough in describing what will be offered in terms of knowledge, but the meat of the work isn’t included in this document. It was all menu and no entree. So essentially, the document tells you what you will be tested on, but doesn’t teach any of the concepts. While they clearly state that: “Taking these four courses is not required to sit for the certification test.” those that do will be much better prepared than I was. I know that these courses are incredibly valuable and students rave about their success, but most professionals like myself don’t have the time to endure them – despite their value.
So, I already ranted about the preparation materials and the costs above, but the Ugly for me was determining if I would actually re-take this test if I failed. The feedback that I received from the WAA did contain results for the four sections that were included in the test (Analytical Business Culture, Case Studies, Marketing Campaigns, and Site Optimization) and my scores for each section. Yet, this was the extent of the feedback on my performance. It was up to me to decipher which questions may have been within each of the four categories and where I needed to focus my efforts to better prepare for a re-test. To the credit of the Association, most standardized tests are scored this way and offer similar amounts of feedback – but most tests of this magnitude also have test preparation courses that teach the skills of taking the test and offer extensive feedback on skills necessary to score well on the exam. Thus, it was ugly for me because I can sincerely admit that I wouldn’t have paid to retake this test because I do not know how I would have prepared for a second exam.
The bright spot in this potentially ugly situation is that the WAA Board is committed to endorsing organizations that choose to develop WAA Certification Exam training programs. Since this test is still very new, these programs have yet to emerge, but the opportunity is out there. I want the WAA Certification Program to succeed for the WAA and for our industry. If the test-takers are better prepared to take the test through the help of a training program, then that’s a win-win. This type of prep course would offer me the confidence I needed to take the test again if I had failed…or for those of you taking the exam for the first time. Stay tuned for more news on this front as it develops.
This post is already getting long in the tooth and I’ve said a lot. The bottom line for me is that this exam is a strong indication of the digital measurement skills that an individual brings to his or her organization. Passing the WAA Certification Exam means that an individual is an expert in the field of web analytics. It’s an accomplishment that anyone in our industry should be proud of, and one that should receive accolades on top of accolades.
But that’s enough of my rant…What do you think?
I look forward to starting a long-term dialog on this topic, so please comment, email me or otherwise shout your opinions from the rooftops.
Baseball fans across the nation were smiling this weekend with opening day games around the league. Those of you who know me, recognize that I’m a raging Red Sox fan, but last night’s 8pm start time against the Yankees was just too late for me to catch the entire game. So, upon checking the scores this morning I got to see this cool new interactive game summary on Redsox.com.
The spark lines show Tweet volume with mouseovers that offer details on each individual tweet. And the highlights are actual video clips that fire off a new window right from the summary page. Way to go MLB.com for delivering a simple, yet innovative mix of professional and consumer generated content.
Oh, yeah…the Sox beat the Yankees 9 – 7 in the opener if you’re wondering.
Being a change agent for web analytics requires taking calculated risks, standing up for what you believe, and working diligently to make our industry stronger. I left my job at Forrester Research in part to become a change agent for web analytics and my bid for a seat on the WAA board of directors is the next big step in my journey. But, this quest I cannot fulfill alone – I need your vote. I’ve never run for elected office before so to illustrate my conviction, I borrow words from John F. Kennedy’s 1960 presidential nomination speech and added a few of my own…
“With a deep sense of duty and high resolve, I accept your nomination.” I’ve stated several times before that there is no industry better than web analytics. Our colleagues within web analytics – the practitioners, the vendors, the leaders and gurus – are by and large friendly, approachable, and always willing to lend a hand. It makes working within analytics gratifying and fun. My hope is to elevate these positive attributes of our industry by aligning under the professional organization that we can call our own.
“The times are too grave, the challenge too urgent, and the stakes too high…”
Despite my positivism, we are facing turbulent times as an industry. We need to strengthen our association at a global scale to ensure that we speak with a common voice in all countries and in all languages to distinguish the Web Analytics Association as the undisputed resource for education, standards, research, and advocacy.
“…if we open a quarrel between the present and the past, we shall be in danger of losing the future.” It has recently come to my attention, despite proof from some members that not everyone receives value from the WAA. If elected to the Web Analytics Association board I will dedicate my term to proving the value of our association to members at every level, from student to vendor to advanced consultant. If we cannot recognize our own value, how then can we expect outsiders to accept our mission with the credibility and respect it deserves?
“I believe the times demand new invention, innovation, imagination, decision.” Those of you who know me recognize that I am not one to dwell on the mistakes of our past. Instead, I look to the future to determine how we can improve our situation and our position within the industry. These are exciting times for web analytics but times that will regale us to obscurity if we fail to demonstrate our vision through genuine contributions. We must think differently about how measurement technologies can be applied to today’s challenges and illustrate how the WAA is defining these efforts by taking a decisive leadership stand.
“This is a platform on which I can run with enthusiasm and conviction.”
It is this industry, its people and our collective challenges that I want to champion as a representative of the Web Analytics Association. I’ve stated my intentions here on the WAA web site, but will reiterate the most important. It’s time to stop making excuses and start delivering value to the members of the WAA. A vote for me will get you a dedicated evangelist who is willing to shoulder the burden of hard work and diligence that’s required to orchestrate change in this industry.
I welcome your thoughts and comments about how to improve our industry and will guarantee an open mind throughout my tenure on the Web Analytics Association board of directors if elected. Thanks for reading, Now Get Out And Vote!!
A few weeks ago, my business partner Eric and I attended a basketball game in Minnesota. Eric purchased the tickets a few days ahead of time and I really didn’t have any expectations going into the game except to have a great time. Much to my surprise, our seats were incredible! We were sitting immediately behind the announcer’s table in the first row. Now, keep in mind, I’m a Boston sports guy and even when the Celtics were struggling through the 90’s and the early part of this decade, you still couldn’t get a seat behind the announcer’s table or anywhere near the first row without taking out a second mortgage on your house. But, this was Minnesota and the Timberwolves are not necessarily a big market team.
Anyway, as we enjoyed the game we struck up a conversation with the woman sitting immediately in front of us who was a coordinator for the announcers. Sitting on either side of her were two official NBA scorers recording all the action into their computers and generating reports at nearly ten-minute intervals. These reports were printed and handed to the announcers, which ended up in a big pile on their desks in front of them. After a while our friendly coordinator began handing Eric and I her extra copy of these Official Scorer’s Reports. So, like any good Web Analysts would do we took a look and gave the report a critical review (see the image below).
We were astounded by how poorly constructed the reports were. Sure, they contained all the critical information on each player like minutes played, field goals, field goal attempts and total points. Yet, there were no indicators of which metrics were moving, who was playing exceptionally well, or even shooting percentages for individual players. The announcers were undoubtedly skilled at their jobs, because these reports did nothing (or at least very little) to inform them of what to say to their television audiences. Clearly the NBA could benefit from some help from @pimpmyreports.
So, here is where I get to the point about telling a story with your data. Sometime during the middle of the fourth quarter a young aspiring sportscaster came running down to the announcer’s row and handed off a stack of paper that offered some new information. Finally! His 4th-Quarternotes recap was the first written analysis we’d seen that actually placed the statistics and metrics recorded during the game into meaningful context (see image below). The 4th-Quarternotes showed that:
A win could bring the T’wolves to 3-3 in their last six games.
Al Jefferson was having a good night – approaching a career milestone for rebounds – and posting his 9th double-double of the season.
Rookie, Jonny Flynn was about to post his first double-double (which only five rookie players have accomplished), needing only one more assist.
Ryan Gomes was once again nearing a 20 point game with a 58.6% field goal percentage in the past five games.
This method of reporting used all of the same data that was contained within the Official Scorer’s Report but added historical context, which really brought the data to life. This was interesting stuff! Now T’wolves fans and casual observers alike could understand the significance of Jefferson’s 16 points and 28:27 minutes on the floor – or that Jonny Flynn needed just one more assist to achieve a significant feat. After reading this, (even as a Boston sports fan) I was invested in the game and had something to root for – Go Flynn!
So here’s the moral of the story:
If you’re going to produce generic reports with no visual cues – do not show them to anyone because they won’t use them – and make sure you hire some damn good analysts that can interpret these reports and give a play-by-play.
If you do want to distribute your reports widely – take the time to format them in a way that highlights important metrics and calls attention to what’s meaningful so that recipients can interpret them on their own.
And most importantly – place your data and metrics in context given historical knowledge; significant accomplishments; or some other method to bring the data to life. Give your executives and business stakeholders something to cheer about!
Finally, if you ever have an opportunity to sit behind the announcer’s table, make sure you befriend the coordinator so you can get a copy of the reports for yourself.
So it’s that time of year again when commercialism runs rampant, people spend with reckless abandon, and at any moment there could be fisticuffs at your local Wal-Mart. But alas, this is Holiday Season in America, so be joyous about it!
Earlier this month, I gave a presentation at the Columbus Web Group meet-up that I titled Mythbusters: Analytics Edition. The more I worked on the presentation — beating the same drums and mounting the same soapboxes I’ve mounted for years — the more I realized that the Discovery Channel show is actually a pretty useful analog for effective digital analytics.
Based on the very successful roll-out of our Advanced Analytics Education offering at ACCELERATE 2013 Web Analytics Demystified is delighted to announce our “Adobe Intensive” sessions in Portland, Oregon April 23rd and 24th, 2014. We will be packing decades of knowledge into two days of Adobe-centric training and covering SiteCatalyst, ReportBuilder, Discover, and Test & Target, all for one low price.
Over the past year or so, I’ve had the opportunity to see some "do's" and "dont's" when implementing a tag management system. I thought today I’d share some thoughts on the most important item on the "do" list: Every good TMS implementation I’ve seen is supported by a carefully planned, well-documented data layer.
A lot has been written about “big data” in the past two or three years — some say too much — and it is clear that the idea has taken hold in the corner offices and boardrooms of corporate America. Unfortunately, in far too many cases, “big data” projects are failing to meet expectations due to the sheer complexity of the challenge, lack of over-arching strategy, and a failure to “start small” and expand based on demonstrated results.