Q and A with CMO and Brion

“Why are companies not testing as much as you would think?”

There are a few companies out there (just a few) that are testing.  Some do it well, but considering that there are thousands of internet companies with the traffic levels to test, I don’t know why more are not testing. But I have some thoughts. Website Optimization and testing methods are new, and with every new technology comes skepticism. Hey, when the first Television came out, people thought it was the devil’s work, and even when color TV was invented, most ranted that it “killed the silver screen”–hate to see what they did to the man who invited fire. So, believe it or not, after nearly a decade of availability, CEOs and decision makers still look at Website Testing and Optimization with scorn, skepticism and doubt.  And despite the countless surveys in its favor, are they participating? Emphatically, not. (John Lovett from a recent Forrester Research case study shared that “right now, only 41 percent of search marketers optimize landing pages” 2009-2010)

My opinion relates to the metaphor that most CEOs should view business as going to war; leading a team of employees towards financial growth much in the manner of winning in combat.  The commonality between, say, a Commander and Chief of the US Military and a CEO is the complete dedication towards procuring intelligence.  Intelligence, or “intel” is what serves the military force as the full understanding of an opponent’s tactics. By owning this information, counterattacks can be effectively maneuvered, deathly blows can be avoided, and a swift resolution can be reached. Now, when a CEO rejects testing and optimization, he is saying “I don’t want intel, I don’t want to use the latest technologies to advance my team, I don’t want information about my battle or enemy”. Each time he makes a change on his website without testing it first, it is like sending his men to battle unarmed.  But in this scenario he won’t instantly lose his front-line, he instantly loses his bottom-line.

“What does it take to have optimization work at a company?”

There are a few key elements that need to exist for optimization to work, some obvious some not. Below are some helpful tips, and these suggestions can vary, as every company is different.

1: Engineering and Implementation

You will need a nimble and fast team of engineers for optimization.  Typically one dedicated engineer will do, but it’s common for others engineers to show interest just because it’s a cool powerful technology with a simple implementation, especially if you are using Test & Target (Omniture/Adobe).  The initial implementation won’t take much time at all, from 1 day to 2 weeks depending on the extent of your implementation. If you don’t have a fast executing engineering team, you have bigger problems than implementing optimization, and if you have a bloated marketing team, I would suggest making some shifts.  With about 4 to 5 short lines of Java Script you are able to run a wide variety of testing: from simple A/B campaigns to sophisticated Multivariate Testing and so on.

2: Designer

One of the most fundamental elements that is often overlooked with optimization and will drive your operation to a complete stop, is not having access to a good designer.
The designer needs to be able to draft concepts in photoshop and it is an added plus to have the same person generate the HTML. If you already have an in-house designer, outsource a few other designers to get a better perspective of how drastic different designers are– some are better than others and you need understand this fast. Also, it is to your advantage to have more than one designer to have different variations to test. Having a great designer in place will drastically alter your process of optimization.

3: Optimization Strategist

There are different sizes of companies: some big, some very big, and some small. All of which will need a point person to run optimization. Now let me make this very clear.  You are hiring someone to run optimization and if you are a VP or CEO who likes to get your hands dirty, that’s fine– keep your hands dirty with what you were doing before, there is no reason for a career change.  You became the CEO or VP because you were talented at something— keep doing that. Your ONLY concern at this point is to ensure your team has what it needs to execute optimization. If you have comments, lightly make your suggestions on process or ideas for recipes, other than that, stay out of it. The more involved you get, the more likely you will become an Oatmeal comic and achieve nothing – Example: http://theoatmeal.com/comics/design_hell < we don’t want this.

Hiring someone to do optimization is tricky because, well, there just aren’t a lot of people who know the technology and the people who know the technology don’t always know much about strategy and to find someone who knows both technology and strategy (and other finer details like Multivariate Testing and Behavioral Targeting) that is even more uncommon. In the current market where the average person stays at a company between 1 to 2 years it is to your advantage to figure out how to keep your optimization strategist.

If and when you find someone, lets say, with 2 years of experience– continue to groom them and send them to every conference or certification possible, tell them to take off for an hour and read optimization books, (Saw them doing this in the Google office in Australia) do everything you can to ramp this person up and to continue to think about optimization 100% of the time. The reason behind this is because with optimization there are millions of scenarios to test, millions of things that can go wrong with optimization but also and most importantly there are millions of dollars to be made.

If you find someone with zero experience, and perhaps optimization ends up being a part time responsibility of your SEM guy, don’t be surprised when at the end of the year, tallying up your progress from optimization, you find part time results.

Want to go full service optimization? It can be expensive but not always,  however, this can be the best option to ramp up a new team.  But be aware that this won’t be as fast as bringing it in-house. By fast I mean fast iterative testing, which is how you increase conversions across all portions of your site.

If you find someone with 4+ years of experience, he will be able to optimize and increase conversion rates at every aspect of your site, especially traffic sources that cost money such as: Retentions Programs (email), Paid Search (SEM), Display (Banner Ads) etc. You have someone who is a seasoned expert and it will be very wise to keep him– if not, he will just go make someone else rich: your competitor? I hope not.

3: Results

From any level, optimization can be very addicting. Finding an increase in conversions in one area of your site and seeing that reflected in your sales only has you asking for more and more faster. How to get more faster is achieved by having a few key players on your team.  An analyst will help dig through some of the data, namely, generate annual increases in incremental revenue from successful campaigns, or dig around for large volumes of traffic that are ideal for testing. Include a QA team that is able to go through and review recipes, (new content to be tested, landing pages or other site pages) and check if they work and look appropriate on different browser types and versions.  Lastly, you will want to have someone assist with setting up and managing your optimization campaigns, as the end goal of a true optimization-focused company is to have multiple campaigns running all at the same time (which in my experience, can range from 5 tests to 100+).  Having all of these key players on your team, keeps the process of optimization not only running constantly but running consistently, fast and annually– each providing insight to driving increases in conversion rate.

Every company is organized differently from a getting-started point of view, so in the end do what is best for you and your group.

– Brion Hickey


6 thoughts on “Q and A with CMO and Brion

  1. Totally agree that companies don’t test their site as much as they should. I have worked with lots of companies that view engineers (fronend/backend) as a means to get the product out. Everything else will then be handled by the marketing department. However, most marketing departments have not clue as to any kind of A/B, let alone multi-variate testing. However, I have also worked at company which tested everything. Each and every landing page was tested. The difference I think was that some companies are run by engineers and other are run by MBAs and we know which group is more likely to test.

    Know any good tools/methodologies for multi-variate testing? I have only used Taguchi method in the past and I wanted to how do other methodologies compare (pros and cons).

    • brion says:

      Taguchi is pretty rock solid. I refuse to split hairs between the argument of Fractional-Factorial vs. Full-Factorial Multivariate testing. To your point, it is better to worry about more important things such as, do you have a team that is lean and mean that allows for consistent testing annually?

    • brion says:

      Omniture has great training classes if you are using TnT. Marketing Experiments has a formulaic approach to optimization that you will find interesting. Tim Ash, has a book that dives into LPO- list go on and on.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s