Metrics — nPost

Out of the cesspool and into the sewer: A/B testing trap

June 21, 2010 By Jason

Your A/B tests are trapped in a cesspool when they should be in the sewer.

Do you really care why A/B testing is analogous to unwanted liquids? Not yet, so I’d better get right to the point.

On the rare occasion that it rains in Austin we get these deep puddles in the backyard. Of course it would be better if the water would flow out into the street and into the sewer, but that’s not how gravity works.

Water “seeks” the lowest point in the yard, but it’s narrow-minded. It doesn’t survey the environment, locate the lowest area, and head there. Rather, at each point along its path it chooses whatever direction is lowest in the immediate vicinity. Water doesn’t “know” that if only it made the effort to hop over the fence, it could get much lower, like in the sewer.

In mathematical terms, water doesn’t “globally optimize” for getting to the lowest possible point, but rather “locally optimizes” at each step. If you enjoy clichés, water misses the forest for the trees.

Maybe your A/B tests are missing the forest for the trees too.

A typical A/B test looks like this: You start with a baseline, then you make a change. Maybe the title changes from “Sour Cream Getting you Down?” to “Don’t know when Sour Cream Goes Bad?” You test that for a while and one wins, and then you try another variation: “Is this Sour Cream Good or Bad?”

And so on, inching your way through incremental improvements. A little here, a little there, and — you believe — soon it adds up to real money.

Except, often it doesn’t.

Often what happens is you get to a point where small changes aren’t doing anything. It can be hard to recognize this effect which is why you need to (horrors!) use math to decide empirically whether anything’s actually happening.

At this point you might be tempted to give up, but that’s wrong too.

What’s happened is that you’ve found what mathematicians call a “local minimum” and what I just called a “cesspool” (and what more tasteful writers call a “watershed.”) Your test is the water in the backyard — you’ve flowed into the lowest point, but you’re still in the backyard!

Completely changing your perspective, your message, your layout, your value proposition, your colors, or your target audience might reveal an entirely new, discontinuous, non-incremental change. The real fun is in the sewer; you need to jump over the backyard fence.

In fact, because looking in completely new places has the potential to yield far more results than incremental improvement, you need to be looking for discontinuous results from the start.

The best idea is to do both: Instead of just running A versus incremental-change A₂, also run a B version that’s radically different from A. Thus you reap the straightforward benefits of incremental improvements while also searching for something that could radically improve your revenue.

Better still, if a radically different message gets you massively better results, perhaps all your messaging should change accordingly. Maybe your idea of what the market wants should shift. Maybe your entire business can change for the better.

Why poop along with minor variations when you could be toying with new ideas and new identities?

Play!

What strategies do you use for tests? Leave a comment and join the conversation.

Join me in my new startup WPEngine: Super-fast, super-secure WordPress hosting.

Filed Under: branding, Essays, lead-gen, Metrics, positioning, testing

The Pattern-Seeking Fallacy

June 14, 2010 By Jason

What do these have in common?

“This pitcher has retired 5 of the last 7 batters.”
“We tried 10 AdWord variants and combination D is the clear winner.”
“The Bible Code predicted the Sept 11 attacks 5,000 years ago.”
“We sliced our Google Analytics data every which way, and these 4 patterns emerged.”

All are examples of a common fallacy that I’m dubbing the “Pattern-Seeker.”

You probably laugh at Nostradamus, yet it’s likely you’re committing the same error with you own data.

Patterns in Chaos

It’s commonly said that basketball players are “streaky” — they get on a roll hitting 3-pointers (have a “hot hand”) or develop a funk where they can’t seem to land a shot (“gone cold”). These observations are made by fans, announcers, pundits, and the players themselves.

In 1985 Thomas Gilovich (featured in the entertaining book Innumeracy) tested whether players really did exhibit streaky behavior. It’s simple — just record hits and misses in strings like: HMHHMMMMHMMHH, then use standard statistical tests (specifically autocorrelation) to measure whether those strings are typical of a random process, or whether there was something more systematic going on.

Turns out players are not streaky; simply flipping a coin produces the same sort of runs of H’s and M’s. The scientists gleefully explained this result to basketball pundits; the pundits remained non-plussed and unconvinced. (Surprised?)

So they tried the same experiment backward: They created their own strings of H’s and M’s with varying degrees of true streakiness and showed those to pundits and fans, asking them to classify which were streaky. Again they failed spectacularly.

We perceive patterns in randomness, and it extends beyond casual situations like basketball punditry, plaguing us even when we’re consciously trying to be analytical.

Take the “interesting statistic” given by the baseball announcers in the first example above. Sure the last 5 of 7 batters were retired, but the act of picking the number “7″ implies that number 8 got on base. Maybe number 9 did too. Of course saying he “retired out 5 of 9 batters” doesn’t sound as impressive even though it’s the same data!

But unlike the basketball example, the baseball announcer’s error runs deeper, and following that thread will bring us to marketing data and the heart of the fallacy.

Seeking combinations

Baseball records a dizzying array of statistics which announcers — or more correctly, staff statisticians — eagerly regurgitate. Maybe it’s because baseballers are a little OCD (just look at pre-bat and pre-pitch rituals) or maybe it’s because they need something to soak up the time between pitches, but in any case the result is a mountain of data.

Announcers exploit that data for the most esoteric of observations:

“You know, Rodriguez is 7 for 8 against left-handed pitchers in asymmetric ballparks when the tide is going out during El Niño.”

This is the epitome of Pattern-Seeking — combing through a mountain of data until you find a pattern.

Some statistician combed through millions of combinations of player data and external factors until he happened across a combination which included a “7 of the last 8,” which sure sounds impressive. Then he proudly delivered the result as if it were insight.

So what’s wrong with stumbling across curious observations? Isn’t that exactly how you make unexpected discoveries?

No, it’s how to convince yourself you’ve made a discovery when in fact you’re looking at pure randomness. Let’s see why.

Even a fair coin appears unfair if you’re Pattern-Seeking

The fallacy is clearer when you look at an extreme yet accurate analogy.

I’m running an experiment to test whether a certain coin is biased. During one “trial” I’ll flip the coin 10 times and count how often it comes up heads. 5 heads out of 10 would suggest a fair coin; so would 6 or even 7, due to the usual random variations.

What if I get 10 heads in a row? Well a fair coin could exhibit that behavior, but it would be rare — a 1 in 1024 event. So if my experiment consists of just one trial and I get 10 heads, the coin is suspect.

But suppose I did a lot of trials, like 1000. A fair coin should still come up heads 3-7 times per trial, but every once in a while it will come up 9 or 10 times. Those events are rare, but I’m flipping so much that rare events will naturally occur. In fact, in 1000 trials there’s a 62% chance that I’ll see 10 heads at least once.

This is the crux of the fallacy. When an experiment produces a result that is highly unlikely to be due to chance alone, you conclude that something systematic is at work. But when you’re “seeking interesting results” instead of performing an experiment, highly unlikely events will necessarily happen, yet still you conclude something systematic is at work.

Bringing it home to marketing and sales data

Let’s apply the general lesson of the coin-flipping experiment to Google Analytics.

Take Google Analytics. There’s a hundred ways to slice and dice data, so that’s what you do. If you compare enough variables enough ways, you’ll find some correlations:

“Oh look, when we use landing page variation C along with AdWord text F, our conversion rate is really high on Monday mornings.”

Except you sound just like the baseball announcer, tumbling combinations of factors until something “significant” falls out.

Except you’re running 1000 coin-flip trials, looking only at the trial where it came up all heads and declaring the coin “biased.”

Except you’re seeing streaks, hoping that this extra-high conversion rate is evidence of a systematic, controllable force.

So what’s the answer?

The fallacy is that you’re searching for a theory in a pile of data, rather than forming a theory and running an experiment to support or disprove it.

So:

Instead of running multiple AdWords variants each against multiple landing page variants each feeding a different website funnel, run just one experiment at a time, one variable at a time.
Instead of using a thesaurus to generate 10 ad variants, decide what pain-points or language you think will grab potential customers and test that theory specifically.
Instead of rooting around Google Analytics hoping to find a combination of factors with a good conversion rate, decide beforehand which conversion rates are important for which cohorts, then measure and track those only.

More!

Do you have more examples of what to do or what not to do? Leave a comment and join the conversation.

Filed Under: Essays, fallacies, Metrics

Win/loss analysis: your process to more closed deals

February 11, 2010 By Nathan Kaiser Leave a Comment

This week I sat in all too familiar customer meeting with the CEO, VP of Sales and the Marketing Director to do a pipeline review. The Sales VP discussed the deals they won and which opportunities were lost. And you can probably guess my first two questions. Why did we win? Why did we lose? And you can probably imagine the answers. If “we won because the sales person has a great relationship with the buyer” and “we lost because our price was too high” were on your list, you guessed correctly. Unfortunately these kinds of answers don’t cut it in today’s environment where every deal matters. There’s no time like now to initiate a win/loss analysis for your company.

Win/loss analysis isn’t about determining who is “at fault” for a lost deal; it is about gaining insights to improve results and bolster revenue. The value of win/loss analysis and data is in its ability to affect sales training, define marketing strategies, and prioritize product development efforts all aimed at one thing – improving your organization’s competitive advantage. When done properly win/loss analysis provides clarity and insights into customers’ perceptions of your product, experience throughout the sales cycle, and expectations created by your company messaging. Win/loss analysis is not a customer satisfaction study. It is a process for differentiating why one sales effort wins and others fall short of the mark in order to adjust go-to-market strategies and tactics. The purpose of win/loss analysis is to learn the pros, cons, likes, dislikes, competitive advantages and disadvantages from the specific people responsible for the purchase decision.

When conducted properly a win/loss analysis helps a company answer these questions:

1. Why do customers select your products and/or services?

2. Why did your prospects select your competitors’ products and/or services and why they didn’t select yours?

3. How do your competitors position themselves when they compete with you?

4. How do your customers and prospects perceive your sales and marketing efforts?

5. How do your customer’s and prospects perceive competitors and their products/services?

6. What are the most important criteria a customer looks for when selecting products and/or services in your category?

7. How effective is your marketing and sales team in presenting your company, your value proposition, and your products and/or services?

Many companies think they know the answers to these questions based on anecdotal information from their sales organization. Perhaps you’ve even heard something similar to this from a sales person: “We could have won this deal if we had X feature in the product.” Maybe adding the feature is the right thing to do, but maybe it isn’t. Adding a feature for a market of one is a very expensive undertaking. Using anecdotal information creates a reactive rather than a proactive process. The implications of win/loss analysis extends beyond your sales team and should provide a complete picture into your enterprise’s and the competitions product, services, price, sales channel and marketing and the prospect/customer evaluation process.

Topics to Include In The Discussion:

Because you are conducting the win/loss analysis to learn why you are winning and why you are loosing in order to understand what are doing right and correct what you are doing wrong you’ll want to be sure you analysis includes the following topics:

1. How they found out about your company and the product category.

2. What problem were they trying to solve with your product or service?

3. Who they listened to or went to for advice during the buying process

4. Whether there were any breakdowns during the sales cycle and if so what were they and where.

5. What the competition doing right and if they won, why?

6. What you would need to do in terms of technology, service, selling, product, etc to win have won the deal.

When and How to Conduct the Analysis:

Win/loss analysis should be performed shortly after the deal is completely closed – about 2-4 weeks after the deal is concluded. By closed we mean either the competition or you have solidly won the deal (the contract is signed and the purchase order has been issued). Avoid doing the analysis before this point because you don’t want to throw any wrenches into the process. And you don’t want to wait to long after the deal is closed because the buyer’s memory will fade and the conversation will turn more toward what they are experiencing with the product/service now rather than their experience and thinking during the buying process. Remember, the goal is to measure what happened during the buying process.

Successful analysis rests on being able to capture unbiased in-depth information with all the key decision makers and influencers If the analysis can be conducted in person that is ideal, however, this is often impractical so phone interviews are the most common approach and are by far better than just having these people complete a written survey. Written surveys sometimes supplement telephone interviews when you require more detailed help in ranking customer wants and needs.

It helps to think of the analysis consisting of three steps: pre-interview where you define the key questions, develop the interview list and schedule the interviews; the interview; and post-interview where you analyze and report on the findings and debrief the team.

Should you fly solo? If you are attempting this solo, be sure to explain upfront that the purpose of the interview is to learn as much as possible about the customer or prospect’s perceptions and experience during the recent sales process so your organization can continually improve and to state that all the individual feedback is confidential. Oftentimes companies have sales or marketing people perform the win loss analysis and this can result in skewed data. Relying on an outside organization to conduct win/loss analysis is a good idea because it allows for more candid and detailed responses. The reasons offered by customers and prospects for winning or losing are surface-level only such as price, feature set or lack of budget. It takes an experienced interviewer to glean the underlying reasons and then proper analysis to identify the strengths and weaknesses of your competition (beyond their product) and the patterns that you can use for setting strategic and tactical direction on sales, marketing and product development hiring, training and management. While using an outside party may appear more expensive the benefits outweigh the cost. Prospect and customers tend to be willing and candid with a third party and can really provide the individual’s confidentiality. Third parties don’t have a vested interest in a particular answer, their goal is to seek the truth and as a result bring the perception of objectivity to the process. An additional benefit is that an experienced third party brings expertise including framing the questions, analyzing the results, and identifying the patterns that will affect key decisions.

By integrating win/loss analysis as an ongoing process you will have data in real-time that sales, marketing, and product development can use to act and adjust more quickly to offset problems and exploit advantages. Institutionalizing win/loss analysis will contribute requirements to product development, feedback about messaging to marketing, and may help uncover new sales strategies and initiatives. For win/loss analysis to be beneficial it needs to be done in a timely fashion with accuracy and objectivity.

VisionEdge Marketing, Inc, is a leading data-driven metrics-based strategic and product marketing firm located in Austin, Texas. The company specializes in consulting and learning services that help organizations use data to make fact based decisions to address market, customer, and product opportunities and to improve and measure marketing performance. For more information, go to www.visionedgemarketing.com.

Filed Under: Laura Patterson, Metrics Tagged With: marketing, sales

Measuring Pipeline Velocity – An Important Sales and Marketing Metric

January 14, 2010 By Nathan Kaiser Leave a Comment

We’re often asked, what is a good metric other than pipeline contribution and conversion rates we should consider. Here’s a metric that is often overlooked – Pipeline Velocity. Pipeline Velocity is a metric many marketing and sales organizations could benefit from using because it measures the rate of change within your pipeline—both in speed and direction. If you haven’t calculated pipeline velocity before, you may want to consider using this metric as a way to provide insight into how Marketing is affecting the business. To calculate your pipeline velocity you are going to need to know the following four variables:

Pipeline Volume
Sales cycle movement
Length of the sales cycle
Average dollars/sale

By taking these four variables, sometimes metrics in and of themselves, and summing them into a single number you can assess whether your sales are accelerating, decelerating, or remaining status quo. You may also need to review how your collecting data related to the number of opportunities and conversion rates so that you can properly calculate the Pipeline Velocity metric.

Other information helpful in assessing your Pipeline Velocity include:

a. Annual Revenue Goal ($)

b. Avg. Length of Sale (days)

c. Sales Stage Probability (%)

d. Opportunities per Stage (#)

e. Sales Booked To Date ($)

f. Stage Conversions Made (#)

g. Months Remaining In Fiscal Year

Once you begin to track your pipeline velocity, it may become evident that you need to revisit your marketing and sales processes to determine whether any adjustments are needed.

Filed Under: Laura Patterson, Metrics Tagged With: marketing, Metrics

The Strategic Plan Serves as the Beacon for How to Measure Marketing

November 19, 2009 By Nathan Kaiser Leave a Comment

How do you know whether your marketing plan is on target? In today’s environment the more on target we are the better. If your marketing plan is based on the organization’s strategic plan then odds are you’re heading in the right direction. If there is a disconnect between the marketing plan and your enterprise’s strategic plan, then odds are you are setting you and your team up for trouble. Some people tell us that their organization doesn’t have a strategic plan. If that’s your situation, then insist on one before developing the marketing plan.

Why? Because the strategic plan is what binds all the different parts of the organization together so that each group knows what It needs to do to move the business forward. It is what the entire leadership team should use to define what success looks like in the future. This picture of the future is derived from a shared base of knowledge created by analyzing market, customer and competitive trends and the organization’s strengths and weaknesses. The process provides a disciplined approach for looking at external forces, such as economies, markets, competition, customers, suppliers, etc, considering a variety of potential scenarios, and exploring new opportunities for growth. As part of the process, the organization can determine how to best use its strengths while mitigating any weaknesses and examine the impact of new markets, products and services in order to achieve future success. By formulating a picture of the future, the organization is defining success and how success will be achieved and measured.

Based on the analysis and the vision, we recommend that the leadership team select a manageable number of key objectives – typically no more than 7 to 10, and less can definitely be more – to accomplish over the next 18 months to 3 years. These are the key objectives that if accomplished will enable the organization to achieve success. These mission-critical objectives provide the foundation for the work to be performed and the parameters by which success is measured. They define the company’s priorities and enable the rest of the organization to allocate resources accordingly.

These objectives in the strategic plan become the business outcomes and serve as the stakes in the ground around which each part of the organization builds its operational plan. Whether you’re in sales, marketing, engineering, manufacturing, customer services, these initiatives are the basis for your plan. This is why the strategic plan is so important – it is the cornerstone for action. Many of you tell us you are revisiting your marketing plans in light of the current economic environment. As you solidify your plan, be sure you have the strategic plan front and center so you the marketing objectives and performance metrics are aligned around what matters most to the organization.

Filed Under: Laura Patterson, Metrics Tagged With: marketing, Metrics

Getting Marketing and Sales to Tango

September 10, 2009 By Nathan Kaiser Leave a Comment

Spend a day inside most business-to-business organizations and you’ll come to the conclusion that Sales and Marketing need to be better aligned. The source of frequent friction and open-conflict between the two revenue-generating departments is almost invariably traceable to the topic of “sales leads.” Ask the Sales Manager and he’ll complain that Marketing isn’t providing enough high-quality leads. Spend a few minutes around the water cooler with the Marketing Manager and you’ll hear tales of how Sales practices lackadaisical lead follow-up and poor record keeping. This situation is widespread and extremely detrimental to the profitability of organizations both large and small. Serious money is being spent every quarter on lead generation programs and headcount within Sales and Marketing departments. The financial health of the organization depends on these two organizations working together productively.

So, what’s a CEO to do?

First, ask your Sales Manager if he/she knows how many deals need to be closed in order to reach the revenue objectives for the next two quarters. Odds are pretty high that the Sales department isn’t thinking of number of deals, so they don’t have a solid handle on the amount of effort necessary to generate the required level of deals. Many companies, sadly, don’t even know the average selling price of their products.

Next, stop by the Marketing Manager’s office and ask him/her how many contacts their programs need to generate to deliver a closed order. Chances are good that your Marketing ace may have data related the conversion of impressions to leads, but struggle when it comes to knowing how many qualified leads convert to prospects. So, the Marketing department won’t be able to say how much effort (lead generation) is required to get a deal either.

It’s time for the CEO to insist that Sales and Marketing integrate the lead pipeline with the sales pipeline into a “buying pipeline”, a customer-centric planning and management tool that keeps the company steadfastly focused on the customer at every stage of the cycle. The buying pipeline tracks the entire course from target to contact, to suspect, to lead, to qualified lead, to prospect and, finally, to customer. Using a buying pipeline approach encourages Marketing and Sales to work together to understand where each opportunity is in the pipeline and who has the primary responsibility to move it forward. A useful guideline is for Marketing to own the processes from target to qualified lead; and Sales to own the processes from qualified lead to customer. Ownership does not mean that the two functions work in isolation. Unfettered communication between the functions is essential for both developing and honing the pipeline.

A solid pipeline tool also details the many steps and the consequent time involved in each phase. It’s an easy leap to see how the pipeline can be used to create a dashboard for measuring marketing and sales progress toward revenue goals.

If you embrace this approach, start the buying pipeline as far back as the target, rather than later with a qualified lead, the company can gauge which programs are most effective in reaching viable targets. Beginning the analysis at the qualified lead stage reveals only half of the story.

Each stage of the pipeline has different demands for the company and product. For example, at the target and contact stage marketing must create awareness for the company and product. After all, people buy from people they know. At the suspect and lead stage, marketing is focuses on getting the audience more familiar with the company and product. People buy from people they like. At the lead stage, Marketing and Sales collaborate to determine which leads are qualified so the best opportunities can be pursued. It is at this transition stage that sales takes the helm in pursuing qualified leads while marketing needs to nurture those opportunities that are not ready to be harvested.

At each stage along the way monitor your progress. You’ll want to keep tabs on how many people/companies populated each stage and how many advanced to the next stage. With a little history, a company can assess its conversion ratio and begin to fine-tune its efforts, propelling more opportunities to conversion faster. Over time, employing a buying pipeline provides a consistent way for both small and large companies to track conversion ratios, identify bottlenecks, and monitor the sales cycle. Most important of all, a buying pipeline enhances a company’s ability to win new customers–and thrive.

Filed Under: Laura Patterson, Metrics, Startups Tagged With: alignment, marketing, pipeline, sales

Applying Six Sigma to Marketing to Grow Revenue

August 13, 2009 By Nathan Kaiser 1 Comment

As someone who worked for Motorola from the early 80s until the late mid-90s I had an opportunity to be a part of the Six Sigma Era. Even though Six Sigma as a measurement standard originated in the 1920s, Motorola is credited with applying the methodologies and coining the term “Six Sigma.” The philosophy behind the Six Sigma approach is if you can reduce process variation, you can improve organizational effectiveness and efficiencies. According to General Electric (GE)–an early adopter of the program–Six Sigma is a “disciplined methodology of defining, measuring, analyzing, improving and controlling the quality in every one of the company’s products, processes and transactions–with the ultimate goal of virtually eliminating all defects.” Originally used to improve engineering and manufacturing, the Six Sigma approach has expanded to include all aspects of organizational performance, including marketing.

Six Sigma enables companies to improve the marketing’s strategic, tactical and operational processes as a way to enhance the top line to drive revenue. By applying Six Sigma to marketing you can develop a lean efficient marketing workflow, identify leading indicators of growth and become proactive about performance improvement. Measurement of performance is one of the five fundamental phases in the Six Sigma methodology. Once you begin measuring marketing performance, you can begin to make modifications and improvements. Six Sigma provides both a methodology for process improvement and a way to prove its value.

One of the key methodologies associated with Six Sigma is DMAIC. DMAIC is used to improve existing business processes. DMAIC includes five steps: define roles, goals, and deliverables consistent with customer demands and the organization’s strategy; measure current performance and processes, and collect relevant data for future comparison and improvement, analyze the relationship and causality factors; improve the process to eliminate defects; and control and correct any variances before they result in defects and thereby improve performance. The five steps for DMADV include: define the goals of the design activity, measure and identify the critical quality, product/process capabilities, analyze to develop and design alternatives to determine the best design, design the process and verify the design. By using the methodology you can create a data-driven, systematic approach to solving business problems that will have a positive impact on customers.

Let’s consider how we can apply the DMAIC process to marketing to grow revenue.

1. Define: The role of marketing is to create predictable streams of revenue growth by enabling the organization to profitably identify and secure new customers, and to keep and grow the value of these customers. Therefore, a key ingredient in this step is for marketing to establish goals and deliverables designed to achieve these three outcomes. To fully realize these three outcomes, the various marketing functions will need to be integrated to create a comprehensive and integrated workflow process. This integrated workflow process will then need to be mapped. Once these three elements are completed, new metrics that tie marketing to the business outcomes must be defined and standardized across the marketing organization for the purpose of providing insight into performance and facilitating strategic decisions.

2. Measure: There is no escaping the fact that to be successful in measurement marketing will need data. Without data, performance cannot be measured and improvements cannot be made. The first step in measuring and improving performance is to determine what data exists, where that data is, what data is needed, and how to obtain the data. Customer purchase activity, marketing program results and conversion rates, actual costs for programs and people, lead quality data and lead cost, win/loss ratios, and defections that occur in the buying process are examples of some of the data that will be needed. Once the metrics are defined, the team should use the data to establish a baseline of past expense and performance.

The metrics should be defined not just in terms of the cost but also in terms of how these investments contributed to the company’s ability to achieve its goals and generate profitable revenue. The marketing metrics are contingent upon knowing the business outcomes. It is imperative that the business outcomes be clarified and specified before the marketing metrics are established. Business outcomes for example may be related to the specific number of customers to be acquired and at what cost, the specific rate of customer acquisition, the specific lifetime value of a customer, customer loyalty, and specifically how quickly customers adopt new products. By knowing the business outcomes, marketing knows what objectives it needs to achieve and within what parameters. Marketing can now establish the metrics, the performance targets and processes, and measure its performance. Tying marketing metrics to business outcomes forces marketing to transform from a transactional function to a strategic contributor.

3. Analyze: Simply measuring performance will not make it improve. Performance improvement results from deriving insight through the analysis of the data. By analyzing the data and understanding what it means, marketing can determine the degree of impact it is having on the organization, and redesign processes that will improve performance. Creating a dashboard of key business initiatives can help process the data and make it easier to visualize both the impact and opportunities for improvement. Analysis leads right into the improve step.

4. Improve: A performance-driven organization welcomes opportunities for improvement. The main purpose of applying Six Sigma to marketing is to determine how to improve performance and processes. Data analysis should result in valuable insights that generate possibilities for improvement. These possibilities for improvement can include enhancements in tools, systems, processes, and skills. Even though change is disruptive, developing new ways to approach the market enables the marketing organization to play a more strategic role.

5. Change and Control: Because marketing prides itself on its creativity, it has often sacrificed control. But the time has come for marketing to document its processes and best practices and to apply these consistently in order to optimize marketing execution.

Applying Six Sigma to marketing will increase marketing’s ability to deliver on market requirements, improve the efficiency and effectiveness of the marketing planning process, successfully manage marketing operations, provide transparency into marketing processes, and improve the collaboration between marketing and other groups within the business.

Filed Under: Laura Patterson, Metrics, Startups Tagged With: marketing revenue, ROI

Does your marketing team have the right stuff?

August 4, 2009 By Nathan Kaiser 1 Comment

The 1979 book (Tom Wolfe) and 1983 movie (Philip Kaufman), The Right Stuff chronicled the transition from breaking the sound barrier to the Mercury space expeditions. The book and subsequent movie explored why the Mercury astronauts accepted the danger of space flight and the mental and physical skills required of them to do their job, that is the right stuff. The inspirational story provided insight into the character and caliber of these dedicated professionals. The current economic challenges and ever-present market dynamics beg the question for companies of all types, “Does Our Marketing Team have the Right Stuff?”

Here are four skills and four tools, the right stuff that will enable every marketing organization to fulfill the charter to be a business driver.

Enhancing the skills of existing talent
Regardless of company size and industry, marketing teams (whether a team of one or more) are under increased pressure to drive top-line growth and profitable revenue. For many organizations this means acquiring new skills related to marketing performance measurement and management, analytics, benchmarking, and customer engagement. Let’s review these four specific skills every marketer should have under their belt.

Metrics and performance target setting – with greater demand for marketing to be more accountable, solid metrics, performance target setting, measurement and reporting skills are a must. Be sure your marketing folks know how to set measurable goals and track results.

Analytics – the ability to derive insights from data. If growing valuable customer relationships and being able to forecast sales from future marketing activities are important, then analytics ought to be among your marketer’s top skills.

Benchmarking – the process of comparing what your company does to another that is widely considered to be an industry standard or best practice. If you don’t know what the standard how will you know what to strive for when it comes to such things as win/loss ratios, marketing key performance indicators, share of preference, product adoption rates, and so on. Benchmarks are essential to any organization that believes continuous improvement is critical to the pursuit of excellence.

Customer Experience Management – If business exists to produce and serve a customer and marketing’s job is to create, communicate and deliver value to customers, then marketing is your organization’s ultimate steward of the customer experience. Marketers need to be sure they have the skills necessary to improve customer engagement and touch point effectiveness, respond to changes in the buying cycle, and conduct voice-of-customer research in order to retain customers, create loyalty, and transform customers into advocates for the company.

Driving operational efficiencies
Marketing operations refers to infrastructure, that is, the tools, systems and processes in place to facilitate customer-centricity. For many organizations achieving these operational efficiencies requires infrastructure changes and improvements. With limited resources, where can you get the best bang for your buck? Here are four areas for investment consideration.

Operational Process Alignment – When was the last time you mapped your operational processes and verified marketing alignment with the sales, product, service, and other parts of the business? All of us get into routines and habits. How many times have you said something such as, “that’s the way we do it here?” Reviewing processes and updating them may be time consuming, but if you are looking for ways to reduce inefficiencies internally, this is necessary step.

Market/Business Intelligence – There is an art and science to using external information to drive business strategy. Business intelligence applications enable the collection, integration, analysis and presentation of competitive, channel, product and customer information to derive trends and insights. The value of having such a tool is that when used properly it enables you to begin to conduct scenario analysis and anticipate the future. With the insights derived from business intelligence there is the potential to anticipate the development of new markets, technological turning points, and how competitors will react.

Customer Relationship Management (CRM) – If the marketing organization is responsible for the relationship between the company and the customer, then it stands to reason the organization needs tools to facilitate this relationship. CRM systems automate the processes an organization uses to organize and tracks contact with its current and prospective customers. As you can see from this definition CRM is both a process and a tool. There are a range of CRM tools, so selecting the right one can be a daunting task. Even so, in today’s environment a company can’t afford to operate without a formal approach to customer relationship management. Of course, once you have the tool the next biggest hurdle is using it.

Performance Management – The ability to use analytics, reporting, and dashboards to assess marketing’s effectiveness, efficiency, financial contribution, and progress toward achieving predetermined goals is performance management. This means marketing needs to be able to report on performance, impact and ROI from the program level up.

Progress doesn’t come without missteps, misfires and failures. Winners look for ways to overcome challenges and continuously improve. They seek outside help, new ideas and new skills. In Wolfe’s story, the national heroes of the Mercury space program were not necessarily the truest and best, what they possessed was the right stuff, the skill and courage to “push the outside of the envelope.” Does your marketing team have the right stuff?

Filed Under: Laura Patterson, Metrics Tagged With: marketing, marketing team

Some well worn advice for any startup

July 30, 2009 By Nathan Kaiser 2 Comments

by John Dietz of Adometry

When we started our company last year, we got a fair amount of advice, and most of it we even asked for. Among the usual stuff like “Build a business, not a product” and “Everything takes longer than you think it will”, we also heard the standard “Get to market as fast as you can, even with a limited feature set.” We knew that we didn’t have all the answers, so we’ve appreciated the advice and help we’ve gotten from a number of sources, notably adjusting our development schedule to get to market as fast as we could.

Based on the idea of getting to market quickly, we had a UI prototype complete that we showed to as many people as possible, potential customers (advertisers and agencies), related companies (various publishers), other technology companies, and perhaps even some competitors. Our goal here was to get feedback to validate or evolve the direction of the product and business we were building. We followed that quickly with a functional beta product while continually getting feedback from our customer base.

Of course to get to this point quickly, like a lot of startups we took shortcuts. We chose what our priorities would be (scale, validity of data), but left some of the detail work for later. The architecture isn’t necessarily what we want to end up with, but we wanted to spend our early development efforts proving we could collect and analyze the data we were pitching.

Then one day it happened. We’d been chugging along happily for a few months with some small beta customers, when we finally got the signed contract from a very large Fortune 100 company that was very interested in using our system to show the value and efficiency of their online advertising purchase. The same afternoon we received the signed contract we had the email outlining the campaign they wanted to run. It wasn’t a huge campaign in the overall scheme of things, when I was at Disney we would easily serve a billion ads per day, but this campaign would represent a magnitude more traffic than we were seeing with our earlier betas. It was go time, or perhaps go go time since a couple of days later we got agreement from our next major customer who would generate for us another large volume of requests.

As everyone knows, success is not a bad problem to have, but our engineers and I worked hard and late for the next several days scaling out our Amazon EC2 tiers and doing some extra load testing to make sure we could handle the traffic. Because we have a good relationship with these customers, we convinced them to do incremental campaign analysis, just as an extra precaution. Our focus at this point was entirely on our customer experience. Whatever happened on the back end, or whatever extra effort we put in, our customers needed to see and trust the data we were providing.

To prepare for the traffic we took several steps:

Moved key files to a CDN (Content Delivery Network) for quick delivery
Validated multi-tiered environment with load balancers and failover for most important services
Generated traffic and data sizing projections
Performed load testing
Server resource monitoring

At 2 PM on a recent Monday, the fire hose opened and we watched carefully. Things looked good and traffic was climbing. Our servers were running fine and data was showing up in our UI with about a 3 minute lag from real time. We were watching error logs, server utilization, and log sizes. At 2:32 one of our logging servers failed, unfortunately due to some processes left running on that box from when we ran the entire system on that box, but the failover process worked perfectly and we lost no data (I would rather have not had the failure, but it was a nice production test of our failover ability).

We quickly found another bug in the data parsing and were able to resolve in quickly, again with no data loss. In all, we spent the next several hours watching, tweaking, and on edge. This time may be the most exciting time for a startup.

The system is still running just fine, and we are projecting traffic based on this data for bringing on the rest of this campaign, and for the start of our second big customer, scheduled to go online shortly.

A couple of last thoughts and learnings:

Advice about getting to market fast is right on. The feedback we got from early customers was fantastic and has helped us build a better product. Had we gone heads down for 12 months to build what we thought was the perfect product would have likely missed the mark

It’s okay to take shortcuts, but understand your core value, and don’t skimp in those areas. Had we not built for scale, we would have had more problems and might have lost data

Design for system failure. Although we didn’t expect things to fail, we planned for it and I’m glad we did when one of our logging servers locked up.

We were fortunate to have some very good data to use when forecasting traffic, and we spent a lot of time forecasting optimistically and pessimistically to make sure we understood what would happen in each case.

Don’t be afraid of success. When we first got that contract back quickly followed by the size of the initial campaign, I’ll admit to a short period of panic. I still wanted to call this a beta, I knew there were going to be problems. Fortunately we had planned well and focused on what we knew was important.

John Dietz (LinkedIn profile) is a co-founder of Adometry, a startup focused on online advertising metrics and writes about online advertising metrics atblog.adometry.com.

Filed Under: Entrepreneurship, John Dietz, Metrics, Startups

Baseball, Cherry-picking, Sample-size, and Startups

May 13, 2009 By Nathan Kaiser Leave a Comment

On May 5th 2009, the Seattle Mariners stood in first place in the American League West, with 15 wins and 11 losses. How excited should we be about the success of the M’s in Seattle? I suppose I temper some of my own excitement by looking at the data. One of the reasons that I love baseball is that we store very detailed situational data for baseball games, with some historical data going back 100 years. One of the easy things to look at is OPS (On-base Plus Slugging), a nice simple number for a team that correlates fairly nicely with a team’s ability to score runs over a long period of time. The M’s have a team OPS of .707 (as of May 5th), ranking them 13th out of 14 teams in the American League, not very good. Looking at the numbers more closely, perhaps there’s another reason (besides great pitching) that the M’s are winning. When you look at the numbers with runners in scoring position (runners on 2nd and/or 3rd), the M’s rank 5th in the league with an OPS of .847. The question that obviously follows is, can the M’s maintain that performance over the course of the year? And are your early successes and failures for your startup likely to continue? But first, a little more baseball…

Baseball is a great forum for statistics. Have you ever watched a baseball game on TV and heard the announcer say about a batter that this guy is hitting .400 on Tuesdays, or is hitting .350 against a certain pitcher? I certainly have, and that’s the beauty and curse of baseball. Baseball people, and announcers in particular, frequently fall into two of the biggest pitfalls of statistics: cherry-picking and sample sizes. In 2008, Ichiro had a .367 batting average against teams in the AL Central Division (his overall average was just .310 last year. Does this mean that Ichiro should never take a day off against the AL Central? Adrian Beltre (Mariner’s third-baseman) batted .316 when batting 3rd in the lineup, but only .258 in other spots, should Beltre always bat 3rd for the M’s? In both of these cases, I can find this detailed information (thanks to ESPN.com), but I’m specifically picking data points that make a point and have relatively small sample sizes. If I look back over several years these trends tend to level out as the sample size gets larger. Going back to how the Mariner’s are doing this year, they are very unlikely to maintain that big an improvement on batting with runners on base for the entire season, I can look at a hundred years of historical data to back up my assertion.

Like with baseball, startups can fall into these same traps, with access to detailed data and the urge to use that data to drive strategic decisions. The key is to recognize the data that really indicates a trend and data that is an anomaly due to small sample size. Unfortunately most of the data we collect doesn’t fall nicely into a sample size calculation that assumes a random sample of data (our data always has multiple variables). If you are looking at some sales data, conversion data, web growth, etc., here are some ideas for identifying real trends:

Look for external causes – If my web site suddenly sees a lift in new registrations on Tuesdays, is it because the last two Tuesdays my site happened to get some press coverage? Perhaps there was some Twitter buzz growing that traffic.
Check for segmentation – If my sales are disproportionately high in Nevada, I can try to further segment my customers to see if there is a trend that makes sense
Increase your sample size – If my conversion rate from 2-6 PM is double other times of day, I will likely try to pull more historical data to see if there is a significant and consistent historical lift.
More advanced trending – If there really is a trend here, I don’t want to ignore it. If you have the chops for it, you can apply some statistical trending models (it’s actually not too hard, Excel has some built in).

Even if you don’t have to time to do mathematical models, you can get a benefit from trying to understand the variables that affect your metrics, and if people really are more receptive to your message on Thursday from 4-7 PM you might want to think about advertising during those times.

As for the Mariner’s, I’ll still root for them and look for signs of real success. I don’t think Junior is going to hit .190 all season, he’s due.

John Dietz (LinkedIn profile) is a co-founder of Adometry, a startup focused on online advertising metrics and writes about online advertising metrics at blog.adometry.com.

Filed Under: John Dietz, Metrics

Out of the cesspool and into the sewer: A/B testing trap