From A Smart Bear: Startups and Marketing for Geeks

I remember “disruptive” when it was called “paradigm shift.” That phrase died during the tech-bubble along with “portal” and “think outside the box,” yet the concept has returned. Don’t follow along.

When I get pitched — usually by someone raising money — that they “have something disruptive,” a little part of me dies. You should be worrying about making something useful, not how disruptive you can be.

“Disruptive” is the in-vogue word for the opposite of “incremental improvement.” A disruptive product causes such a large market shift that entire companies collapse (the ones who don’t “get it”) and new markets appear.

Disruptive is fascinating, disruptive changes the world, disruptive makes us think. Disruptive also sometimes generates billions of dollars, which is why venture capitalists have always loved it and always will.

But disruptive is rare and usually expensive. It’s hard to think of disruptive technologies or products that didn’t take many millions of dollars to implement. Most of us don’t have access to those resources, and many of us don’t care, because we’d rather work on an idea we actually understand and can build ourselves, an idea that might make us a living and be useful to people.

There’s nothing wrong with incremental improvement. What’s wrong with doing something interesting, useful, new, but not transcendental? What’s wrong with taking a known problem with a known market and just doing it better or with a fresh perspective or with a modern approach? Do you have you create a new market and turn everyone’s assumptions upside down to be successful? Should you?

I’m not so sure. Here’s my argument:

1.  It’s hard to explain the benefits of disruption.

Have you tried to explain Twitter someone? Not the “140 characters” part — the part about why it’s a fundamental shift in how you meet and interact with people?

Hasn’t the listener always responded by saying, “I don’t need to know what everyone had for lunch. Who cares? What’s next, ‘I’m taking a dump?’” They don’t get it, right? But it’s hard to explain.

There are ways to elucidate the utility of Twitter, but even the good ones are lengthy and require listeners with patience and open minds — two attributes in short supply.

“It’s hard to explain” should not be a standard part of your sales pitch. “You just need to try it” and “trust me” don’t cut it. That may be OK for Twitter — today — but what about the 100 other social-networking-slash-link-sharing networks that didn’t survive? Ask them about selling intangible benefits.

2.  It’s hard to sell disruption, because people don’t want to be disrupted.

If you’re reading this you’re probably more open to new ideas and new products than most, because you’re inventing a new product, starting a company, or you’re just ruffled because I’m pissing on “disruptive” and you’re looking for nit-picky things to argue with me about.

But most people are creatures of habit. They don’t want their lives turned upside down. They launch into a tirade of obscenities if you just rearrange their toolbar. When they hear about a new social media craze they cringe in agony, desperately hoping it’s a passing fad and not another new goddamn thing they’ll be aimlessly paddling around in for the next decade.

Change is hard, so a person has to be experiencing real pain to want change. Selling a point-solution for a point-problem is easier than getting people to change how they live their lives. Identifying specific pain points and explaining how your software addresses those is easier than trying to tap into a general malaise and promising a better world.

3.  Most technology we now consider “disruptive” wasn’t conceived that way.

Google was the 11th major search engine, not the first. Their technology proved superior, but “a better search engine” was hardly a new idea. In retrospect we say that Google transformed how people find information, and further, how advertising works on the Internet.

Disruptive in hindsight, sure, but the genesis was just “incrementally better” than the 10 search engines that came before.  (Or 18.)

Scott Berkun gives several other examples in a recent BusinessWeek article. He highlights the iPod — an awesome device, but not the first of its kind. Rather, there were a bunch of crappy devices that sold well enough to prove there was a market, but no clear winners. Here an innovation in design alone was enough to win the market. Not inventing new markets, not innovative features, not even improving on existing features like sound quality or battery life — just a better design, unconcerned about “disrupting” anything else.

Setting your sights on being disruptive isn’t how quality, sustainable companies are built. Disruption, like expertise, is a side-effect of great success, not a goal unto itself.

4.  The disruptors often don’t make the money.

The construction of high-speed Internet fiber backbones and extravagant data centers fundamentally changed how business is conducted world-wide both between businesses and consumers, but many of the companies who built that system went bankrupt during the 2000 tech bubble, and those who managed to survive have still not recovered the cost of that infrastructure. They were the disruptors, but they didn’t profit from the disruption.

Disruptive technology often comes from research groups commissioned to produce innovative ideas but unable to capitalize on them. Xerox PARC invented the fax machine, the mouse, Ethernet, laser printers, and the concept of a “windowing” user interface, but made no money on the inventions. AT&T Bell Labs invented Unix, the C programming language, wireless Ethernet, and the laser, but made no money on the inventions.

Is it because disruptors are “before their time,” able to create but not able to hold out long enough for others to appreciate the innovation? Is it because innovation and business sense are decoupled? Is it because “version 1″ of anything is inferior to “version 3,” and by the time the innovator makes it to version 2 there are new competitors — competitors who don’t bare the expense of having invented version 1, who have silently observed the failures of version 1, and can now jump right to version 3?

“Why” is an interesting question, but the bottom line is clear: Disruption is rarely profitable.

5.  Simple, modest goals are most likely to succeed, and most likely to make us happy.

It’s not “aiming low” to attempt modest success.

It’s not failure if you “just” make a nice living for yourself. Changing the world is noble, but you’re more likely to change it if you don’t try to change everything at once.

I made millions of dollars at Smart Bear with a product that took an existing practice (peer code review) and solved five specific pain points (annoyances and time-wasters). Sure it wasn’t worth a hundred million dollars, and it didn’t turn anyone’s world inside-out, but it enjoys a nice place in the world and it is incredibly fulfilling to see people happier to do their jobs with our product than without it.

Had I tried to fundamentally change how everyone writes software, I’m sure I would have failed.

I made less money personally at ITWatchDogs, but the company was profitable and sold for millions of dollars. We took a simple problem (when server rooms get hot, the gear fails) and provided a simple solution (thermometer with a web page that emails/pages you if it’s too hot). There were many competitors, both huge (APC with $1.5 billion market cap), mid-sized (NetBotz with millions in revenue and funding), and small (sub-$1m operations like us). We had something unique — an inexpensive product that still had 80% of the features of the big boys — but nothing disruptive.

Had we tried to fundamentally change how IT departments monitor server rooms, I’m sure we would have failed.

There’s nothing wrong with modesty. Modest in what you consider “success,” and modest in what you’re trying to achieve every day:

My daughter convinced me that insisting something be Deeply Meaningful With Purpose can sometimes suck the joy from it.  –Kathy Sierra

Of course it’s wonderful that disruptive products exist, improving life in quantum leaps. And it’s not wrong to pursue such things! But neither is it wrong to have more modest goals, and modest goals are much more likely to be achieved.

You must have your own thoughts on this subject!
Leave a comment and let’s continue the conversation.

From Tony Wright dot com

15 years ago, you couldn’t even BEGIN to look for a house without a real estate agent (who takes 6-7% of the purchase price from the buyer). Today, the internet has changed that. 10 years ago, someone starting a small business had to eat a cost of thousands of dollars to get a solid looking logo– often more if they didn’t want to roll the dice on just using a solo designer (of if their first designer didn’t create something that they loved). Today, a small business can get dozens of designers working in a public forum for $500. I think that’s AWESOME. But like real estate, there are casualties. And, like real estate, there is anger. But to me, “transactional design” (the kind of design that can take a few hours to net a good product and doesn’t require a lot of consultation) is an inevitable casualty of the global economy and the evolution of the internet (see 99Designs).

It’s a Global Village Now

I was in India for 3 weeks last year and was STUNNED at the cost of labor. We rode in taxis for the entire trip and spent less on them than the 1 way trip home from the airport in Seattle. Talented tailors would throw in manpower of tailoring a shirt if you just bought the cloth. If it’s unfair to pay $500 for a logo, was it unfair for me to pay Indian market rates for a taxi ride (usually less than a buck or two)?

The $300 bounty for a winning logo design is a kings ransom for a young designer in most of India (and the rest of the world). Guess what, Western world? You’ve got to compete– and Walmart has taught us over and over again that consumers aren’t going to pay 10% more (much less the 1000% more that an onshore hourly designer would cost) just so they can feel good. Some of them will- but most of them won’t. We can’t put the genie back in the bottle here. You’re better off trying to find creative ways to compete than bemoaning the unfairness of it all– it’s like a cottage seamstress complaining about the existence of the new textile factory down the road– technology changes markets.

For a rural Indian designer, entering 10 contests per week and winning one for $500 might be a huge win (and he doesn’t have to write a single proposal!). And that designer might be damned talented. How different is this than a services business investing $500k in sales effort on 10 different $10m RFPs and ultimately winning one? In fact, isn’t this just a different sales investment/risk than costly networking, proposal writing, advertising, etc., etc? Heck, the designer doesn’t even have to issue a Net-30 invoice– 99Designs drops the money to the winner pretty instantly.

So I’m assuming that the gripe with design contests isn’t that people are getting paid LESS than they used to, but rather that they could get paid NOTHING even after expending the time and effort of producing a logo. Which brings me to my next point:

Whether you are a Business or Freelancer – getting paid requires that you risk time and money.

If you want paying work without spending time/money or taking risks, you should go find a job with a paycheck.

My first business (a technology consultancy) was CONSTANTLY investing staggering amounts of money and time to get customers…. We had sales guys, who made healthy base salaries and some commissions. We went to networking events to establish relationships with people who could be customers someday. We took existing clients to lunch to chat about projects on the horizon. We sent out custom holiday cards to every client every year to keep us visible. We built and maintained a web site with a rich and updated portfolio. We had snazzy business cards that had to be kept up to date. We had really nice business clothes for the clients that cared about such things. We cooked up gorgeous custom proposal documents for customers– and these proposals required considerable analysis work and consultation with the customer (spec work!). We even responded to RFPs sometimes (rarely). All of these efforts can come up empty, of course. Many of them did, but in aggregate, my business grew like gangbusters. Software is no different. I heard that Salesforce.com spends 60-70% of their topline on sales/marketing. Much of that is probably wasted, but I’m sure they are in a constant state of making their marketing spend more efficient (just like 99Design entrants are probably in a constant state of gauging the kinds of contests that will net them the most bang for their effort).

In short, getting paying work cost TONS of time, money, and risks (how many freelancers do you know who average 100% billability in a 40 hour work week over a year?).

If you are a fresh-off-the-boat designer (or a rural one), you should expect your costs and risk here to be higher than if you’re not. You’ll have to invest more and get less as you build up relationships, your skills, and a portfolio. If there are too many designers eager for work (as I believe there are right now– the design world is NOT growing as fast as were churning out design grads), the market is going to make this harder for you. Don’t like markets? Get a paycheck-job or go learn Ruby on Rails (then you can fall out of bed and land on 2-3 lucrative freelance offers).

The nature of design

The best work general comes from seasoned professionals who engage in a deep discovery process, run through a lot of iterations, and work closely with the client. That being said, you can see flashes of brilliance without all of this, especially in the world of “transactional design”. Some of the stuff on 99Designs is GOOD. For a logo, book cover, or smallish web site design (especially for a smallish business) the difference in value received between a $30,000 engagement and a $500 contest is not worth $29,500. In fact, the contest might (on some occasions) yield better results faster. Even if it doesn’t, it’s CERTAINLY faster and can help with brainstorming. From a purely economic point of view, rolling the dice with a contest is a quick experiment to run that might yield exceptional results. I could design a good from-the-hip book cover in a few hours and it MIGHT be great… Design can be random and certain design tasks are 90% inspiration and 10% perspiration rather than the inverse. The bigger the design project, the less this is true, obviously. Again, I think logos (for small businesses) is the sweet spot.

Supply & Demand

As a business, we try to be as fair as possible with vendors, but we’re in business to be profitable. If I look at the winning designs on 99Designs and I generally like them more as much as any designer’s portfolio, is eschewing the cheaper option really the way to go? Paying bottom dollar prices CAN mean that someone somewhere is being exploited. I’ve seen no evidence that the 99Designs designers are exploited however, though it’s obvious that there are designers with higher costs of living in the US who simply can’t compete on transactional design services.

If you answered “yes, as a matter of principal” to the last question, how do you feel about internships (unpaid or crappy pay)? How do you feel about buying sneakers that were made in a Chinese factory with awful working conditions (check your feet, please)? How do you feel about the fact that the average Google employee generates over $1m per year in revenue but gets paid less than 10% of that #? Shopping for the best dollar-to-value ratio generally means that someone gets a disproportionate cut of the wealth in the transaction (even just a little bit)… Though are Google employees really getting screwed? Is an Indian designer getting screwed if she’s pulling down $20k year on 99Designs? And where is the outrage about things like iStockPhoto? Or 99Designs’ Logo Store? Is responding to a clear need in a design contests for a speculative chance at pay really that different from a photographer tossing up a speculative photo on iStockPhoto and hoping that someone might eventually buy it? The ones that have great photos make a ton of money. The ones that suck probably need to take photography classes. Heck, is it really that much different from my startup, where I spent a big (expensive) chunk of my live to launch something hoping that someone would want to buy it? Isn’t a startup in the “spec-work” category?

Design contests are a meritocracy in the extreme– good designers can probably make good money and (with a track record of winning and a great portfolio), eventually graduating to less-speculative lead generation if they so desire (though I bet GREAT designers could net thousands a day on 99Designs). Bad ones don’t and have to seek other marketing avenues or other lines of work. Again, welcome to business. Given the huge number of designers that enter contests OVER AND OVER again, clearly many have decided that they’d rather roll those dice than roll the dice associated with RFPs, Adwords, hiring salesfolks and other lead-generation efforts.

These are just some thoughts. As a designer, I’ve never done spec work (unless proposals count– they probably should). As a business, I’ve never asked for it… But from either side of the table, I’m not sure I have an ethical problem with it. So from one (admittedly kinda mediocre) designer to the rest of you– how are design contests “damaging” designers beyond the way that Google News is “damaging” newspapers?

From Futuristic Play by @Andrew_Chen

Just a quick FYI on an upcoming conference – here’s the details if you’re interested:

Startup Lessons Learned is the first event designed to unite those interested in what it takes to succeed in building a lean startup. The goal for this event is to give practitioners and students of the lean startup methodology the opportunity to hear insights from leaders in embracing and deploying the core principles of the lean startup methodology. The day-long event will feature a mix of panels and talks focused on the key challenges and issues that technical and market-facing people at startups need to understand in order to succeed in building successful lean startups.

• Conference agenda
• Speaker list
• Where it is

I’ll be on a panel on Minimum Desirable Product with Dave McClure and others. We’ll be talking about the dynamics of incorporating design into a lean startup methodology, with all the difficulties and tradeoffs that entails.

25% discount if you use the link below:

Register for the conference.

From Startup Whisperer

From How to Start a Two-Bit Operation: Small Business Tips

A few days ago we asked the supportive community over at Hacker News for some feedback on PickFu. We were extremely pleased to see that the overall sentiment was positive and that there were other entrepeneurs out there who have the same need for quicky and cheap polling.  In the spirit of some of the comments, we now offer a few more options for the number of reponses (50, 100, and 200) and for some basic demographic targeting (gender and age group).

If you’ve got some new logos you need to a/b test go give it a try!

Justin

From Startup Whisperer

I was talking to a friend of mine recently who runs a major national brand.  We were talking about the perils of running brand ads online.  His concern was that he didn’t have the ability to really understand where his ads were being placed or whether they were being seen by the right audience.  The folks at Mpire rolled out a service last year called AdXpose.  It asks as sort of the “Omniture for online advertising.”  Its an all-in-one solution to provide brand verification and campaign optimization for online ads.  The team put together a research white paper that you can request here that shows over 50% of online advertising is being wasted.

The chart below looks at the cost of delivering an ad impression.  As you can see for a standard $1.00 CPM remnant ad the cost of delivering it is typically .$10 when you load in all of the manual cost and technology.  This is basically a fixed cost so a higher CPM brand campaign ($10-$40 CPM ad) barely feels this cost.  Knowing that there is so much waste in advertising today – brands and agencies are starting to figure out that they have to tackle transparency and accountability.  The price of this brand protection is relatively small especially considering that there is so much waste.  Plus, the opportunity cost is huge since there is so much upside in having consumer brands move their budgets online.  If you no that 50% of your advertising is being wasted and/or harmful to your brand, then it would seem to make sense that you’d want to track your campaign(s) with a microscope.

A recent PriceWaterHouse Coopers study indicated that nearly 1 in 3
ads is never even seen because they are below the fold.  Today, only 7%
of the global US media budget is spent online.  80% of that volume is
distributed thru indirect channels like ad networks or ad exchanges.
The answer to the question on what is the price that a brand manager
would be willing to pay for brand protection – its priceless.

Some interesting thoughts from Mpire’s Kirby Winfield from this post on Adotas.  Kirby is always excited about talking to potential customers.

From cdixon.org

Anyone who has pitched VCs knows they are obsessed with market size.  If you can’t make the case that you’re addressing a possible billion dollar market, you’ll have difficulty getting VCs to invest. (Smaller, venture-style investors like angels and seed funds also prioritize market size but are usually more flexible – they’ll often invest when the market is “only” ~$100M).  This is perfectly rational since VC returns tend to be driven by a few big hits in big markets.

For early-stage companies, you should never rely on quantitative analysis to estimate market size. Venture-style startups are bets on broad, secular trends. Good VCs understand this. Bad VCs don’t, and waste time on things like interviewing potential customers and building spreadsheets that estimate market size from the bottom-up.

The only way to understand and predict large new markets is through narratives. Some popular current narratives include: people are spending more and more time online and somehow brand advertisers will find a way to effectively influence them; social link sharing is becoming an increasingly significant source of website traffic and somehow will be monetized; mobile devices are becoming powerful enough to replace laptops for most tasks and will unleash a flood of new applications and business models.

As an entrepreneur, you shouldn’t raise VC unless you truly believe a narrative where your company is a billion dollar business. But deploying narratives is also an important tactic. VCs are financiers — quantitative analysis is their home turf. If you are arguing market size with a VC using a spreadsheet, you’ve already lost the debate.

From cdixon.org

I recently tweeted:

New early-stage start up trend: get big quietly, so you don’t tip off potential competitors.

Chris Sacca agreed:

@cdixon Agreed. As of this morning, I have four companies who don’t want investors mentioning that they’ve been funded.

Business Insider took these tweets to mean “Stealth mode is back.”  But that’s actually not what I meant.  The companies I’m referring to (and I think Chris is referring to) are publicly launched, acquiring users and generating revenue. They are modeling themselves after Groupon, where the first time the VC community / tech press gets excited about them, they are already so successful that it’s hard for competitors to jump in.

This trend strikes me as a response to the fact that 1) raising money from certain investors can be such a strong signal that it triggers massive investor/tech press excitement, 2) things are “frothy” now – meaning lots of smart people are starting companies and easily raising lots of money, 3) word seems to travel faster than ever about interesting startups, and 4) there are big companies like Facebook and Google who are good at fast following.

I don’t know what to call this but it’s not stealth mode.  Maybe “underhype” mode?

From A Smart Bear: Startups and Marketing for Geeks

In the quest for optimization, A/B tests, metrics, and funnels, we’re in danger of losing the fun and value of creative work.

When we demand overwhelming customer outcry before committing to the slightest product change, we’re in danger of losing the value of creating a cool feature that takes too much effort but people just love.

When we do the minimum necessary to get the job done, we’re efficient but not thrilling. We’re “lean” but we’re not stirring hearts. We’re effective but not playful.

I’m as excited as everyone else about Lean principles gaining traction, and sure most companies are erring on the side of too little objective feedback rather than too much. Still, every article I read turns the creative process of business and product design into Vulcanian objectivity.

Sometimes, you should do something just because it’s cool.

Look at this incredible display of affection IHumanable has for his computer:

This is one of the reasons I love my new iMac, it’s just a beautiful magic floating screen filled with win.

You couldn’t ask for a stronger endorsement. This is even better than “It saved me $725,231.” This is beyond utility — this is love.

Does love come from feature bullet points? Do you earn love through A/B tests and implementing features off the top of GetSatisfaction? Or is this something else, something deeper, something less incremental, less data-driven, more gut feel, more emotional?

My first product at Smart Bear had a non-optimal, floating-in-win invention called the “mini-viewer.” Here’s its story.

Code Historian was my first product. It was the first file difference viewer with built-in support for version control systems, letting you view various historical versions of a file side-by-side. You could switch between which versions you were comparing with one click:

The thing to focus on is that user interface element in the bottom-right corner. That’s the “mini-viewer,” and in every measurable sense it’s a terrible business decision.

The mini-viewer summarized the modifications — the lines added, changed, and removed — so the user could easily see how many changes there were and where they’re located. Sounds useful, right?

Right, except it’s a really wasteful, expensive way to do it. Many competitors used a different technique I call “boogers,” because to me it looks like someone shot snot rockets all over the screen, and also because it’s fun to deride competitors, because it feels good to make fun of other people who (appear to) have more revenue than you do.

But don’t you agree they look like boogers?

The boogers are placed next to the scrollbar, indicating where you’d need to scroll to see differences between the two versions of the file.

Now by all of the usual arguments for Lean, Agile, and minimimalism, I should have used boogers too:

1. Boogers were already semi-standardized. User interfaces should follow the principle of “least surprise” — if people are used to a certain metaphor, icon, or behavior, you should honor that so people understand your product immediately. No one else had a mini-viewer.
2. Boogers occupy minimal screen real-estate. It’s just a thin strip no wider than a scrollbar; in fact some products put the boogers on top of the scrollbar. The mini-viewer is not only larger, it has significant width, which means you have to occupy the rest of the right side of the screen with other crap.
3. Boogers appear right next to the scrollbar, which is where you look anyway when navigating the file.
4. Boogers take less effort to compute than the algorithm for determining color variations in the mini-viewer.
5. Boogers take less effort to draw. Boogers are drawn on the screen once, and don’t change unless the window is resized — an infrequent operation. The mini-viewer however indicates your current scroll position in the file (those black brackets) so when you’re scrolling around the file the speed at which you can recompute and redraw the mini-viewer matters. If you draw directly on the screen it will flicker, so you need off-screen buffering. In short, the mini-viewer is a lot more programming effort with a lot more chance for bugs.
6. The mini-viewer doesn’t convey more information than boogers do.

And yet, everyone loved the mini-viewer. People sent emails saying they used Code Historian just because of the mini-viewer. Some developers wrote in asking how I was able to render it so efficiently. It was always a high point in product reviews.

The mini-viewer was wasteful, but fun. It wasn’t optimal and had no measurable benefit to usability, but it was “filled with win.”  It took extra effort but it was endearing — an important attribute not easily captured with metrics and spreadsheets.

Now sure, there are many of aspects of business and product development where it’s best to stop obsessing and just cut corners. Often we can and should accept 80% of the benefit if it means 20% of the effort. Customers generally prefer the right features over more features.

But sometimes it’s your job to fill the screen with joyous win.

From A Smart Bear: Startups and Marketing for Geeks

They say “statistics lie,” but they don’t. People do.

Well, that’s a little harsh. Sure some people intentionally skew numbers and selectively pull data, but most folks misinterpret data by accident.

Why do you care if you’re not a scientist?  Because you collect data about your business all the time — web traffic, revenue sources, expenses, customer behavior — and make decisions based on your (mis)understanding of that data.

Here’s a few basic mistakes I encounter constantly.

Statistics don’t tell the whole story

It’s easy to boil a data set down to a single number, like an average. Easy — and often shortsighted.

Single numbers feel powerful; you feel able to wrap your mind around a lot of data. Sometimes that is indeed useful, but it can also obscure the truth.

Consider Anscombe’s Quartet, four graphs that have identical statistical properties, yet clearly represent four distinct processes:

Since the statistics are identical in each case, it’s clear that statistics alone don’t describe what’s actually happening with the data.

The true story of each graph:

1. The process is mostly linear. The best-fit line is handy in describing the relationship, but there are other possibly-random factors at work as well.
2. The data are perfectly related, but not linear. Applying typical linear statistics is just wrong.
3. The data are perfectly linear, with one outlier. Probably the outlier should be ignored, and the best-fit line should reflect the other points.
4. The data don’t vary at all in the x direction, except for an outlier which probably should be ignored. All the standard statistical numbers are useless.

Lessons:

• Processes can’t be boiled down to a single number.
• Blindly applying statistics doesn’t explain what’s happening.
• Charts can help.

“Average” is often useless

You can’t open an analytics tool without being attacked by averages: Average hits/day, average conversion ratios, average transaction size, average time on site.

Trouble is, the average is often not only useless, but misleading. Take “average time-on-site,” a typical web analytics metric. It’s important enough in Google Analytics that it appears on the top-level site information dashboard, as shown in this real example from my blog:

Of course it’s better if a visitor spends longer on a site because it means they’re engaged.

Is 1:33 good? Actually that’s the wrong question. The true story becomes clear when you break this single number into pieces:

The “average” time-on-site of 93 seconds is useless when trying to explain user behavior. The correct way to think about time-on-site is:

1. Most visitors “bounce” off the site without really looking at it.
2. About a third of the visitors stayed long enough to read some articles.

Furthermore, the way you optimize #1 and #2 are completely different:

1. Bouncing can indicate that the traffic source is poor (i.e. we attracted eyeballs, but they weren’t the right eyeballs) or the landing page was poor (i.e. we attracted the right eyeballs, but we failed to lure them into reading further).
2. Getting a few minutes of time on a blog is already “success.” Trying to get someone to stay even longer (e.g. 10 minutes instead of 5) probably isn’t useful. So the better question is: How do we get more people into this category, rather than trying to “increase the average” in this category?

So not only is the average value 1:33 useless in describing reality, it’s useless in deciding what to do next.

Lessons:

• The simple “average” is often meaningless.
• Using a single number to describe a process obscures the truth.
• Using a single number to describe a process prevents you from learning how to improve and optimize.

The dangers of “Top 10″ and “Others”

Whether it’s a cover article in Cosmo or a web analytics report, people love to read “Top 10″ lists.

Top lists can be useful. For example, here are the origins of search engine traffic to my blog:

There are several “other” search engines (e.g. Ask and AOL), but the traffic doesn’t amount to a hill of beans (as we say in Texas). It’s useful to cut those out of the chart because they’re just noise.

The trouble starts when “Other” isn’t so trivial. The following chart is a real report from a board meeting I was in years ago (only the names and layout have been altered):

Looks fine. But later I was poking around the data myself and decided to add a category for “Others”:

Some people call this the Long Tail — a pattern wherein a few big players are far larger than any other single player, but when you add up all the little players they collectively match or — as in this case — tower over the big players.

If you discover a Long Tail in your data, there’s several ways to react. Consider the case of having a Long Tail in sales of your product line, as with iTunes and Amazon who have a few blockbuster hits plus a long tail containing millions of products that sell infrequently. Here are four opposing viewpoints of how you could approach the situation:

• The Long Tail is too expensive to sell into because it requires reaching a lot of people, each of whom don’t give us much money, so it won’t be cost-effective.
• The Long Tail is the least expensive way to sell because it means reaching under-served markets, which means cheap ads and hungry customers.
• Addressing the Long Tail means we have to be all things to all people, and that means we’re unfocused. Instead, let’s try to be #1 at one thing.
• Trying to be #1 at anything is hard, and often the spoils go to those with the most money, not to the smartest or most passionate. Rather than fight the 800-pound gorilla, let’s address the rest of the market that gorillas ignore, but which contains a ton of potential business.

No one of these views is automatically correct. For example, iTunes gets most of their revenue from the big players (contrary to “common” knowledge), but other companies like Beatport make millions of dollars off the Long Tail of niche music markets (electronic music, in their case).

The only wrong thing is to ignore your “Others” column.

Lessons:

• “Top 10″ lists can hide important data.
• Any time you truncate data you must first be certain you’re not throwing away important information.
• Data patterns like the “Long Tail” aren’t “good” or “bad” per se. There are usually many equally-viable ways for you to react.

Metrics and statistics “rules” cannot be applied blindly

Consider the following (intentionally unlabelled) chart:

It’s tempting to start making observations:

• The average value is 57. (But you already know this is crap, right?)
• The value is generally increasing as we move to the right.
• Some data is missing. Maybe they should be discarded.

Unfortunately, even these basic observations are assumptions, and could be wrong depending on context. Consider these scenarios:

1. These are a student’s test scores over time. The student was failing and not turning in assignments. However, in the middle of the period the student hired a tutor. Results improved, and by the end of the class the student had mastered the material.This student should probably be awarded an A or B because of the clear improvement and steady results at the end of the year when tests are hardest. The student should not receive a grade of 57 — the average.
2. Each result represents a survey of one person on the effectiveness of a certain advertisement.The “zero” rating from subjects #2 and #4 is real data, and it’s a bad sign. This could indicate something drastically wrong with the ad, for example being offensive. We need to dig in with those participants and learn more about this failure.In general the average value — including the zeros — is probably a useful indication of the ad’s overall effectiveness. It’s curious that the results “improved” so much in later trials since the participants were supposed to be randomized.  This may indicate a bias in the test itself.

An interesting result of #1 is that in order to obtain a useful “average” value we ought to throw away almost all our data points! The opposite is true with #2.

The point is that the context for the data determines how the data is interpreted. You can’t blindly apply a “rule,” such as which data points can be ignored.

Lessons:

• You have to interpret results in context, not blindly apply formulas.
• Form a theory first, then see whether the data supports or invalidates your theory.

Formulas are not a substitute for thinking.

Like any tool, statistics is useful when used properly and dangerous otherwise. Like any algorithm, garbage in yields garbage out.

Yes this means “metrics analysis” is harder than it looks. Yes this means you have to take time with your data and verify your thought process with others.

But what’s the alternative? Thinking about your processes incorrectly and then wasting time on senseless “solutions?”

Final lesson: Since metrics are hard and take time and effort to get correct, don’t attempt to measure and act on 100 variables. Pick just a few you really understand and can act on, and optimize with those alone. You’re more likely to make a genuine, positive change in your business.

What tips do you have? Leave a comment and join the conversation.