Correlation and Causation: A new look

26 06 2008

Correlation does not imply causation. The logical fallacy that correlation implies causation will be starkly pointed to if you falsely make a cause-and-effect determination based on your observation of the world, your observation of data, or your study findings of a correlation.

Chris Anderson, editor-in-chief and famed writer of The Long Tail, has tackled the idea that the deluge of data makes the scientific method obsolete. That correlation is sufficient in the Petabyte Age as he calls it, and the practice of causation only leads us to realize that we understand even less about the world.

Jackson West rightly points out the dangers of throwing the scientific method under the bus for the sake of applied mathematics and relegating the creation of models of the world to high and mighty, “pansy” theoreticians. And he’s generally right to be critical of a loss of strict rationality and too much focus on faith.

Data can be manipulated to show what you want. Not always in an evil manner, but methodology and the very collection of data itself can affect the outcome. You can get what you ask for, or perhaps deserve as Mr. West concludes.

Take it further and a strict reliance on data leaves room for dangerous regimes. Tyranny of the majority, self-fulfilling prophecies: these are all dangers that are left on the table when strict rationality goes out the window.

Yet Anderson is certainly not suggesting this, only pointing out a real observation of the world in which we live. And yes applied mathematics can be profited from, theoretical mathematics not so much. Ian Ayres, a Yale Professor, is the author of Supercrunchers whose very title is number crunching is the way to be smart.

The examples are largely the same: information (Google), airline predictions (Farecast), legal discovery, predicting the outcome of an election. It’s easiest to see manipulation possible in the last example.

Oh well, so be it. I’ve written that blind faith is basically idiotic. A strict reliance on numbers may not mean you’re determining a causation, but perhaps correlation is enough. And for those who operate in a gray area where luck plays a part, where luck is better described as a set of circumstances beyond your control, Anderson’s piece could be comforting.

But more importantly, it outlines a real path to be smart and succeed. And so you might be wise to heed the words that correlation is enough, and that correlation supersedes causation in a world dictated by Petabytes of information ready to crunched.


Contradictions, Changing the World & Success

24 06 2008

The contradictions and absurdities that exist in the city always crack me up. I could honestly go on and on.

Another interesting week in San Francisco. A stranger tracks your contact information down, and mails you your a stranded credit card as simple courtesy. An accidental bump when running down a busy boardwalk with a group leads to a lecture about “running in single file line” as the proper formation when going for a leisurely run. Or at a free concert: defensiveness over a piece of grass when there was no outward intention to steal the free grass, yet a willingness just minutes later to scoot in closer to your two children just to make your young neighbors a little more comfortable.

This is generally a business blog, but it’s also just my ramblings about how to tie life and business together and a process I find unendingly enjoyable.

There was a short interview with Paul Saffo, a technology forecaster, on GigaOM. There are elements here of benevolence as a effective business strategy. The essence, without agreeing completely with some overly save the world sentiments, is that changing the world with a side consequence of making a lot of money is today’s recipe for success both in life and work.

This is not the touchy-feely argument either. There are many reasons by focusing on “get rich! get rich!” will lead you down the wrong path. Most simply, you’re not going to innovate. And you’re just another rider just looking to steal a position too late to ride a wave of money, ready to get pummeled in the white water. Ride the wave you’re meant to ride in a strong yet principled way, and you might ride the wave of your life.

My question in today’s world is: which of these above people are you and do you want to be? Seems like a simple choice to me.

Yahoo and Importance of Business at Scale

13 06 2008

Put quite simply, building a business that scales is what separates the pros from the amateurs.

Every business hits a brick wall. The business will not grow without drastic changes, and really at that point, you are simply using your existing business structure to go into a BRAND NEW business as a means to growth.

You can scale up vertically within your choice market and vertical. There are a number of ways to then scale horizontally: different applications, different verticals, different markets.

In every conversation I have with would-be and startup entrepreneurs, possibly the number one piece of advice that frankly many ignore is to choose a business that will scale. In other words, choose a big sandbox to play in even if you’re playing in just a very small corner to start.

Investors tend to intuitively understand this. Maximization of returns requires scalability. It still surprises me sometimes, and sometimes doesn’t, that many entrepreneurs don’t think about top-level scale from the get go. Maybe it’s a byproduct of experience. Most often, it’s a byproduct of understated expectations.

But as an entrepreneur, building a scale business gives you more leeway for being wrong. Things will almost never go your way – those are the odds. Building a scale business means that you can move into different directions in a seamless, and yes scalable, way. When your business scales effortlessly, you stand a better chance of hitting numbers you can be proud of. When you scale, you also “let a thousand flowers blossom” and you might be surprised what comes of it.

It can be hard to spot scalability sometimes. For example, it was possibly hard to see why Facebook could be a scalable business when looking at walled gardens of colleges. Someone was thinking bigger from the beginning, even if it seemed otherwise.

I’m not talking about server scalability. It’s almost always a mistake to build for too much scalability there. You can rebuild your server farm. You can’t quite as easily rebuild your business whose very success is dictated by moment-over-moment momentum.

Competing in an undifferentiated manner within a commoditized market is not scalable. Selling someone else’s wares under your banner with no discernible value-add is not scalable.

Maybe Yahoo can get its mojo back, but many people don’t think it’s possible. Not understanding the importance of something as fundamental as business at scale at this stage of Yahoo’s company history for many is inexcusable.

Internet Famous (and Successful) in San Francisco

4 06 2008

Max Levchin, co-founder of PayPal and Founder/CEO of Slide, walked into the coffee shop I was posted up at this afternoon.  He was talking something about widgets, RockYou and such, but I didn’t encroach enough to gather much more than that or anything of substance.

Cool though… Yet another gotta love SF Tech story!