March 13, 2013

Ran into a very nice article by Colleen Taylor @ TechCrunch about Code.org’s video A/B testing strategy. A particular paragraph stands out:

Using a Facebook ad campaign, Code.org ran a 36-way ad campaign on Facebook with each combination of six different headlines and six different thumbnails, settling on “What most schools don’t teach” because it performed nine percent better than average — and 15 percent better than Code.org’s initially favorite headline, “Wizards of the future.” In terms of thumbnail images, photos of Bill Gates and Mark Zuckerberg actually performed 17 percent better than a thumbnail image of an attractive blonde female programmer, which Partovi also said went against Code.org’s initial instincts.

If you’d have presented those two headlines and thumbnails to me as “from the gut” options, I’d have picked the two more optimal ones. And I think any competent writer on Buzzfeed’s team would probably have called it right too. “What most schools don’t teach” has a wider reach, and tugs at an experience most of us have had at some point in our lives, and a majority of the population wouldn’t call “class time” the most fruitful use of their young lives. That headline wins because it appeals to a highly shared experience, one that is usually memorable (positively or negatively) and one which many people have a sense of “schadenfreude” about. 

As for the thumbnails, that’s a pretty easy one to dissect. This isn’t a Google Glass pitch, where the focus is on an object that adorns a face. Having a random pretty face is going to move eyeballs, but it is unlikely to get them to click on “Play” especially when the headline is something about Wizards. Gates and Zuckerberg are multi-generational rockstars, glorified in media, and wear the “drop out” tag with pride. The emotional tie-in of school, high name recognition and outlier element combine for a winning strategy. 

The goal here isn’t to gloat or call into question the value of A/B testing. One can posit that these outcomes seem obvious to us all because they are being retrospectively analyzed. But I think there’s more to it than that. 

Those who appear to have “good instincts” are those who’ve acted on them and thus honed them. Consider it to be a personal A/B test you run over a lifetime. You are going to screw some calls up, but the more situations you attempt to deduce, the better your mind gets at breaking down the problem and coming up with better appropriations on “where the puck will be”.

You don’t have to make these calls for decisions that directly impact your life. You can “practice” by making calls at a distance and seeing how they pan out based on decisions actually taken by those directly involved. It might sound like a futile exercise, but it helps because it keeps your mind flexible and capable of situating itself quickly into the heart of a problem. Life is a fractal, and consequently, so are the situations it fosters.

For the foreseeable future, we remain in a world where decisions need to be made in tight windows of time and without access to experimentation and resulting evidence. So get out there hone that instinct. It’ll do you and the world some good. 

Previous post Death Because I could not stop for Death,He kindly stopped for me;The carriage held but just ourselvesAnd Immortality. We slowly drove, he knew no Next post Chutzpah, it's good for you. In 1910, a young Parsi assistant accountant with the Bank of India in Bombay had the far-fetched idea that he could set up a bank of his own. When