I was just listening to Hold On a RadioLab podcast.
There is a section where they ran an A/B test of the hold music for the national suicide hotline.
Note: Suicide is a serious matter and I am sure there are trained professionals in charge of it making the experience as optimal as possible. If you are feeling suicidal, please dial 988 from any phone.
The result was a 0.7% increase in getting people to stay on the line. This is not insignificant, but I feel that the test could have been more ambitious.
The problem for me was that the test variants were too similar. They tested corporate Jazz vs corporate Mood music. Of course, they were going to be similar results.
Incremental vs Fundamental Changes
I learned a good deal about testing from the analytics master Avinash Kaushik in 2006 at Intuit. I learned about DoE (Design of Experiments), P-Values, Standard Deviations, Statistical Confidence, controlling your variables, and many other testing topics.
Joke for engineers:
Yo Mama is so mean…
She has no standard deviation!
One of the first things I experienced was the difference between an incremental and fundamental changes.
Incremental changes are what RadioLab described. One type of music versus another. This can improve the results, but usually by small percentages.
Fundamental changes are when you change something deeper. For example, if I was testing the hold audio for the suicide prevention hotline I might imagine:
- Interactive music selector (press 1 for 80’s, press 7 to go back a song, press 9 to skip)
- Stories by people who have been through suicidal thoughts. Maybe by celebrities?
- Explaining how many people call and how many people can talk. Then telling them how to lobby their representatives to get more people and how to volunteer to help others.
Interactive systems are very easy to implement these days. All of these options are feasible. The key difference is that you aren’t testing an incremental change but rather a whole new approach.
I remember testing something for Turbotax in 2006. I wildly changed the entire page. Everything was in a different place and the look and feel was totally different. The result was shockingly (at the time) a statistical tie. Literally, my changes made no difference.
The problem was that I didn’t change the model or approach. I just changed the look and feel. This was incremental despite being very different to the eye.
The realization in that case was that users showed up and wanted to do their taxes. As long as I had the information on the page, they were motivated to do their task.
Making the difference
When I was designing Marketo, I noticed that the way all of the competing systems worked had a Visio diagram approach. I changed that key element to a non-visio approach called a Smart Campaign. It honestly wasn’t “better” – it was just different. That difference was enough for people to latch onto.
Differentiation is a key element for a startup to be successful. It can be a user interface difference, but not always. Sometimes it can be how you partner, how you build, or how you deliver or sell the service. There are many ways to differentiate
Salesforce’s slogan was “no software” which was a statement about installing on-premises software like Seibel or Oracle. The “cloud” was a new concept in 1999 when they were founded. Today, SaaS is common place, but they made the key difference.
Fun fact: Salesforce and others modeled this change by seeing a product called Quickbase. It was incredibly influential.
The difference of customization and extensibility is the reason Salesforce remains the dominant CRM.
Design Driven Inspiration
Rather than thinking about the UI or a particular feature, I believe designers should be thinking more broadly at the model of value delivery.
Designers have two particular superpowers. The first is that we have excellent imaginations. This allows us to have creative thinking for problem solving. The second is that we know how to draw pictures of these ideas.
It is the responsibility of a designer to proactively imagine, to proactively draw pictures early in the process. The first version will be wrong, but it will drive the right conversations.
Of course, you need to know the details. No one is going to pay attention to your ideas if you don’t understand the business and technology on par with the engineers and product managers.
Summary
This post could be a whole book, I think. I will cut it short here. But the bottom line is that making a difference means thinking creatively about the fundamental models of what you are working with and being proactive to show how it could be better.
Different will yield FUD (Fear, Uncertainty, and Doubt). You have to stay firm, build bridges, and keep designing. No real change is possible without someone sticking their neck out. I hope this post gives you a little help and nudge to make a difference.
