The Unbelievable Power of the Hypothesis Catalog
What if there was a simple way to:
Organize all of your marketing efforts
Incorporate genius from around your organization
Improve the overall creativity of your advertisements
I believe you can achieve all of those things with what I call a Hypothesis Catalog.
Introduction
In this post, I want to share with you how I use the Hypothesis Catalog that I've developed to deliver substantial returns. I hope that by the end, you will have a full understanding of the value of a hypothesis catalog and a roadmap for the development of your own.
What you will learn
My definition of a Hypothesis Catalog
How a hypothesis catalog is different from a Kanban Board
Why I believe a Hypothesis Catalog is useful
How I use the Hypothesis Catalog
Why the Hypothesis Catalog works
How you can use a Hypothesis Catalog in your marketing
What is my definition of a Hypothesis Catalog?
I'm sure there are many definitions of a hypothesis catalog. Still, for me, it is an electronically stored (e.g., database or spreadsheet) list of ideas for new marketing assets, ads, or campaigns framed as experiments that are accessible by all marketing stakeholders (e.g., marketing, CEO, sales, BD, etc.).
Let's break this down to make it easier to understand:
Accessible
Database or Spreadsheet
Housing assets, ads, or campaign ideas
Framed as experiments
Accessible
Accessible
A hypothesis catalog gains its potential, in part, because it provides a means for people from around the organization to contribute their ideas and knowledge.
Database or Spreadsheet
To effectively prioritize and review data, a hypothesis catalog needs to quickly sort, search, and automatically update its idea list.
Housing assets, ads, or campaign ideas
The core of a hypothesis catalog is ideas for new creative assets, ads, or campaign ideas.
Framed as experiments
Finally, the power of a hypothesis catalog comes from seeing each new idea as an experiment to be prioritized & tested.
In my opinion, marketing is an experiment. If you put out an ad, you have no idea if it's going to work. Further, if you put out the same ad twice, and you have no idea if it's going to perform the same both times. Markets are dynamic and chaotic.
Instead of believing that we can accurately predict what the market will do, I've found it's wiser to assume we don't know what the market will do, put forth the best guess, monitor the results, and use those results to update a model of our customers. Therefore the ultimate goal of the hypothesis catalog is increasing our customer intelligence.
If this sounds like science, it should. The scientific method is, in my opinion, the best way humans have devised to learn about uncertain and hard to predict phenomenon.
So what does it mean to frame something as an experiment?
Well, this is where the term: hypothesis catalog gets its name. Each idea stored in the catalog is a "best guess" for an asset, ad, or campaign. We call these best guesses, hypotheses.
For example: "we believe that this ad will increase sales by 5% because we believe it represents female empowerment, and we believe our target demographic has a unique desire to feel powerful."
In this hypothesis, we are calling out:
The test subjects: our female demographic
The aspect of our customer model we are testing: the need to feel powerful,
A secondary hypothesis: that our ad accurately represents female empowerment
Our primary measure for validation: sales increase.
The only thing not represented in this statement are two topics we will cover later:
Statistical significance or the number of customers that must experience the ad vs. the control condition
The importance of this test, or phrased differently: is this the highest impact experiment that we can be running, or would a different test be more valuable.
By housing all of this information, your hypothesis catalog can reimagine how an individual or team thinks about their marketing.
How is this different from a Kanban Board?
There are similarities and differences from my definition of a Hypothesis Catalog and a Kanban Board.
A quick definition of a Kanban Board
A Kanban Board is a tool used in agile development to track the state of a project. It also provides an easy way for team members to contribute ideas and manage their tasks.
In my experience of its use, anyone can submit ideas to "the board." These ideas get collected in a prioritization stack. The submitter provides a rationale for introducing their concept, and then a developer will assess the cost.
The team can then collectively evaluate if the value of the idea is worth the cost.
All of the approved features are moved into the "To Do" column. Developers can then pick the features they want to build. As they create them, developers move tasks to the "In Progress" column. Once finished, tasks usually move to a "Quality Assurance" or "Validation" column. Then once the code is successfully tested, tasks are transferred to the "Done" or "Completed" column. This process progresses until every feature is tested and done.
At this point, developers have thoroughly tested and released the new upgrade, and that sprint is considered complete. Developers then start again selecting features for the next "sprint."
Similarities to a Kanban Board
The Hypothesis Catalog and a Kanban have four things in common:
Both a Hypothesis Catalog and a Kanban allow you to collect ideas from around the company.
Both allow for the prioritization of tasks. Prioritization is done by evaluating the feature (Kanban) or experiment (Hypothesis Catalog) cost to value ratio.
Both view the world experimentally. The Kanban board tests features as elements of a release, which developers can roll back if the release or feature performs poorly. The Hypothesis Catalog treats each campaign as an experiment that the marketer can end if it fails to perform.
Both allow those following teams to track the progress of their different projects through their life cycle. In the case of the Kanban board, the status progresses from "idea" to "pending" to "in-progress" to "QA" to "done." In the case of the Hypothesis Catalog, the status progresses from "idea" to "pending" to "developed" to "live" to "statistically significant" to "mature" to "ended."
Differences between a Hypothesis Catalog and a Kanban Board
The primary difference between a hypothesis catalog and a kanban board is that a hypothesis catalog explicitly connects each marketing experiment to at least one element of your customer model.
This difference reflects a philosophical difference between the two approaches. The Kanban Board looks at data and says, "did these features improve performance: yes or no?" The Hypothesis Catalog goes deeper and seeks to understand why performance improved at the level of the customer's psychology.
This increase in customer insight is a significant advantage of the Hypothesis Catalog. In a hypothesis catalog, every marketing activity paints a clearer picture of some element or elements of a customer segment.
As a result, the hypothesis catalog allows your marketing team to not only evaluate a marketing initiative on the grounds of its strategic, market, and financial impact but on its ability to increase learning or phrased differently, its Learning Return on Investment (LeROI).
Why I believe a Hypothesis Catalog is useful
In the above sections, I touched on some benefits of a hypothesis catalog. In this section, I will delve into the benefits in greater detail, which include:
Better prioritization from an improved evaluation approach.
Increased team morale from the ability to use ideas from around the org and customers.
Increased customer knowledge from directly tying all marketing activities to populating a customer model
Increased team intelligence from increased access and contribution to customer knowledge
Improved creativity due to a focus on model populating criteria
Better Prioritization
As I discussed above, the Hypothesis Catalog allows you to prioritize your marketing efforts in three ways, through an estimation of the:
Expected revenue
Expected cost
Learning potential
With all this information, you can calculate various metrics to help you optimize based on your current priorities. For example, if profit is your primary goal, you can prioritize marketing that has the highest ratio of revenue-to-cost. However, if learning is more important, you can prioritize marketing that has the highest learning-to-cost. Taking the time to assess this data accurately pays off in giving your team tremendous flexibility to objectively prioritize your marketing activities.
Increased Team Morale
Throughout my career, I have been consistently surprised by a fantastic marketing idea that came from a completely different area of the company.
Great marketing ideas come from everywhere, and our customers are an especially valuable source of ideas.
A hypothesis catalog can become a suggestion box on steroids. A place where anyone can submit an idea, have it objectively evaluated, and then put into the production queue.
A robust hypothesis catalog should include suggestions from customers, but I find that marketers rarely ask customers about the type of advertisements they want to see.
The other benefit of this "Super Suggestion Box" is that it does not presume that any idea is valuable. Instead, it makes objective evaluations and then lets the data decide.
Increased Customer Knowledge
A customer model that is available to the entire department can improve a marketing team's performance almost overnight. The collectively available knowledge helps your group make better marketing decisions across the board, including ideas for creative, new placement suggestions, better product features, and new product ideas.
The incredible power of a validated customer model is why I believe the primary value of a Hypothesis Catalog is its ability to systematically and strategically populate a customer model.
Increased Team Intelligence
Teams that work effectively share the common attribute of continuous, real-time, effective communication. Both the Hypothesis Catalog & Customer Model show the state of a marketing team at any given moment. This real-time knowledge allows for better long-term decisions and more intelligent responses to short-term issues and opportunities.
Improved Creativity
Increased Creativity via Focus
Creativity exists best when it works within limitations. Painting within the bounds of a canvas, for example.
In my own creative endeavors, if I'm clear on the problem I'm trying to solve for, I always come up with better solutions. A Hypothesis Catalog & Customer Model allows individuals and teams to focus their creativity around narrow problems. If we are using a hypothesis catalog + customer model, the problem we are focused on solving is accurately populating and updating the customer model.
For example, we may see data gaps in our Customer Model. We may not be sure we know how our customers represent their drive for stability in a particular context.
All of a sudden, our creative goal becomes very clear, and our creativity can be unleashed on a very specific mission: testing creative to understand better how customers represent stability.
Increased Creativity via Ego Reduction
In my 12+ years as a marketer, I know there were many times where my ego got in the way.
I have wanted my beautiful & perfect campaign idea to work so much that I justified bad results and let it run longer than I should have.
A Hypothesis Catalog can take the ego out of marketing decisions and ultimately make them better because it represents a more profound reality than my ego's belief that it knows best. Hypothesis Catalogs force you to realize that all marketing decisions are ultimately just best-guesses or, phrased more scientifically: hypotheses.
Unlike accounting, where cash is either there or not, marketing decisions rely on a chaotic and ever-changing group of humans—a group of sometimes predictably irrational decision-makers. What worked for the last 2 years can change tomorrow.
A Hypothesis Catalog represents this truth. As a result, it keeps you and your team honest.
For example, Let's say you come up with a marketing idea that doesn't test some aspect of the customer model. When filling out the hypothesis catalog, some elements will be unpopulated. This incompleteness will be a signal to you that you should either: not implement the idea or update the idea design.
The downside of using a Hypothesis Catalog
There are a few downsides to using a hypothesis catalog:
There is a learning curve
Getting buy-in can be difficult
It takes extra time
Learning Curve
It's not steep, but if you aren't practiced thinking scientifically and instead take a more run & gun, "gut-based" approach, it can be difficult to transition to a mindset of marketing-as-learning.
I've found the key to overcoming this resistance is consistency. The hypothesis catalog format I use acts as it's own training mechanism. It forces me and the others who use it to think through the testing process rigorously.
Getting Buy-In
If you work in an organization that has a history of "gut-based," non-data driven decisions, you might have a difficult time introducing rigor into the decision making process. This resistance seems especially entrenched in organizations that fancy themselves as "creative." In my experience, agencies tend to be the biggest culprits because they tend to get clients based on awards won and not results earned (if only, clients every found out how most of those awards worked).
If you experience resistance, there are multiple change-management approaches that you can try. I think one of your most potent tools is being able to demonstrate how a hypothesis catalog can bring people together and be useful for a team. Reluctances seems to reduce when something can increase collaboration.
It takes time
Not a single person wants to work longer. I get it. Using a hypothesis catalog does add several additional steps to your marketing work. It also requires greater rigor as you think through the experimental success criteria, determine what you're going to test, and update the results.
However, I have never regretted my investment of time.
On a personal level, the process has grounded me and unlocked greater creativity. On a team level, it has connected me to others and allowed me to source ideas from their genius. On a corporate level, it has allowed me to perform at a high level.
Zero regrets.
How I use the Hypothesis Catalog
It should be clear that a Hypothesis Catalog is more than just an idea dropbox. It is a way of managing and effectively populating your customer model, which is a critical part of making truly informed decisions.
Here I'm going to lay out step-by-step how I use my Hypothesis Catalog so you can get a better idea of how you might develop and use your own.
Step 1 | Have a customer model
Since the primary job of a hypothesis catalog is to populate a customer model, the first step in developing and using your hypothesis catalog is to make sure your customer model is ready to use.
Step 2 | Give each marketing activity a unique ID
Secondly, I give every marketing activity I'm testing (e.g., campaign, ad, feature) a unique ID so it can be referenced by others and used in my other models. The fact that it is a unique id reflects the fact that every experiment is different.
At times, I've used a particular ID structure to communicate to myself and others, various aspects of the experiment. For example, I might use an id like "GLG_AW - 0046" for Google Adwords experiment 46. Using a specific code isn't necessary, but if everyone on your team follows the format, it can make life a little easier.
Step 3 | Create a name & description for your campaign
Next, you need to give your campaign or asset a name and a description, so others using the hypothesis catalog an understanding of the goals of the campaign, the targeting of the campaign, etc. These labels are critical when you are receiving ideas from around the organization or from customers.
Depending on how extensive your marketing efforts are, you might want to supplement the description with separate flags, tags, columns, or entries (depending on the technology you are using).
So, for example, you could have a column for the media used: OOH, Facebook, Google, etc. Or a column (or tag) for the type of ad: banner, text, video, etc. Additionally, you could have a label for the kind of marketing program: campaign, asset, or advertisement.
These columns or tags will allow you to sort and resort your projects during the evaluation phase quickly.
Step 4 | Determine your target market
In most cases, you develop an asset or campaign with at least some idea about who your marketing efforts are targeting.
Since your customer model ties directly to your hypothesis catalog, this approach requires that we make our target market explicit. We cannot learn more about our target market if we don't know who we are targeting.
Step 5 | Determine what part of your customer model you are testing
In this stage, we look again at our customer model. Based on the customer segment we are addressing for this experiment, we choose a specific aspect of the model we will test.
For example, if we decide to target middle-class mothers for our campaign, we might then elect to test what goals they believe are essential in their role as a parent. Alternatively, we might choose to test, what feelings they seek, in association with their role as protector.
It is also important to note that one marketing activity (e.g., a banner ad campaign) can contribute to validating multiple aspects of the model. In this case, I will make sure that I reflect that in the hypothesis catalog.
This step is a make-or-break for your Hypothesis Catalog. If you don't connect the marketing activity to your customer model, you are wasting your time using a hypothesis catalog. You would be better off using an easier to learn and less time-consuming management approach.
Step 6 | Estimate the size of the market you are testing
Once you've chosen your customer segment and the aspect of their psychology, you want to learn more about, you are ready to estimate the size of the market. You should have this data from previous marketing activities and your corporate research.
Getting this number correct is crucial. You will use this number to determine your sample size and estimate the value of this project.
The total size of the market can become a point of debate during the Go/NoGo discussion. For example, if you've misestimated this number or based it on flakey data, your later defense of the project could be in jeopardy.
Step 7 | Determine the sample size you need
There are several online calculators to determine the required sample size. All of them require you to select your desired confidence level and confidence interval. The statistics behind this choice require more time and are more interesting than I can cover here, but in short:
The more confident you want or need to be, the higher the number of people you need to sample.
It is easier to get a higher sample size for larger campaigns or for tests that focus on answering questions like: "can we cut through the noise?" where you know you are going to receive a lot of impressions. However, it can be challenging for smaller campaigns or aspects of your campaign that are going to have fewer interactions—for example, a promotion-specific landing page or for a seasonal sales script.
In these cases, you may need to extend the length of time you are testing or tolerate the uncertainty for that test while finding additional ways of testing those particular customer model elements.
Step 8 | Call out the control
Most experiments have what's called a "control" that generates baseline results. Those results are then compared to the results of the experimental condition to determine if there was an effect. In this section of the hypothesis catalog, you are labeling, which marketing campaign, ad, or asset you will use as your control.
Most marketers are familiar with an A/B or A/B/N Test. In this section, you call out your current "Champion" or control ad.
Ideally, you will run your control simultaneously with your experiment. If not, say you had to run the control condition then run the experimental condition, you run the risk of something influencing one set of results and not the other.
For example, let's say you ran a special for your brick & mortar store. You launched Banner Ad A during the first week and Banner Ad B during the second. Well, it rained all of the second week, so was Banner Ad B worse, or do people just avoid brick and mortar stores during the rain? You might be able to get some insight from past results, but you'll never really know.
Step 9 | Estimate the Revenue & Cost of the Project
Whether its a click, lead, or sale, you should have an idea of the value of each marketing interaction. If not, you should be able to make some reasonable estimates. These are then loaded into your hypothesis catalog to be used during the Go/NoGo review to evaluate the economic value of the experiment.
Step 10 | Update the Status to "Pending"
Once you've completed uploading your experiment into the Hypothesis Tracker, you can now change the status of that experiment to "Pending."
Depending on your team's process, any projects with the status: "Pending" should initiate a review.
Step 11 | Review all of the Pending experiments
Your team will then review all of the pending experiments. The experiments should be considered based on their: financial, learning, and strategic impacts.
Depending on the size of your team and how creative they want to be with the selection process, your experiment review can take several different approaches:
You could have a table discussion where the whole group reviews & discusses each idea
You could have an independent committee assigned
You could do peer review, where the submitter cannot be involved in their idea's analysis
You can have a "Dissertation Defense" format where the submitter has to defend their suggestion.
Whatever approach you choose, the end goal is the same: you decide which projects will be goes & no-gos.
Step 12 | Change the Status and Run the Experiment
Once a project is a-go, the status on that project turns to "Live" or "In Progress."
This update gives anyone with access to the hypothesis catalog the power to check on the status of all marketing related activities.
This visibility is particularly powerful for inviting and rewarding ideas from around the corporation. If non-marketing employees or customers can see that the marketing team has implemented their concept, they are going to feel a certain amount of pride, and they are going to want to provide additional help.
Step 13 | Check-ins
Experiments can run for a while, so I find it is helpful to give status updates of the marketing experiments from the hypothesis catalog regularly (at least once per week). I've used this as part of a collective brainstorming process to gain inspiration for new ideas. It can also take the form of a stakeholder newsletter.
One thing to avoid: do not change the experiment midstream because the campaigns are performing poorly. The instinct might be high, but in my experience, it's best to avoid a midstream change unless it's an emergency. You never know why it's happening, so it may rebound, and you don't want to lose the learning.
If, however, you find an error, then obviously fix that and begin the experiment again.
Step 14 | Review Results
Your experiment is over.
It's time to analyze the results and see if the data will support updating your customer model. To do this, you will compare the results of your control test with the results of your experiment. The math you will use is beyond the scope of this test.
However, in short, we want to know three things:
1. Is there a meaningful difference between the control and experimental groups?
If not, it tells us that our control and our experimental group marketing materials are equally motivational to our customers. This lack of effect may be due to our campaigns having insubstantially differences in the mind of our customers.
For example, let's say we changed the hero image of a landing page. For our experimental group, we tested a picture we believe represented nurturing to our parent prospects and tested it against an image we believe represents safety.
We run the test, and there is no real difference. We can infer one or two scenarios. Either the hero image does not influence our customers, or one or both of our pictures don't represent what we think they do.
FYI - There exist tests that can validate whether or not a customer associates an image, word, or phrase with another image, words, or phrase. It's called a Reaction Time test, and I will cover it in another post. I will post the update here. You can subscribe below to be notified of future posts.
2. There is a difference; what do we do?
If there is a statistically significant effect from our test, does that imply that we should update our model or not?
It depends.
If the results show the experimental condition is valid, we update our model. If not, it shows that the experiment performed worse than the control, and you would not update the model.
One note, even if the experiment was successful, we may not entirely want to eliminate the control from our marketing program. Further, just because it was unsuccessful doesn't necessarily mean we want to forget the new idea.
Why?
The reason: most marketing tests that we run will have a difficult time narrowly selecting for segments.
For example, we may want to update our customer model on middle-income Moms, but wealthy and poor Moms will also likely see and have the opportunity to react to our advertisement.
Also, people are reacting to our entire multichannel experience and our brand. One experimental change may be muted in the overall scheme of our marketing.
This difficulty with cutting through the noise is why multiple, valid tests are needed, and why we must continue to test over time.
3. How big of an effect does the experimental idea have?
Mathematically, the impact is demonstrated by the size of the difference between the two tests' metrics: control and experiment. The larger the difference, the more comfortable you can be that the new idea is meaningful. And, the more it makes sense to update your model.
For example, let's say we run two experiments; both are testing a new landing page configuration against the current landing page, which converts to sales at 5%.
Landing page A focuses on communicating a feeling of strength and control. Landing page B focuses on how our product helps you feel belonging.
Both improve the conversion rate. Landing Page A converts at 10%, and Landing Page B converts at 7%.
We can draw several conclusions from these results. First, we see that communicating a "sense of control" is far more impactful than a "sense of belonging." As a result, we can say that for that market segment, it is a conversion driver.
But let's examine the nuances of this example.
We might have our sales reps (or whatever lead-to-sale process we are using) include a survey. We might then go on to find that most of the converts of Landing Page A were new parents, and most Landing Page B were parents of children 2 years old or older.
Like with any experiment, we need to be careful and allow the data to help us better understand our customers.
I think this example nicely summarizes why I use a Hypothesis Catalog and Customer Model, versus just a series of regular A/B/N Test.
It's substantially quicker and easier just to throw out a bunch of test ideas and see which performs better. However, at the end of those tests, all you have is a knowledge of which test performed better. You haven't gained deeper insights.
With a Hypothesis Catalog + Customer Model, your approach is rigorous and more systematic. As a result, you gain a deeper understanding of your customers. You gain the power to understand "why" results happen and not just "what" happened.
And this leads to:
Increased reach on future campaigns
Increased return on future campaigns
Happier employees from greater buy-in
Even better products through a far more profound understanding of the customer.
This approach does not prevent traditional brainstorming
One of the questions I get, especially from people perceiving themselves as creative or people who say their ideas "come from the gut," is "Does this approach change the way we should brainstorm?"
The answer is yes and no.
If you don't want to change your brainstorming process, this approach does not mandate it.
If data doesn't inform your creative approach, then using a hypothesis catalog acts as merely one more extra tracking step.
However, if data does inform your creative approach, then your customer model can inspire your ideas, and your hypothesis catalog can make sure those ideas are grounded and tracked.
A hypothesis catalog is a beautiful way of validating your idea. If you cannot figure out how this idea validates your model, then it might be just a cool idea, but one without marketing value.
This has happened to me several times.
I had an idea so brilliant, it just has to happen. I put it up against the model, it failed, but I pushed ahead, only to find out that while the idea was cool, it didn't drive actual customer results.
In Summary
Thanks for sticking with me until the end. I threw a lot out at you, and you might have questions. If so, feel free to reach out to me.
Also, I will be updating this and other posts regularly, if you want to get alerted to the updates I've put a subscription bar at the top of this page.
All the Best,
Sean