How to Determine Your A/B Testing Sample Size & Time Frame I remember running my first A/B test after college. It wasn’t till then that I understood the basics of getting a big enough A/B test sample size or running the test long enough to get statistically significant results. But figuring out what “big enough” and “long enough” were was not easy. Googling for answers didn’t help me, as I got information that only applied to the ideal, theoretical, and non-marketing world. Turns out I wasn't alone, because asking how to determine A/B testing sample size and time frame is a common question from our customers. So, I figured I'd do the research to help answer this question for all of us. In this post, I’ll share what I’ve learned to help you confidently determine the right sample size and time frame for your next A/B test. Table of Contents A/B Test Sample Size Formula A/B Testing Sample Size & Time Frame How to Determine Sample Size for an A/B Test How to Choose the Right Timeframe for Your A/B Test for a Landing Page Choosing the Right Timeframe for Your A/B Test for Email A/B Test Sample Size Formula When I first saw the A/B test sample size formula, I was like, woah!!!! Here’s how it looks: Image Source n is the sample size 𝑝1 is...

15 of the Best A/B Testing Tools for 2023 Every company has a unique set of customers, so there’s no one-size-fits-all formula for designing an optimal website, crafting the most compelling copy, or building the most effective product. This is where A/B testing tools come in, where you can use them to test and optimize your website or app design, copy, product, and, most importantly, create an experience tailored to customer needs. Read on to discover high-quality A/B testing tools that will help you discover what your unique set of customers prefers.  What makes a great A/B testing tool? Before we jump into the top A/B testing tools, let’s talk about the features you should look for in an A/B testing tool. Look out for:  A/B Tests: The tool should, as a baseline, offer A/B testing. Some have additional capabilities, so consider what else is offered if you’re looking for a more inclusive tool.  Required Skills: Some tools require technical knowledge to build and launch tests, while others have easy-to-use builders for any ability. Some offer both but make sure your tool supports your skillset.  Segmentation Capabilities: Your A/B test tool should offer segmentation and targeting abilities to target your preferred audience groups.  Statistical Analysis and Reporting: The significance of your test results is an essential part of your test, so you want the tool you use to calculate the significance of your results and provide other metrics...

9 A/B Testing Examples From Real Businesses Whether you're looking to increase revenue, sign-ups, social shares, or engagement, A/B testing and optimization can help you get there.But for many marketers out there, the tough part about A/B testing is often finding the right test to drive the biggest impact — especially when you're just getting started. So, what's the recipe for high-impact success? Truthfully, there is no one-size-fits-all recipe. What works for one business won't work for another — and vice versa. But just because you can't replicate the same test and expect the same result doesn't mean you can't get inspired by other companies' tests. In this post, let's review how an hypothesis will get you started with your testing, and review excellent examples from real businesses using A/B testing. While the same tests may not get you the same results, they can get you inspired to run creative tests of your own. A/B Testing Hypothesis Examples A hypothesis can make or break your experiment, especially when it comes to A/B testing. When creating your hypothesis, you want to make sure that it is:  Focused on one specific problem you want to solve or understand Able to be proven or disproven Focused on making an impact (bringing higher conversion rates, lower bounce rate, etc.) When creating a hypothesis, following the “If, then” structure can be helpful, where if you changed a specific variable, then a particular result would happen. Here are some examples...

The Key Difference Between Multivariate Testing & A/B Testing There’s seemingly no end to what you can test in your marketing — conversion rates, offer placements, and even which titles perform better.  There’s also no end to the type of test you can run, but two players take center stage: A/B and multivariate testing. Is there a huge difference between them, though? And will my results be affected if I choose the wrong one?  Yes, there is a difference, and yes, your results will be affected. Not to fear, though; in this post, we’re going to break down the difference between A/B tests and multivariate tests and tell you exactly when to use each, so your tests run smoothly and your inbound marketing can go from working pretty well to amazingly well.  The critical difference is that A/B testing focuses on two variables, while multivariate is 2+ variables. As the difference between both tests can be seen visually, let’s go over an example.  Multivariate vs. A/B Testing Example   In the image above, the A/B test is simply two different versions of the same with minute changes, while the multivariate test looks at multiple different page elements (variables) in different positions on the page.  Given their differences, let’s learn more about each one and when to leverage each test in your marketing.  What Is an A/B Test? When you perform...

How to Understand & Calculate Statistical Significance Have you ever presented results from a marketing campaign and been asked, “But are these results statistically significant?” As data-driven marketers, we’re not only asked to measure the results of our marketing campaigns but also to demonstrate the validity of the data — exactly what statistical significance is. While there are several free tools out there to calculate statistical significance for you (HubSpot even has one here), it’s helpful to understand what they’re calculating and what it all means. Below, we’ll geek out on the numbers using a specific example of statistical significance to help you understand why it’s crucial for marketing success. In marketing, you want your results to be statistically significant because it means that you’re not wasting money on campaigns that won’t bring desired results. Marketers often run statistical significance tests before launching campaigns to test if specific variables are more successful at bringing results than others. Statistical Significance Example Say you’re going to be running an ad campaign on Facebook, but you want to ensure you use an ad that’s most likely to bring desired results. So, you run an A/B test for 48 hours with ad A as the control variable, and B as the variation. These are the results I get: Ad Impressions Conversions ...

How to Conduct the Perfect Marketing Experiment After months of hard work, multiple coffee runs, and navigation of the latest industry changes, you've finally finished your next big marketing campaign. Complete with social media posts, PPC ads, and a sparkly new logo, it's the campaign of a lifetime. But how do you know it will be effective? While there's no sure way to know if your campaign will turn heads, there is a way to gauge whether those new aspects of your strategy will be effective. If you want to know if certain components of your campaign are worth the effort, consider conducting a marketing experiment. Marketing experiments give you a projection of how well marketing methods will perform before you implement them. Keep reading to learn how to conduct an experiment and discover the types of experiments you can run. What are marketing experiments? A marketing experiment is a form of market research in which your goal is to discover new strategies for future campaigns or validate existing ones. For instance, a marketing team might create and send emails to a small segment of their readership to gauge engagement rates, before adding them to a campaign. It's important to note that a marketing experiment isn't synonymous with a marketing test. Marketing experiments are done for discovery, while a test confirms theories. Why should you run a marketing experiment? Think of running...

How to A/B Test Your Pricing (And Why It Might Be a Bad Idea) Choosing the right pricing for your product is a little bit like Goldilocks. Too high, and you risk alienating a large majority of your potential customers. Too low, and you likely won't have enough revenue to run a sustainable business. Plus, consumers might not value your product or brand as highly if they see a much lower cost than competitors'. But how can you get it just right? That's what we're going to explore in this post. Let's dive into the pros and cons of A/B testing your pricing — and how to do it. Plus, some alternatives to A/B testing your pricing if you've determined the weaknesses outweigh the strengths.  Product pricing is undeniably one of the most important decisions for your company. Your price can determine how consumers see you in the marketplace — for instance, Ray Bans' expensive sunglasses suggests they're higher-quality than the ones I can find at CVS. Sure, the price might limit the amount of total consumers Ray Ban attracts, but the price also attracts high-intent prospects based on perceived value. This premise is known as value-based pricing: a strategy that chooses pricing based on how much a consumer believes a product is worth. I believe Ray Ban sunglasses are high-quality, and more importantly, I have a good perception of the brand, which makes me feel the sunglasses are worth the hefty price. Value-based pricing is most impactful...

5 Email Testing Tools to Try (& What to Test on Them) A/B testing is one of those techniques that, if you have enough volume to give you significant results, is pretty much guaranteed to generate better results from your marketing. Email marketers have known this for ages, but what drives me nuts is that they waste their time on tiny little tests -- instead of tackling some of the bigger, more exciting tests that yield real insights and improvements. In fact, MarketingSherpa's email survey found that subject lines are still the most commonly tested element in email marketing. Meaning that those few words that get your subscribers to open your emails and see your wonderful offers are what marketers focus on most in their attempts to optimize their email marketing. While I'm sure this strategy can end up getting you the most tested, optimized subject line that will ever reach an inbox, the impact of these tests are minimal compared to all the other things an email marketer could be testing. So, are you ready to run some big, exciting tests? In this blog post, we'll highlight the top email testing tools to try and what to test on them.  A/B testing is a great way to test two different newsletter formats that promote the same content or two newsletters with slightly different design elements, such as different images or types of CTAs. ...

The Ultimate Guide to Social Testing As marketers, we know the importance of making data-driven decisions. The more data we know about our audience — from how many are being reached to how many engage with our content — the more we're able to make effective marketing moves. Having the numbers to back up marketing strategy is almost as important as the strategy itself. If numbers aren't your favorite thing to calculate, that's not a problem (thankfully). There are so many strategies and automation tools to back up those tough marketing choices effectively — no math needed. For instance, let's say you're running a social media campaign, and one of your Facebook posts includes a video. You've never run a video ad, so you need data to prove it's the right move for engaging audiences. To get that data, you decide to run a social test for engagement. In your test, one Facebook ad will show a short cut of the video and the other, a longer cut. If one of the videos reaches 50% of your benchmark engagement goal, you'll know the length is a good choice to capture the attention of users. Social media tests, like the one explained above, let you run an experimental campaign before investing in the real thing. They show different versions of the same ad to the desired audience and allow you to pick the winning ad based on the campaign's objectives and what you know your audience wants to see....

Bayesian A/B Testing: A More Calculated Approach to an A/B Test What are some of the reasons you run an A/B test? When I think of the benefits of A/B testing, I think of one of the most popular and concrete ways to experiment with ad designs that are effective for target audiences. I think of how changing one simple element can be the deciding factor for customers, and that running a test will help me figure out the preferred design. Up until recently, I thought that there was only one kind of A/B test. After all, the definition itself is pretty straightforward. Then, I came across a different kind of A/B test. This method still involves testing variants to discover the preference of an audience, but it involves more calculation, and more trial and error. This method is called Bayesian A/B testing, and if you want to take a more specific, tactical approach to your ad testing, this might be the answer. But first, let's talk about how Bayesian A/B testing is different from traditional A/B tests. Bayesian A/B Testing There are two types of A/B tests: Frequentist and Bayesian. Every A/B test has the same few components. They use data, based on a metric, that determines variants A and B. For example, a metric can be the amount of times an ad is clicked. To determine the winner, that metric is measured statistically. Let's apply this to an example of using the frequentist, or traditional, approach....