Angler AI

Unlocking Advertising Campaign Effectiveness: Different Methods to Measure Incrementality

Incrementality is a crucial metric for measuring the effectiveness of advertising campaigns. It involves comparing the difference in desired outcomes between two audience groups: one that has been exposed to an ad and another that hasn't. There are many philosophies on how to measure incrementality. However, regardless of which tactic you use, this measurement has become increasingly challenging in recent times due to data protection regulations and the limitations of online click-paths. This article explores popular methods for measuring incrementality, along with their respective pros and cons.


Multi-Touch Attribution: The Death of Click-Path Attribution

In the past, multi-touch attribution (MTA) was seen as the holy-grail of for measuring ad effectiveness. It promised that marketers could track customers by assigning appropriate credit to every advertising touchpoint on their path to conversion. While MTA was challenged with the lack of ad exposure level data from walled gardens, such as Meta, YouTube, the rise of data privacy regulations and policies has made MTA solutions even difficult to execute. Tracking users’ click-path across multiple devices and channels, once a cornerstone of multi-touch attribution, is becoming less and less possible. In response to this, marketers have turned to other methods as their go-to solution. Some of the most popular alternatives include Meta's conversion lift study, geo-matched market testing, and audience-based incrementality studies. However, these methods also have limitations.


Ad Platform’s (e.g. Meta) Conversion Lift Study

Meta’s Conversion Lift study involves “comparing the actions of real people in randomized test and control groups to measure the additional online, offline, or mobile app business driven by Facebook, Instagram, and Audience Network ads.” Conversion Lift can be very helpful in understanding ad performance across multiple devices and conversion events, whether they occur online, offline, or in-app. Meta builds control audiences with people who would have won the auction of that advertiser, ensuring that the test group and the ‘intent to treat’ control group are statistically comparable. Moreover, you are not limited by attribution windows, as any conversion during the test in your control or experiment cells will be counted. This is true regardless of how long it has been since the converter clicked or saw one of your ads.

Conversion Lift, however, is not a perfect fit for all campaigns. Firstly, it takes a good amount of time to get a final readout, and statistically significant results require large enough spend. Secondly, it limits how many Conversion Lift campaigns you can run simultaneously, and to avoid contamination you should have as little overlap as possible with existing campaigns which also limits what you can test at one time. Ideally, you should have a cool off period between such lift studies. Thirdly, Conversion Lift is not applicable for measuring the impact of integrated campaigns. It does not run cross-channel and is designed to provide insight into Facebook's impact only. Finally, the control group doesn’t receive any ad exposure, reducing the scale of your campaign which can can sometimes result in a smaller-sized test audience with comparatively higher CPMs.


Pros:

  • Provides a comprehensive view of ad performance across Facebook, Instagram, and Audience Network

  • Allows for insight into ad performance across multiple devices and conversion events, regardless of where they occur

  • Not limited by attribution windows


Cons:

  • Takes time to get a readout, and statistically significant results require a large enough spend. Requires cool-off period between lift studies

  • Requires platform support and is not easy to set up

  • Meta limits how many Conversion Lift campaigns can be run simultaneously

  • Not applicable for integrated campaigns or cross-channel campaigns

  • Holdout group doesn't receive any exposure, reducing scale of the campaign


Meta's Conversion Lift Study, from Meta for Business


Geo-Matched Market Testing

Geo-matched market testing is an effective method for measuring the impact of media on any metric that can be collected at the geographic level. This cohort-based method is particularly useful when it is not possible to target a specific audience at the user level due to limitations in platform targeting and segmentation, or with non-addressable media. Since geo-matched market testing does not require user-level data, it can help businesses comply with data protection regulations.

Furthermore, geo-testing is a powerful tool for testing the relationship between activations across multiple media channels. Unlike Meta's Conversion Lift study, geo-testing can measure the incremental impact of a single channel (such as Meta) or a combination of channels (such as Meta and Google Ads).

While geo-matched market testing can offer valuable insights for businesses, it also has limitations. As a cohort-based method, it may not provide businesses with the detailed insights they need to refine their marketing strategies at a granular level like user-based methods can. Although it can provide broad insights into marketing trends and patterns across specific geographic regions, businesses may struggle to identify specific user behaviors that drive the success or failure of particular marketing campaigns. Furthermore, isolating experiments for specific channels and pinpointing specific campaigns remains a challenge.


Pros:

  • Effective for measuring impact of media at geographic level

  • Useful when unable to target specific audience at user level

  • Does not require user-level data, helps with data protection compliance

  • Can test relationship between activations across multiple media channels: measures incremental impact of single or combined channels


Cons:

  • As a cohort-based method, it may not provide detailed insights needed for granular marketing strategy refinement

  • Difficult to identify specific user behaviors that drive success/failure of campaigns

  • Challenges isolating experiments for specific channels/campaigns

    Diagram inspired by SegmentStream


Audience-Based Incrementality Study

Audience-based incrementality studies are another effective method for measuring incrementality. These studies involve creating audience groups outside of ad platforms walled gardens, which allows you to control your user-level test at a much more granular level. This method utilizes holdout groups, where one audience is exposed to ads from the advertiser while the other is not. Unlike Meta's Campaign Lift study, both groups are exposed to all other media and variables, allowing for more precise testing of specific, tactical strategies at the user level for both single and multichannel campaigns.

One of the major benefits of this method is its ability to test the effectiveness of integrated marketing efforts. In integrated marketing, the goal is to create a "surround sound" of marketing efforts that reach your audience across multiple channels and touch points. By using audience-based incrementality studies, businesses can gain insight into which marketing strategies and channels are most effective in driving incremental lift, and which are not. This information can then be used to optimize marketing efforts and increase overall ROI.

However, one potential limitation of this method is that it requires audiences to be created outside of ad platforms' walled gardens. Named audiences cannot be applied to walled garden lookalike (LAL) and broad audiences. Additionally, this method requires a significant amount of planning and coordination to ensure that the holdout groups are created properly, and that the experiment is executed in a way that produces accurate results. Traditionally, this is carried out by an internal data science team, a resource not all brands possess.

Despite these limitations, audience-based incrementality studies can provide businesses with valuable insights into the effectiveness of their marketing efforts, and help them refine their strategies for maximum impact. And with Angler AI, you can easily create predictive audiences and measure their incremental impact (using the audience-based method), taking the burden off marketers and brands, especially those without data science infrastructures.


Pros:

  • Measured at the user-level which allows for precise testing of specific, tactical strategies for both single and multichannel campaigns.

  • They can provide valuable insights into the effectiveness of integrated marketing, measuring the surround sound effect of your marketing

  • This method traditionally requires an internal data science team, which may not be accessible for all brands, however with Angler AI marketers can execute and measure through our platform


Cons:

  • Audience-based incrementality studies require audiences to be created outside of ad platforms' walled gardens and need careful planning and coordination to produce accurate results.
    Diagram inspired by SegmentStream


Which Method is Best for Measuring the True ROI of My Campaigns?

Measurement in advertising has undergone significant disruption over the last few years. Marketers can no longer rely on a single source of truth, such as the last click in a customer's journey, to represent the impact of their marketing budget. To measure the true impact, utilizing various methods and combining directional signals from each is essential. By using a combination of different measurement methods, advertisers can get a better understanding of what is truly driving success.

Start with the basics by establishing what you value and what your ultimate objective is. For many this is top-line revenue and profitability. There is no one silver bullet, but many resources exist, such as Angler AI, that can combine all your data sources and see what's driving that value and show how strong the relationships are between the different marketing channels.

Run surveys to get feedback from your customer directly, in the process allowing you to collect zero-party data which can be used to measure the future value of those prospects and customers. Utilize incrementality studies (such as Anglers) and geo-lift studies to measure at both the user- and cohort-level. Finally, look at performance on platform, but know that it is not representational of your marketing performance in aggregate or reflective of your total marketing budget.

Overall, the key to effective measurement in modern advertising is to be open to using multiple approaches and combining different data sources to get a more complete understanding of what's working and what's not. By combining all these methods, you can get a much more comprehensive picture of how your campaigns are driving value for your business.