‘Test and Learn’ is an old adage in marketing.  Try something.  Start small. Measure.  Learn.  Grow. Repeat.   It is a proven process that can bring great results.  But Test and Learn can also be slow and inefficient in a world where companies are trying to adapt quickly.  That cycle time can be lost opportunity.  Where possible, it is better to ‘Learn and Confirm.’

A Learn and Confirm approach leverages alternate data sources and data science to predict success in the market in the absence of direct historical test results. Use Exploratory Analysis to make educated guesses that peek into the future. Insights are then confirmed in live market tests to reinforce learning.  This approach can give a business a jump start in their continuous improvement journey.  

The benefits of a Learn and Confirm approach include:

• Competitive advantage in speed to market
• Reduce the total cost of programs by eliminating multiple redundant testing
• Improved customer or prospect experience, in more relevant touchpoints

Data Sourcing and Machine Learning Targeting are examples where Learn and Confirm can be successfully deployed:

‍‍

Data Sourcing: Selecting Purchased Outside Data Sources

‍

Data quality and reach need to continue to evolve to feed businesses.  When evaluating new list sources or data appends, evaluating list samples and summary statistics can give significant learnings into what is most likely to yield value. (See Case Study: Blend360)

‍

Vertical List Evaluation

• Footprint Evaluation (Are you expanding or enhancing existing base files?)
• Penetration (What percent is customer coverage vs. prospect?)
• Correlation (How does a vertical list sample correlate with existing insights you have (model ranks, etc.)  Are you gaining new leads, or just additional sources for existing leads?)

·Compiled List Evaluation

• Completeness

            - Percent of records matching your file (at what confidence level)?  Does it expand your file?

            - Of the variables of interest – what percent have valid values?

• Quality: Use some elements your business already knows to evaluate the quality of the lists in general (i.e., if your business has good data on age and gender – compare the existing information against an evaluation sample).  
• Differentiation: How is the list collected?  Can the list providers detail advantages in speed, accuracy, or coverage that are compelling and that can be confirmed?
• Feature Impact: Compare models built only with existing data to those which add in the data being evaluated.  What types of lift are expected?

Machine Learning Targeting

Marketing analysts ideally have historical campaigns which are used to build targeting models. A deep change to a modeling approach might require a head-to-head market test. These are good ideas, but the sales-cycle time may cause long delays in learning. Here are a couple of scenarios where up-front learning can give you a head start:

1. No campaign history to leverage to build a response model?

Rather than launching a small canvass campaign or leveraging basic segmentation targeting based on intuition, consider a clone model (aka a look-a-like model) with existing customers. A clone model can differentiate between people who look like the target and those who do not.

• For cross-sell opportunities, this becomes a model comparing those who already have the product to those who do not.  The model can identify key features that are associated with customers who are likely to respond.
• For prospecting this takes the form of modeling those who are already your customers against a sample of a ‘universe’ file.  What attributes make people likely to become your customer?

While true response modeling and/or uplift modeling is the gold standard, there are several learning opportunities and steps that can advance your progress while history and volume are acquired to build a full response model.

‍

2. Existing campaign history – Improve models more quickly and with more confidence.

Where businesses already have campaign history and are working to improve their approach, leveraging existing campaigns to build, and also back-cast and validate, on out-of-time (OOT) previous campaigns, provides a more assured path forward.  

Not all Machine Learning algorithms are built equally. Each approach has different strengths and weaknesses.  Some models look very promising when built, but do not perform well in market. A solid test/train methodology will help to avoid some pitfalls, but not all. Some common tricky points:

• Building the initial modeling file without bias
• Avoiding over and underfitting
• Determine a good cutoff for contact depth

While even seasoned Data Scientists can get caught in some of these traps, a Learn and Confirm approach can help avoid waiting for a full market cycle to detect these issues.  

Back-casting and out-of-time validation are key tools to ensure that the new models being built perform well not-only on the sample they were built on, but that they can be reasonably expected to do well in market.  

A Learn and Confirm approach focuses on up front analytic work and leverages alternate data sources to predict success in the market in the absence of direct historical test results.  Data Sourcing and Machine Learning Targeting are two of many areas that will give you a peek into the future success of your strategies.  Where can you jump ahead with a Learn and Confirm strategy in your business?

‍