What is a Confidence Level?

You may have heard your CSM talking about "data science model confidence" and perhaps your wondering what it means.

Our data science models look for statistically significant sample sizes and a convergence of results to build a level of confidence.  If Cortex sends a lot of something and it performs consistently, we will have a high level of confidence in that level of performance.  If Cortex gets a bunch of new content it will have no confidence and will begin with an even split test.

The graphs below show how our confidence level changes over time when new content is introduced.  For the subject lines below, the X-axis represents the expected performance range (open rate), and the Y-axis represents confidence.  As more sends go out, the expected performance narrows while confidence grows.

Image 1: The subject line in red is freshly introduced.  Cortex assumes average performance, but with very low confidence.

Image 2: As performance resolves through a month of sends Cortex's level of confidence in the new subject line grows.  The expected performance range has begun to narrow.

Image 3: The image below shows one older subject line with high confidence, but low performance.  To the right are three newer (higher performing) subject lines with lower confidence.

This data is used in Cortex to determine the highest performing content, which Cortex then uses to split and prioritize send volume.