New arrivals: 14 Scout case studies

Internal exploration · Merchant agent

Tracing the origin of every order on a sales day

On the day a store clearly beats its target, every tool claims a share, and the claims added together do not match the actual orders. For the merchant agent we are building, we reconstructed order by order, in a semi-manual investigation that we ran and checked ourselves, the origin of the sixteen sales on a Black Friday launch day. This investigation sets the method the agent will have to follow: start from the orders, find the contacts that came before them, separate the retained origin from contributions and say what remains uncertain.

AIAttributionE-commerce

In early November, the online store of a brand we work with had a day well above its usual level: sixteen orders, thirteen of them first orders, on the day it sent its Black Friday launch email. It was the only day in the last thirty when the merchant's daily target was met. The question seemed simple: where did these sales come from? Email, advertising or search? The answers given by each platform did not settle it.

Start from the orders, not the dashboards

Our first choice of method was to take the Shopify store as the reference. The orders recorded by the store, not the conversions reported by the platforms, form the basis of the investigation. Ad platform conversions are not unique orders: the same sale can be counted by several tools, and a conversion may not match any identifiable order.

For each of the sixteen orders, we therefore looked for the available traces: journey information kept by the store, visits recorded by our measurement tool Signal, email clicks and opens logged by Klaviyo, ad clicks from Google Ads and Meta, and the codes used at checkout. We cross-checked these sources order by order and noted, for each retained origin, how many pieces of evidence supported it. An unknown origin stayed unknown: we never classified it as direct access by default.

Put the contacts before the purchase in order

An order is rarely preceded by a single contact. Someone may click an email in the morning, come back through a Google search in the afternoon, then buy. The investigation therefore consists of placing the contacts in chronological order, then separating the last identifiable contact before the purchase from the contacts that contributed earlier.

By last contact, the day breaks down as follows: five orders after a click on a Google Ads ad, five after an unpaid Google result, four after the Black Friday launch email, one after the welcome email sent automatically to new subscribers, and one after a Meta ad. Google, paid and unpaid together, therefore appears as the last contact for ten orders out of sixteen.

Contributions complete this reading. Six orders had an email click before the purchase: five where the email is the last contact, and one where a Google ad took over after several clicks on the launch email. The launch email was therefore clicked before five orders, not only before the four where it is the last contact. That still does not entitle it to claim all of the day's sales. Likewise, the order attributed to Meta is not the only one where Meta appears: another one included a visit from a Meta ad in the morning, before a purchase that followed a click on a Google ad.

What the platforms report, and why it does not add up

We then compared this reconstruction with the figures reported by each tool. Meta reported two purchases. Our data does show two orders touched by Meta, but only one with Meta as the last click; the other was completed after a click on a Google ad, which Meta cannot see. Meta's figure is therefore not contradicted, but it may cover one last click and one contribution rather than two exclusive sales. Google Ads reported about four conversions. That number is consistent with the four orders whose Google click is confirmed by two of our sources, out of the five we attribute to it by last contact. In both cases, no exact join is possible: nothing tells us which orders each platform counted. Klaviyo, for its part, can count an order after a simple email open, with no click or visit. We took a click followed by a visit as evidence, not an open.

Each of these reports is consistent with its own rules. The problem comes from adding them up, or from reading any one of them as the truth about the day. The merchant agent will have to present the platforms' reports as reports, next to the origin reconstructed from the orders.

The journeys we had not resolved

A first version of our reconstruction, based on the store, the email tool and our measurement tool, had eleven known origins, three mixed ones and two unresolved orders. We then looked at PostHog, a journey analytics tool already installed on the store. For the two unresolved orders, its recordings showed an arrival from Google, with a marker that Google adds to the links in its unpaid results and no trace of an ad click. We attached them to unpaid Google, noting that this origin rests on a single source. For one mixed order, PostHog found the Google ad click from the day before, then the return to the site on the day of purchase: its origin became known.

The investigation also corrected a reading we were tempted to make. Thirteen of the sixteen orders were first orders, which suggested an influx of new audiences who had little to do with email. Yet six of these first-time buyers had clicked an email before buying: they were existing subscribers buying for the first time. In the end, fourteen origins are known and two remain mixed, with conflicting clues that we described rather than settled.

What the investigation cannot tell us

Several limits remain written in the case file. Seven customer accounts were created that day, which does not mean seven new people. Six orders used a welcome code: we read these codes as purchase accelerators, not as an origin. The detailed archives of our own measurement tool for that day were not reviewed in full, and the journey search was limited to thirty days: some contacts may have escaped us. Above all, this reconstruction describes journeys; it does not measure what would have happened without the email or without the ads. No incrementality was measured.

What we take from it for the agent

We ran and checked this investigation ourselves, semi-manually. It serves as a reference for what the agent will have to produce, at its own scale, whenever a day warrants it. It will have to start from the orders, cross-check several independent sources and say how much evidence supports each origin, put the contacts in order, separate origin from contribution, and name the unresolved journeys along with the reason for that uncertainty.

We have started building these rules into the test bench that replays past days: an explanation of the day based on advertising or email must now rest on a matching of the orders, otherwise it stays undecidable. Moving to a reconstruction produced automatically by the agent has yet to be built and tested. What the method has produced so far is a reading of this day that depends on no particular platform, and that says where what we know stops.

All case studies →

Something to dig into? Add this article to your office hours discussion.

See my discussion