Internal exploration · Merchant agent
Designing a morning review a merchant can read in a few minutes
A serious investigation into a sales day produces dozens of figures, caveats and leads. What the merchant needs every morning is a clear judgement, a few figures and three actions. For the merchant agent we are building, we designed a morning review meant to be sent by email: a short message that says where the day stands against the target, why, with what degree of certainty and what to do next. The evidence remains available in an appendix. Today this review exists as a mock-up; it has not yet been sent to anyone.
We work with the online store of a brand that has set itself a daily sales target. Since it was switched on on 12 November, an automatic summary produced by Signal, the measurement platform we are developing, has been sent every night to three recipients. It covers the previous day's orders, email statistics and Google Ads data, and shows as missing any data it could not obtain. This summary provides figures, but it does not investigate or judge anything. The morning review we are designing has to go further: turn the agent's investigation into something the merchant can read in a few minutes, then discuss with their team.
A judgement, not an investigation report
Our first choice was to separate the deliverable from the investigation. The investigation checks the measurement, compares with previous days, follows the leads and matches orders to their journeys. It is necessary, but it is not what the merchant reads first. The body of the email answers a single question: why is the day above or below target, and what should be done?
To answer it, the review always separates two gaps. The first is the gap from the merchant's target. The second is the gap from normal, calculated over the previous seven and thirty days. A day can be below target while remaining normal: the problem is then the usual level of sales, not an incident. A normal day does not rule out an incident either. These two situations call for different actions, and the review has to say which one applies.
The events that matter
On the mock-up we built for a November day, the review opens with a verdict. That day, five orders had been placed, about 37% of the target. The day was close to the week's average, and the target had been met only once in thirty days. A bar shows the distance to the target, followed by three figures: orders, average basket and the day's ad spend. A short series of the last seven days, compared with the target, shows at a glance whether the day is exceptional.
Next come the events that explain the day, ranked by likely weight. A Meta campaign was bringing in many clicks with no observed add-to-cart. A Google acquisition campaign had been paused during the day. The order button could block until early afternoon. Everything else, such as the dozens of measurements examined during the investigation, stays in the appendix. The body of the email must stay between 250 and 400 words; the mock-up has about 400.
State the finding and its degree of certainty
Each finding carries a level of evidence: measured, cross-checked by several sources, hypothesis or undecidable. We set a demanding rule: uncertainty is not a way to dodge the diagnosis. The agent has to take a position, in the form “probably this, then that; size uncertain”, and say in one sentence what would make it change its mind.
Several kinds of wording are forbidden: splitting the gap from the target between causes with made-up precision, promising a gain, or presenting a correlation as a cause. A measurement problem must be flagged separately, in a visible alert. That day, our traffic collection had been interrupted from the early evening of the day before until mid-morning. The review has to say so before any reading of the traffic, and make clear that sales, recorded by the store, are not affected.
Three actions, with their reasons
The review ends with three actions at most. For each one, the agent states the expected effect on the gap from the target, its confidence level and the condition that triggers it, or the bounded test that will settle the question. It does not recommend what already exists, and it proposes no budget decision based on a single last click.
For the November day, the three proposed actions were the following:
A seven-day comparative test on the Meta campaign that brought clicks without carts, with the total budget unchanged, judged on the store's carts and orders.
A Black Friday email with a clear offer and tracked links, to make its contribution measurable.
Reconnecting access to the Google Ads data before it expired, so as not to lose the measurement.
These are proposals. The email states that nothing is changed and no send is triggered; the decision rests with the merchant and the team that runs the campaigns.
Evidence you can consult without weighing things down
Everything that supports the judgement is kept in an appendix that can be opened from the email. On the mock-up, it first had nine blocks: the day's position over seven and thirty days, traffic and choice of denominators, consent, Meta and Google presented separately, emails, the journey before and after the checkout fix, offer and prices, strong days, possible levers. Two blocks were added when the acquisition analysis was taken further: one on purchases attributed by Meta, still provisional, the other on measurement limits.
We apply a few principles there. Conversions from Meta, Google and the email tool are never added together. Missing data appears as missing, never as a zero. No customer personal data appears in the email or the appendix: no address, no identifier, no order number.
Where we stand
The rules for the agent's judgement are written down and checked by automated tests on our local test bench. They have not yet been tested on reviews produced entirely by the agent. The mock-up was written from a semi-manual investigation, carried out by a language model under our direction, then reviewed and corrected: a one-page session figure in particular turned out to be wrong, and the email's subject line was rewritten. It has not been sent to anyone. The automatic daily summary does work, but it remains a separate building block, with no investigation or judgement.
Email remains a reporting channel. The agent's ground is the data from the store and its tools. Before we let the agent send the review itself, we want to check that it can produce, day after day, a judgement the merchant can challenge, with the evidence to back it. That is the criterion that will decide what comes next, before any promise of time or sales gained.