How a record is calculated

Methodology · v1.1

The same rules for every store. Every change is versioned and published.

Principles

Only facts from sources the store does not control. Disputes opened by the buyer's bank, transactions from the payment processor and timestamps from the platform's own system. Never reviews, tags, notes or data from apps the store has installed.

No volume figures. Volume is published only in bands rounded down ("more than 50,000"), never the exact figure nor counts it could be inferred from.

Only corroborated payments. Orders charged through a payment processor count. An order marked as paid by hand (manual cash bogus cash on delivery (cod) bank deposit money order) proves nothing and counts towards no metric.

Records from before installation. They are used going back up to 18 months, only from records the store cannot rewrite, and every record says so.

The "Verified store" status

A store is shown as verified when its observed rate of service-related bank disputes stays below the public threshold of 5 per 1,000 orders, with at least 200 orders measured. The minimum is 1 divided by the threshold: with fewer orders, a single dispute already reaches it and the percentage means nothing. The rule is the same for a store with 400 orders as for one with 50,000. The confidence interval is published in every record for reference, not as a filter. If the rule is not met, the store is shown as "under observation" and its figures remain public.

Exit criterion. If a verified store stops meeting the rule, it keeps the status for 30 days and the record declares it "under review". If it still does not meet the rule at the end, it moves to "under observation".

For reference: the Visa (VAMP, Europe) and Mastercard (ECM) monitoring programmes start at around 15 per 1,000 counting every dispute, fraud included. This threshold counts service reasons only, which makes it several times stricter, and it applies to small stores too.

Purchases disputed with the bank

Refunds money

Refund records where the money went back through the same processor it was charged with, measured against every one of the refund records in the window. These count as "no money proven": store credit (shopify_store_credit gift_card), vouchers or codes issued by apps outside the processor, exchanges, failed attempts and refunds through processors with no corroboration (manual cash bogus cash on delivery (cod) bank deposit money order). That way no voucher can be left out of the percentage. A window of 90 days; it is published with at least 10 records.

A "Yes" requires two proofs. First: bank disputes for refunds not processed (CREDIT_NOT_PROCESSED) stay below the public threshold of 2.5 per 1,000 orders (observed rate, with at least 400 orders measured). Second: the store really does refund money, with at least 3 refunds that returned money to the original payment method and that account for at least 10 % of all the refund records. That bar is deliberately low: offering a voucher the customer accepts is legal and is not penalised. If either proof is missing, the record shows "Unconfirmed", never "No"; with fewer orders or refunds, "Not enough data".

Other figures in the record

Orders charged: orders charged through a payment processor in the last 365 days that still stand (not cancelled, not fully refunded, and shipped), in bands. Cancellations: measured against the orders of the last 365 days. Order preparation: median from the order to the first recorded shipment, over a 90-day window ending 14 days before the calculation; orders not yet shipped count as slow. This figure is recorded by the store itself (level 2) and is shown as such.

Payment processors and coverage

Bank disputes can currently only be read from shopify_payments. Dispute percentages are calculated over the orders charged through those processors, and the share of orders they cover is stated. If that share is below 70 %, nothing is claimed about disputes or about "Refunds money", the store is listed as "no verifiable payment processor" and the product-page module is not offered.

Integrity signals

Every record publishes, with no personal data: the peak order count of the last 3 months against the median of the previous 9; the share of orders more than 30 days old with no shipment; the share of orders fully refunded within 7 days or less; and whether orders are missing from the months snapshotted at the first measurement. An order's creation date is set by the platform and cannot be backdated.

Questions to the buyer

Orders that the carrier reports as delivered receive an email with two closed questions about that order: whether everything arrived and whether it matched the listing. A third one —whether the store replied— appears only when something went wrong. There are no stars, no comments and no free text, and no answer is published on its own: only proportions over the whole. holdmark sends it, not the store, and only if the store asks for it: it is off by default. Each order is asked once and there is no reminder.

When the question is sent. 3 days after the carrier reports the complete order as delivered: enough time to open the box and look at what is inside. If the order came in several shipments and one of them never arrives, the question is sent 8 days after the first. More than 60 days after delivery nothing is sent: nobody remembers the details, and an answer from memory is not a fact.

To publish any proportion, two conditions must hold at once: at least 30 answers and at least 5 % of those asked must have answered. Neither is redundant, because they protect against different things. The minimum number of answers protects against noise: with fewer, a single answer moves the figure by several points. Coverage protects against bias: if only a very small share answers, the people who answer stop resembling the people who bought, and that is not fixed by accumulating answers. «The store replied when something went wrong» is measured only among those who had a problem and did ask, so it has its own denominator and its own minimum.

Declared bias: only delivered orders are asked. Someone who never received theirs cannot answer, so these figures always come out better than reality. What never arrived is measured by the carrier, separately and over all orders.

Transparency about these rules

Anyone can pick a threshold after seeing their own figures and put it wherever suits them. The only thing that separates a rule from an excuse is putting in writing when it was set and what was known at that moment. Here it is, including the times it was set with the data already in hand.

• 14 Sep 2026Set before seeing the data

  Bank claims threshold

  Set before the dispute reasons of the first store measured were computed. What was already known about it: its aggregate dispute rate, not the breakdown by reason.

• 14 Sep 2026With the data in hand

  90-day window for refunds

  Set after seeing the first store’s data, for a structural reason: when a store changes its process or its apps, its refunds from a year ago no longer describe how it works today.

• 15 Sep 2026Set before seeing the data

  Threshold for refunds not processed

  Set before that figure was looked up in any store.

• 15 Sep 2026With the data in hand

  Minimum of refunds in money

  Set with the first store’s aggregate percentage already known.

• 15 Sep 2026Correction

  Refund percentage

  It used to count only refunds that moved money through the payment processor, leaving out credit notes issued by external apps. It now counts every record.

• 15 Sep 2026Rule change

  Observed rate with a minimum of 1 over the threshold

  So that stores of any size are measured by the same yardstick.

• 15 Sep 2026Rule change

  Processor coverage, manual payments, volume and integrity

  A minimum processor coverage was added, manually marked payments were excluded, volume began counting only orders still standing, and the integrity signals were added.

• 30 Sep 2026Set before seeing the data

  Questions to the buyer: minimum answers and coverage

  Set before a single answer was received. The 5% coverage figure is an empirical floor —post-purchase emails are answered at rates between 5% and 15%— and not a theoretical result: it will be revised once there are real answers from several stores, and the change will be recorded here.

• 2 Oct 2026Rule change

  When the question is sent: from 4 days to 3

  The wait after delivery drops from 4 days to 3, and the one for an incomplete order from 12 to 8, the day before real questions start going out. It was not dropped to 2, the other option on the table, because of a risk that cannot be measured with what we store: the carrier says «delivered» without saying whether it was handed over in person or left at a pickup point, and someone who has not collected it yet would be asked about a box they do not have. A late answer is only late; a false one dirties the registry. It will be reviewed with real answers.