Journal · August 2025
Cohort windows that lie
A window is a promise about time. Stretch it after you see the chart and you have not analysed the product. You have edited it.
The lie is usually small. Someone says “users who returned within a week,” then filters for eight days because the weekend sits awkwardly. Or “activated in the first session,” then includes sessions that began at 23:50 and spilled into the next calendar day in UTC while the user was in London. App Analytics tools will comply. They are not referees.
A more elegant lie is the moving window on an experiment readout. Control and treatment are compared on day 3 while the test is young, then on day 14 when the lift appears, then back to day 7 when a reviewer asks for the original spec. Each number can be true. The sequence is not an analysis; it is a hunt.
In our flagship we make the window a contract with a date. You may change it, but you file an amendment: why, who asked, and which previous charts are now incomparable. Students find this bureaucratic until they watch a stakeholder try to reopen day 7 in week five. The paper is then a relief.
Time zones deserve a dull paragraph. If your cohort is “installed on Monday,” specify the timezone of the clock. Mobile apps collect timestamps in several dialects. If Android uses device time and your warehouse converts to UTC at ingest, Monday in Glasgow and Monday in the database can be different days. Funnel Forensics spends a session on this; it is not glamorous; it explains more “drops” than copy changes do.
If you want the short version for a sceptical colleague: freeze the window before you generate the figure. If the figure disappoints, that is information. Changing the window is a different product question, and it needs a new title.