Why this exists

Schema drift is a structural problem - a column changes type, a table moves, a field goes missing, and the pipeline usually breaks loudly enough to get noticed. Semantic drift is quieter and more dangerous: the schema stays exactly the same, the SQL defining the metric doesn't change a single line, and the dashboard still runs without error. What moves is the real-world condition a value is supposed to represent - and nothing in the warehouse is positioned to notice that, because from its perspective, nothing happened.

This is what that looked like the specific time it mattered: a single metric, "Active Users," that started telling a noticeably different story with no schema change, no failed test, and no one on the data side having touched anything.

The starting point

Weekly Active Users stepped up by a meaningful margin between one reporting week and the next, with no product launch, no campaign, and no seasonal pattern to explain it. The metric's SQL - COUNT(DISTINCT user_id) FROM events WHERE event_type = 'active_session' - hadn't been touched in months, confirmed by checking the dbt model's own change history. The underlying event table's schema was identical: same columns, same types, same event name showing up in the same place.

By every signal analytics normally watches - schema, SQL, pipeline logs, test results - nothing had changed. And yet the number had.

What was actually happening

The event itself hadn't changed. What had changed was the condition under which the mobile app decided to fire it.

  • active_session had always required a minimum ten-second foreground session before it fired - a threshold meant to filter out accidental opens.
  • A mobile release, shipped by a different team for an unrelated reason, needed a general "app opened" signal for a push notification feature and reused the existing active_session event instead of introducing a new one, dropping the duration threshold in the process.
  • The event name, its schema, and its column in the warehouse never changed. What changed was the population of real-world behavior that now qualified to write a row into it.
  • Nothing about that change touched anything analytics owned or reviewed - it lived entirely in application code, several layers upstream of the warehouse, with no migration and no schema diff to flag it.

The metric wasn't computing anything wrong. It was correctly counting exactly what it was defined to count. The definition itself just wasn't the same definition anyone thought it was anymore.

The decision log

The instinct when a metric jumps is to check the pipeline first. That instinct is usually right - just not this time - and ruling it out in a specific order is what eventually pointed at the real cause.

Decision 1: Rule out the warehouse before questioning the event

Schema, SQL history, and load logs were checked first, in that order, because a structural change is the more common cause and the fastest one to confirm or eliminate. All three came back clean, which is what shifted the investigation from "what broke downstream" to "what's actually landing in the table."

Decision 2: Look at the shape of the change, not just its existence

The jump was a sustained step-change on a specific date, not a gradual drift and not a one-day spike. That shape looks like a release, not a data-quality issue, which is what justified checking app release dates next instead of continuing to inspect the pipeline.

Decision 3: Plot the raw event, segmented by app version

Breaking active_session volume down by client app version - rather than looking at the aggregated daily metric - showed the increase was isolated entirely to sessions from the new release. Older app versions kept producing the event at the old rate throughout, which made this a behavior change tied to a specific release, not a warehouse-side issue.

Decision 4: Confirm with the team that owns the event, then decide how to handle the seam

The mobile team's release notes confirmed the duration threshold had been dropped for an unrelated feature. The remaining decision wasn't technical - it was whether to silently patch the SQL to approximate the old definition, or to treat this as a genuine redefinition of "active" and document the cutover date so the trend line's discontinuity had a known, stated cause instead of an invisible one.

Before and after

Before

  • "Active" defined implicitly by whatever condition the app happened to enforce
  • No record connecting a metric's history to the event contract behind it
  • A firing-condition change shipped with no analytics review
  • The trend line's step-change with no documented explanation

After

  • "Active" defined explicitly in a written event contract, reviewed on both sides
  • The redefinition dated and annotated directly on the dashboard's trend line
  • Event-firing-condition changes flagged as a required review step before mobile releases
  • Raw event volume by client version tracked as a standing check, not a one-off investigation

What this actually took

Nothing in this required a pipeline fix, because nothing in the pipeline was broken. What it required was accepting that "the schema didn't change" and "the metric didn't change" are two different claims, and only one of them had actually held. A metric built on top of an event inherits whatever the event currently means - and that meaning lives in application code the warehouse has no visibility into and no reason to alert on.

The lasting fix wasn't a query change. It was treating the definition of "active" as something that needs its own explicit contract and its own change log, separate from the SQL that happens to compute it today. Catching drift like this before it reaches a dashboard is core to the Analytics & Data work done for every client engagement.

Frequently asked questions

How is this different from schema drift?

Schema drift is when the structure underneath a metric changes - a column type, a table's shape, a field going missing - usually visible as a broken pipeline or a failed test. Semantic drift is when the structure stays completely intact and only the real-world meaning behind the data shifts, which means none of the usual structural checks have any reason to fire.

How would I know if this is happening to one of my own metrics?

Watch for a metric that step-changes on a specific date with no corresponding schema change, deploy, or campaign on the analytics side. Segmenting the underlying raw event by app or platform version, rather than looking at the aggregated metric, usually isolates it quickly if the cause is a release-driven behavior change.

Can this be prevented, or only caught after the fact?

It can be reduced, not fully prevented, by treating any event that feeds a reported metric as having an explicit contract - documented firing conditions, reviewed by both the team that owns the event and the team that reports on it - so a change to what qualifies gets flagged as a decision instead of shipping as an implementation detail.

Not sure if one of your own metrics has quietly redefined itself?

A short conversation is usually enough to tell whether there's a quick answer or something deeper worth tracing.