hadobakht Independent thinking · Worldwide
Contact ↗
← Back to all writing

Data lineage is the history of a decision

If the meaning of a metric changes, a company must be able to discover which choices were made with the old meaning.

⌁DATA INTELLIGENCE · HN / 18

Lineage is usually introduced as a diagram showing where data originated and which transformations it passed through. That is useful for debugging, but the deeper problem is organizational memory. A financial forecast, a licensing decision or a customer segmentation may have relied on a version of a metric that no longer exists. If the company cannot reconstruct the meaning available at the time, it cannot responsibly explain why the decision seemed reasonable.

02 / PERSPECTIVE

A mature lineage record should connect source events, processing steps, definition versions, query context, access policy and outputs. For automated decisions it should also identify the acting system, approval, model or rule version and the resulting business event. This does not mean storing every intermediate byte forever. It means preserving enough evidence to reconstruct the relevant chain under explicit retention and security policies.

03 / PERSPECTIVE

Lineage has practical limitations. Event volume can make complete retention expensive, and revealing the entire graph to every operator would violate least privilege. Teams should distinguish high-consequence decision records from disposable operational traces. Cryptographic digests, immutable summaries, sampled diagnostic traces and appropriately protected evidence may be combined when they serve the relevant legal and operational need. A beautiful graph is not a substitute for usable evidence.

04 / PERSPECTIVE

The product question is whether a manager can move backward from a surprising decision to the underlying meaning and forward from a changed definition to affected reports, agents and customers. This two-way navigation turns lineage into a business capability. It lets organizations learn from corrections instead of silently rewriting their history. In intelligent systems, remembering the reason behind a decision may become as important as recording the decision itself.

Research and design credits ↗
Continue readingObservability needs a business unit, not just a log line↗