The business argument for data lineage Written by Ross Turk on June 1, 2021

There’s value in data, so the common wisdom goes. So your organization has started to collect, store, process and analyze data from any and all available sources. Over the past few years, the scale of these processes has steadily increased.

But there’s a frustrating little paradox you have to overcome if you want the most possible value out of it: the more data you have, the more difficult it becomes to understand. As pipelines become bigger and more expansive, with multiplying sources, a diverse set of use cases and greater volume than ever before, the whole ball of wax increases in value. However, it also grows hard to tame and even harder to trust.

IDC predicts global data volume will expand more between 2021 and 2024 than in the previous 30-year period. Dealing with this expansion in a structured and deliberate way is difficult, but it’s essential if your organization wants to make the most of its data assets.

Making sense of your data in 2021 and beyond requires a comprehensive understanding where it all comes from and how it was created. You need a strong metadata management layer, that, in turn, must be built on a strong foundation. Data lineage is that foundation. By learning what data lineage is and how to trace and analyze it, you are building capabilities that can keep you thriving, even as the data world becomes ever more complex.

What is data lineage and how do you identify it?

Datasets are moved through an organization by a complicated network of jobs. These jobs combine datasets together, transform them, and aggregate them. Data lineage is an understanding of this movement , acquired by tracing the execution of individual jobs and performing centralized analysis. Knowing the lineage of a dataset can help you understand:

• The provenance of your data; i.e., where it is being generated or how it is being acquired
• What data transformations occur, and how they affect key data characteristics
• Which personnel and applications are using data to make key decisions
• What the exact dependencies are between various datasets and transformations
• The business logic underlying each instance of data transformation and use

Data observability is the key benefit behind data lineage. Your data engineering teams need to be able to trace the path of each dataset; correlating this information allows them to find, fix, and prevent issues that would otherwise escape notice.

While tracing data lineage seems like a tall order in today’s sprawling and fast-moving environments, the consequences of proceeding without it can be severe. A lack of clarity about data’s context can lead to a lack of trust in that data. That mistrust undermines efforts to analyze and optimize business processes.

Tracing data lineage: How it works

Tracing accurate and complete data lineage for your organization requires the deployment of a new, end-to-end solution for data visibility. This solution, a lineage repository, captures operational metadata about pipeline jobs that run and the datasets they consume and produce. Tracing the connections between every job and dataset becomes automatic, rather than something data engineers have to do by hand in the midst of a harrowing troubleshooting process.

As each job executes, it passes key metadata to the lineage repository, which tracks all dependencies between each dataset and job, as well as the business logic generating those changes. If the lineage of a dataset changes suddenly, the lineage repository can report the anomaly and provide a historical view to show the potential causes. By maintaining flexible metadata on the observed relationships, the lineage repository can become the foundation for many other data operations and compliance systems.

Automatically tracing data lineage is so important because today’s sprawling data landscapes are so large, with so many complex dependencies and connections that it’s impossible to keep track of them manually. An average data practitioner already spends half the day chasing after quality and reliability issues — data lineage solutions are designed to make their workload easier to manage, not harder.

Once you have a lineage repository in place, and integrated with your data pipeline, your team can add a new dimension of visibility to their existing workflows, performing analytics and other information-intensive activities with a new level of confidence and trust in their data.

Data lineage offers so many benefits, and yet it doesn’t require a ton of heavy lifting to put it into place. Specifically, there is no need to migrate your data pipelines, move datasets around, or make changes to your application code. Data lineage can be traced in previously-existing pipelines without creating unnecessary operational overhead, and data lineage solutions should integrate with existing data ecosystem tools. That’s important, because the main objective in understanding data lineage is to create clarity, rather than add complexity.

Now that you’ve seen the way to trace data lineage through a lineage repository, it’s worth circling back to the central question: why? What are the likely outcomes of this effort?

Why is tracing data lineage important?

Within the data science space, the importance of data lineage has been long understood. As industry expert Matt Turck points out, data lineage is one of the major areas of DataOps technology businesses should be looking to strengthen. Matt attributes this need to increasing pipeline complexity, as well as the demands of advanced algorithms such as machine learning tools.

dAs organizations grow more digitally-native and data becomes more deeply ingrained in strategic decision making processes, a gap may open up between the haves, who can confidently map their data lineage, and the have-nots, forced to grapple with uncertainty.

The following are a few specific reasons to focus on data lineage, now and in the years ahead.

Trustworthy data is essential for your core business

Data plays an important role in the success of every product, project and decision. Take a second to think about some of the ways in which your organization may use data.

• Marketers use real-time customer information to power dashboards that will help them plan their next campaigns
• Executives make budget adjustments based on the past quarter’s company-wide performance
• An automated digital service draws on user behavior data to make an intelligent suggestion

There are dozens of such use cases scattered throughout all kinds of departments in every organization. One thing every one of them shares is a need for trustworthy data. Making internal decisions based on inaccurate, outdated or incomplete data negates the power of analytics, and may lead stakeholders to make less informed decisions.

Using poor-quality data in customer-facing services can be even worse, leading to a bad product experience. People may lose confidence in your business, all because your data is unreliable.

The stakes are high for ensuring data is trustworthy, and this process depends on having established a solid, reliable data lineage repository. Tracing data provenance and transformation is essential. There should be no unknown relationships between your datasets or programs. If something is off, a data engineer or data scientist should be able to determine the reason why, right away.

As data becomes a more central element in decision-making, driving business-critical processes, the quality and accuracy of data have become impossible to ignore. An issue with unreliable data could be extremely disruptive. Trust is fundamental to making data work for you, and establishing that trust depends on data lineage.

Your data management layer depends on data lineage

There is a horizontal layer within your business, connecting raw data and the analytics and operational functions described above. This is the data management layer, a central location consolidating metadata about datasets and transformations. This layer exists in the background so that data engineers can manage the complex data environment effectively

These data engineers’ day-to-day activities include the following, each of which has a role to play in making data useful across your entire organization:

• Data cataloging: Is all relevant information being catalogued? Data discovery, inventory and dataset usage depend on this process.
• End-to-end availability and quality assurance: Can users in every department have faith in the content they are using? If there is a solid underlying data lineage, they can.
• Security audits: Are you able to keep unauthorized users out of datasets? Data lineage must be established to determine who is accessing which dataset.
• Data governance and compliance management: Can the system track the flow of private user data? Complying with regulations such as the General Data Protection Regulation (GDPR) or California Consumer Privacy Act (CCPA) can be a major challenge if your team cannot trace the relationships between information.

To get all these processes right, you have to understand where data is being generated, stored and used. A deep understanding of end-to-end data lineage provides additional context to your data management layer. That, in turn, allows your organization to operate with more reliable and trustworthy data.

Data’s scale and role are only expanding

Data has been big in the recent past, it’s huge today and, in the future, it’s poised to become greater still. This sense of scale applies to everything data-related, from the variety of data sources available to the myriad applications and platforms, as well as the volume of information flowing in.

With IDC predicting 26% compound annual growth in data between 2020 and 2024, there’s a clear need for data management tools to keep up the pace. The researchers’ estimate for data created, captured, copied and consumed in 2020 was over 59 zettabytes.

Trends such as increased use of video communications and the internet of things sensor deployment are ensuring the numbers keep moving upward. As the scale and importance of data both tick up, you can be empowered rather than daunted by your information — if your technology choices keep pace with the state of the art.

The past decade has seen a revolution in information management, from Hadoop to Spark to data warehouses in the cloud. This process isn’t slowing down. This means your data lineage solution should easily integrate each new program or dataset, while also maintaining a connection with the possible multiple legacy selections in use. This should all occur without the need for strenuous manual processes by your data engineers.

How does Datakin help your organization thrive?

The data landscape of tomorrow is taking shape. Information is larger and more complex than ever before, and used in more contexts. Dealing with all this complexity can be daunting when there is a lack of clarity in the relationship between sources, uses and datasets.

Implementing the end-to-end solution offered by Datakin provides the vital perspective you need to start unraveling your data environment. Everything from data governance to analytics and AI relies on the context a solid foundation of data lineage provides.

If your organization is ready to discover the power of tracing data lineage firsthand, you can sign up for Datakin today.