Eliminating Waste Across the Data Factory

The time spent collecting and validating data can account for up to 80% of the total effort required to manage data. This estimate was shared by several speakers at the DGIQ 2022 conference in San Diego. We had previously estimated this “waste tax” to be in the 20%–30% range. Regardless of the exact figures, this waste is a huge problem and it’s getting worse. At DGIQ, there was also considerable discussion of the bureaucratic burden modern Data Governance has brought into corporations. Another source of waste. Aside from references to “Data Governance 3.0” as the next new thing, we did not hear much in the way of practical solutions.

This brings us to Lean Governance as a way forward. MetaGovernance has dedicated this column to the Art of Lean Governance. Disclaimer: Lean is not a synonym for Agile. In this piece, we are going to revisit the factory walkthrough as a process for detecting and eliminating waste.

Lean Thinking began in automotive manufacturing. Lean Manufacturing and the emerging Theory of Constraints revolutionized this industry. The hallmark of this approach was to build quality into products using the least time and materials. The goal was zero defects. As this approach gained acceptance, company after company began the journey to lean with simple walkthroughs of their manufacturing floor to look for waste. Among the first problems to be spotted were the growing piles of work-in-process inventory next to a machining workstation. Someone upstream was working faster than the process could handle. That explained the bottlenecks and production delays. Factories that went lean managed existing resources – equipment, inventory, and people – more effectively. Less truly was more. It still is.

What does this have to do with Data Governance? Everything. We can learn a lot from the factory movement, which leaned by cutting procedures – including bureaucracy – that add no value to the customer. Back then, quality metrics were redefined. Inventory management was king. Factories returned to the basics of manufacturing a quality product at the lowest cost. All it took was a massive paradigm shift.

We see Enterprise Information Management (the modern data factory) as the complete set of processes and technologies used in. This includes all the procedures, controls, and safeguards built in to reduce risk and waste. Despite their stated goals of data quality and regulatory compliance, many data factories today are failing due to bureaucratic burdens.

Below, we show where our data factory walkthroughs uncovered waste in two key areas: manual data reconciliations and incomplete awareness. These examples were based on interviews and general observations from a broad group of stakeholders.

Manual Data Reconciliations

Show us a spreadsheet used for data collection or validation, and we will show you waste. Given the time and money companies spend on data quality tools, it is still mind-boggling how often we observe manual data reconciliation. This process usually involves multi-tab spreadsheets designed to validate data used for reporting or operations. They persist because data consumers do not trust the data, or their control procedures mandate evidence. Why aren’t these reconciliations automated, and why aren’t the results shared? At one client, we counted the exact same reconciliation occurring across 6 departments daily. In this case, the added cost was estimated at over 200 hours per month. The solution was to leverage existing technology for the reconciliation and publish and share the results.

Incomplete Governance Awareness

People are trying to do their jobs without sufficient detail on owners, consumers, the system of record, definitions, and data copies. We call it Awareness. The massive projects to build glossaries and lineage can become outdated due to reorganizations or system changes. Beyond the sheer amount of wasted effort spent chasing down answers, the bigger cost is exposure to risk.

This was highlighted recently when a treasurer was horrified to learn that a market risk department had been using the wrong data source for model validation for more than two years. Although the treasurer knew that the market data vendor could not be trusted for this class of instruments, not all data consumers did. Everyone became aware of the problem after an adverse regulatory finding. This damaged the organization’s reputation. Higher audit fees and tighter scrutiny added insult to injury. All because the true system of record was unknown.

The Solution: Know Your Inventory

MetaGovernance uses an Awareness Matrix as part of our Lean Governance methodology. Using the factory analogy, the key to success is knowing your inventory. Rather than parts and widgets, your inventory is data. Both factories require inventory classification systems. When working with dangerous materials, factories require safety labels and handling instructions. In the data factory, confidential and PII data require the same. In data and information, this last point is a hallmark of Information Governance.

In the world of data, the sheer volume of moving parts requires a precise classification system. Across many clients, we see upwards of 20,000 individually named columns, many of which are synonyms due to spreadsheets. It is virtually impossible to maintain awareness at this level. Instead, we group data into like subject areas, or domains. Depending on client industry and desired level of precision, we typically see fewer than 100 subject areas. We then establish governance awareness at the subject area level. This awareness is tracked as metadata and includes owners, consumers, security classifications, retention, system of record, and risk levels. For critical or sensitive data used in controls or that is more vulnerable, we track lineage and the sheer number of known copies across structured and unstructured data.

Invitation to Participate in Lean Governance Project

This level of awareness and control makes a measurable difference in delivering value to the data customer while reducing risk and cost. MetaGovernance is seeking joint research projects for a book we are writing on the Art of Lean Governance. We welcome the opportunity to explore (confidentially) ways to eliminate potential waste and risk in your data factory at no cost. Please contact us at metagovernance.com if interested.