Skip to main content
Logical Data Management
Logical Data Management

Overview

We live in a world of accelerating change, disruption, and technological breakthroughs, which are transforming how we live and work. To be prepared, an organization’s secret weapon is their data.

The main goal of data management is to improve insights and organizational effectiveness through data. It strives to deliver data:

  • In the forms that a variety of stakeholders need, including both business and technical users and, increasingly, artificial intelligence (AI) models
  • At the speed needed by the business
  • In a secure, compliant manner.

However, organizations manage many applications and systems focused on different aspects of the business, from supply chain management to marketing automation, as well as a variety of data systems, such as data warehouses, data marts, and data lakes.

To be able to deliver data that meets the above requirements, organizations need to efficiently manage these complex data landscapes. They have turned to two different approaches:

  1. Centralized data strategies, based on physical data consolidation, and
  2. Logical data strategies, based on consolidated views of data, across disparate systems.
Logical Data Management

Centralized data architecture and management

Centralized data architecture and management rely on physically copying data from multiple systems into a single repository (e.g., data warehouse/data lake/house). This is the traditional and more common approach. It is relatively easy to get started with this approach, however, with ever-increasing numbers of data sources and ever-rising data volumes, it is just not practical and lacks agility.

When data is constantly produced in massive quantities and is always in motion and constantly changing (e.g., IoT platforms and data lakes), attempts to collect all this data are neither practical nor viable. This is driving an increase in demand for connection to data, not just the collection of it (Data Virtualization).”

Gartner Critical Capabilities for Data Integration Tools, August 2022

Logical Data Management

Challenges of a centralized approach

One size never fits all

Operational databases, noSQL, graphs, key-value pairs, data lake engines, enterprise data warehouses, etc., exist to address different data requirements. No single data system can meet all data needs, so data must be continually copied to and from these different systems.

Take significant time and effort

Complex data pipelines and expensive data replication scripts must be designed for each new data need, and re-designed to accommodate infrastructural changes.

Consolidation may be prohibited

Regional data privacy regulations can often limit where data can be physically stored after being copied.

Does not reduce data silos

For 30+ years, organizations have tried consolidating all data into a single repository, including enterprise data warehouses, big data lakes, and now cloud data “lakehouses,” but the outcome has not changed, and silos remain.

Logical data management

Logical data architecture and management enable access to multiple, diverse data sources while appearing as one “logical” data source to users. It is about unifying data that are stored and managed across multiple data management systems, including traditional data sources like databases, enterprise data warehouses, data lakes, etc., and other data sources like applications, big data files, web services, and the cloud to meet every analytics use case.

Logical data management allows practices like data discovery, access, security, integration, and sharing to be performed through a logical (or virtual) representation instead of directly on each physical source system. Logical data management practices enable the consistent implementation of policies and practices to manage, integrate and use an organization’s data, regardless of each source system's nature, location, and capabilities.

The market has realized that those data integration tools that do not balance “collect”- with “connect”-based data management architecture strategies will always result in data silos and/or poorly integrated infrastructures.”

Gartner Magic Quadrant for Data Integration Tools, August 2022

Logical Data Management

Benefits of logical data management

illustration easy use

Ease of use

Consumers have a single location to access any data.

Agile data integration options

One platform to support full range of data integration options from full replication and transformations, caching and real-time federation options.

Agile Data Integration

Centralized security and governance

Access control and policy implementation are done consistently in a single location.

Futureproof

Decoupling from data location and schemas allows for technology evolution and infrastructure changes.

illustration future proof
The Role of a Logical Architecture in Modern Data and Analytics

The role of a Logical Architecture in Modern Data and Analytics

Although there are clear advantages to logical approaches, organizations sometimes need to employ a monolithic approach to meet a particular business need, both approaches can be deployed together as part of an overall agile data management strategy.

Logical Data Management

The Benefits of Logical Data Architecture and Management

Investments in logical data architecture and management pay off quickly. By implementing a logical data layer, organizations can access any data source, connect data silos, and provide 360-degree views of their data to anyone across the enterprise. This holistic data integration and management approach have a wide range of benefits, including improved collaboration between data managers/owners and data users/consumers. Below are just a few of these benefits from the business and IT perspective.

Business Perspective

Business Perspective

  • One place to find all your data: A logical data layer centralizes your data, enabling real-time access to all data, including data stored on-premises and in the cloud, with semantic consistency.
  • Business-friendly views: With a logical data layer, organizations can easily create business-friendly, self-service views of their data. This is done by eliminating the traditionally complex data integration technologies, formats, protocols, and location details, making data more approachable and user-friendly.
  • Improved data utilization: Enable less-technical users to quickly find, access, integrate, and share data. Enable subject matter experts in the business to become a part of the data modeling process and reduce the cycle time required to create ready-to-use data.
IT Perspective

IT Perspective

  • Real-time data: A logical data layer enables stakeholders to access the most current data on demand, anytime, in preferred formats to support a wide variety of business use cases. Businesses no longer need to wait for a nightly batch run to refresh their datasets.
  • Reduced data sprawl and duplication: By eliminating unnecessary data replication with a logical data layer, organizations can increase agility, mitigate risk, and reduce costs.
  • Faster data fulfillment: Logical data layers enable accelerated provisioning of new datasets, enabling organizations to keep pace with ever-changing data requirements.
  • Centralized security and governance: With a single layer embracing access to all enterprise data, enforcing security and managing data controls for every user across all datasets becomes much easier.
Logical Data Management

The Required Capability for Logical Data Management

To successfully implement a logical data architecture and management approach, organizations need only one capability:

Data virtualization is a data integration and data management technology that leverages metadata to enable organizations to access all enterprise data in real-time, and to discover, catalog, provision, combine, share, and govern data to meet a wide variety of use cases.

Learn more about data virtualization
LDF Architecture