Business Data Lake Conceptual Framework

Existing IS landscape

Delivering Insights outside of the Business Data Lake aims at directly or indirectly (through people) influence Enterprise Business Processes.

Relevant ways to deliver insights at the point of action within the existing Enterprise Landscape include:

• 
  Injecting Insights into a external Data Store
  
• 
  Sending messages to external Applications using their native APIs
  
• 
  Providing an API to extract the insight or involve an analytic to drive insight on the fly using data in the BDL.
  

4. 
   Unified Data Management
   

Unified Data Management exploits metadata to manage the data lifecycle, data quality, access policies and services for Master Data Management (MDM), and Reference Data Management (RDM) and metadata management.

1. Master Data Management

Master Data represents a single source of common, basic business data objects that can be used by the BDL distillation and real-time analysis processing to verify, enrich and correlate data.

If Master Data Management practices and tools are already deployed in the Enterprise, the Business Data Lake should be built on the side at first. This is typically a case that illustrates how the BDL is deployed complementary to other IT services.

2. Reference Data Management

Reference Data contains authoritative lists of values or entities. These lists are generally massively re-used and widely “referenced” by other data or metadata. Country codes or Calendars constitutes typical examples of Reference Data. They also can be considered as Master Data from internal or external standards organizations.

3. Audit & Policy Management

The Big Data Lake standard should be implemented to accommodate the audit controls (e.g. COBIT 5.0) and the centralized application of information policies for security and information governance including provisioning, de-provisioning, access logs, data quality actions, authentication, authorization, encryption, filtering, log-ins, and single sign-on.

4. Privacy and Protection

Data in a BDL implementation may come from numerous sources that residebe in different jurisdictions, each with different privacy, retention and appropriate use legislation. This is especially true in large multinational companies. Architects have to be aware of the legislation and ensure that the appropriate controls can be implemented in the BDL.

5. Information Security

Information Security shall be architected from the beginning, including the labeling, handling and access to data over time (i.e. the sensitivity of data can vary over time such as a report to shareholders which becomes common knowledge after release).

5. 
   Unified Operations
   

Unified Operations concern the ability to provision, configure, monitor and manage the whole Business Data Lake from a single, unified environment that abstracts the distributed infrastructures and the multiple integrated services.

1. System Monitoring

System monitoring shall consolidate information from multiple levels, at least:

• 
  Infrastructures (disk, memory and network usage)
  
• 
  Operating System
  
• 
  Data storage
  
• 
  Processing workflows
  

The BDL itself can be used to get Insights from the logging data extracted from all the layers and services of the BDL.

2. System Management

The Business Data Lake System Management mainly consists in

• 
  A resource manager for the provisioning of BDL Elastic Infrastructures. It also takes care of failures that can happen among the cluster nodes.
  
• 
  A workflow manager that executes Batch Processing Workflows.
  

The Resource Manager generally has control over the processing engines, so that the Business Data Lake is as scalable as possible.

System Management must take in account the diversity of Business Compartments, especially regarding the elasticity (or not) of the underlying infrastructures and priorities for processing workflows.

Index

1^ From Real-World Data Mining: Applied Business Analytics and Decision Making,by Dursun Delen. Publisher: Pearson FT Press, December 2014

2^ Davenport and Harris "Competing on Analytics: The New Science of Winning" Harvard Business School Press ©2007 Page 7

3^ Definition from the Open Group Information Architecture Whitepaper

4^ "Big Data: The Management Revolution" Andrew McAfee and Erik Brynjolfsson, Harvard Business Review October 2013

5^ Ibid

6^ Ibid

7^ IEEE xxx

8^ "Big Data: A Revolution That will Transform How We Live, Work and Think" Viktor Mayer-Schoenberger and Kenneth Cukier ©2013 published by Houghton Mifflin Harcourt ISBN 978-0-544-00269

9^ Ibid

10^ "The Rise of Big Data: How It is Changing the Way We Think About The World" Kenneth Cukier and Viktor Mayer-Schoenberger published in Foreign Affairs May/June 2013

11^ Canadian Government Critical Infrastructure Protection Seminar 4-5 Nov 2013

12^ See the referenced material by Gartner at http://www.gartner.com/it-glossary/data-warehouse downloaded 30-Aug-2015

13^ Definition from the Open Group Information Architecture Whitepaper

14^ The Association for the Advancement of Artificial Intelligence (AAAI) and academia have been involved in these fields since the 1980s.

15^ Refer to ISO/IEC 2382-1: 1993.

16^ Refer to ISO/IEC 2382-1: 1993.

17^ NASCIO "Do You Think or Do You Know ? Part 2 - The EA Value Chain, The Strategic Intent Domain and Principles" ©2010 NASCIO

18^ Definition from the Open Group Information Architecture Whitepaper

19^ Data Management Association (DAMA): Dictionary of Data Management; refer to www.dama.org

20^ Data Management Association (DAMA): Guide to the Data Management Body of Knowledge (DMBOK) V1; refer to www.dama.org

21^ Government of Canada: Meta-data Directive, 2010

22^ Definition from the Open Group Information Architecture Whitepaper

23^ See the Open Group Information Architecture Guide for structured/unstructured/non-structured data definitions.

Directory: bigdata
bigdata -> Use Cases from nbd(nist big Data) Requirements wg
bigdata -> Use Cases from nbd(nist big Data) Requirements wg 0
bigdata -> Nist big Data Public Working Group (nbd-pwg) nbd-pwd-2015/6a,DW. abbreviated rr (M0444) Source: nbd-pwg status: Draft Title: Big Data Use Case #6 Implementation, using nbdra author: Afzal Godil
bigdata -> Nist special Publication 1500-4 draft: nist big Data Interoperability Framework: Volume 4, Security and Privacy