Integration of heterogeneous data in the data vault model

Abstract

Data increases tremendously with respect to volume, velocity, and variety. Nowadays, most of these data are unstructured like text documents, images, videos, Internet of Things (IoT) data, etc. Especially in enterprises, the analysis of semi-structured and unstructured data together with traditional structured data can add value. For example, semi-structured email data can be combined with structured customer data to keep a complete record of all customer information. Likewise, unstructured IoT data can be combined with structured machine data to enable predictive maintenance. Thereby, heterogeneous data need to be efficiently stored, integrated, and analyzed to derive useful business insights. The traditional modeling techniques like Kimball’s approach and Inmon’s approach are primarily focused on modeling structured data. Due to vast amounts of data being collected and agile project execution, scalability and flexibility become more essential characteristics in data modeling. However, especially regarding flexibility, the traditional data modeling approaches used in data warehousing face some limitations. Therefore, Data Vault modeling was developed to overcome these limitations. However, the Data Vault model was designed for structured data. To combine these structured data with semi-structured and unstructured data, the Data Vault model therefore needs to be adapted. However, there exists no comprehensive approach to do so for both semi-structured and unstructured data. This thesis, therefore, focuses on developing various modeling approaches to integrate semi-structured and unstructured data along with structured data into the Data Vault model. To this end, multiple use cases from different areas like Customer Relationship Management (CRM), Manufacturing, and Autonomous Car Testing that produce and use heterogeneous data are taken into consideration. Using examples from these areas, the different approaches are implemented and their advantages and disadvantages are discussed. In addition, the developed concepts are evaluated to check whether they fulfill the Data Vault characteristics.