Evaluation of MLOps approaches and implementation of a data product development pipeline

Abstract

Recent advancements in Machine Learning (ML), particularly with Large Language Models (LLMs), have significantly impacted the tech industry, prompting companies to integrate these technologies into their products and services. However, many organizations lack the expertise to effectively develop or incorporate ML models, facing challenges in both development and operation. Despite the emergence of Machine Learning Operations (MLOps) paradigms intended to streamline the handling of Artificial Intelligence (AI)-enabled applications, selecting appropriate tools and constructing cohesive MLOps pipelines remain significant hurdles due to the vast and fragmented landscape of available tools. This master’s thesis investigates how a holistic MLOps pipeline can enhance the development and operational processes of data products, even in business contexts without extensive technical expertise. Conducted within a department of the Robert Bosch GmbH, the research is structured into three phases. The first phase involves a comprehensive evaluation of state-of-the-art MLOps architectures, platforms, and tools through a Rapid Review, establishing a foundation for the pipeline’s design. The second phase focuses on designing and implementing an MLOps pipeline tailored to the specific requirements and constraints of the case study environment. The developed solution integrates with existing infrastructure on Azure and Development and Operations (DevOps) processes using GitHub and Terraform. It is built on the Databricks platform, utilizing integrated components such as Unity Catalog and MLflow, and is extended with the tool Langfuse and libraries like Giskard and Ragas. In the third phase, the implemented pipeline is evaluated in a practical setting through two hackathon-like use tests. Feedback from these tests assesses the pipeline’s effectiveness in improving the efficiency and quality of data product development. The evaluation revealed successful application in data product development and identified areas for further improvement, such as enhancing the pipeline’s documentation. Overall, this thesis presents a complete process of developing an MLOps pipeline from initial evaluation to practical implementation and assessment in a real-world context. The findings contribute to the field by providing practical insights into effectively implementing MLOps pipelines, offering valuable guidance for organizations seeking to adopt MLOps practices to optimize their data product development.