DataOps as a Service

DevOps revolutionized Software engineering by adopting agile, lean practices and fostering collaboration. The same need exists in Data Engineering.

In this talk, Antoni will go over how to adopt the best DevOps practices in the space of data engineering. And the challenges in adopting them considering the different skill sets of the data engineers and the various needs.

– What is the API for Data? – What types of SLOs and SLAs do data engineers need to track? – How do we adapt and automate the DevOps cycle – plan, code, build, test, release, deploy, operate, and monitor data? – Those are challenging questions, and the data engineering space has no good answer yet.

Antoni will demonstrate how a new open-source project Versatile Data Kit (https://github.com/vmware/versatile-data-kit), answers those questions and helps introduce DevOps practices in data engineering.