LocalStack without LocalStack - BigData pipelines on your laptop

The benefits of testing cloud based BigData pipelines locally are many and obvious and therefore the idea behind it is not something new. Thus tools such as the AWS specific LocalStack and similar solutions for other cloud providers have appeared – closing the gap between the cloud infrastructure and your local machine. In this session we will outline a generic and cloud agnostic approach of emulating BigData processing engines with readily available and free technologies. We will go through some of the challenges of combining those into a CI/CD being able to test the correctness of your pipeline at build time. Our end goal will be to encourage you to try it and see the benefits for the stability and quality of the applications being built especially in a you-build-it-you-own-it mentality in which most of us live in today.