Cloud Architecture for Threat Hunting on Petabytes of Data

Modern cloud architectures often rely on storing and analyzing “big data” where common recurring problems are cost, scale and searchability. This has been traditionally solved with proven No-Sql technologies such as ElasticSearch, Cassandra, Hadoop, data lake, etc. Our team within Vmware CarbonBlack explored these technologies and found limitations with respect to the data ingress volume, latency, data privacy, cost, scale and search flexibility. In this talk we explore the alternative solution built: a novel way to leverage existing Lucene search technology in conjunction with data lake approach to store, index and search many petabytes of real-time streaming data in a cost-efficient way.