Building Data Flows Using Distributed Key-Value Stores

Abstract

Social communication features on most of today’s largest websites require propagating the data inside the database/key-value store leading to massive data flows. In this paper we study alternative architectures to build data flows using distributed key-value stores. We compare programming model, execution model, failure model, and scalability highlighting a problem of the state-of-the-art architecture based on an external queue: non-optimal resource utilization. As part of this study, we propose an optimization of this approach by integrating queues into the key-value store. It results in better resource utilization and, thus, more cost-effective scalability; as well as easier programmability and lower maintenance cost. Our experimental study confirms these findings.