![]() When we designed and built the initial EDI, our team had the mission statement to “provide infrastructure for teams at Spotify to reliably collect data, and make it available, safely and efficiently.” The use cases we focused on were well supported, such as music streaming and application monitoring. Now our incomplete and low-quality data was degrading the productivity of the Spotify data community. Our internal users had feature requests and needed more from the system. However, with that high adoption and traffic increase we discovered some bottlenecks. This increased the total volume of data which we ingested daily to nearly 70TB! (Figure 1).įigure 1: Average total volume (TB) of events stored daily by our ETL process (after compression). The peak traffic increased from 1.5M events per second to nearly 8M, and we were ready for that massive scale increase. We also improved operational stability and the quality of life of our on-call engineers. We then extended it to adapt to the General Data Protection Regulation (GDPR), we introduced streaming event delivery in addition to batch, and we brought BigQuery to our data community. Our design was optimized to make it quick and easy for internal developers to instrument and log the data they needed. Not everything went as planned, and we wrote about our learnings from operating our cloud-native EDI in Part IV. In 2016, we redesigned the EDI in Google Cloud Platform (GCP) when Spotify migrated to the cloud, and we documented the journey in three blog posts ( Part I, Part II, and Part III). Throughout this blog post we make a distinction between the internal users of the EDI, who are Spotify Engineers, Data Scientists, PMs and squads, and end users, who use Spotify as a service and audio platform. ![]() ![]() We instrument and log data across every surface that is running Spotify code through a system called the Event Delivery Infrastructure (EDI). We log a variety of data, from listening history, to results of A/B testing, to page load times so we can analyze and improve the Spotify service. ![]()
0 Comments
Leave a Reply. |