Devnil from SDF's Voyager team presents the first version of Stellar's data analytics system, built on BigQuery and Kubernetes to enable fast historical queries and network insights. The system powers internal business analytics and will soon support real-time data, improved observability, and public visualizations.
Devnil, a software engineer at SDF's Voyager team, discusses the design and implementation of Stellar's new cloud-scale data analytics pipeline. The system addresses limitations of Postgres and Horizon for historical analytics by using Google BigQuery for fast queries, Google Cloud Storage for intermediate data, and Metabase for visualization. Key design goals included supporting internal stakeholders, maintaining open access, daily updates, Postgres integration, and easy deployment on Docker and Kubernetes. The talk covers implementation details, sample SQL queries demonstrating network analysis, and next steps including real-time ingestion via Isaiah Turner's command-line tool, Airflow task management for better observability, public-facing market visualizations, and future data science platform capabilities.
