Hubble's Stellar ETL has been refactored to use the Composable Data Platform (CDP), replacing Captive Core with Galexie and a Ledger Metadata Store. This architectural change delivers 93% faster processing, 90% cost reduction, and significantly simplified maintenance.
The article presents a detailed case study of how Hubble, Stellar's open-source historical dataset, refactored its data ingestion pipeline (Stellar ETL) to leverage the new Composable Data Platform. The original architecture tightly coupled data extraction and transformation, requiring a new Captive Core instance for every ledger range processed, which resulted in 15+ minute startup times, high compute costs, and operational complexity. By decoupling Captive Core and replacing it with CDP components—specifically Galexie for data processing and a Ledger Metadata Store—the team achieved dramatic improvements: processing time reduced from 15+ minutes to under 1 minute per ledger range (93% reduction), monthly costs dropped from ~$100 to ~$10, and machine requirements decreased from 3.5 vCPU/20GB to 0.5 vCPU/1GB. The refactoring also eliminated maintenance burden by removing the need to manage separate Captive Core configurations and enabled parallelized backfilling, reducing full network history backfill time from months to under 4 days.
