Stellar network experienced a 16-hour outage on September 20, 2014, caused by validator nodes running out of RAM, halting consensus and all transactions. The foundation addressed communication gaps and began work on scalability improvements to prevent recurrence.
On September 20, 2014, multiple Stellar validator nodes failed due to memory exhaustion, causing the Linux OOM killer to terminate processes and preventing the network from reaching consensus. The outage lasted approximately 16 hours, during which most nodes lost network connectivity and ledger synchronization. Root causes included legacy codebase scalability limitations and potential memory leaks in the nodestore. The foundation's response was delayed due to an off-site team meeting and internet connectivity issues at their location. Immediate remedial steps included rebooting failed nodes, reducing node sizes, and restarting stellard services. Preventative measures underway include rebuilding validators with more RAM, improving community communication channels, investigating alternative nodestore backends, and continuing the stellard rewrite to address fundamental scalability issues.
