Coinbase Post-Mortem on July 14 Service Outage: Configuration Error Caused 50-Minute Downtime, User Funds Unaffected

Odaily Odaily reports that Coinbase has released a post-mortem report on the service outage that occurred on July 14, stating the incident was caused by a misconfiguration of a critical network component triggered by a routine configuration update. The impact lasted approximately 50 minutes, and user funds remained safe throughout the period.

Coinbase stated that at 12:34 Beijing time (12:34 AM Eastern Time) on July 14, the company deployed a routine configuration change to a shared production Kubernetes cluster hosting core infrastructure services. The update was originally intended to migrate to a new service deployment model to enhance system performance and reliability.

However, due to a resource name conflict that was not detected by the pre-production environment's detection mechanisms, this configuration modification unexpectedly affected the Istio Ingress Gateway-related Kubernetes resources within the cluster, rendering the network component unavailable. Approximately 3 minutes later, all inbound traffic to the cluster was interrupted, preventing internal services from accessing several infrastructure components.

Coinbase noted that because numerous asynchronous workflows rely on these infrastructure services, including processes such as transaction settlement, fund transfers, and debit card transaction authorizations, multiple business lines were affected. The impacted services included:

Retail users were unable to complete off-chain transactions, deposits, and withdrawals; some pending transactions appeared stuck and were completed after the service was restored;

Coinbase Card debit card transactions temporarily failed, credit card payments were unaffected, but some card management functions were unavailable;

Coinbase DEX's on-chain swap services on Base and Solana were suspended;

Coinbase Exchange and Prime institutional customers experienced failed or delayed transfers and settlements;

Coinbase Developer Platform customers were unable to complete account creation, fund transfers, or deposit services.

Coinbase stated that the ingress gateway was restored at 13:20 Eastern Time, and the incident was mitigated by 13:23. Subsequently, the system began processing the backlog of tasks, with most services returning to normal quickly; the remaining queues were processed over the following hours. During the recovery process, Coinbase encountered two additional challenges. First, the deployment tools themselves relied on the affected ingress gateway to operate, preventing the standard rollback process from being executed, creating a circular dependency where the "fault impacted the remediation tools." Second, while the emergency break-glass access process was effective, the permission verification and manual review procedures increased the recovery time.

Coinbase stated that there was no risk to asset security in this incident, but it exposed areas requiring continuous optimization in automated deployment, system dependencies, and disaster recovery mechanisms for large-scale financial infrastructure. The company will further strengthen system resilience to reduce the probability of similar events occurring again.