The zkSync Era network shared a detailed post-mortem of yesterday's downtime incident from 1:52–6:02 AM CET. The team said issues are quickly identified and resolved, and the system is now healthy and operating 100% as expected.

According to the report, the issue began when the database for the block queue failed, causing block production to halt. Despite this, the server API remained unaffected, allowing transactions to continue being added to the mempool and queries to be served normally. Since the API was functional, none of the existing monitoring, logging, and alerting systems were triggered.

The team implemented a fix within 5 minutes of discovering the problem. They also took several immediate measures to address similar issues in the future. 

"Ultimately, the only long-term solution for liveness and availability is decentralization. Decentralized systems are inherently more resilient, and the decentralization of the sequencer (and subsequently the prover) is a critical priority for our engineering team." the team said in announcement.

Read full thread:

TL;DR:

1) Fix took 5 min.

2) Monitoring significantly improved.

3) Please remember, Era is in Alpha.

4) Decentralizing the sequencer is the long-term solution.

We'd like to apologize to everyone affected by this incident. Era's performance and availability are of utmost importance to us, and we take any incident extremely seriously. Although the system is still in Alpha, this doesn't imply any reduction in our level of care.

@zksync Era, still in Alpha, in only one week has already become the 3rd most popular L2 by activity. We are deeply humbled by your trust and see it as our immense responsibility to justify and protect it. https://l2beat.com/scaling/activity

We are constantly refining and improving the system. While we cannot completely rule out occasional hiccups in the short term, we are confident that their frequency will diminish as the system continues to mature and evolve.

Here's what happened:

1. The database for the block queue failed, causing block production to halt. The database health alert did not trigger because it could not connect to it to collect metrics.

2. Despite this, the server API remained unaffected. Transactions continued to be added to the mempool, and queries were served normally.

3. We have comprehensive monitoring, logging, and alerting in place across all components. However, since API was functional, none of them were triggered. It's challenging to predict and cover all edge cases at once, as situations like "no blocks for a while" can be quite normal.

4. At the time of the incident, the entire team was together at an offsite. Our team is usually widely distributed, with engineers online in various time zones across the world. In this case, it was around 2 am for everyone when the problem occurred!

The fix was implemented within 5 minutes.

Here are the further immediate measures we have taken to address similar problems:

1. We have assigned a special role to our database monitoring agents, enabling them to connect to the database and continuously gather metrics, even during database issues, similar to the one we experienced this time.

2. We have introduced alerts to signal when our database monitoring agent is malfunctioning or unable to establish a connection with the database for metric collection.

3. We addressed numerous additional corner cases that may indicate potential failures. Our on-call team is notified via multiple channels 24/7 in case of a critical escalation.

4. We have deactivated the auto-resolution feature for all alerts associated with our database's "golden metrics" within our on-call platform. This ensures that even if a critical alert temporarily resolves itself, the on-call engineer is still required to investigate the issue.

Ultimately, the only long-term solution for liveness and availability is decentralization. Decentralized systems are inherently more resilient, and the decentralization of the sequencer (and subsequently the prover) is a critical priority for our engineering team.