Celo validator roundup, September 2020

After a little detour in the #DeFi land we are back at covering Celo network updates. September marked two big events for the Celo ecosystem. CELO token got listed by the CoinbasePro exchange, which has had large positive consequences for the Celo stability protocol. Valora app is now available on both the Google play and on the iOS App store for all users.

We will cover CoinbasePro listing of CELO and review the current state of the Celo stability protocol in a separate post. You can subscribe to our newsletter if you want to be notified immediately when that article is released.

Overview

Celo mainnet has now been live for 160 epochs. Target APY earned on locked and voting CELO continues to decrease slowly but still at sizeable 5.78%.

As we discussed in the last issue, key factor to focus on and improve in Celo’s protocol right now is its lightweight identity protocol and attestations.

Attestations

First, lets look at some stats around adoption of the Celo network. First graph shows increase in “registered accounts”. Registered accounts includes all ValoraApp accounts but it also includes all the addresses that might register for other reasons (i.e. to lock CELO, or to run as a validator or become a validator group and so on). Total number of the registered accounts seems like a good proxy to track overall adoption of the Celo network.

Second graph shows total number of unique users that have issued attestation requests, aggregated over a 24h time period.

As we can see from the graphs, the numbers are still too small to do too much statistics on them. There are around 20-30 new registered accounts per day on average, but that is too small of a number to be able to distinguish testing noise from the actual new users.

Nonetheless, we can still look at overall success of attestation requests. The graph below shows percentage of unique users that have failed to complete even a single attestation, aggregated over a 24h time period. Note that for a user to successfully verify with the Valora App they need to complete 3 attestation requests, thus this graph is potentially underestimating percentage of users that have failed to go through the verification flow. We chose to track this metric because if a user fails to complete even a single attestation, it is more likely that the individual validators aren’t at fault and there are large issues in the system that prohibit users to successfully verify (i.e. general Twilio/Nexmo delivery issues, app issues, users aborting verification flows, etc…)

We can see that overall attestation completion rates are quite bad. And more importantly, they aren’t getting better. Because actual per day sign ups are so low, it makes it hard to distinguish noise from the actual large scale issues, so these numbers should still be taking with a grain of salt. But at least the preliminary view seems to be that there are quite a bit of issues still with the attestation flow that is independent of individual validators having issues with their attestation setups.

For alternative set of similar stats, thecelo.com has also added a new tab to cover state of the attestations: https://thecelo.com/?tab=attestations

Validator Performance

September was a rough month for a lot of the validators. An unintentional burst of transaction volume caused out of memory issues for a lot of the validators, causing them to crash and corrupt the local disk state. You can read more about the issue and the post mortem here.

Only one validator crossed the slashing threshold, which was part of the `Alive29` validator group.

Below is the list of the validator groups that had quicker recovery from their issues:

And also the list of groups that had longer issues (or multiple issues through out the month):

As you can see the list is a lot larger compared to previous more quiet months. As we have been predicting, it is very likely that as the Celo network picks up more real usage (and especially if there will be malicious actors involved), that we will see more operational issues with the validators overall.

We are all still running in the easy mode, still serving less than 1TPS on average. If and when the actual usage picks up, that is when we will truly see how stable the network is overall and which validators can keep up with the demands of a growing network.

Coming up next…

Most important development for the validators to look forward to in next month is related to critical upgrades for the blockchain client and for the core contracts. More information about this in this post: https://forum.celo.org/t/governance-proposals-to-make-the-protocol-safe-and-easy-to-upgrade/6

Celo network has had mixed success with previous much simpler upgrades. While the upgrades have gone more or less ok (with some downtimes for individual validators), the proposal to upgrade minimum client version to 1.0.1 has actually failed and has been postponed indefinitely for now. This not the greatest sign for a young network, but hopefully this new set of upgrades goes more smoothly. And hopefully we can successfully upgrade minimum client version in October.