Stealth qPoS Internal Testnet Week 1

Today, I describe progress and challenges related to Week 1 of the qPoS Internal Testnet. Most of my time this week was spent tracking down multithreading issues. Each issue took nearly two days to diagnose but about 20 seconds to fix. But most notably (from an end-user perspective), I had to change the block index from the original (genesis) design. The bulk of today’s post is dedicated to describing this change.

–––––

Block Index Change: Block Signer Will Be Stored in the Block Index from Genesis

The very first problem I encountered running the Internal Testnet this week was related to reading block headers. This problem was very nuanced and took almost three days to debug and fix. During this process I realized I needed to change how block indices are stored. To describe this change, I first need to explain some key differences between PoW, PoS, and qPoS blocks, and why they differ.

In PoW blocks, the block signer is identified by the coinbase transaction, which contains the signer’s public key as a destination for the block reward. This public key is used to verify the block signature. In PoS blocks, the block signer is identified by the coinstake transaction, that similarly contains the signer’s public key as both a destination and a source of stake.

To optimize blockchain storage, qPoS blocks differ from PoW and PoS blocks in that qPoS blocks do not contain the signer’s public key. Instead, the signer’s public key is held in the registry and indexed by a unique staker ID. To verify a qPoS block, clients use the staker ID stored in the block header to look up the staker’s public key in the registry (a very fast operation), then verify the block signature with this public key. For each block, this is a savings of at least 29 bytes (33 byte public key minus the 4 byte staker ID). Compared to PoS blocks, this qPoS optimization saves even more storage, adding to over 463 MB of blockchain storage per year (assuming 5 second block times).

Of course PoW and PoS block headers have no field for storing a staker ID, which means that upon switching to qPoS, the staker ID needs to be added to the block header. This change increases the block header by 4 bytes (while reducing the transaction storage by 77 bytes). The logic of this transition is simple enough, and can be written in pseudocode as “if qPoS has started, write the 4 byte staker ID to the header”. Indeed the actual storage of the staker ID poses no problem whatsoever.

However, the problem comes with reading a block index. Block indices, derived from block information, are stored with the following structure:

• Block header
• Transactions

The hangup comes when reading a transaction. To access a transaction, the very low-level database API known as LevelDB starts reading right after the block header. To know where to start reading transactions, LevelDB must know the length of the header.

Here is the catch, if block headers vary according to block height (where blocks coming after the qPoS transition are 4 bytes longer than blocks coming before the transition) then LevelDB won’t know where the transactions start without reading the block to determine its height.

This problem has a few solutions:

1. Store the header length independently of the rest of the block, keyed by the block hash.
2. Read the block to determine its version, then calculate where to start reading the transactions.
3. Keep all headers the same length, storing the staker ID in a special transaction.
4. Keep all headers the same length, storing the staker ID beginning at block 0.

Each solution has tradeoffs. For example, the first two solutions mean that the disk storage must be accessed twice to determine where to start reading the transactions. The first solution has an additional disadvantage that not only must the 1 byte length information be stored but also the 32 byte block hash must be stored as a key, adding 33 bytes of index storage per block.

For these two solutions, the big killer, in my opinion, is not additional storage, but additional disk I/O. In the first solution, the additional I/O is to read the header length, and in the second solution the additional I/O is to read the entire block header to read the block version to deduce the header length.

The third solution requires no additional disk I/O operation and requires no significant effort from users, services, or exchanges. The one downside is that it needs 48 additional bytes per block, or an additional 289 MB of blockchain storage per year.

The fourth solution is most efficient, requiring no additional disk I/O. It requires a single one-time addition of about 11 MB to the disk index (but not the actual blockchain). This requirement is basically negligible. The huge drawback of the fourth solution is that it requires all nodes to rewrite history (i.e. re-index their chains upon upgrading to the qPoS client). The problem here is that the Stealth chain is over 2.5 million blocks long, and so re-indexing can take between one and three days, depending on one’s hardware. While I feel like this is significant burden for users, block producers, third parties, and especially exchanges, it makes the most sense in the long-run. It does mean that each exchange may be down between 4 and 6 days (because they usually run on VPS and not bare metal servers), but this is a one-time requirement that will greatly improve the storage efficiency of qPoS.

Re-indexing does have another upside in that it will clean up the network significantly. As everyone is well aware, a blockchain is not a single data structure sitting on a single computer. A blockchain exists in numerous copies distributed across the world. And although the best blockchain can be determined by the consensus mechanism (PoW, PoS, qPoS, etc.), each separate copy has storage variation in the form of orphans and ordering. These variations can cause issues, such as when synchronizing a chain from the network. When nodes are required to reindex, they can be encouraged to use a specific bootstrap, which itself can be cleaned of orphans and ordered through a process called linearization.

In the end, even though re-indexing will pose a significant burden for users, third party service providers, and exchanges, and may cause service disruptions, it will ultimately be good for the Stealth network and enable the optimal approach to indexing block signers in qPoS.

–––––

Internal Testnet Week 1 Milestones

This week I was able to achieve the following milestones on the internal testnet:

1. Full bootstrap (synchronization) to the mainnet blockchain.
2. Genesis on the testnet blockchain.
3. Mining on the testnet blockchain.
4. Staking on the testnet blockchain.
5. Sending/receiving on the testnet blockchain.

Future goals required for successful completion of the Internal Testnet include, but may not be limited to:

• Register stakers.
• Produce blocks by qPoS, keeping target times.
• Send and receive under qPoS block production.
• Use delegation.
• Claim balances.
• Test enabling/disabling of stakers.
• Attempt to fork the network.
• Attempt timestamp attacks.
• Transfer staker ownership.
• Test other constraints, like disallowed names, re-registering an alias, etc.

One question that may arise is: why not jump straight to testing qPoS? Why “waste” a week testing what should already work (chain sync, PoW, PoS, sending)? The simple answer is that one must test what should already work to ensure it did not break. An additional, and perhaps less obvious answer is that testnet must recapitulate mainnet. In other words, a testnet must be able to do all those things a mainnet has done or will do, in the order the mainnet did or will do them. Operationally, the most common need for recapitulation is that simply that new clients must replay the blockchain to validate it, and (in the case of qPoS) to know the state of the registry.

–––––

Hondo

https://www.stealth.org