This implements the agenda for voting on the sequence lock fixes as
defined in DCP0004 along with consensus tests and mempool acceptance
tests to ensure its correctness.
It also modifies the mempool to conditionally treat all transactions
with enabled sequence locks as non standard until the vote passes at
which point the will become standard with the modified semantics
enforced.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 6
- New block version on testnet is version 7
- Introduce a convenience function for determining if the vote passed
and is now active
- Enforce modified sequence lock semantics in accordance with the state
of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0004 in accordance with the state of the vote
- Make all transactions with enabled sequence locks non standard until
the agenda vote passes
- Add tests to ensure acceptance and relay behave according to the
aforementioned description
- Add tests for determining if the agenda is active for both mainnet and
testnet
- Add tests to ensure the corrected sequence locks are handled properly
depending on the result of the vote
This adds a new definition for the upcoming agenda vote to enable the
corrected sequence locks behavior as defined by DCP0004. It does not
include any code to make decisions or bump the versions. It is only the
definition.
Also, bump the chaincfg module to v1.3.0 so the new definitions are
available to consumers.
This adds infrastructure to the mempool test harness to allow the faked
chain to handle sequence lock calculation for use in upcoming sequence
lock tests.
In particular, it modifies the existing CalcSequenceLock function on the
faked chain to perform the full sequence lock calculation versus simply
returning a lock that is always valid.
Since by-time sequence lock calculation in CalcSequenceLock depends on
the ability to calculate the median time for the block before the block
that contains the input being spent and the test harness only provides
an incomplete faked chain, this introduces a map which tracks the median
times per utxo and provides two new functions named FakeUxtoMedianTime
and AddFakeUtxoMedianTime which the tests can make use of to properly
fake the necessary data to perform the calculations.
Finally, since sequence locks require v2 transactions, the pool harness
policy now allows a maximum transaction version of 2.
This modifies CreateSignedTx in the mempool test harness to accept munge
functions similar to how NextBlock in chaingen works and updates the
callers accordingly. This allows the test code to more easily mutate
the transactions as desired prior to them being signed which provides
greater flexibility.
While here, it also sets the input amount to the proper value in the
generated transaction. While there is not currently anything in the
tests that relies on that value being accurate, it is more desirable to
create transactions in the test code which are fully valid in case they
end up being used where it does matter in future tests.
This adds tests to ensure the incorrect sequence lock behavior of the
current consensus rules is preserved.
In addition, in order to allow similar style tests in the future, this
adds a function named quickVoteActivationParams which provides a unique
set of chain parameters which allow the tests to more quickly activate
an agenda while still creating a fully valid test chain that consists of
full blocks.
The changes to reverse the utxoset semantics also had the side effect of
correcting the behavior for sequence locks when inside of a block to the
expected behavior instead of the actual current consensus behavior.
In order to avoid that, this modifies the blockchain consensus rules to
properly preserve the old incorrect behavior of sequence locks within
blocks and also updates the mempool to to reject transactions
accordingly thus providing parity between the two.
The behavior needs to be corrected, but since it constitutes a change to
the consensus rules, a vote will be necessary.
This adds a checkpoint to mainnet at height 295940 and testnet at height
83520.
Also, bump the chaincfg module to v1.2.1 so the new checkpoint are
available to consumers.
This bumps the various module versions as follows:
- github.com/decred/dcrd/dcrec/secp256k1@v1.0.1
- github.com/decred/dcrd/dcrjson@v1.1.0
- github.com/decred/dcrd/database@v1.0.3
- github.com/decred/dcrd/blockchain/stake@v1.1.0
- github.com/decred/dcrd/mining@v1.1.0
- github.com/decred/dcrd/certgen@v1.0.2
- github.com/decred/dcrd/connmgr@v1.0.2
- github.com/decred/dcrd/mempool@v1.1.0
In addition, it serves as a base for tagging releases of the following
module versions that have previous been bumped since the last release,
but not yet tagged:
- github.com/decred/dcrd/wire@v1.2.0
- github.com/decred/dcrd/chaincfg@v1.2.0
- github.com/decred/dcrd/dcrutil@v1.2.0
- github.com/decred/dcrd/txscript@v1.0.2
- github.com/decred/dcrd/blockchain@v1.1.0
- github.com/decred/dcrd/hdkeychain@v1.1.1
- github.com/decred/dcrd/peer@v1.1.0
- github.com/decred/dcrd/rpcclient@v1.1.0
Finally, it updates all of the dependencies for every module accordingly,
adds a few missing overrides for transitive dependencies, and tidies up
some of the go module sum files.
This adds support for the getblockchaininfo RPC to rpcclient,
adding asynchronous and blocking functions to rpcclient.Client.
The rpcclient module version is bumped to v1.1.0.
This commit performs the necessary modifications to hook the fee
estimation facility added by the previous commit into the node software.
The block manager is modified to notify the estimator of all new blocks,
such that their transactions can be accounted for, and the mempool is
modified to relay transactions entering and leaving it, so that those
transactions can be tracked.
The results of the estimator can be queried by issuing estimatesmartfee
rpc commands to the node.
This adds the first version of the fees package, responsible for
performing fee estimation of network transactions.
The main goal of fee estimation is to allow the usage of dynamic fees
by wallets, contingent on block contention and the desired confirmation
range for a given transaction.
This version was based on bitcoin core fee estimation.
This corrects an ultra rare and nearly impossible to hit edge case where
the case of r == N in a deterministic signature could lead to an invalid
signature being produced versus returning an error as desired.
This bumps the major version of the JSON RPC server to reflect the
breaking change of reordering the reorg and block connected
notifications. The reorg notification now indicates a successful reorg
was completed rather than initiated.
This modifies the forceHeadReorganization function to remove checks that
are now duplicates due to the recently updated conversion of the
reorganization logic.
Also, correct the exported comment while here to explain what the
function does as opposed to referring to an unexported function that
won't show up in the generated documentation.
This modifies the version 5 migration code to update the database
version atomically with setting a reindex tip and resetting the best
chain tip back to the genesis block so the chain can be reindexed.
This is necessary because, unlike the modifications that come before it,
storing the current chain tip and resetting it is not idempotent, so
lack of atomicity in those updates can lead to failure to recover in
unclean shutdown circumstances.
This adds code to migrate the database to the new schema required by the
recent utxo set semantics reversal changes.
It involves removing the existing utxo set, spend journal, ticket
database, various indexes, resetting the best chain tip back to the
genesis block, and finally performing a full reindex. The process can
be interrupted at any point and future invocations will resume from the
point it was interrupted.
In addition, some new infrastructure to remove the ticket database and
support for index versions is added to support the migration.
This modifies the chain reorganization logic to directly perform the
reorg one block at a time with rollback in the case of failure, as
opposed to the existing memory-based two-step approach, so that it is
more optimized for the typical case, better handles large reorgs, gives
the ability to implement better caching strategies, and helps provide a
path to decouple the chain processing and connection code from the
download logic. It also removes the cached stxos from the view since
the aforementioned changes make them no longer necessary.
A side effect of these changes is that it is no longer possible to know
if a reorg will succeed before actually performing it, so the
NTReorganization notification is now sent after a successful reorg. The
notification really should have been sent after the reorg before anyway.
Prior to these changes, chain reorganization used a two-step approach
such that the first step involved checking all of the blocks along the
reorg path in memory and then actually performing the reorg in a second
step if those checks succeeded. While that approach does have some
benefits in terms of avoiding any intermediate mutation to the current
best chain for failed reorgs, and thus not requiring a rollback in that
case, it also has some disadvantages such as not scaling well with large
reorgs, being more difficult to make use of different caching
strategies, and hindering the ability to decouple the connection code
from the download logic.
In a certain sense, the approach this replaces assumed that a reorg
would fail and took measures to detect that condition prior to
performing the reorg, while the new approach assumes the reorg will
succeed and rolls back the changes in the very rare case it doesn't.
This is an acceptable and safe assumption because the proof-of-work
requirements make it exceedingly expensive to create blocks that are
valid enough to trigger a reorg yet ultimately end up failing to
connect, thus miners are heavily disincentivized from creating such
invalid blocks and attackers are also unable to easily create such
blocks either. Even in the case of attack, the only result would be
nodes performing slightly more database updates than the existing
approach.
The following results show the difference between performing the large
reorg full block tests before and after these changes:
before: 4.3GB memory usage, 2m18.629s to complete
after: 2.8GB memory usage, 2m04.056s to complete
As can be seen, the new approach takes much less memory and is also a
bit faster as well.
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
This separates the block and header context checks to make it more
explicit exactly which checks require the availability of the full data
for all ancestors and which checks do not.
Currently, all ancestor data is always available due to the way the
download and block processing logic is implemented, however, ultimately,
the plan is to decouple the chain processing and connection code from the
download logic. In order to help pave the way towards that goal, this
more clearly delineates the checks by redefining the current meaning of
the concept of contextual checks and creating a new concept of
positional checks.
Positional checks are defined as those which depend on the position of a
block or header within the block chain and have all of the ancestor
block headers available, but do NOT have all of the full block data for
all of the ancestor blocks.
Contextual checks are now defined as those which depend on having the
full block data for all of the ancestors available, which also implies
all of the ancestor block headers are available too.
To that end, this introduces two new functions named
checkBlockHeaderPositional and checkBlockPositional, moves all of the
checks which adhere to the aforementioned semantics to the new
functions, and updates all call sites which invoke the contextual
variants to call the positional variants first.
This modifies the code which enforces a maximum number of peers to
always allow whitelisted inbound peers even when they would exceed the
maximum number of peers to ensure that operators can always allow their
own clients.
When an accepted block is attached to the main chain, potentially
three notifications are emitted: NTBlockConnected,
NTSpentAndMissedTickets, and NTNewTickets. This change moves
NTBlockConnected to being the first notification, followed by
NTSpentAndMissedTickets and NTNewTickets.
This is important for applications which react to the stake
notifications since they will have already been notified of the most
recently connected block. For example, dcrwallet uses the
spentandmissedtickets JSON-RPC notification to create revocation
transactions for missed votes. However, if the missed tickets are
notified before the connected block, a race is possible where the
revocations are rejected as they double spend a vote on what wallet
believes is still the main chain tip block.
This is considered a patch bump to the JSON-RPC API semantic version
due to affecting and improving the ordering of the blockconnected and
spentandmissedtickets notifications.
This reworks the stake node handling logic a bit to clean it up,
slightly optimize it, and to make it easier to decouple the chain
processing and connection code from the download logic in the future.
To accomplish this, the fetchStakeNode dependence on the
getReorganizeNodes function is removed in favor of directly iterating
the block nodes in the function as needed. Not only is this more
efficient, it also allows the function to return stake nodes for
branches regardless of their validation status. Currently, this is
irrelevant due to the connection and download logic being tightly
coupled. However, it will be necessary in the future when those are
separated.
Also, the flushing and pruning logic is modified to no longer rely on
the download logic being tightly coupled to the the connection logic.
In particular, a new function named flushBlockIndex is added as a
wrapper to the index flush function which populates stake information in
a node before flushing it to the database as needed, and all flushing
invocations use the wrapper instead.
Finally, the stakeUndoData field is removed since it is only used to
avoid some database loads on reorgs that are large enough to exceed the
pruning depth, which never happens in practice anyway, so there is no
point in using up extra memory for it.
This updates the code which deals with marking block nodes modified when
setting and clearing status flags to only mark them modified if the
status flags actually change.
This is more efficient as it prevents the need to write nodes that
haven't actually changed to the database.
This simplifies the CheckConnectBlockTemplate by making use of the fact
that a template may only build off the current tip or the parent of the
current tip in order to remove the reorganization detach loop and all
code to attach nodes.
This can be done because there can never be any blocks that need to be
attached given the aforementioned constraints, and the only two
possibilities for the blocks that need to be detached are none in the
case the the template is building on the current tip, or the current tip
itself in the case the template is building on its parent.
In other words, there will always be 0 nodes to attach and either 0 or 1
node(s) to detach. Consequently, it is not necessary to include any
attach code nor have a detach loop.
This modifies the getblock and getblockheader RPC results to include the
total number of hashes expected to produce the chain up to the requested
block in hex and updates the JSON-RPC API documentation accordingly.
This modifies the getblock and getblockheader RPC results to be a little
more consistent in their ordering, to include the extra data field in
the getblockheader results, and to update the JSON-RPC API documentation
for both to match reality. The documentation was missing several fields
and had some typos.
This removes the CheckWorklessBlockSanity function since nothing uses
it anymore. It was used for testing templates and unit tests in the
past, but the former is now handled by CheckConnectBlockTemplate and the
latter is handled by the full block tests.
It should be noted that this is a breaking change to the API and thus
will need a major version bump of the blockchain module to be published
before the next release.
This cleans up code recently added to support the --altdnsnames flag and
DCRD_ALT_DNSNAMES environment variable to match the project standards,
simplify it, and correct some issues with standalone test binaries.
- Gets rid of the setup func which had several issues:
- It was creating a file that was not needed (and without even
checking the error)
- Since it was parsing flags and missing with os.Args in a func, it
was not possible to individual run the test funcs without failure
due to the additional flags
- Performs the flag parsing and removal in an init func that only runs
once when the tests are run (and before TestMain)
- Avoids creating empty objects when nil will suffice
- Uses full sentences and periods in the comment and properly spaces
them out so they are consistent with the rest of the code
- Adds comments to all of the added test functions as required by the
code contribution guidelines
- Closes the create temp files before trying to write to them in the cert
generation
- Defers the removal of the temp files directly after they are created so
they are removed properly if the next one fails to create
- Uses consistent camel case in the variable names
This renames the unexported and exported versions
of ThresholdState to NextThresholdState as well as
related test helpers.
NextThresholdState better describes the function
because the threshold state being returned is for the
block after the provided block hash.
- rpcserver: Adds ability to allow alternative dns names for TLS.
- Adds AltDNSNames config for passing additional hostnames to the
TLS certificate generate method. Without this change there is no
way to pass in alternative dns names for the rpc server.
- Additionally this commit allows the user to set 'DCRD_ALT_DNSNAMES'
as an environment variable when starting dcrd. This is useful for
docker and other app runtimes that require apps to be 12 factor.
