BgBlkTmplGenerator represents the background process that
generates block templates and notifies all subscribed clients
on template regeneration. It generates new templates based
on mempool activity for vote and non-vote transactions and
the time elapsed since last template regeneration.
This also adds a template pool to the background block
generator for recreating submitted blocks.
Instead of waiting for a second inv to process what the first inv
may have requested, just send it all at once. This allows dropping
of a request queue for each server peer.
This changes the behavior of peer disconnection logic, such that
permanent peers are actually removed from the connection manager when
requested, instead of just being disconnected and therefore triggering a
future retry.
Previously, this prevented the addnode [addr] remove from correctly
removing permanent peers. A connection retry attempt would always be
performed.
This updates the main module to use version 2 of the mempool module.
The following is a summary of changes:
- Update all imports to use mempool/v2
- Update module requirements to include new module and remove no longer
needed old versions
- Add an override for the v2 module so CI is always builds with the
latest code
- Update rpcserver getrawmempool to make use of new mempool
VerboseTxDescs instead of the removed RawMempoolVerbose
- Update rpcserver getrawmempool to properly return the parameter
provided for the transaction type instead of the type of variable in
the case an invalid type is provided
- Update rpcserver getrawmempool to return the supported transaction
types in the error when an invalid type is provided
This implements the agenda for voting on the sequence lock fixes as
defined in DCP0004 along with consensus tests and mempool acceptance
tests to ensure its correctness.
It also modifies the mempool to conditionally treat all transactions
with enabled sequence locks as non standard until the vote passes at
which point the will become standard with the modified semantics
enforced.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 6
- New block version on testnet is version 7
- Introduce a convenience function for determining if the vote passed
and is now active
- Enforce modified sequence lock semantics in accordance with the state
of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0004 in accordance with the state of the vote
- Make all transactions with enabled sequence locks non standard until
the agenda vote passes
- Add tests to ensure acceptance and relay behave according to the
aforementioned description
- Add tests for determining if the agenda is active for both mainnet and
testnet
- Add tests to ensure the corrected sequence locks are handled properly
depending on the result of the vote
This commit performs the necessary modifications to hook the fee
estimation facility added by the previous commit into the node software.
The block manager is modified to notify the estimator of all new blocks,
such that their transactions can be accounted for, and the mempool is
modified to relay transactions entering and leaving it, so that those
transactions can be tracked.
The results of the estimator can be queried by issuing estimatesmartfee
rpc commands to the node.
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
This modifies the code which enforces a maximum number of peers to
always allow whitelisted inbound peers even when they would exceed the
maximum number of peers to ensure that operators can always allow their
own clients.
This modifies the infrastructure for the server inventory relay to allow
the caller to specify the inventory should be announced immediately
versus using the typical trickle mechanism.
It also updates all callers in the repository and changes the relay of
accepted blocks use the new ability.
This introduces a new internal package named version to house the
version information and updates the various code to use it. This allows
both dcrd and dcrctl to make use of the package so the version
information only needs to be bumped in one place and also ensure that
any link-time overrides to the pre-release and/or build metadata apply
to both.
Also, while here, include the OS and architecture in the version log at
start of day as well as the output from the version CLI flag.
This adds a new parameter named UserAgentComments to the peer Config
struct so that callers may specify additional comments to add to the
user agent string.
It also updates the server to include the prelease portion of the
semantic version as a comment when creating peers.
This modifies the BlockByHash function to return blocks regardless of
whether or not the block is on a side chain and removes the
FetchBlockByHash function since this change makes it redundant.
This is being changed to simplify the API and because there are no
current cases where calls into chain to retrieve a block need to be
limited to the main chain and if the need should ever arise, the caller
can simply make use of the MainChainHasBlock function to verify.
This reverts a change made in 4f6ed9afdb
which resulted in a headers response to not be created for a
getheaders request if there were no more discovered headers. This was
a deviation from the previous behavior of queueing a headers message
with an empty vector of headers.
This was a protocol change without a protocol bump, so the behavior
must be reverted.
I also cross referenced the Bitcoin Core source code and, just like
the now-restored behavior, headers messages were always sent even when
no new headers were discovered. I don't see any reason why Decred
should deviate from this behavior.
This modifies the getcfheaders handling code to use the recently
refactored block locator code that is now available in blockchain
instead of manually performing the task.
Not only is the code in chain more efficient, it also helps ensure the
handling for block locators is consistent.
While here, it also modifies a few cfilter-related comments for
consistency and corrects a near bug which was limiting the response to
the maximum allowed block headers per message as opposed to maximum
allowed cfilter headers per message. Since the two constants are
currently the same value, this is no functional change in this regard.
This moves the function definition for enforceNodeCFFlag so it are more
consistent with the preferred order used throughout the codebase. In
particular, the functions are defined before they're first used and
generally as close as possible to the first use when they're defined in
the same file.
While moving, it also reformats the code to match the same spacing as
the other CF functions for consistency.
There are no functional changes.
This modifies the OnVersion handler for server peers to use a local
variable for the remote address of the peer in order to avoid grabbing
the mutex multiple times.
There are no functional changes.
This changes the server peers OnVersion handler to only advertise the
server as a viable target for inbound connections when the server
believes it is close the best known tip.
This modifies the OnVersion handler for server peers to use a local
variable for the inbound status of the peer in order to avoid grabbing
the mutex multiple times.
While here, it also does some light cleanup. There are no functional
changes.
This adds code to update the address manager services for a known
address to the services advertised by peers when they are connected to
via an outbound connection. It is only done for outbound connections to
help prevent malicious behavior from inbound connections.
This modifies the OnVersion callback to allow a reject message to be
returned in which case the message will be sent to the peer and the peer
will be disconnected.
This modifies the negotiation logic to ensure the callback has the
opportunity to see the message before the peer is disconnected and
improves the error handling when reading the remote version message.
It also has the side effect of ensuring the protocol version is
negotiated before sending reject messages with the exception of the
first message not being a version message since negotiation is not
possible in that case.
This is being changed because it is useful for the server to see the
message regardless in order to have the opportunity to things such as
update the address manager and reject peers that don't have desired
services.
This modifies the OnGetAddr handler to prevent the server from
responding to getaddr messages more than once per connection which helps
reduce unhelpful traffic and fingerprinting attacks.
This refactors the code that locates blocks (inventory discovery) out of
server and into blockchain where it can make use of the fact that all
block nodes are now in memory and more easily be tested. As an aside,
it really belongs in blockchain anyways since it's purely dealing with
the block index and best chain.
In order to do this reasonably efficiently, a new memory-only height
to block node mapping for the main chain is introduced to allow
efficient forward traversal of the main chain. This is ultimately
intended to be replaced by a chain view.
Since the network will be moving to header-based semantics, this also
provides an additional optimization to allow headers to be located
directly versus needing to first discover the hashes and then fetch the
headers.
The new functions are named LocateBlocks and LocateHeaders. The former
returns a slice of located hashes and the latter returns a slice of
located headers.
Finally, it also updates the RPC server getheaders call and related
plumbing to use the new LocateHeaders function.
A comprehensive suite of tests is provided to ensure both functions
behave correctly for both correct and incorrect block locators.
This change begins the work of bringing committed filters to the
network consensus daemon. Committed filters are designed to enable
light wallets without many of the privacy issues associated with
server-side bloom filtering.
The new gcs package provides the primitives for creating and matching
against Golomb-coded sets (GCS) filters while the blockcf package
provides creation of filters and filter entries for data structures
found in blocks.
The wire package has been updated to define a new protocol version and
service flag for advertising CF support and includes types for the
following new messages: cfheaders, cfilter, cftypes, getcfheaders,
getcfilter, getcftypes. The peer package and server implementation
have been updated to include support for the new protocol version and
messages.
Filters are created using a collision probability of 2^-20 and are
saved to a new optional database index when running with committed
filter support enabled (the default). At first startup, if support is
not disabled, the index will be created and populated with filters and
filter headers for all preexisting blocks, and new filters will be
recorded for processed blocks.
Multiple filter types are supported. The regular filter commits to
output scripts and previous outpoints that any non-voting wallet will
require access to. Scripts and previous outpoints that can only be
spent by votes and revocations are not committed to the filter. The
extended filter is a supplementary filter which commits to all
transaction hashes and script data pushes from the input scripts of
non-coinbase regular and ticket purchase transactions. Creating these
filters is based on the algorithm defined by BIP0158 but is modified
to only commit "regular" data in stake transactions to prevent
committed filters from being used to create SPV voting wallets.
This finishes separating the mining code from the mempool that was
partially done in commit 9031d85 by extending the TxSource interface to
include the new functionality required by Decred.
The following is an overview of the changes to accomplish this:
- Move the VoteTx struct out of the mempool package in the mining
package and rename it to VoteDesc instead to signify it is a vote
descriptor and be consistent with the naming of TxDesc
- Update mempool to use the new mining.VoteDesc struct
- Rename CheckIfTxsExist to HaveAllTransactions to be more consistent
with HaveTransaction in the TxSource interface as it now lives
alongside it
- Add VoteHashesForBlock, VotesForBlocks, and IsTxTreeKnownInvalid
to the TxSource interface
- Switch the mining code to call the functions on the TxSource interface
instead and remove the reference to the concrete mempool
This replaces the GetGeneration function which allowed getting the
entire generation (all children stemming from the same parent) of an
arbitrary bock with TipGeneration which only returns the same
information for the tip block.
This is being done because the function is only used for mining purposes
to get the generation of the current tip. The code is simplified by
reducing its scope to its actual purpose as an initial benefit. It also
provides much better optimization opportunities later.
This propagates the interrupt channel through to blockchain and the
indexers so that it is possible to interrupt long-running operations
such as catching up indexes.
This renames the mempool.Config.RelayNonStd option to AcceptNonStd which
more accurately describes its behavior since the mempool was refactored
into a separate package.
The reasoning for this change is that the mempool is not responsible for
relaying transactions (nor should it be). Its job is to maintain a pool
of unmined transactions that are validated according to consensus and
policy configuration options which are then used to provide a source of
transactions that need to be mined.
Instead, it is the server that is responsible for relaying transactions.
While it is true that the current server code currently only relays txns
that were accepted to the mempool, this does not necessarily have to
be the case. It would be entirely possible (and perhaps even a good
idea as something do in the future), to separate the relay policy from
the mempool acceptance policy (and thus indirectly the mining policy).
This makes the semantics of block fetching much more explicit by
introducing a new function named fetchMainChainBlockByHash which only
attempts to the load a block from the main chain block cache and then
falls back to the database. While currently only blocks in the main
chain are in the database, the intention is for future commits to
allow the database to house side chain blocks as well, so having clearly
defined semantics will help make that transition easier to verify.
While here, it also renames {f,F}etchBlockFromHash to {f,F}etchBlockByHash
to be consistent with the naming used by the other existing functions
and updates the comments to better describe the function semantics.
