This updates all code in the main module to use the latest major modules
versions to pull in the latest updates.
A more general high level overview of the changes is provided below,
however, there is one semantic change worth calling out independently.
The verifymessage RPC will now return an error when provided with
an address that is not for the current active network and the RPC server
version has been bumped accordingly.
Previously, it would return false which indicated the signature is
invalid, even when the provided signature was actually valid for the
other network. Said behavior was not really incorrect since the
address, signature, and message combination is in fact invalid for the
current active network, however, that result could be somewhat
misleading since a false result could easily be interpreted to mean the
signature is actually invalid altogether which is distinct from the case
of the address being for a different network. Therefore, it is
preferable to explicitly return an error in the case of an address on
the wrong network to cleanly separate these cases.
The following is a high level overview of the changes:
- Replace all calls to removed blockchain merkle root, pow, subsidy, and
coinbase funcs with their standalone module equivalents
- Introduce a new local func named calcTxTreeMerkleRoot that accepts
dcrutil.Tx as before and defers to the new standalone func
- Update block locator handling to match the new signature required by
the peer/v2 module
- Introduce a new local func named chainBlockLocatorToHashes which
performs the necessary conversion
- Update all references to old v1 chaincfg params global instances to
use the new v2 functions
- Modify all cases that parse addresses to provide the now required
current network params
- Include address params with the wsClientFilter
- Replace removed v1 chaincfg constants with local constants
- Create subsidy cache during server init and pass it to the relevant
subsystems
- blockManagerConfig
- BlkTmplGenerator
- rpcServer
- VotingWallet
- Update mining code that creates the block one coinbase transaction to
create the output scripts as defined in the v2 params
- Replace old v2 dcrjson constant references with new types module
- Fix various comment typos
- Update fees module to use the latest major module versions and bump it v2
This commit introduces a new major version of the dcrjson module which
removes all dcrd RPC type support, instead focusing only on method and
type registration. The dcrd methods and types are moved to the
github.com/decred/dcrd/rpc/jsonrpc/types module.
In order to improve backwards compatibility with dcrjson/v2, the API
has been modified to register methods as interface{} instead of
string. This allows different method string types to be used to key
parameter types during registration and lookup, and will allow
dcrjson/v2 to forward registrations of RPC methods to v3 without
causing duplicate registrations errors for incompatible types.
With the introduction of the new types package, the RPC API has been
modified to replace concatenated hash blobs to JSON string arrays of
hash strings. The RPC API major version is bumped to reflect this
change.
A future update to dcrjson/v2 will add additional registrations,
forwarding the registrations to v3 and replacing command types with
type aliases where possible. Unfortunately, this can not be done
entirely in a single commit due to dcrjson/v2 and dcrjson/v3 sharing
the same directory in the source tree, and a branch will need to be
used for this update.
Module replacements are temporarily used to enable the changes for the
main module, including dcrctl. After the aforementioned update to
dcrjson/v2 and a forthcoming update to dcrwallet's RPC types package,
these replacements will be removed.
BgBlkTmplGenerator represents the background process that
generates block templates and notifies all subscribed clients
on template regeneration. It generates new templates based
on mempool activity for vote and non-vote transactions and
the time elapsed since last template regeneration.
This also adds a template pool to the background block
generator for recreating submitted blocks.
Add DifficultyRatio to GetBlockChainInfoResult, and bump the minor rpc
version accordingly. This is to maintain consistency with GetDifficulty
RPC. Also deprecate the existing Difficuly field in
GetBlockChainInfoResult, to be removed in a future major version bump.
This updates the main module to use version 2 of the mempool module.
The following is a summary of changes:
- Update all imports to use mempool/v2
- Update module requirements to include new module and remove no longer
needed old versions
- Add an override for the v2 module so CI is always builds with the
latest code
- Update rpcserver getrawmempool to make use of new mempool
VerboseTxDescs instead of the removed RawMempoolVerbose
- Update rpcserver getrawmempool to properly return the parameter
provided for the transaction type instead of the type of variable in
the case an invalid type is provided
- Update rpcserver getrawmempool to return the supported transaction
types in the error when an invalid type is provided
This commit performs the necessary modifications to hook the fee
estimation facility added by the previous commit into the node software.
The block manager is modified to notify the estimator of all new blocks,
such that their transactions can be accounted for, and the mempool is
modified to relay transactions entering and leaving it, so that those
transactions can be tracked.
The results of the estimator can be queried by issuing estimatesmartfee
rpc commands to the node.
This bumps the major version of the JSON RPC server to reflect the
breaking change of reordering the reorg and block connected
notifications. The reorg notification now indicates a successful reorg
was completed rather than initiated.
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
When an accepted block is attached to the main chain, potentially
three notifications are emitted: NTBlockConnected,
NTSpentAndMissedTickets, and NTNewTickets. This change moves
NTBlockConnected to being the first notification, followed by
NTSpentAndMissedTickets and NTNewTickets.
This is important for applications which react to the stake
notifications since they will have already been notified of the most
recently connected block. For example, dcrwallet uses the
spentandmissedtickets JSON-RPC notification to create revocation
transactions for missed votes. However, if the missed tickets are
notified before the connected block, a race is possible where the
revocations are rejected as they double spend a vote on what wallet
believes is still the main chain tip block.
This is considered a patch bump to the JSON-RPC API semantic version
due to affecting and improving the ordering of the blockconnected and
spentandmissedtickets notifications.
This modifies the getblock and getblockheader RPC results to include the
total number of hashes expected to produce the chain up to the requested
block in hex and updates the JSON-RPC API documentation accordingly.
This modifies the getblock and getblockheader RPC results to be a little
more consistent in their ordering, to include the extra data field in
the getblockheader results, and to update the JSON-RPC API documentation
for both to match reality. The documentation was missing several fields
and had some typos.
This renames the unexported and exported versions
of ThresholdState to NextThresholdState as well as
related test helpers.
NextThresholdState better describes the function
because the threshold state being returned is for the
block after the provided block hash.
- rpcserver: Adds ability to allow alternative dns names for TLS.
- Adds AltDNSNames config for passing additional hostnames to the
TLS certificate generate method. Without this change there is no
way to pass in alternative dns names for the rpc server.
- Additionally this commit allows the user to set 'DCRD_ALT_DNSNAMES'
as an environment variable when starting dcrd. This is useful for
docker and other app runtimes that require apps to be 12 factor.
This modifies the ChainTips function in blockchain to return the results
using a type defined in the blockchain package itself instead of
directly returning a dcrjson type. This is preferable since nothing
else in the module depends on dcrjson and therefore allows the
dependency on the dcrjson module to be broken.
It also updates the RPC server that makes use of the function to perform
the necessary conversions.
Finally, the associated module hierarchy documentation is updated to
properly remove the no longer required dependency.
It should be noted that this is a breaking change to the API and thus
will need a v2 major version bump of the blockchain module to be
published before the changes can be externally consumed.
This removes the handler for the createrawssgen RPC and deprecates the
associated dcrjson structures.
This is being done because it doesn't make any sense for votes to be
created over the RPC interface. In order for voting on the previous
block to have any semblance of relevance, the software producing the
vote clearly must have code to deserialize and understand the relevant
data structures.
this updates the function signatures of minimumMedianTime and
medianAdjustedTime and removes all chainState usage in
rpcserver.go in favour of using BestState.
Port of upstream commit 2274d36 (subset). In preparation of the
removal of chainState.
This modifies the results provided by the getrawtransaction and
searchrawtransactions RPCs to make them consistent with each other.
In particular, it adds the transaction expiry, block height, and block
index fields to the searchrawtransactions result and ensures that both
omit the block height when it is zero so that it is not included in the
results for mempool transactions.
Since populating the block index requires additional details to be
stored to the transaction index to efficiently provide, only the
infrastructure is added and the value is left as the default for now,
which means it will be omitted and therefore not change the visible
behavior.
Finally, it also updates the RPC server help to provide consistent
descriptions for the fields and also updates the JSON-RPC documentation.
This introduces a new internal package named version to house the
version information and updates the various code to use it. This allows
both dcrd and dcrctl to make use of the package so the version
information only needs to be bumped in one place and also ensure that
any link-time overrides to the pre-release and/or build metadata apply
to both.
Also, while here, include the OS and architecture in the version log at
start of day as well as the output from the version CLI flag.
This modifies the getcfilter and getcfilterheader RPC handlers to return
an error when an unknown block is provided as opposed an empty filter in
the case of the former and an all-zero hash in the case of the latter.
It also modifies the error returned when an unknown filter type is
specified to indicate an invalid parameter as opposed to a misc error so
it is more consistent with the other handlers.
Finally, it removes the debug logs in the error cases since the errors
are now returned to the caller.
This refactors the code that relies on the database main chain index to
use the in-memory block index now that the full index is in memory.
This is a major optimization for several functions because they no
longer have to first consult the database (which is incredibly slow in
many cases due to the way leveldb splits all of the information across
files) to perform lookups and determine if blocks are in the main chain.
It should also be noted that even though the main chain index is no
longer used as of this commit, the code which writes the main chain
index entries to the database in connectBlock, disconnectBlock, and
createChainState have not been removed because, once that is done, it
will no longer be possible to downgrade and thus those changes must be
performed along with a database migration and associated version bump.
An overview of the changes are as follows:
- Update all code which previously used the db to use the main chain by
height map instead
- Update several internal functions which previously accepted the
hash to instead accept the block node as the parameter
- Rename fetchMainChainBlockByHash to fetchMainChainBlockByNode
- Rename fetchBlockByHash to fetchBlockByNode
- Rename dbFetchBlockByHash to dbFetchBlockByNode
- Update all callers to use the new names and semantics
- Optimize HeaderByHeight to use block index/height map instead of db
- Move to chain.go since it no longer involves database I/O
- Optimize BlockByHash to use block index instead of db
- Move to chain.go since it no longer involves database I/O
- Optimize BlockByHeight to use block index/height map instead of db
- Move to chain.go since it no longer involves database I/O
- Optimize MainChainHasBlock to use block index instead of db
- Move to chain.go since it no longer involves database I/O
- Removed error return since it can no longer fail
- Optimize BlockHeightByHash to use block index instead of db
- Move to chain.go since it no longer involves database I/O
- Optimize BlockHashByHeight to use block index instead of db
- Move to chain.go since it no longer involves database I/O
- Optimize HeightRange to use block index instead of db
- Move to chain.go since it no longer involves database I/O
- Remove several unused functions related to the main chain index
- dbFetchHeaderByHash
- DBFetchHeaderByHeight
- dbFetchBlockByHeight
- DBFetchBlockByHeight
- dbMainChainHasBlock
- DBMainChainHasBlock
- Rework IsCheckpointCandidate to use block index
- Modify the upgrade code to expose the old dbFetchBlockByHeight
function only in the local scope since upgrades require the ability to
read the old format
- Update indexers to fetch the blocks themselves while only querying
chain for chain-related details
This modifies the BlockByHash function to return blocks regardless of
whether or not the block is on a side chain and removes the
FetchBlockByHash function since this change makes it redundant.
This is being changed to simplify the API and because there are no
current cases where calls into chain to retrieve a block need to be
limited to the main chain and if the need should ever arise, the caller
can simply make use of the MainChainHasBlock function to verify.
This adds an amount field to TransactionInput intended for specifying the prev out amount when creating a transaction. createraw/ssrtx/sgentx/transaction rpcs are also updated accordingly as well as associated tests.
A typo in assignField is also corrected.
This optimizes the FetchHeader function to make use of the fact that all
block nodes are now in memory and therefore it is no longer necessary to
consult the database for them.
It also renames the function to HeaderByHash so it is consistent with
other similar functions such as HeaderByHeight, BlockByHash, and
BlockByHeight and updates all callers in the repo accordingly.
This speeds up fetching all of the headers in the chain by roughly 3x to
4x for a chain height of 246,000 headers. Longer chains will benefit
more.
For example, here is some timing information before and after this
commit plus the full block index in memory commit for 246,000 headers:
7200 RPM HDD:
-------------
Previous fetch time: ~15.76s
New fetch time: ~4.17s
SSD:
----
Previous fetch time: ~12.63s
New fetch time: ~4.17s
This refactors the code that locates blocks (inventory discovery) out of
server and into blockchain where it can make use of the fact that all
block nodes are now in memory and more easily be tested. As an aside,
it really belongs in blockchain anyways since it's purely dealing with
the block index and best chain.
In order to do this reasonably efficiently, a new memory-only height
to block node mapping for the main chain is introduced to allow
efficient forward traversal of the main chain. This is ultimately
intended to be replaced by a chain view.
Since the network will be moving to header-based semantics, this also
provides an additional optimization to allow headers to be located
directly versus needing to first discover the hashes and then fetch the
headers.
The new functions are named LocateBlocks and LocateHeaders. The former
returns a slice of located hashes and the latter returns a slice of
located headers.
Finally, it also updates the RPC server getheaders call and related
plumbing to use the new LocateHeaders function.
A comprehensive suite of tests is provided to ensure both functions
behave correctly for both correct and incorrect block locators.
This corrects the handler for the rebroadcastwinners RPC to return the
hash of the block the tickets are voting on when there are multiple
candidates at the current tip as opposed to all claiming to vote on the
current best block.
This removes the CheckConnectBlock function in favor of a new function
named CheckConnectBlockTemplate which more accurately reflects its
purpose of testing block template proposals and uses this fact to be
more restrictive about the allowed inputs and to avoid performing the
proof of work check.
In particular, the provided block template must only build from the
current tip or its parent or an error is returned.
All mining code has been updated to call the new function accordingly.
Finally, it also adds a full suite of tests for the new function using
the chaingen framework in order to ensure proper functionality.
