This removes the getblocktemplate and its helpers from the codebase.
Ongoing mining updates focused on the voting/block validation process
with respect to generating block templates for getwork makes it the
better option for decred. Also getblocktemplate rpc was buggy and
has been disabled for a while.
Some lint related issues have been addressed as well.
During 32-bit nonce iteration, if a block solution wasn't found, the
iterator variable would overflow back to 0, creating an infinite loop,
thus continuing the puzzle search without ever updating the extra
nonce field. This bug has never been triggered in practice because
the code in question has only ever been used with difficulties where
a solution exists within the regular nonce space.
The extra nonce iteration logic itself was also imperfect in that it
wouldn't test a value of exactly 2^64 - 1.
The behavior we actually want is to loop through the entire unsigned
integer space for both the regular and extra nonces, and for this
process to continue forever until a solution is found. Note that
periodic updates to the block header timestamp during iteration ensure
that unique hashes are generated for subsequent generations of the
same nonce values.
This updates all code in the main module to use the latest major modules
versions to pull in the latest updates.
A more general high level overview of the changes is provided below,
however, there is one semantic change worth calling out independently.
The verifymessage RPC will now return an error when provided with
an address that is not for the current active network and the RPC server
version has been bumped accordingly.
Previously, it would return false which indicated the signature is
invalid, even when the provided signature was actually valid for the
other network. Said behavior was not really incorrect since the
address, signature, and message combination is in fact invalid for the
current active network, however, that result could be somewhat
misleading since a false result could easily be interpreted to mean the
signature is actually invalid altogether which is distinct from the case
of the address being for a different network. Therefore, it is
preferable to explicitly return an error in the case of an address on
the wrong network to cleanly separate these cases.
The following is a high level overview of the changes:
- Replace all calls to removed blockchain merkle root, pow, subsidy, and
coinbase funcs with their standalone module equivalents
- Introduce a new local func named calcTxTreeMerkleRoot that accepts
dcrutil.Tx as before and defers to the new standalone func
- Update block locator handling to match the new signature required by
the peer/v2 module
- Introduce a new local func named chainBlockLocatorToHashes which
performs the necessary conversion
- Update all references to old v1 chaincfg params global instances to
use the new v2 functions
- Modify all cases that parse addresses to provide the now required
current network params
- Include address params with the wsClientFilter
- Replace removed v1 chaincfg constants with local constants
- Create subsidy cache during server init and pass it to the relevant
subsystems
- blockManagerConfig
- BlkTmplGenerator
- rpcServer
- VotingWallet
- Update mining code that creates the block one coinbase transaction to
create the output scripts as defined in the v2 params
- Replace old v2 dcrjson constant references with new types module
- Fix various comment typos
- Update fees module to use the latest major module versions and bump it v2
This modifies the CPU miner to make the speed monitor optional by making
use of non-blocking sends to its update channel and then changes
GenerateNBlocks, which is used specifically for mining a discrete number
of blocks for testing purposes, to avoid launching it.
This updates the cpuminer code for Decred to use the extra data space in
the header for the extra nonce space instead of putting it in the
coinbase as is required in the upstream code.
The current code creates block templates with coinbase transactions that
have 4 uint64 extra nonces, but only ever uses one of them. Rather than
wasting the extra space, this modifies the code so it only uses a single
uint64 extra nonce.
It should be noted that realistically there isn't even a real need for
an extra nonce in the coinbase at all because there is extra space in
the header specifically for that purpose and miners can't modify the
coinbase via getwork anyways.
However, it is still necessary to insert for the current code in order
to ensure every block template has a unique merkle root since that is
what is used to uniquely identify the block template.
This modifies all files in the main package so that function definitions
are on the a single line as is the style used in the upstream code as
well as throughout its code base. This helps minimize the differences
between them and the upstream code in order to facilitate easier syncs
due to less merge conflicts as a result of superfluous changes.
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
Upstream commit a7b35d9f9e.
Also, the merge commit contains necessary decred-specific alterations.
It also has the side effect of correcting a couple of min reduction bugs
even though they can't currently ever be hit due to the min reduction
interval flag being disabled for all networks.
Contains the following commits:
- 711f33450c
- b6b1e55d1e
- Reverted because Travis is already at a more recent version
- bd4e64d1d4
Also, the merge commit contains the necessary decred-specific
alterations, converts all other references to sha to hash to keep with
the spirit of the merged commits, and various other cleanup intended to
bring the code bases more in line with one another.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
Upstream commit 2b6a9a56e5.
NOTE: This is only merging in the upstream changes, so while they are
useful for Decred as well, this merge commit does fully separate the
mempool as upstream does due to the new functionality provided by
Decred.
This simply exports and adds some comments to the fields of the
BlockTemplate struct.
This is primarily being done as a step toward being able to separate the
mining code into its own package, but also it makes sense on its own
because code that requests new block template necessarily examines the
returned fields which implies they should be exported.
This creates a skeleton mining package that simply contains a few of the
definitions used by the mining and mempool code.
This is a step towards decoupling the mining code from the internals of
btcd and ultimately will house all of the code related to creating block
templates and CPU mining.
The main reason a skeleton package is being created before the full
blown package is ready is to avoid blocking mempool separation which
relies on these type definitions.
This introduces the concept of a new interface named TxSource which aims
to generically provide a concurrent safe source of transactions to be
considered for inclusion in a new block. This is a step towards
decoupling the mining code from the internals of btcd. Ultimately the
intent is to create a separate mining package.
The new TxSource interface relies on a new struct named miningTxDesc,
which describes each entry in the transaction source. Once this code is
refactored into a separate mining package, the mining prefix can simply
be dropped leaving the type exported as mining.TxDesc.
To go along with this, the existing TxDesc type in the mempool has been
renamed to mempoolTxDesc and changed to embed the new miningTxDesc type.
This allows the mempool to efficiently implement the MiningTxDescs
method needed to satisfy the TxSource interface.
This approach effectively separates the direct reliance of the mining
code on the mempool and its data structures. Even though the memory
pool will still be the default concrete implementation of the interface,
making it an interface offers much more flexibility in terms of testing
and even provides the potential to allow more than one source (perhaps
multiple independent relay networks, for example).
Finally, the memory pool and all of the mining code has been updated to
implement and use the new interface.
This introduces the concept of a mining policy struct which is used to
control block template generation instead of directly accessing the
config struct. This is a step toward decoupling the mining code from
the internals of btcd. Ultimately the intent is to create a separate
mining package.
Introduce an ECDSA signature verification into btcd in order to
mitigate a certain DoS attack and as a performance optimization.
The benefits of SigCache are two fold. Firstly, usage of SigCache
mitigates a DoS attack wherein an attacker causes a victim's client to
hang due to worst-case behavior triggered while processing attacker
crafted invalid transactions. A detailed description of the mitigated
DoS attack can be found here: https://bitslog.wordpress.com/2013/01/23/fixed-bitcoin-vulnerability-explanation-why-the-signature-cache-is-a-dos-protection/
Secondly, usage of the SigCache introduces a signature verification
optimization which speeds up the validation of transactions within a
block, if they've already been seen and verified within the mempool.
The server itself manages the sigCache instance. The blockManager and
txMempool respectively now receive pointers to the created sigCache
instance. All read (sig triplet existence) operations on the sigCache
will not block unless a separate goroutine is adding an entry (writing)
to the sigCache. GetBlockTemplate generation now also utilizes the
sigCache in order to avoid unnecessarily double checking signatures
when generating a template after previously accepting a txn to the
mempool. Consequently, the CPU miner now also employs the same
optimization.
The maximum number of entries for the sigCache has been introduced as a
config parameter in order to allow users to configure the amount of
memory consumed by this new additional caching.
This commit converts all block height references to int32 instead of
int64. The current target block production rate is 10 mins per block
which means it will take roughly 40,800 years to reach the maximum
height an int32 affords. Even if the target rate were lowered to one
block per minute, it would still take roughly another 4,080 years to
reach the maximum.
In the mean time, there is no reason to use a larger type which results
in higher memory and disk space usage. However, for now, in order to
avoid having to reserialize a bunch of database information, the heights
are still serialized to the database as 8-byte uint64s.
This is being mainly being done in preparation for further upcoming
infrastructure changes which will use the smaller and more efficient
4-byte serialization in the database as well.
Create GenerateCmd in btcjson v2. Update tests to check GenerateCmd.
Update chaincfg/params.go with a new bool in Params, GenerateSupported,
with true values in SimNetParams and RegressionNetParams and false in
the others.
Create new flag, discreteMining, in CPUMiner struct.
Add GenerateNBlocks function to cpuminer.go and handleGenerate
function to rpcserver.go.
Update documentation for the RPC calls.
This commit removes the error returns from the BlockHeader.BlockSha,
MsgBlock.BlockSha, and MsgTx.TxSha functions since they can never fail and
end up causing a lot of unneeded error checking throughout the code base.
It also updates all call sites for the change.
As pointed out in #189, according to the Go documentation, a ticker must
be stopped to release associated resources. This commit adds a defer call
to stop two tickers there were previously not being stopped as well as
changes two others that were being stopped over to use defer so it's more
consistent.
The other ticker in ScheduleShutdown is replaced and already calls Stop
before replacing it, so it has not been modified.
Closes#189.
ok @jrick
There are certain cases such as getblocktemplate which allow external
callers to be repsonsible for creating their own coinbase to replace the
generated one. By allowing the pay address to be nil in such cases, the
need to specify mining addresses via --miningaddr can be avoided thereby
leaving the payment address management up to the caller.
