This updates the following modules to use the secp256k1/v2 module:
- blockchain
- chaincfg/v2
- dcrutil/v2
- hdkeychain/v2
- mempool/v3
- txscript/v2
- main
The hdkeychain/v3 and txscript/v2 modules both use types from secp256k1
in their public API.
Consequently, in order avoid forcing them to bump their major versions,
secp256k1/v1.0.3 was released with the types redefined in terms of the
secp256k1/v2 module so callers still using v1 of the module that are not
ready to upgrade to the v2 module yet can interoperate by updating to
the latest patch version.
This updates the dcrec/secp256k1 module dependencies and serves as a
base for dcrec/secp256k1/v2.0.0.
The updated direct dependencies in this commit are as follows:
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
The full list of updated direct dependencies since the previous
dcrec/secp256k1/v1.0.2 release are as follows:
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
This ensures any empty/nil data elements in the input array to version 2
filter construction are not added to the filter since empty elements do
not make sense given how the filters are used. It also updates the
tests to ensure proper behavior.
The single match function already failed attempts to match an empty
element as intended, however, prior to this change, it was possible to
match an empty item in the multi-item matching path. This is not
desirable since the matching behavior must be consistent in both the
single and multi-match cases.
This adds the ErrorCode member to TxRuleError, filling it with
appropriate values throughout the mempool package. This allows clients
of the package to correctly identify error causes with a greater
granularity and respond appropriately.
It also deprecates the RejectCode attribute and ErrToRejectError
functions, to be removed in the next major version update of the
package.
All call sites that inspect mempool errors were updated to use the new
error codes instead of using RejectionCodes. Additional mempool tests
were added to ensure the correct behavior on some relevant cases.
Finally, given the introduction and use of a new public field, the main
module was updated to use an as-of-yet unfinished mempool v3.1.0, which
will include the required functionality.
This updates the rpc ask wallet set with missing entries.
The set has been ordered alphabetically and some
entries have been removed because they are yet to be
implemented by the wallet.
This combines the two conditions for the aggressive mining path into a
single condition and does a bit of light cleanup to remove the template
copies that are no longer necessary due to the removal of the old style
template caching.
This removes the UpdateExtraNonce function which updated an extra nonce
in the coinbase transaction and recalculated the merkle root since it is
not necessary and wasteful for Decred due to the extra nonce being
available the block header.
Further, due to the aforementioned and the fact the template doesn't
have a height set, it isn't currently actually being called anyway as
can be seen by diffing the decoded output of subsequent getwork calls
and noting the only thing that is being updated in between full
regeneration of new templates is the timestamp as expected.
$ diff -uNp work1.txt work2.txt
--- work1.txt 2019-09-07 08:18:58.410917100 -0500
+++ work2.txt 2019-09-07 08:19:01.216456300 -0500
@@ -16,7 +16,7 @@
"sbits": 0.00021026,
"height": 98,
"size": 7221,
- "time": 1567862338,
+ "time": 1567862341,
"nonce": 0,
"extradata": "0000000000000000000000000000000000000000000000000000000000000000",
"stakeversion": 0,
This removes all of the code related to setting and updating cached
templates in the block manager since they are no longer used.
It is easy to see this is the case by considering that the only places
that set cachedCurrentTemplate and cachedParentTemplate set them to nil.
This modifies all of the RPC code to use the chain
parameters that are associated with the RPC server
instead of the global activeNetParams and thus
moves one step closer to being able to split the
RPC server out into a separate package.
When a transaction is checked for relevance to a websocket client with
a loaded transaction filter, a call to ExtractPkScriptAddrs is not
enough. Commitments in tickets are encoded in an OP_RETURN output
which require an additional parse of the script to check for a
committed P2PKH or P2SH HASH160.
This removes the getblocktemplate and its helpers from the codebase.
Ongoing mining updates focused on the voting/block validation process
with respect to generating block templates for getwork makes it the
better option for decred. Also getblocktemplate rpc was buggy and
has been disabled for a while.
Some lint related issues have been addressed as well.
This adds an additional read from the ok channel in the peer listener
tests to ensure the version message is consumed as well as the verack so
that the remaining tests line up with the messages that are being tested.
This modifies the order in which the subsidy cache is created when not
provided by a caller to happen before the blockchain instance is created
to be more consistent.
This implements new version 2 filters which have 4 changes as compared
to version 1 filters:
- Support for independently specifying the false positive rate and
Golomb coding bin size which allows minimizing the filter size
- A faster (incompatible with version 1) reduction function
- A more compact serialization for the number of members in the set
- Deduplication of all hash collisions prior to reducing and serializing
the deltas
In addition, it adds a full set of tests and updates the benchmarks to
use the new version 2 filters.
The primary motivating factor for these changes is the ability to
minimize the size of the filters, however, the following is a before and
after comparison of version 1 and 2 filters in terms of performance and
allocations.
It is interesting to note the results for attempting to match a single
item is not very representative due to the fact the actual hash value
itself dominates to the point it can significantly vary due to the very
low ns timings involved. Those differences average out when matching
multiple items, which is the much more realistic scenario, and the
performance increase is in line with the expected values. It is also
worth nothing that filter construction now takes a bit longer due to the
additional deduplication step. While the performance numbers for filter
construction are about 25% larger in relative terms, it is only a few ms
difference in practice and therefore is an acceptable trade off for the
size savings provided.
benchmark old ns/op new ns/op delta
-----------------------------------------------------------------
BenchmarkFilterBuild50000 16194920 20279043 +25.22%
BenchmarkFilterBuild100000 32609930 41629998 +27.66%
BenchmarkFilterMatch 620 593 -4.35%
BenchmarkFilterMatchAny 2687 2302 -14.33%
benchmark old allocs new allocs delta
-----------------------------------------------------------------
BenchmarkFilterBuild50000 6 17 +183.33%
BenchmarkFilterBuild100000 6 18 +200.00%
BenchmarkFilterMatch 0 0 +0.00%
BenchmarkFilterMatchAny 0 0 +0.00%
benchmark old bytes new bytes delta
-----------------------------------------------------------------
BenchmarkFilterBuild50000 688366 2074653 +201.39%
BenchmarkFilterBuild100000 1360064 4132627 +203.86%
BenchmarkFilterMatch 0 0 +0.00%
BenchmarkFilterMatchAny 0 0 +0.00%
This refactors the best chain state and block index loading code into
separate functions so they are available to upcoming database update
code to build version 2 gcs filters.
During 32-bit nonce iteration, if a block solution wasn't found, the
iterator variable would overflow back to 0, creating an infinite loop,
thus continuing the puzzle search without ever updating the extra
nonce field. This bug has never been triggered in practice because
the code in question has only ever been used with difficulties where
a solution exists within the regular nonce space.
The extra nonce iteration logic itself was also imperfect in that it
wouldn't test a value of exactly 2^64 - 1.
The behavior we actually want is to loop through the entire unsigned
integer space for both the regular and extra nonces, and for this
process to continue forever until a solution is found. Note that
periodic updates to the block header timestamp during iteration ensure
that unique hashes are generated for subsequent generations of the
same nonce values.
This modifies the code to support an independent false positive rate and
Golomb coding bin size. Among other things, this permits more optimal
parameters for minimizing the filter size to be specified.
This capability will be used in the upcoming version 2 filters that will
ultimately be included in header commitments.
For a concrete example, the current version 1 filter for block 89341 on
mainnet contains 2470 items resulting in a full serialized size of 6,669
bytes. In contrast, if the optimal parameters were specified as
described by the comments in this commit, with no other changes to the
items included in the filter, that same filter would be 6,505 bytes,
which is a size reduction of about 2.46%. This might not seem like a
significant amount, but consider that there is a filter for every block,
so it really adds up.
Since the internal filter no longer directly has a P parameter, this
moves the method to obtain it to the FilterV1 type and adds a new test
to ensure it is returned properly.
Additionally, all of the tests are converted to use the parameters while
retaining the same effective parameters to help prove correctness of the
new code.
Finally, it also significantly reduces the number of allocations
required to construct a filter resulting in faster filter construction
and reduced pressure on the GC and does some other minor consistency
cleanup while here.
In terms of the reduction in allocations, the following is a before and
after comparison of building filters with 50k and 100k elements:
benchmark old ns/op new ns/op delta
--------------------------------------------------------------
BenchmarkFilterBuild50000 18095111 15680001 -13.35%
BenchmarkFilterBuild100000 31980156 31389892 -1.85%
benchmark old allocs new allocs delta
--------------------------------------------------------------
BenchmarkFilterBuild50000 31 6 -80.65%
BenchmarkFilterBuild100000 34 6 -82.35%
benchmark old bytes new bytes delta
--------------------------------------------------------------
BenchmarkFilterBuild50000 1202343 688271 -42.76%
BenchmarkFilterBuild100000 2488472 1360000 -45.35%
This simply rearranges the funcs so they are more logically grouped in
order to provide cleaner diffs for upcoming changes. There are no
functional changes.
This optimizes the Hash method of gcs filters by making use of the new
zero-alloc hashing funcs available in crypto/blake256.
The following is a before and after comparison:
benchmark old ns/op new ns/op delta
-------------------------------------------------
BenchmarkHash 1786 1315 -26.37%
benchmark old allocs new allocs delta
-------------------------------------------------
BenchmarkHash 2 0 -100.00%
benchmark old bytes new bytes delta
-------------------------------------------------
BenchmarkHash 176 0 -100.00%
Currently, the filters provide two different serialization formats per
version. The first is the raw filter bytes without the number of items
in its data set and is implemented by the Bytes and FromBytesV1
functions. The second includes that information and is implemented by
the NBytes and FromNBytesV1 functions.
In practice, the ability to serialize the filter independently from the
number of items in its data set is not very useful since that
information is required to be able to query the filter and, unlike the
other parameters which are fixed (e.g. false positive rate and key), the
number of items varies per filter. For this reason, all usage in
practice calls NBytes and FromNBytesV1.
Consequently, this simplifies the API for working with filters by
standardizing on a single serialization format per filter version which
includes the number of items in its data set.
In order to accomplish this, the current Bytes and FromBytesV1 functions
are removed and the NBytes and FromNBytesV1 functions are renamed to
take their place.
This also updates all tests and callers in the repo accordingly.
