Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nft index #936

Merged
merged 61 commits into from
Mar 9, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
9944d99
Added new -nftindex param
mxaddict Feb 9, 2022
c6abfe3
Added missing declaration
mxaddict Feb 9, 2022
192a342
Added missing declaration
mxaddict Feb 9, 2022
7f62acf
Updated ALLINDEX
mxaddict Feb 21, 2022
152b3de
Updated some whitespace in rpcwallet
mxaddict Feb 21, 2022
b4be813
Added nftindex flags to blocktree logic
mxaddict Feb 21, 2022
5e21f53
Added txdb functions for nft index
mxaddict Feb 21, 2022
ee4040b
Added nft index update calls to Connect/Disconnect block
mxaddict Feb 21, 2022
5905863
Added nftunspendindex.h
mxaddict Feb 21, 2022
29deaf7
Added nftunspendindex.h
mxaddict Feb 21, 2022
b9cc1c2
Updates based on alex's input
mxaddict Feb 21, 2022
10804bb
Added new DB_NFTUNSPENTINDEX char to txdb.h
mxaddict Feb 21, 2022
af6f3d8
Fixed some references
mxaddict Feb 21, 2022
a1a4a12
Moved a line down that I think should not be on the end of the previo…
mxaddict Feb 21, 2022
e08a92f
Removed tokenId from CNftUnspendIndexValue
mxaddict Feb 21, 2022
e767a3f
Updated nftunspentindex to serialize the blockHeight as big indian
mxaddict Feb 21, 2022
e334121
Changed blockHeight to uint32_t
mxaddict Feb 21, 2022
6415376
Updated nftunspentindex calls to not have the tokenId in value param
mxaddict Feb 21, 2022
09af9b6
Added a Read func for nftunspentindex
mxaddict Feb 21, 2022
c517ba4
Removed tokenId from CNftUnspentIndexValue
mxaddict Feb 21, 2022
ce50c30
Ops
mxaddict Feb 21, 2022
5d20e20
Added -ntfindex=1 to stressor for tokens
mxaddict Feb 21, 2022
65d27f3
Revert "Ops"
mxaddict Feb 21, 2022
b3d81e4
Revert "Removed tokenId from CNftUnspentIndexValue"
mxaddict Feb 21, 2022
60a7c1f
Added UTXO data to listtokens
mxaddict Feb 21, 2022
765f2d2
Replaced tokenId uint256 with tokenId TokenId instead
mxaddict Feb 21, 2022
66a74c3
Replaced tokenId uint256 with tokenId TokenId instead
mxaddict Feb 21, 2022
532baf1
Replaced tokenId uint256 with tokenId TokenId instead
mxaddict Feb 21, 2022
6eab03a
Fixed a crash
mxaddict Feb 21, 2022
430dc4f
Updated the way the token is loaded for adding to the index
mxaddict Feb 21, 2022
18f2ebe
Removed redundant checks
mxaddict Feb 21, 2022
d94b861
Fixed reading method for nft index
mxaddict Feb 21, 2022
4afb994
Changed some logic and stored the index in a more efficient way
mxaddict Feb 21, 2022
a064844
Removed a commit that is no longer needed
mxaddict Feb 21, 2022
82f14bd
Added (with_utxo) to gettoken rpc command
mxaddict Feb 21, 2022
9ea3bdc
Added new viewstate functions for token utxos
mxaddict Feb 24, 2022
dd3337e
Added new viewstate functions for token utxos
mxaddict Feb 24, 2022
7fc6b4b
Updated the nftindex logic
mxaddict Feb 24, 2022
5e03b3f
Updated TokenUtxoModified logic
mxaddict Feb 24, 2022
3fb8cae
Updated TokenUtxoValue
mxaddict Feb 24, 2022
ea0773b
Updated TokenUtxoValue
mxaddict Feb 24, 2022
0ab78f4
Renamed DB_NFTUNSPENTINDEX to DB_TOKEN_UTXO
mxaddict Feb 24, 2022
e580a30
Updated BatchWrite logic for CStateView flush
mxaddict Feb 24, 2022
2fac02a
Updated BatchWrite logic for txdb batchwrite
mxaddict Feb 24, 2022
8e94f31
Fixed a crash and added some logging
mxaddict Feb 24, 2022
583bf48
Removed a log
mxaddict Feb 24, 2022
e2df75f
Fix gettoken
mxaddict Feb 24, 2022
3183371
Check for null entries in TokenUtxoValues
mxaddict Mar 2, 2022
7681f54
Replaced TokenId with unint256 from SerializeHash(TokenId)
mxaddict Mar 2, 2022
c76aec6
Added more logging for now
mxaddict Mar 2, 2022
21c82f7
Removed some logging
mxaddict Mar 2, 2022
34959d7
Updated some logging
mxaddict Mar 2, 2022
cc4634c
Updated some logging
mxaddict Mar 2, 2022
7c18569
Add a call to view.HaveTokenUtxos to reload the cache with old values…
mxaddict Mar 3, 2022
32f1884
Fixed rpcwallet listtokens and gettoken to use latest non-null entry …
mxaddict Mar 3, 2022
3bcfbb5
Moved nftindex call to navcoind
mxaddict Mar 6, 2022
3402ed0
Made rpc function changes discussed with alex, added getnft and listn…
mxaddict Mar 7, 2022
9dbbf58
Updated stressor_token
mxaddict Mar 7, 2022
6e1f7ed
Updates to the stressor for tokens/nfts
mxaddict Mar 8, 2022
589a980
Updates to the stressor for tokens/nfts
mxaddict Mar 8, 2022
bb1adde
Updates to the stressor for tokens/nfts
mxaddict Mar 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,833 changes: 933 additions & 900 deletions contrib/stressor_token.sh

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/Makefile.am
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,7 @@ libnavcoin_consensus_a_SOURCES = \
consensus/programs.h \
consensus/program_actions.h \
ctokens/ctokens.h \
ctokens/tokenutxos.h \
ctokens/tokenid.h \
dotnav/namedata.h \
dotnav/namerecord.h \
Expand Down
122 changes: 117 additions & 5 deletions src/coins.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ bool CStateView::GetConsultation(const uint256 &cid, CConsultation& consultation
bool CStateView::GetConsultationAnswer(const uint256 &cid, CConsultationAnswer& answer) const { return false; }
bool CStateView::GetConsensusParameter(const int &pid, CConsensusParameter& cparameter) const { return false; }
bool CStateView::GetToken(const uint256 &id, TokenInfo& token) const { return false; }
bool CStateView::GetTokenUtxos(const uint256 &id, TokenUtxoValues &tokenUtxos) { return false; };
bool CStateView::GetNameRecord(const uint256 &id, NameRecordValue& height) const { return false; }
bool CStateView::GetNameData(const uint256 &id, NameDataValues& data) { return false; }
bool CStateView::HaveCoins(const uint256 &txid) const { return false; }
Expand All @@ -63,6 +64,7 @@ bool CStateView::HaveConsultation(const uint256 &cid) const { return false; }
bool CStateView::HaveConsultationAnswer(const uint256 &cid) const { return false; }
bool CStateView::HaveConsensusParameter(const int &pid) const { return false; }
bool CStateView::HaveToken(const uint256 &id) const { return false; }
bool CStateView::HaveTokenUtxos(const uint256 &id) const { return false; }
bool CStateView::HaveNameRecord(const uint256 &id) const { return false; }
bool CStateView::HaveNameData(const uint256 &id) const { return false; }
bool CStateView::GetAllProposals(CProposalMap& map) { return false; }
Expand All @@ -78,7 +80,7 @@ uint256 CStateView::GetBestBlock() const { return uint256(); }
bool CStateView::BatchWrite(CCoinsMap &mapCoins, CProposalMap &mapProposals,
CPaymentRequestMap &mapPaymentRequests, CVoteMap &mapVotes,
CConsultationMap& mapConsultations, CConsultationAnswerMap& mapAnswers,
CConsensusParameterMap& mapConsensus, TokenMap& mapTokens,
CConsensusParameterMap& mapConsensus, TokenMap& mapTokens, TokenUtxoMap& mapTokenUtxos,
NameRecordMap& mapNameRecords, NameDataMap& mapNameData,
const uint256 &hashBlock, const int& nCacheExcludeVotes) { return false; }
CStateViewCursor *CStateView::Cursor() const { return 0; }
Expand All @@ -92,6 +94,7 @@ bool CStateViewBacked::GetConsultation(const uint256 &cid, CConsultation &consul
bool CStateViewBacked::GetConsultationAnswer(const uint256 &cid, CConsultationAnswer &answer) const { return base->GetConsultationAnswer(cid, answer); }
bool CStateViewBacked::GetConsensusParameter(const int &pid, CConsensusParameter& cparameter) const { return base->GetConsensusParameter(pid, cparameter); }
bool CStateViewBacked::GetToken(const uint256 &id, TokenInfo& token) const { return base->GetToken(id, token); }
bool CStateViewBacked::GetTokenUtxos(const uint256 &id, TokenUtxoValues &tokenUtxos) { return base->GetTokenUtxos(id, tokenUtxos); }
bool CStateViewBacked::GetNameRecord(const uint256 &id, NameRecordValue& height) const { return base->GetNameRecord(id, height); }
bool CStateViewBacked::GetNameData(const uint256 &id, NameDataValues& data) { return base->GetNameData(id, data); }
bool CStateViewBacked::HaveCoins(const uint256 &txid) const { return base->HaveCoins(txid); }
Expand All @@ -102,6 +105,7 @@ bool CStateViewBacked::HaveConsultation(const uint256 &cid) const { return base-
bool CStateViewBacked::HaveConsultationAnswer(const uint256 &cid) const { return base->HaveConsultationAnswer(cid); }
bool CStateViewBacked::HaveConsensusParameter(const int &pid) const { return base->HaveConsensusParameter(pid); }
bool CStateViewBacked::HaveToken(const uint256 &id) const { return base->HaveToken(id); }
bool CStateViewBacked::HaveTokenUtxos(const uint256 &id) const { return base->HaveTokenUtxos(id); }
bool CStateViewBacked::HaveNameRecord(const uint256 &id) const { return base->HaveNameRecord(id); }
bool CStateViewBacked::HaveNameData(const uint256 &id) const { return base->HaveNameData(id); }
int CStateViewBacked::GetExcludeVotes() const { return base->GetExcludeVotes(); }
Expand All @@ -119,10 +123,10 @@ void CStateViewBacked::SetBackend(CStateView &viewIn) { base = &viewIn; }
bool CStateViewBacked::BatchWrite(CCoinsMap &mapCoins, CProposalMap &mapProposals,
CPaymentRequestMap &mapPaymentRequests, CVoteMap &mapVotes,
CConsultationMap &mapConsultations, CConsultationAnswerMap &mapAnswers,
CConsensusParameterMap& mapConsensus, TokenMap& mapTokens,
CConsensusParameterMap& mapConsensus, TokenMap& mapTokens, TokenUtxoMap& mapTokenUtxos,
NameRecordMap& mapNameRecords, NameDataMap& mapNameData,
const uint256 &hashBlock, const int &nCacheExcludeVotes) {
return base->BatchWrite(mapCoins, mapProposals, mapPaymentRequests, mapVotes, mapConsultations, mapAnswers, mapConsensus, mapTokens, mapNameRecords, mapNameData, hashBlock, nCacheExcludeVotes);
return base->BatchWrite(mapCoins, mapProposals, mapPaymentRequests, mapVotes, mapConsultations, mapAnswers, mapConsensus, mapTokens, mapTokenUtxos, mapNameRecords, mapNameData, hashBlock, nCacheExcludeVotes);
}
CStateViewCursor *CStateViewBacked::Cursor() const { return base->Cursor(); }

Expand Down Expand Up @@ -278,6 +282,23 @@ TokenMap::const_iterator CStateViewCache::FetchToken(const uint256 &id) const {
return ret;
}

TokenUtxoMap::const_iterator CStateViewCache::FetchTokenUtxos(const uint256 &id) const {
TokenUtxoMap::iterator it = cacheTokenUtxos.find(id);

if (it != cacheTokenUtxos.end() && it->second.size() > 0)
return it;

TokenUtxoValues tmp;

if (!base->GetTokenUtxos(id, tmp) || tmp.size() == 0)
return cacheTokenUtxos.end();

TokenUtxoMap::iterator ret = cacheTokenUtxos.insert(std::make_pair(id, TokenUtxoValues())).first;
tmp.swap(ret->second);

return ret;
}

NameRecordMap::const_iterator CStateViewCache::FetchNameRecord(const uint256 &id) const {
NameRecordMap::iterator it = cacheNameRecords.find(id);

Expand Down Expand Up @@ -389,6 +410,15 @@ bool CStateViewCache::GetToken(const uint256 &id, TokenInfo &token) const {
return false;
}

bool CStateViewCache::GetTokenUtxos(const uint256 &id, TokenUtxoValues &tokenUtxos) {
TokenUtxoMap::const_iterator it = FetchTokenUtxos(id);
if (it != cacheTokenUtxos.end() && it->second.size() > 0) {
tokenUtxos = it->second;
return true;
}
return false;
}

bool CStateViewCache::GetNameRecord(const uint256 &id, NameRecordValue &height) const {
NameRecordMap::const_iterator it = FetchNameRecord(id);
if (it != cacheNameRecords.end() && !it->second.IsNull()) {
Expand Down Expand Up @@ -616,6 +646,17 @@ TokenModifier CStateViewCache::ModifyToken(const uint256 &id, int nHeight) {
return TokenModifier(*this, ret.first, nHeight);
}

TokenUtxosModifier CStateViewCache::ModifyTokenUtxos(const uint256 &id, int blockHeight) {
assert(!hasModifier);
std::pair<TokenUtxoMap::iterator, bool> ret = cacheTokenUtxos.insert(std::make_pair(id, TokenUtxoValues()));
if (ret.second) {
if (!base->GetTokenUtxos(id, ret.first->second)) {
ret.first->second.clear();
}
}
return TokenUtxosModifier(*this, ret.first, blockHeight);
}

NameRecordModifier CStateViewCache::ModifyNameRecord(const uint256 &id, int nHeight) {
assert(!hasModifier);
std::pair<NameRecordMap::iterator, bool> ret = cacheNameRecords.insert(std::make_pair(id, 0));
Expand Down Expand Up @@ -773,6 +814,21 @@ bool CStateViewCache::AddToken(const Token& token) const {
return true;
}

bool CStateViewCache::AddTokenUtxo(const uint256 &id, const TokenUtxoEntry& utxo) const {
if (cacheTokenUtxos.count(id)) {
cacheTokenUtxos[id].erase(
std::remove_if(cacheTokenUtxos[id].begin(), cacheTokenUtxos[id].end(),
[&utxo](const TokenUtxoEntry & o) { return o.first == utxo.first && o.second.IsNull(); }),
cacheTokenUtxos[id].end());
cacheTokenUtxos[id].push_back(utxo);
} else {
cacheTokenUtxos.insert(std::make_pair(id, TokenUtxoValues()));
cacheTokenUtxos[id].push_back(utxo);
}

return true;
}

bool CStateViewCache::AddNameRecord(const NameRecord& namerecord) const {
if (HaveNameRecord(namerecord.first))
return false;
Expand Down Expand Up @@ -826,6 +882,28 @@ bool CStateViewCache::RemoveToken(const uint256 &id) const {
return true;
}

bool CStateViewCache::RemoveTokenUtxo(const TokenUtxoKey &key) const {
if (!HaveTokenUtxos(key.id))
return false;

if (cacheTokenUtxos.count(key.id))
{
TokenUtxoValues temp;

for (auto& it: cacheTokenUtxos[key.id]) {
if (it.first == key.blockHeight) {
temp.push_back(TokenUtxoEntry(key.blockHeight, TokenUtxoValue()));
} else {
temp.push_back(it);
}
}

cacheTokenUtxos[key.id] = temp;
}

return true;
}

bool CStateViewCache::RemoveNameRecord(const uint256 &id) const {
if (!HaveNameRecord(id))
return false;
Expand Down Expand Up @@ -960,6 +1038,11 @@ bool CStateViewCache::HaveToken(const uint256 &id) const {
return (it != cacheTokens.end() && !it->second.IsNull());
}

bool CStateViewCache::HaveTokenUtxos(const uint256 &id) const {
TokenUtxoMap::const_iterator it = FetchTokenUtxos(id);
return (it != cacheTokenUtxos.end() && it->second.size());
}

bool CStateViewCache::HaveNameRecord(const uint256 &id) const {
NameRecordMap::const_iterator it = FetchNameRecord(id);
return (it != cacheNameRecords.end() && !it->second.IsNull());
Expand Down Expand Up @@ -1016,7 +1099,7 @@ void CStateViewCache::SetBestBlock(const uint256 &hashBlockIn) {

bool CStateViewCache::BatchWrite(CCoinsMap &mapCoins, CProposalMap &mapProposals, CPaymentRequestMap &mapPaymentRequests,
CVoteMap& mapVotes, CConsultationMap& mapConsultations, CConsultationAnswerMap& mapAnswers,
CConsensusParameterMap& mapConsensus, TokenMap& mapTokens, NameRecordMap& mapNameRecords,
CConsensusParameterMap& mapConsensus, TokenMap& mapTokens, TokenUtxoMap& mapTokenUtxos, NameRecordMap& mapNameRecords,
NameDataMap& mapNameData, const uint256 &hashBlockIn, const int &nCacheExcludeVotesIn) {
assert(!hasModifier);
assert(!hasModifierConsensus);
Expand Down Expand Up @@ -1121,6 +1204,13 @@ bool CStateViewCache::BatchWrite(CCoinsMap &mapCoins, CProposalMap &mapProposals
mapTokens.erase(itOld);
}

for (TokenUtxoMap::iterator it = mapTokenUtxos.begin(); it != mapTokenUtxos.end();) {
TokenUtxoValues& entry = cacheTokenUtxos[it->first];
entry.swap(it->second);
TokenUtxoMap::iterator itOld = it++;
mapTokenUtxos.erase(itOld);
}

for (NameRecordMap::iterator it = mapNameRecords.begin(); it != mapNameRecords.end();) {
NameRecordValue& entry = cacheNameRecords[it->first];
entry.swap(it->second);
Expand All @@ -1141,7 +1231,7 @@ bool CStateViewCache::BatchWrite(CCoinsMap &mapCoins, CProposalMap &mapProposals
}

bool CStateViewCache::Flush() {
bool fOk = base->BatchWrite(cacheCoins, cacheProposals, cachePaymentRequests, cacheVotes, cacheConsultations, cacheAnswers, cacheConsensus, cacheTokens, cacheNameRecords, cacheNameData, hashBlock, nCacheExcludeVotes);
bool fOk = base->BatchWrite(cacheCoins, cacheProposals, cachePaymentRequests, cacheVotes, cacheConsultations, cacheAnswers, cacheConsensus, cacheTokens, cacheTokenUtxos, cacheNameRecords, cacheNameData, hashBlock, nCacheExcludeVotes);
cacheCoins.clear();
cacheProposals.clear();
cachePaymentRequests.clear();
Expand All @@ -1150,6 +1240,7 @@ bool CStateViewCache::Flush() {
cacheAnswers.clear();
cacheConsensus.clear();
cacheTokens.clear();
cacheTokenUtxos.clear();
cacheNameRecords.clear();
cacheNameData.clear();
cachedCoinsUsage = 0;
Expand Down Expand Up @@ -1397,6 +1488,27 @@ TokenModifier::~TokenModifier()
}
}

TokenUtxosModifier::TokenUtxosModifier(CStateViewCache& cache_, TokenUtxoMap::iterator it_, int blockHeight_) : cache(cache_), it(it_), blockHeight(blockHeight_) {
assert(!cache.hasModifier);
cache.hasModifier = true;
prev = it->second;
}

TokenUtxosModifier::~TokenUtxosModifier()
{
assert(cache.hasModifier);
cache.hasModifier = false;

if (it->second.size() == 0) {
cache.cacheTokenUtxos[it->first].clear();
}

if (!(prev == it->second))
{
LogPrint("token", "%s: Modified %stoken utxo %s\n", __func__, blockHeight > 0 ? strprintf("at block height %d ", blockHeight) : "", it->first.ToString());
}
}

NameRecordModifier::NameRecordModifier(CStateViewCache& cache_, NameRecordMap::iterator it_, int height_) : cache(cache_), it(it_), height(height_) {
assert(!cache.hasModifier);
cache.hasModifier = true;
Expand Down
Loading