Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate renterd subscriber to coreutils types #1098

Merged
merged 99 commits into from
May 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
a1ba198
chain: introduce subscriber
peterjan Mar 25, 2024
d2fbe63
stores: add TestApplyChainUpdate
peterjan Mar 25, 2024
fb39286
bus: fix interface
peterjan Mar 25, 2024
a8560e5
testing: fix TestWalletTransactions
peterjan Mar 25, 2024
9358313
subscriber: commit update on close
peterjan Mar 25, 2024
8e91afd
test: fix TestUploadDownloadSpending
peterjan Mar 25, 2024
4735104
stores: update interface
peterjan Mar 26, 2024
0e80949
chain: update subscriber
peterjan Mar 26, 2024
c6a1898
testing: remove TODOs
peterjan Mar 26, 2024
de5402e
subscriber: get rid of OnReorg
peterjan Mar 27, 2024
0b012bc
subscriber: re-add OnReorg
peterjan Mar 27, 2024
452c39e
subscriber: get rid of sync interval entirely
peterjan Mar 27, 2024
ad998b4
chain: handle state elements
peterjan Mar 28, 2024
ba97b55
chain: get rid of update in favour of exposing a transaction from the…
peterjan Apr 5, 2024
1acaabd
stores: remove chain test
peterjan Apr 5, 2024
13d5415
worker: undo change
peterjan Apr 5, 2024
2a03ff4
debug: add logging
peterjan Apr 5, 2024
e341c92
Tmp commit
peterjan Apr 16, 2024
4fe7400
Merge branch 'its-happening' of github.com:SiaFoundation/renterd into…
peterjan Apr 16, 2024
b1065c1
Merge branch 'pj/default-db-logger' into pj/subscription-api
peterjan Apr 16, 2024
2174d81
node: change start order
peterjan Apr 16, 2024
b04ef80
subscriber: add waitgroup
peterjan Apr 16, 2024
0f88914
testing: fix TestBusRecordedMetrics
peterjan Apr 16, 2024
eb055f2
subscriber: check for close inside retry loop
peterjan Apr 16, 2024
22e93ea
testing: avoid NDF in TestWalletRedistribute
peterjan Apr 16, 2024
df62a81
node: sleep after closing listener
peterjan Apr 16, 2024
4b20878
testing: fix TestBusRecordedMetrics
peterjan Apr 16, 2024
d0518d3
stores: add rollback
peterjan Apr 16, 2024
c133e85
testing: add debug logger
peterjan Apr 17, 2024
37b4a8e
testing: configure MySQL
peterjan Apr 17, 2024
4571ceb
subscriber: update concurrency
peterjan Apr 17, 2024
1a88963
go.mod: update coreutils dep
peterjan Apr 17, 2024
bf0cf3b
testing: test for contracts length
peterjan Apr 17, 2024
0923817
subscriber: fix index
peterjan Apr 17, 2024
b87aa78
testing: fix TestBusRecordedMetrics NDF
peterjan Apr 17, 2024
a523bed
testing: add logging
peterjan Apr 17, 2024
21fe4ba
debug: add logging
peterjan Apr 17, 2024
f249ee8
node: pass WithPeerDiscoveryInterval
peterjan Apr 17, 2024
74e248f
node: add syncer opts to bus config
peterjan Apr 17, 2024
1427824
testing: lower ap heartbeat
peterjan Apr 17, 2024
bdeeb92
testing: add debug logger
peterjan Apr 17, 2024
8e77a8d
testing: default to debug logger
peterjan Apr 17, 2024
3bdfc80
testing: update TestBlocklist
peterjan Apr 17, 2024
a962725
subscriber: update way we handle rollbacks
peterjan Apr 17, 2024
38ee174
testing: trigger autopilot on failure
peterjan Apr 17, 2024
9ce29bb
testing: wait for peers
peterjan Apr 17, 2024
2168a00
internal: use shorter tx retry intervals in testing
peterjan Apr 17, 2024
88e8bd2
testing: fix TestBusRecordedMetrics
peterjan Apr 17, 2024
1a24d21
testing: fix TestNewTestCluster NDF
peterjan Apr 17, 2024
9627b67
debug: add logging
peterjan Apr 17, 2024
8880d6b
subscriber: add logging
peterjan Apr 18, 2024
029f1f5
Merge branch 'its-happening' of github.com:SiaFoundation/renterd into…
peterjan Apr 18, 2024
733bc31
testing: fix TestBusRecordedMetrics
peterjan Apr 18, 2024
310fbda
go.mod: upgrade coreutils dependency
peterjan Apr 18, 2024
5b47b7e
subscriber: add logging
peterjan Apr 18, 2024
fbd4c60
testing: retry stored data check
peterjan Apr 18, 2024
4af3312
testing: add micro sleep in MineBlocks
peterjan Apr 18, 2024
3cc5a9d
testing: update TestEphemeralAccounts
peterjan Apr 18, 2024
15adf40
stores: only update if revision number is smaller, fix NDF
peterjan Apr 18, 2024
4305ba2
stores: update contract
peterjan Apr 18, 2024
9820426
stores: improve perf in UpdateStateElements
peterjan Apr 19, 2024
8d52431
Merge branch 'its-happening' into pj/subscription-api
peterjan Apr 30, 2024
bfd11a6
stores: update ChainStore interface
peterjan Apr 30, 2024
f0cceb3
stores: remove height and block_id from wallet outputs
peterjan Apr 30, 2024
44613b2
ci: revert test.yml
peterjan Apr 30, 2024
c8cd78a
chain: rename manager.go to chain.go
peterjan Apr 30, 2024
c70c754
chain: make applyChainUpdate mirror revertChainUpdate
peterjan Apr 30, 2024
ecc005c
stores: fix UpdateStateElements performance
peterjan Apr 30, 2024
25c3ff1
ci: run test in loop and add logging
peterjan May 2, 2024
2a2198a
ci: fix test.yml
peterjan May 2, 2024
30b2611
stores: add logging
peterjan May 2, 2024
5379e82
stores: add logging to chainUpdateTx
peterjan May 2, 2024
2b7f6ba
subscriber: add debug logging
peterjan May 3, 2024
04b14f2
deps: update coreutils
peterjan May 3, 2024
2cc097c
stores: remove contract subscriber pattren
peterjan May 3, 2024
706c9d7
chain: avoid the error handling in ForEach
peterjan May 3, 2024
e189460
testing: use default logger in TestUnconfirmedContractArchival
peterjan May 3, 2024
8b22b75
testing: remove all NoOp loggers
peterjan May 3, 2024
094c962
testing: add TestContractExists
peterjan May 3, 2024
d189483
chain: add duration
peterjan May 3, 2024
0020c7a
testing: add context to sync issues
peterjan May 3, 2024
fe1fb70
autopilot: expose ContractConfirmationDeadline and use 144 in testing
peterjan May 3, 2024
8fc37a0
contractor: refresh contract metadata
peterjan May 3, 2024
85bf378
all: cleanup PR
peterjan May 3, 2024
893501e
Merge branch 'its-happening' of github.com:SiaFoundation/renterd into…
peterjan May 3, 2024
fae31b0
ci: tmp disable stores test
peterjan May 3, 2024
5532b8e
chain: update subscriber
peterjan May 3, 2024
7485ceb
stores: add logging
peterjan May 3, 2024
42ca8b2
stors: add logging
peterjan May 3, 2024
fbb6b5a
bus: fix store contract update, add sync height
peterjan May 3, 2024
5366be3
lint: disable builtinShadow
peterjan May 3, 2024
00ffb04
ci: re-enable store tests
peterjan May 3, 2024
9671cc0
test: block after mining
peterjan May 6, 2024
7c6d1a0
test: upgrade host
peterjan May 7, 2024
98940a7
Merge branch 'its-happening' of github.com:SiaFoundation/renterd into…
peterjan May 7, 2024
783f480
all: implement CR remarks
peterjan May 7, 2024
accc3a1
testing: always mine and sync
peterjan May 10, 2024
620c9f7
chain: fix chain.db location and pass genesis block id
peterjan May 10, 2024
4560593
bus: use maxSyncTime to decide whether the bus is synced or not
peterjan May 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
host port: 3800
mysql version: '8'
mysql root password: test
- name: Test
- name: Test Stores
uses: n8maninger/action-golang-test@v1
with:
args: "-race;-short"
Expand Down
4 changes: 3 additions & 1 deletion autopilot/contractor/contractor.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ type Bus interface {
AncestorContracts(ctx context.Context, id types.FileContractID, minStartHeight uint64) ([]api.ArchivedContract, error)
ArchiveContracts(ctx context.Context, toArchive map[types.FileContractID]string) error
ConsensusState(ctx context.Context) (api.ConsensusState, error)
Contract(ctx context.Context, id types.FileContractID) (api.ContractMetadata, error)
Contracts(ctx context.Context, opts api.ContractsOpts) (contracts []api.ContractMetadata, err error)
FileContractTax(ctx context.Context, payout types.Currency) (types.Currency, error)
Host(ctx context.Context, hostKey types.PublicKey) (api.Host, error)
Expand Down Expand Up @@ -615,6 +616,7 @@ func (c *Contractor) runContractChecks(ctx *mCtx, hostChecks map[types.PublicKey
"toArchive", len(toArchive),
"toRefresh", len(toRefresh),
"toRenew", len(toRenew),
"bh", bh,
)
}()

Expand Down Expand Up @@ -650,7 +652,7 @@ LOOP:
toArchive[fcid] = errContractMaxRevisionNumber.Error()
} else if contract.RevisionNumber == math.MaxUint64 {
toArchive[fcid] = errContractMaxRevisionNumber.Error()
} else if contract.State == api.ContractStatePending && bh-contract.StartHeight > contractConfirmationDeadline {
} else if contract.State == api.ContractStatePending && bh-contract.StartHeight > ContractConfirmationDeadline {
toArchive[fcid] = errContractNotConfirmed.Error()
}
if _, archived := toArchive[fcid]; archived {
Expand Down
8 changes: 4 additions & 4 deletions autopilot/contractor/hostfilter.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ import (
)

const (
// ContractConfirmationDeadline is the number of blocks since its start
// height we wait for a contract to appear on chain.
ContractConfirmationDeadline = 18

// minContractFundUploadThreshold is the percentage of contract funds
// remaining at which the contract gets marked as not good for upload
minContractFundUploadThreshold = float64(0.05) // 5%
Expand All @@ -23,10 +27,6 @@ const (
// acquirable storage below which the contract is considered to be
// out-of-collateral.
minContractCollateralDenominator = 20 // 5%

// contractConfirmationDeadline is the number of blocks since its start
// height we wait for a contract to appear on chain.
contractConfirmationDeadline = 18
)

var (
Expand Down
38 changes: 28 additions & 10 deletions bus/bus.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,17 @@ import (
"go.sia.tech/renterd/api"
"go.sia.tech/renterd/build"
"go.sia.tech/renterd/bus/client"
"go.sia.tech/renterd/chain"
"go.sia.tech/renterd/object"
"go.sia.tech/renterd/webhooks"
"go.sia.tech/siad/modules"
"go.uber.org/zap"
)

// maxSyncTime is the maximum time since the last block before we consider
// ourselves unsynced.
const maxSyncTime = time.Hour

// Client re-exports the client from the client package.
type Client struct {
*client.Client
Expand Down Expand Up @@ -205,7 +210,7 @@ type (
Redistribute(outputs int, amount, feePerByte types.Currency) (txns []types.Transaction, toSign []types.Hash256, err error)
ReleaseInputs(txns ...types.Transaction)
SignTransaction(txn *types.Transaction, toSign []types.Hash256, cf types.CoveredFields)
SpendableOutputs() ([]wallet.SiacoinElement, error)
SpendableOutputs() ([]types.SiacoinElement, error)
Tip() (types.ChainIndex, error)
UnconfirmedTransactions() ([]wallet.Event, error)
Events(offset, limit int) ([]wallet.Event, error)
Expand All @@ -232,6 +237,7 @@ type bus struct {
w Wallet

as AutopilotStore
cs chain.ChainStore
eas EphemeralAccountStore
hdb HostDB
ms MetadataStore
Expand Down Expand Up @@ -450,7 +456,11 @@ func (b *bus) syncerConnectHandler(jc jape.Context) {
}

func (b *bus) consensusStateHandler(jc jape.Context) {
jc.Encode(b.consensusState())
cs, err := b.consensusState(jc.Request.Context())
if jc.Check("couldn't fetch consensus state", err) != nil {
return
}
jc.Encode(cs)
}

func (b *bus) consensusNetworkHandler(jc jape.Context) {
Expand Down Expand Up @@ -1771,19 +1781,23 @@ func (b *bus) paramsHandlerUploadGET(jc jape.Context) {
})
}

func (b *bus) consensusState() api.ConsensusState {
cs := b.cm.TipState()
func (b *bus) consensusState(ctx context.Context) (api.ConsensusState, error) {
index, err := b.cs.ChainIndex(ctx)
if err != nil {
return api.ConsensusState{}, err
}

var synced bool
if block, ok := b.cm.Block(cs.Index.ID); ok && time.Since(block.Timestamp) < 2*cs.BlockInterval() {
block, found := b.cm.Block(index.ID)
if found && time.Since(block.Timestamp) <= maxSyncTime {
synced = true
}

return api.ConsensusState{
BlockHeight: cs.Index.Height,
LastBlockTime: api.TimeRFC3339(cs.PrevTimestamps[0]),
BlockHeight: index.Height,
LastBlockTime: api.TimeRFC3339(block.Timestamp),
Synced: synced,
}
}, nil
}

func (b *bus) paramsHandlerGougingGET(jc jape.Context) {
Expand All @@ -1809,7 +1823,10 @@ func (b *bus) gougingParams(ctx context.Context) (api.GougingParams, error) {
b.logger.Panicf("failed to unmarshal redundancy settings '%s': %v", rss, err)
}

cs := b.consensusState()
cs, err := b.consensusState(ctx)
if err != nil {
return api.GougingParams{}, err
}

return api.GougingParams{
ConsensusState: cs,
Expand Down Expand Up @@ -2422,12 +2439,13 @@ func (b *bus) multipartHandlerListPartsPOST(jc jape.Context) {
}

// New returns a new Bus.
func New(am *alerts.Manager, hm WebhookManager, cm ChainManager, s Syncer, w Wallet, hdb HostDB, as AutopilotStore, ms MetadataStore, ss SettingStore, eas EphemeralAccountStore, mtrcs MetricsStore, l *zap.Logger) (*bus, error) {
func New(am *alerts.Manager, hm WebhookManager, cm ChainManager, cs chain.ChainStore, s Syncer, w Wallet, hdb HostDB, as AutopilotStore, ms MetadataStore, ss SettingStore, eas EphemeralAccountStore, mtrcs MetricsStore, l *zap.Logger) (*bus, error) {
b := &bus{
alerts: alerts.WithOrigin(am, "bus"),
alertMgr: am,
webhooks: hm,
cm: cm,
cs: cs,
s: s,
w: w,
hdb: hdb,
Expand Down
2 changes: 1 addition & 1 deletion bus/client/client_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ func newTestClient(dir string) (*client.Client, func() error, func(context.Conte

// create bus
network, genesis := build.Network()
b, shutdown, _, err := node.NewBus(node.BusConfig{
b, shutdown, _, _, err := node.NewBus(node.BusConfig{
Bus: config.Bus{
AnnouncementMaxAgeHours: 24 * 7 * 52, // 1 year
Bootstrap: false,
Expand Down
26 changes: 26 additions & 0 deletions chain/chain.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package chain

import (
"go.sia.tech/core/consensus"
"go.sia.tech/core/types"
"go.sia.tech/coreutils/chain"
)

type (
Manager = chain.Manager
HostAnnouncement = chain.HostAnnouncement
ApplyUpdate = chain.ApplyUpdate
RevertUpdate = chain.RevertUpdate
)

func TestnetZen() (*consensus.Network, types.Block) {
return chain.TestnetZen()
}

func NewDBStore(db chain.DB, n *consensus.Network, genesisBlock types.Block) (_ *chain.DBStore, _ consensus.State, err error) {
return chain.NewDBStore(db, n, genesisBlock)
}

func NewManager(store chain.Store, cs consensus.State) *Manager {
return chain.NewManager(store, cs)
}
10 changes: 0 additions & 10 deletions chain/manager.go

This file was deleted.

Loading
Loading