Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: process image cache in stages for memory optimization #234

Merged
merged 10 commits into from
Aug 20, 2024

Conversation

rafaelcr
Copy link
Collaborator

@rafaelcr rafaelcr commented Aug 15, 2024

Also:

  • Add profiler server
  • Add option to turn queue automatic start on/off

Copy link

github-actions bot commented Aug 15, 2024

Vercel deployment URL: https://token-metadata-7q7mua307-hirosystems.vercel.app 🚀

Copy link

codecov bot commented Aug 15, 2024

Codecov Report

Attention: Patch coverage is 43.82022% with 100 lines in your changes missing coverage. Please review.

Files Patch % Lines
src/token-processor/images/image-cache.ts 41.96% 62 Missing and 3 partials ⚠️
src/admin-rpc/init.ts 48.38% 16 Missing ⚠️
src/index.ts 0.00% 14 Missing ⚠️
src/token-processor/queue/job-queue.ts 76.92% 3 Missing ⚠️
src/api/@types/fastify/index.d.ts 0.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@rafaelcr rafaelcr changed the title fix: try to use native sharp pipeline fix: process image cache in stages for memory optimization Aug 20, 2024
@rafaelcr rafaelcr marked this pull request as ready for review August 20, 2024 19:37
@rafaelcr rafaelcr merged commit 617573b into refactor/chainhooks Aug 20, 2024
8 of 9 checks passed
@rafaelcr rafaelcr deleted the fix/sharp branch August 20, 2024 19:40
rafaelcr added a commit that referenced this pull request Aug 21, 2024
…ct stacks api connection (#200)

* chore: add chainhook client

* chore: single predicate draft

* fix: remove old code

* fix: test builds

* test: get contract abi

* test: first sip-019 notif

* test: print event chainhook observer

* feat: consume deployments

* test: chain tip

* fix: unify chainhook events processors

* feat: rollback contract deployments

* feat: rollback sft mint

* chore: draft for sip-019 storage

* feat: keep sip-019 notifs

* fix: notifications table inserts

* fix: update code

* fix: some tests

* fix: admin rpc

* fix: rollback contracts and sip-013 mints

* test: divide into suites

* ci: codecov

* chore: progress

* chore: update deps

* fix: remove abi

* fix: block cache

* chore: migrations

* fix: upgrade chainhook client

* chore: upgrade api toolkit

* chore: divide test files

* test: ft, nft

* test: sft

* fix: notification tests

* fix: dynamic refresh

* fix: token queue tests

* fix: admin rpc tests

* fix: api tests

* chore: remove contract import admin rpc

* feat: start validating with chainhook

* fix: eslint

* fix: admin rpc tests

* ci: run matrix tests properly

* fix: skip ingested blocks

* feat: return asset_identifier in FT responses (#229)

* test: zero token nft

* fix: refactor GCS image cache upload script to typescript (#232)

* fix: image cache ts

* test: more

* fix: code

* build: use v2 docker compose

* fix: admin tests

* fix: try finally

* test: debug logs

* test: only 1

* test: one more

* fix: not the timeout

* fix: different ports

* fix: improve test servers

* fix: add cause to error message

* fix: try 127.0.0.1

* fix: try timeout sleep

* fix: standard close

* test: more

* fix: improve message

* fix: echo status

* fix: detect

* fix: finally

* fix: x

* fix: timeouts

* fix: revert debug stuff

* chore: remove old test

* fix: store chainhook predicate in local file for reuse after restart (#233)

fix: add tests

* fix: process image cache in stages for memory optimization (#234)

* fix: try to use native sharp pipeline

* fix: job queue controls, profiler

* fix: tests

* fix: upgrade toolkit

* fix: improve error propagation

* fix: listen to stream errors

* fix: process cached image step by step

* fix: report errors retried immediately

* fix: image stream

* fix: wait for sharp stream correctly

* chore: upgrade to node 20 (#235)

* feat: provide detailed error responses for tokens with user/contract errors (#236)

* chore: demote httperror to usererror

* fix: add errors

* fix: custom messages

* feat: add testnet support
blockstack-devops pushed a commit that referenced this pull request Aug 21, 2024
## [1.0.0-beta.1](v0.7.0...v1.0.0-beta.1) (2024-08-21)

### ⚠ BREAKING CHANGES

* use chainhook to listen for chain events instead of a direct stacks api connection (#200)

### Bug Fixes

* use prometheus port configured in ENV ([c769d29](c769d29))

### Code Refactoring

* use chainhook to listen for chain events instead of a direct stacks api connection ([#200](#200)) ([2ddb2c7](2ddb2c7)), closes [#229](#229) [#232](#232) [#233](#233) [#234](#234) [#235](#235) [#236](#236)
blockstack-devops pushed a commit that referenced this pull request Aug 26, 2024
## [1.0.0](v0.7.0...v1.0.0) (2024-08-26)

### ⚠ BREAKING CHANGES

* use chainhook to listen for chain events instead of a direct stacks api connection (#200)

### Features

* convert data: image uris into image files and upload to cdn ([#245](#245)) ([903b0aa](903b0aa))

### Bug Fixes

* catch econnreset errors ([#247](#247)) ([51347d6](51347d6))
* set maximum job timeout ([#244](#244)) ([3444917](3444917))
* take only first page of gif images ([#241](#241)) ([334f8c5](334f8c5))
* use bignumber to handle FT supplies ([#239](#239)) ([053d622](053d622))
* use google cloud library for image uploads ([#238](#238)) ([c7f1b43](c7f1b43))
* use prometheus port configured in ENV ([c769d29](c769d29))

### Code Refactoring

* use chainhook to listen for chain events instead of a direct stacks api connection ([#200](#200)) ([2ddb2c7](2ddb2c7)), closes [#229](#229) [#232](#232) [#233](#233) [#234](#234) [#235](#235) [#236](#236)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

1 participant