Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: refactor GCS image cache upload script to typescript #232

Merged
merged 25 commits into from
Aug 15, 2024

Conversation

rafaelcr
Copy link
Collaborator

@rafaelcr rafaelcr commented Aug 9, 2024

To improve error handling when running the job queue, the Image Cache processing code was moved from a .js file that was spawned in an external process into a TypeScript file that we can integrate within our token processing code.

After this, we are now able to discern errors based on different image processing situations and decide whether or not we should retry the image caching later.

Copy link

github-actions bot commented Aug 9, 2024

Vercel deployment URL: https://token-metadata-eq383fedh-hirosystems.vercel.app 🚀

Copy link

codecov bot commented Aug 9, 2024

Codecov Report

Attention: Patch coverage is 58.84477% with 114 lines in your changes missing coverage. Please review.

Files Patch % Lines
src/token-processor/images/image-cache.ts 41.39% 109 Missing ⚠️
src/token-processor/util/metadata-helpers.ts 91.80% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

@rafaelcr rafaelcr marked this pull request as ready for review August 12, 2024 16:46
@rafaelcr rafaelcr merged commit 29bdf2c into refactor/chainhooks Aug 15, 2024
8 of 9 checks passed
@rafaelcr rafaelcr deleted the fix/image-cache-ts branch August 15, 2024 17:04
rafaelcr added a commit that referenced this pull request Aug 21, 2024
…ct stacks api connection (#200)

* chore: add chainhook client

* chore: single predicate draft

* fix: remove old code

* fix: test builds

* test: get contract abi

* test: first sip-019 notif

* test: print event chainhook observer

* feat: consume deployments

* test: chain tip

* fix: unify chainhook events processors

* feat: rollback contract deployments

* feat: rollback sft mint

* chore: draft for sip-019 storage

* feat: keep sip-019 notifs

* fix: notifications table inserts

* fix: update code

* fix: some tests

* fix: admin rpc

* fix: rollback contracts and sip-013 mints

* test: divide into suites

* ci: codecov

* chore: progress

* chore: update deps

* fix: remove abi

* fix: block cache

* chore: migrations

* fix: upgrade chainhook client

* chore: upgrade api toolkit

* chore: divide test files

* test: ft, nft

* test: sft

* fix: notification tests

* fix: dynamic refresh

* fix: token queue tests

* fix: admin rpc tests

* fix: api tests

* chore: remove contract import admin rpc

* feat: start validating with chainhook

* fix: eslint

* fix: admin rpc tests

* ci: run matrix tests properly

* fix: skip ingested blocks

* feat: return asset_identifier in FT responses (#229)

* test: zero token nft

* fix: refactor GCS image cache upload script to typescript (#232)

* fix: image cache ts

* test: more

* fix: code

* build: use v2 docker compose

* fix: admin tests

* fix: try finally

* test: debug logs

* test: only 1

* test: one more

* fix: not the timeout

* fix: different ports

* fix: improve test servers

* fix: add cause to error message

* fix: try 127.0.0.1

* fix: try timeout sleep

* fix: standard close

* test: more

* fix: improve message

* fix: echo status

* fix: detect

* fix: finally

* fix: x

* fix: timeouts

* fix: revert debug stuff

* chore: remove old test

* fix: store chainhook predicate in local file for reuse after restart (#233)

fix: add tests

* fix: process image cache in stages for memory optimization (#234)

* fix: try to use native sharp pipeline

* fix: job queue controls, profiler

* fix: tests

* fix: upgrade toolkit

* fix: improve error propagation

* fix: listen to stream errors

* fix: process cached image step by step

* fix: report errors retried immediately

* fix: image stream

* fix: wait for sharp stream correctly

* chore: upgrade to node 20 (#235)

* feat: provide detailed error responses for tokens with user/contract errors (#236)

* chore: demote httperror to usererror

* fix: add errors

* fix: custom messages

* feat: add testnet support
blockstack-devops pushed a commit that referenced this pull request Aug 21, 2024
## [1.0.0-beta.1](v0.7.0...v1.0.0-beta.1) (2024-08-21)

### ⚠ BREAKING CHANGES

* use chainhook to listen for chain events instead of a direct stacks api connection (#200)

### Bug Fixes

* use prometheus port configured in ENV ([c769d29](c769d29))

### Code Refactoring

* use chainhook to listen for chain events instead of a direct stacks api connection ([#200](#200)) ([2ddb2c7](2ddb2c7)), closes [#229](#229) [#232](#232) [#233](#233) [#234](#234) [#235](#235) [#236](#236)
blockstack-devops pushed a commit that referenced this pull request Aug 26, 2024
## [1.0.0](v0.7.0...v1.0.0) (2024-08-26)

### ⚠ BREAKING CHANGES

* use chainhook to listen for chain events instead of a direct stacks api connection (#200)

### Features

* convert data: image uris into image files and upload to cdn ([#245](#245)) ([903b0aa](903b0aa))

### Bug Fixes

* catch econnreset errors ([#247](#247)) ([51347d6](51347d6))
* set maximum job timeout ([#244](#244)) ([3444917](3444917))
* take only first page of gif images ([#241](#241)) ([334f8c5](334f8c5))
* use bignumber to handle FT supplies ([#239](#239)) ([053d622](053d622))
* use google cloud library for image uploads ([#238](#238)) ([c7f1b43](c7f1b43))
* use prometheus port configured in ENV ([c769d29](c769d29))

### Code Refactoring

* use chainhook to listen for chain events instead of a direct stacks api connection ([#200](#200)) ([2ddb2c7](2ddb2c7)), closes [#229](#229) [#232](#232) [#233](#233) [#234](#234) [#235](#235) [#236](#236)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants