Table of Contents
The image service allows any utils service to send files to be publicly hosted. Through an utils interface, such as Directus for example, a user can utils an image and obtain a public URL to use to access their file. The user can either use the original image via a specific URL or process the image for optimization.
The image service (more generally files) consists of a first Nginx server whose configuration contains 3 types of locations:
- /catalog
- /assets/media/*
- /purge/assets/media/*
This first web server is caching resource 1m, then proxy pass to the nginx purge server that contains the same locations but configured with proxy_purge_module. This server caching 30d and can invalidate cache with purge location.
To retrieve an uploaded file, 3 ways :
- /assets/media/original/
image_name.extension
: no processing on the image - /assets/media/full/
image_name.extension
: transform to webp and serve as webp - /assets/media/optimise/200x0/
image_name.extension
: resizing to with 200 and transform to webp and serve as webp
And finally an Express Node API whose roles are:
- providing the list of files exposed by the catalog/reference on a GET route
- uploading new files
- updating existing files
An important point is also the notion of catalog. The catalog serves as a reference & truth for the express API, containing information indicating not to serve it (either it has been deleted from the catalog, or it has expired for example).
To track the images that we have uploaded/deleted/updated we use a Redis server and serve catalog list to json format. Here's a glimpse of it's structure:
[
{
"uuid": "e080a953-5300-427b-bd39-6e235d8238a2",
"version": 1,
"namespace": "DEV",
"public_url": "http://localhost:8080/palpatine/assets/media/full/image/DEV/default.webp",
"unique_name": "/DEV/default.webp",
"filename": "default.webp",
"original_filename": "default.webp",
"base_url": "http://localhost:8080/palpatine/assets/media",
"external_id": null,
"expired": false,
"expiration_date": null,
"information": null,
"original_mimetype": "image/webp",
"mimetype": "image/webp",
"signature": "ca71754acda70e41cb23e465fbb5ecc683186cf779a2bae2cbf290527b1f6671",
"size": 16730
},
{
"uuid": "d26a191f-1087-4169-b6cd-3db96f38ece4",
"version": 1,
"namespace": "DEV",
"public_url": "http://localhost:8080/palpatine/assets/media/full/image/DEV/error.webp",
"unique_name": "/DEV/error.webp",
"filename": "error.webp",
"original_filename": "error.webp",
"base_url": "http://localhost:8080/palpatine/assets/media",
"external_id": null,
"expired": false,
"expiration_date": null,
"information": null,
"original_mimetype": "image/webp",
"mimetype": "image/webp",
"signature": "368ba95afb311edfe0cb7f3b4a221e8b2a3edeb4e16fc5683762791f9619b28a",
"size": 10614
}
]
2 daily jobs associated with TADA (Transform And Deliver Assets 🎉):
- a job to synchronize the state of our API in relation to YOUR delegated_storage: if the image is in the catalog, but not in your storage, it deletes the image from the catalog.
- a catalog publication job on your delegated storage: the status of the catalog once a day is published on your storage which allows you to retrieve the most up-to-date list in the event of a new API instance.
apiVersion: v2
name: chart
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: '0.0.1'
dependencies:
- name: transform-and-deliver-assets
version: latest
repository: oci://registry-1.docker.io/bouyguestelecomcharts/tada
All releases : https://hub.docker.com/r/bouyguestelecom/tada/tags
transform-and-deliver-assets:
local: true
redis: ...
delegatedStorage: ...
s3: ...
mediaProvider: ...
rateLimit: ...
domain: .media
env: media-service
# SEE BELOW ALL VALUES
Clé | Description | Exemples de Valeurs |
---|---|---|
local | Activer ou désactiver le mode local | true / false |
redis.service | Nom du service Redis | 'redis-service' |
redis.dumpFolderPath | Chemin du dossier de dump pour Redis | '/dumps' |
redis.storage.storageClassName | Nom de la classe de stockage pour Redis | 'hostpath' |
redis.storage.resources.requests.storage | Espace de stockage demandé par Redis | '500Mi' |
delegatedStorage.rateLimitWindow | Fenêtre de limitation de débit (ms) | 30000 |
delegatedStorage.rateLimit | Limitation de débit | 5 |
delegatedStorage.host | Nom d'hôte du service personnalisé | 'your_custom_service_api' |
delegatedStorage.routes.readinessCheck | Chemin de vérification de disponibilité | '/readiness-check' |
delegatedStorage.accessToken | Jeton d'accès pour le stockage délégué | 'your_access_token' |
delegatedStorage.storageMethod | Méthode de stockage | 'DISTANT_BACKEND' |
s3.routes.readinessCheck | Chemin de vérification de disponibilité pour MinIO | '/minio/health/live' |
s3.endpoint | Point de terminaison (endpoint) pour S3/MinIO | 'minio' |
s3.port | Port pour S3/MinIO | '9000' |
s3.accessKey | Clé d'accès pour S3/MinIO | 'minioadmin' |
s3.secretKey | Clé secrète pour S3/MinIO | 'minioadmin' |
s3.bucketName | Nom du bucket S3/MinIO | 'media' |
s3.storage.storageClassName | Nom de la classe de stockage pour S3/MinIO | 'hostpath' |
s3.storage.resources.requests.storage | Espace de stockage demandé pour S3/MinIO | '500Mi' |
mediaProvider.service | URL du service média | 'http://media-service' |
mediaProvider.apiPrefix | Préfixe de l'API pour le service média | '/palpatine' |
mediaProvider.routes.healthcheck.get | Chemin de vérification de la santé du service média | '/readiness-check' |
mediaProvider.routes.file.get | Chemin GET pour récupérer des fichiers | '/assets/media/' |
mediaProvider.routes.file.post | Chemin POST pour télécharger un fichier | '/upload' |
mediaProvider.routes.files.post | Chemin POST pour télécharger plusieurs fichiers | '/uploads' |
mediaProvider.routes.catalog.get | Chemin GET pour récupérer le catalogue de fichiers | '/catalog' |
mediaProvider.payloadMaxSize | Taille maximale de la charge utile | '10mb' |
mediaProvider.rateLimit.windowMs | Fenêtre de limitation de débit (ms) | 30000 |
mediaProvider.rateLimit.limit | Limitation de débit | 5 |
mediaProvider.originsAllowed | Origines autorisées | 'localhost,*' |
mediaProvider.methodsAllowed | Méthodes HTTP autorisées | 'GET,POST' |
mediaProvider.storage.storageClassName | Nom de la classe de stockage pour le provider média | 'hostpath' |
mediaProvider.storage.resources.requests.storage | Espace de stockage demandé pour le media provider | '500Mi' |
rateLimit.windowMs | Fenêtre de limitation de débit (ms) | 30000 |
rateLimit.limit | Limitation de débit | 5 |
domain | Domaine du service | '.media' |
env | Environnement du service | 'media-service' |
NAMESPACES_ALLOWED | Espaces de noms autorisés | 'DEV' |
version | Version du chart | '1.0.6' |
You can use our docker image API from docker hub
docker pull bouyguestelecom/tada:api-latest
- docker
- kubernetes
Prerequisites :
- Make installed
- Docker
- Kubernetes
- Launch Makefile
make start
- Stop services
make stop
Prerequisites :
- Helm installed
- Docker
- Kubernetes
- Helm install
helm upgrade --install media-release opensource/. -f opensource/values.local.yaml
- Helm uninstall
helm uninstall media-release
Prerequisites :
- Docker compose
- Docker
- Build and run services (image docker api from docker hub)
docker compose up --build -d
- Build and run services locally
docker compose -f docker-compose.dev.yml up --build -d
- Stop services
docker compose down
Prerequisites:
- npm
- node
- Install NPM packages
npm install --prefix src/api/
- Run api
npm run dev:api
Use this space to show useful examples of how a project can be used.
For more examples, please refer to the Documentation
- Add Changelog
- Add waza
- Add waza 2
See the open issues for a full list of proposed features (and known issues).
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Distributed under the MIT License. See LICENSE.txt
for more information.
Maintainer name - [email protected]
Project Link: https://github.com/
Use this space to list resources you find helpful and would like to give credit to. I've included a few of my favorites to kick things off!