Some conversions are not generated

[chimit]

I use Redis queues (via Horizon) to generate conversions (small, medium, and large). Images are stored in S3 (DigitalOcean Spaces). Everything works fine but sometimes conversions are not generated. And there are no clues because in Horizon such jobs look successful:

One of the skipped examples:

{
    "conversions": {
        "items": [
            {
                "fileNamer": {},
                "extractVideoFrameAtSecond": 0,
                "manipulations": {
                    "manipulationSequence": {
                        "groups": [
                            {
                                "optimize": "{"Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim":["-m85","--force","--strip-all","--all-progressive"],"Spatie\\ImageOptimizer\\Optimizers\\Pngquant":["--force"],"Spatie\\ImageOptimizer\\Optimizers\\Optipng":["-i0","-o2","-quiet"],"Spatie\\ImageOptimizer\\Optimizers\\Svgo":["--disable=cleanupIDs"],"Spatie\\ImageOptimizer\\Optimizers\\Gifsicle":["-b","-O3"],"Spatie\\ImageOptimizer\\Optimizers\\Cwebp":["-m 6","-pass 10","-mt","-q 90"]}",
                                "format": "jpg",
                                "width": "600",
                                "height": "600",
                                "fit": "max"
                            }
                        ]
                    }
                },
                "performOnCollections": [],
                "performOnQueue": true,
                "keepOriginalImageFormat": true,
                "generateResponsiveImages": false,
                "loadingAttributeValue": null,
                "pdfPageNumber": 1,
                "name": "small"
            },
            {
                "fileNamer": {},
                "extractVideoFrameAtSecond": 0,
                "manipulations": {
                    "manipulationSequence": {
                        "groups": [
                            {
                                "optimize": "{"Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim":["-m85","--force","--strip-all","--all-progressive"],"Spatie\\ImageOptimizer\\Optimizers\\Pngquant":["--force"],"Spatie\\ImageOptimizer\\Optimizers\\Optipng":["-i0","-o2","-quiet"],"Spatie\\ImageOptimizer\\Optimizers\\Svgo":["--disable=cleanupIDs"],"Spatie\\ImageOptimizer\\Optimizers\\Gifsicle":["-b","-O3"],"Spatie\\ImageOptimizer\\Optimizers\\Cwebp":["-m 6","-pass 10","-mt","-q 90"]}",
                                "format": "jpg",
                                "width": "960",
                                "height": "960",
                                "fit": "max",
                                "watermark": "/home/example/example.com/public/images/watermark.png",
                                "watermarkOpacity": "50",
                                "watermarkWidth": "10w",
                                "watermarkPaddingX": "18",
                                "watermarkPaddingY": "18"
                            }
                        ]
                    }
                },
                "performOnCollections": [],
                "performOnQueue": true,
                "keepOriginalImageFormat": true,
                "generateResponsiveImages": false,
                "loadingAttributeValue": null,
                "pdfPageNumber": 1,
                "name": "medium"
            },
            {
                "fileNamer": {},
                "extractVideoFrameAtSecond": 0,
                "manipulations": {
                    "manipulationSequence": {
                        "groups": [
                            {
                                "optimize": "{"Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim":["-m85","--force","--strip-all","--all-progressive"],"Spatie\\ImageOptimizer\\Optimizers\\Pngquant":["--force"],"Spatie\\ImageOptimizer\\Optimizers\\Optipng":["-i0","-o2","-quiet"],"Spatie\\ImageOptimizer\\Optimizers\\Svgo":["--disable=cleanupIDs"],"Spatie\\ImageOptimizer\\Optimizers\\Gifsicle":["-b","-O3"],"Spatie\\ImageOptimizer\\Optimizers\\Cwebp":["-m 6","-pass 10","-mt","-q 90"]}",
                                "format": "jpg",
                                "width": "1280",
                                "height": "1280",
                                "fit": "max",
                                "watermark": "/home/example/example.com/public/images/watermark.png",
                                "watermarkOpacity": "50",
                                "watermarkWidth": "10w",
                                "watermarkPaddingX": "24",
                                "watermarkPaddingY": "24"
                            }
                        ]
                    }
                },
                "performOnCollections": [],
                "performOnQueue": true,
                "keepOriginalImageFormat": true,
                "generateResponsiveImages": false,
                "loadingAttributeValue": null,
                "pdfPageNumber": 1,
                "name": "large"
            }
        ],
        "escapeWhenCastingToString": false
    },
    "media": {
        "class": "Spatie\MediaLibrary\MediaCollections\Models\Media",
        "id": 21992,
        "relations": [],
        "connection": "mysql",
        "collectionClass": null
    },
    "onlyMissing": false,
    "connection": "redis",
    "queue": ""
}

In DO Spaces there is one original file without conversions.

The only suspicious thing here is the duration of the job - it's too short comparing other successful conversions. So I can only guess something wrong happens during the conversion and medialibrary is unaware of these failures, marking such jobs as done. Or medialibrary for some reason mistakenly thinks there is nothing to convert for some images and skips the job.

Running php artisan media-library:regenerate --only-missing successfully regenerates such missing conversions.

[FlashWS]

Absolutely similar situation
PHP 8.2.7
spatie/laravel-medialibrary 10.10.1

[Cigoler]

Have you installed the file convertors on your server?

sudo apt install jpegoptim optipng pngquant gifsicle
npm install -g svgo

[chimit]

@Cigoler yes, I have them.

[chimit]

The generated_conversions field is empty for such images if it helps in investigation.

INSERT INTO `media` (`id`, `model_type`, `model_id`, `uuid`, `collection_name`, `name`, `file_name`, `mime_type`, `disk`, `conversions_disk`, `size`, `manipulations`, `custom_properties`, `generated_conversions`, `responsive_images`, `order_column`, `created_at`, `updated_at`)
VALUES
	(26243, 'App\\Models\\Listing', 443, '2c41472b-fd00-4e76-ba9d-d9b035492339', 'shared', 'photo', 'photo.jpeg', 'image/jpeg', 'media', 'media', 12818732, '[]', '[]', '[]', '[]', 1, '2023-08-21 05:16:31', '2023-08-21 05:16:52');

[abdelwahidJobs]

Same here using php 7.4 and ^8.7 not all but some photos are not generated
Can anyone help ?

[freekmurze]

Since there aren't too many complaining about this, I'm assuming it's caused by something at your end. Moving this to a discussion.

[chimit]

I guess just not too many people use S3/DO Spaces. The bug is confirmed by 3 people. I provided all possible information. What else can I do to prove that it's a bug?

[kb6673]

Did anyone solve this? I am also experiencing missing conversions
.

[EpocDotFr]

Also experiencing this issue with millions of images stored on S3. The vast majority of conversions are generated, but a small portion still don't because of an unknown issue, extremely similar to what is talked about in this discussion.

Just adding my 2 cents: generated_conversions may also be empty even if conversions have been generated.

[chimit]

This is clearly an issue. Can it be moved back to issues, please? @freekmurze

[shortcodes]

A also have exacly same issue. S3 and sometimes when Completed PerforConversionsJob lasted suspiciously small amount of time conversions are just not being generated which i think is logical

[shortcodes]

I also created a command which redispatch those allegedly completed thinking that maybe something in payload was build wrong.

But all redispatched jobs did onversions correctly (100%)

There must an issue somewhere. Transfer? Timeout? Async? Maybe someone will figure it out