Thumbnail import compression setting

[benbakelaar]

Hi, I read the details here already:
https://omeka.org/s/docs/developer/configuration/config_reference/#thumbnails

I thought I recalled there was a way to set the ImageMagick compression level - maybe it was in Omeka Classic, but even for that I’m not finding anything while searching.

We have 240k JPEG images that were already compressed and don’t need to be further compressed - if they do, major artifacting happens.

Is there somewhere in the local.config.php that I can tell it to leave it at 100% quality?

Here’s my current config, almost default but I did try to add the “strategy” setting since when I just tested, it was display square images on “full” view in the UI.

return [
    'logger' => [
        'log' => true,
        'priority' => \Zend\Log\Logger::INFO,
    ],
    'http_client' => [
        'sslcapath' => null,
        'sslcafile' => null,
    ],
    'cli' => [
	'phpcli_path' => null,
    ],
    'thumbnails' => [
        'types' => [
            'large' => ['strategy' => default, 'constraint' => 1200],
            'medium' => ['strategy' => default, 'constraint' => 400],
            'square' => ['strategy' => square, 'constraint' => 400],
        ],
	'thumbnailer_options' => [
            'imagemagick_dir' => null,
        ],
    ],
    'translator' => [
        'locale' => 'en_US',
    ],
    'service_manager' => [
        'aliases' => [
            'Omeka\File\Store' => 'Omeka\File\Store\Local',
            'Omeka\File\Thumbnailer' => 'Omeka\File\Thumbnailer\ImageMagick',
        ],
    ],
];

[jflatnes]

I don’t recall there ever having been an option for JPEG quality level in either S or Classic. It’s, to my recollection, a simple flag to ImageMagick, so it certainly could be added (or hacked in).

Do you have an example of a source image and the bad artifacting you’re seeing from S’s thumbnailing (simply a link to an item works, if you have any that are visible)? I’m wondering if there are other settings we can alter or improve here.

[benbakelaar]

I ended up finding the code that calls ImageMagick by grepping for convert and adding in the quality flag.

./application/src/File/Thumbnailer/ImageMagick.php

I’ll get the example images, don’t have it handy - but in my experience ImageMagick is using a compression setting of ~92-93 by default. Also still having an issue where when I import media via URL, the full image is coming out as a square crop, regardless of the settings in local.config.php.

    public function create($strategy, $constraint, array $options = [])
    {
        $origPath = sprintf('%s[%s]', $this->source, $this->getOption('page', 0));

        switch ($strategy) {
	    case 'square':
                $gravity = isset($options['gravity']) ? $options['gravity'] : 'center';
                $args = [
                    '-quality 100',
                    '-background white',
                    '+repage',
                    '-alpha remove',
                    '-thumbnail ' . escapeshellarg(sprintf('%sx%s^', $constraint, $constraint)),
                    '-gravity ' .  escapeshellarg($gravity),
                    '-crop ' . escapeshellarg(sprintf('%sx%s+0+0', $constraint, $constraint)),
                ];
                break;
            case 'default':
	    default:
                $args = [
                    '-quality 100',
                    '-background white',
                    '+repage',
                    '-alpha remove',
                    '-thumbnail ' . escapeshellarg(sprintf('%sx%s>', $constraint, $constraint)),
                ];
        }

[benbakelaar]

Ah and here’s the default setting.

ImageMagick

For the JPEG and MPEG image formats, quality is 1 (lowest image quality and highest compression) to 100 (best quality but least effective compression). The default is to use the estimated quality of your input image if it can be determined, otherwise 92.

[jflatnes]

Omeka S always makes both square-cropped and preserved-aspect-ratio thumbnails of images. I would assume it’s just that your theme or the admin or whatever you’re looking at just happens to be asking for the square ones.

[benbakelaar]

It’s definitely possible, but I did inspect the source so it was a square image inside of /full. Very weird, haven’t had it happen before.

So the same theme with the original media table displays everything fine. If I swap it out for a blank table, then run the import - I get the squares.

Basically we are trying to unhack the media database, we had manually populated the SQL table with entries pointing to the local files that already exist. Now we want them to go through the full import process because editors will be adding new images (or replacing/adjusting) in the future.

[jflatnes]

So just to be clear, the URL for these square images that you’re seeing: it should say what kind of image it is. Does it say “square” in the URL, or, “large,” or something else?

[benbakelaar]

Yup they are in the sub-folders /medium and /large - and square.

Just took some before/after screenshots and placed them in this Google Drive folder:
https://drive.google.com/open?id=13KhxgDnD3O_GfIPnm48nH3vKIVTgL2wo

FWIW, when importing a collection of 240k images into an Omeka system - I wish I had a bit more finer-grained control of the thumbnailing process FileSideload is really a one-by-one image loader, and using CSV Import, once you trigger the process - you’d hate to find out at the end that it wasn’t what you expected! Of course, doing small test imports can also control for that.

As an example, in this case I would tell the thumbnailer that the original = large (because it’s already been compressed, resized for web, etc.), so don’t thumbnail it at all (or duplicate it for that matter) - but DO generate medium and square thumbnails for the purposes of admin, image previews, etc. - at 0% compression / 100% quailty.

[jflatnes]

You said you added those “strategy” keys after you noticed things were coming out square, right? I notice that the config you’ve got is missing the quotation marks that should be there around 'square' and 'default', but I’m not sure that would account for what you seem to be seeing.

I don’t think we’ve ever had any reports of this happening: squared-off thumbnails being created for the “large” and “medium” sizes.

You mentioned the problem being only with things loaded by URL. Is that true? The thumbnail system works identically for URL-loaded, file sideloaded, or direct uploaded images, so that really should be making no difference.

I’m left wondering if perhaps some leftover element of the customizations you are/were using for images is causing this problem.

[benbakelaar]

Yup I fixed those quotes! No worries, I’ll keep experimenting - I had never come across it either. It’s a fresh upgrade from v1.4.0 to v2.1.0, removed all old modules and re-installed new ones.

[benbakelaar]

OK figured it out. Inside the Omeka-S install, I had the file directories symlinked to their original sources - this is the “hack” I had used for both Omeka Classic and Omeka-S to just maintain a single copy of the images (238k of them) on the server.

As to exactly why the import resulted in the description above, not sure. But once I took out the symlinks, it worked as expected.

I’ll be posting/sharing my modified ImageMagick.php, I do see a need for a “skip” option for certain sizes - considering DerivativeImages could do the trick after the initial import of the original images. And in the perhaps unusual case of the files that are being imported already being on the server - it might be useful to have CSV Import have an option under Media to import via “absolute file path” and just use a “copy” command. I had thought that was what FileSideload did, but that’s more about bypassing the server’s “max_upload” parameters with a smaller quantity of files.

[jflatnes]

FileSideload does basically do a “copy” to bring over files. What is it doing that you would prefer it didn’t (or vice-versa)?

[benbakelaar]

Well I might be wrong, but I believe FileSideload requires you to individually attach files from a drop-down to an item? You can upload files to a server location, tell FileSideload where they are, and then it populates a drop-down with all files in that folder. Then you have to go into an individual item and choose an individual filename from the drop-down to attach - so it’s one-by-one.

If there is some bulk “sideload” aspect of it that I missed, that would be great. But if the above is accurate, it can’t work for even 100 images being attached (via CSV Import > Append) to multiple items.

[jflatnes]

You can use File Sideload together with CSV Import. When they’re both installed there’s a “Sideload” choice for Media source when mapping columns.

[benbakelaar]

Ahh OK… I guess I did not notice it when I installed it - and when I checked the documentation, it didn’t mention it either!

So I can have a column in my CSV that has a path similar to the documentation? The documentation shows /websites/omekadev/home/www - so this is the absolute server path (relative to root)?

…

OK, now I see this document:
https://omeka.org/s/docs/user-manual/modules/csvimport/#csv-import-with-other-modules

So I see you just put the filename, and it uses the module config to find the directory. But I do have one other problem The images aren’t in a single directory. They are divided into sub-directories, like /av, /xv, /xy, etc. In the “filename” column, do you think I can just give it a filename like “av/av1928.jpg”?

[jflatnes]

Paths like that with subdirectories included should work fine with the Sideload+CSV combination. You can either give the path as just the path portion underneath the sideload directory, or as an absolute local-server path (but it still must resolve to a path that’s within the sideload directory).

[benbakelaar]

Would it be possible to incorporate two UI-enabled settings for import into a future release?

1. Global admin - “disable thumbnail creation” (core)
2. Global admin - “ImageMagick quality (compression)” (core - default is “unset” which is 92)

And actually, a 3rd:
3. On import - “disable thumbnail creation” (CSV Import)

If yes, where exactly should I file these requests on Github?

[jflatnes]

You can file requests like that as Github issues… the first two would go to the main omeka/omeka-s project, the third to CSV Import.

I’m a little leery to make it so easy to disable the thumbnailing, simply because going back and adding them requires a module. There’s some value in making that kind of setting change a little more deliberate.

[benbakelaar]

@jflatnes

I’m a little leery to make it so easy to disable the thumbnailing, simply because going back and adding them requires a module.

How about an advanced setting to choose the Thumbnailer option from a drop-down? From the list Daniel sent me in the other thread

omeka/omeka-s/blob/develop/application/config/module.config.php#L261