Media identification process (aka problem loading a 12402x12402 2M JPG file)

theCA

I would like to know how to troubleshoot the issue we are having.
We are trying to load a file of 1.9Mb with 12402x12402 resolution and as soon you hit save in the form, ca just die (get secure connection failed in browser, NS_ERROR_NET_RESET in Firefox, ERR_EMPTY_RESPONSE in Chrome). No errors, no logs (even in apache).

I got the same file edited in theGimp, saved again and it got 4.9Mb and when i upload that file, the file is recognized as a binary 1000x1000.
If i see the app/log for that file looks like is not detected as a JPG due the fact i see calls to dcraw, mediainfo, ffmpeg, libreoffice, "gm identify", identify, pdftotext, pdfinfo.
If I run in the same server identify, file or any other comand i get image/jpg or JPEG like it should.

I want to understand how ca identifies the file, and try to track where the error is.
My Guess is the very first stage when it should be identified as a JPEG is failing and i would like to understand best steps to troubleshoot. Which plugin/id process comes first in the list, etc.
Im on Providence 1.7.16 in ubuntu 20.04

seth

Please try uploading your file on the demo site at https://demo.collectiveaccess.org and see if it works there. This will at least help to narrow down whether the issue is with the file or (CA and/or your particular set up)

theCA

It works there, but i need to troubleshoot in my installation. De demo site is 2.0 (dev/php8) and im on stable 1.7.17.
The file looks, fine. I ran the same commands i saw in the logs and im getting JPEG, thats why im trying to figure out in which step identification is failing...

seth

You're running GraphicsMagick with the gmagick PHP extension? Or something else?

theCA

seth
This are my plugins

GD
Provides limited image processing and conversion services using libGD
Didn't load because ImageMagick (command-line) is available and preferred
Didn't load because Gmagick is available and preferred
Didn't load because GraphicsMagick is available and preferred
Not used
Gmagick
Provides image processing and conversion services using GraphicsMagick via the PECL Gmagick PHP extension
Available
GraphicsMagick
Provides image processing and conversion services using GraphicsMagick via exec() calls
Didn't load because Gmagick is available and preferred
Not used
ImageMagick
Provides image processing and conversion services using ImageMagick via exec() calls to ImageMagick binaries
Didn't load because GraphicsMagick is available and preferred
Didn't load because Gmagick is available and preferred
Not used
Imagick
Provides image processing and conversion services using ImageMagick via the PECL Imagick PHP extension
Didn't load because Imagick is not available
Didn't load because Gmagick is available and preferred
Not used
Mesh
Accepts files describing 3D models
Available
Office
Accepts and processes Microsoft Word, Excel and PowerPoint format documents
Available
PDFWand
Provides PDF conversion services using ImageMagick or the Zend_PDF library. Will use Ghostscript to generate image-previews of PDF files.
PDFMiner cannot be found: indexing of text locations in PDF files will not be performed; you can obtain PDFMiner at http://www.unixuser.org/~euske/python/pdfminer/index.html
Available
QuicktimeVR
Provides services for processing of QuicktimeVR files
Available
Spin360
Accepts ZIP archives containing 360 spinnable images in SpinCar format (http://SpinCar.com)
Available
Video
Provides ffmpeg-based video processing
MediaInfo will be used to extract metadata from video files.
Available
XMLDoc
Accepts and processes XML-format documents
Available

Im assuming it must be used primarily Gmagick

robc

You need to check the PHP error log file. A 144 megapixel image is going to require gigabytes of memory to process and most PHP installations are not configured for that scenario.

Workarounds include resizing the image prior to upload or using a ZIP binary. It is rare to need that many megaxpixels displayed on a website.

theCA

robc
I Will check, weird thing is resolution is high but the file is not so big, only 1.9Mb.

php.ini
memory_limit = 1024M

robc

It's not weird, it's a high compression ratio and could easily generate a tile set that would blow out 1024 M of ram and disk space, depending on settings.

Small JPG files with huge dimensions generally don't have any optical value to lose by resizing to 25% or less of the original dimensions. That will help more than anything.

I've had similar issues and brainstormed about it in 1 or 2 other threads.

theCA

robc Small JPG files with huge dimensions generally don't have any optical value to lose by resizing to 25% or less of the original dimensions.

I agree with this.

theCA

I followed the recommendations of @robc and saw in syslog the out of memory as he predicted.

Out of memory: Killed process 240031 (apache2) total-vm:1756952kB, anon-rss:959840kB, file-rss:2572kB

The VM has 2Gb RAM, i have a limitation for php of 1Gb. And as far i understand the total-vm value means around 1.6Gb. I guess i should increase the RAM and set memory_limit to 2048M...

Question for @seth: how much RAM the demo site has?

Now I need to understand what is going wrong with the "identification" of the second image which is not blowing the apache process but is still recognized as binary...

theCA

Today we doubled the RAM to 4GB, and apache keeps getting OOM.
I guess my best option is to try to troubleshoot the image identification problem.