recommendations for large media import (lots of JPGs)?

nobody

Hello,

We've been using caUtils --import-media over the years to import a large set of images into providence. The image set has grown to ₆₀,000 JPGs and we were wondering if there are recommendations for a more efficient import process?

We saw discussion of imports via TUS a few years back - https://support.collectiveaccess.org/d/300715-300715 - is this still supported and is their any documentation for this method?

Previously, we have had the images split into directories: one directory per item with anywhere from 1-500 JPGs inside (the average folder has ₄₀). Running the import on the full set of JPGs/directories took several days.

If I remember correctly the import logs showed a regex check running across ever sub-directory and file in /media matching on each entry in the import mapping (which seemed to slow things down checking 60,000 files, 60,000 times). We experimented with using the mediaPrefix parameter in the mapping to try and limit the regex check to each subdirectory, but didn't have any luck as described in https://github.com/collectiveaccess/providence/issues/1812

I just noticed caUtils import-media has the --match-mode option which I will try to test as well:

--match-mode             Determines how matches are made between media and records. Valid values are
                                 DIRECTORY_NAME, FILE_AND_DIRECTORY_NAMES, FILE_NAME. Set to DIRECTORY_NAME
                                 to match media directory names to target record identifiers; to
                                 FILE_AND_DIRECTORY_NAMES to match on both file and directory names; to
                                 FILE_NAME to match only on file names. Default is FILE_NAME.

Any suggestions are greatly appreciated and many thanks again.

seth

How large are these files? And what software are you using to process them? ImageMagick or GraphicsMagick? And for either option, are you using the command-line integration or the PHP extension?

nobody

seth hi seth, roughly i think the jpgs are 6 to 12 MB each, can get more exact numbers this evening. i believe we've been using the gmagick php extension.

seth

What are the rough specs of the machine(s) you're running this on?

nobody

fairly modest, a 2014 macmini with a 3.0ghz i7 and 16gb ram - https://everymac.com/systems/apple/mac_mini/specs/mac-mini-core-i7-3.0-late-2014-specs.html

/media and /import are symlinked to a USB 3.0 external drive. we've tried imports with /import symlinked to a different USB 3.0 drive with no real difference in speed.

seth

What do you think the bottleneck is? The regular expression or the 12 year old entry-level machine? Or something else?

nobody

I'm not sure, if it's the specs of the test server then I completely understand. If the above is the recommended method for importing media I don't mind the wait, I was mostly curious if I might be doing something wrong or misunderstanding the options.

seth

I guess I'm just asking where you think the time is going? I mean, processing 60,000 images does take time. A 12mb JPEG is actually quite a large file. Decompressed it will take many times more memory to work with. With 16gb of RAM you may well be swapping things out to virtual memory every time you open one of these files. Or maybe not.

Can you try processing an image, resizing it from original size to, say, a 2000px x 2000px JPEG on the command line of your machine using ImageMagick or GraphicsMagick and see how long that takes? From that number you can roughly extrapolate how long 60,000 images might take to run on the machine you have. If the speed on the command line vs. how long it's taking in CA widely diverge then it's worth looking closely at what CA is doing (keeping in mind it's generating several different sizes + a zoomable version, so it's definitely going to take longer than running a single size). If not, then maybe it's worth looking at server software configuration and/or hardware.

nobody

thanks for the info, that makes a lot of sense. i will run the tests as soon as possible and report back, hopefully in the next day or so.

nobody

Here's what I've tried so far: I pulled 95 4000px x 6000px JPGs from the image set (I just grabbed a few random object folders, next time I will use a rounder number). The 95 files have an average file size of 7.98 MB and are stored on the same USB 3.0 external drive as /media. The files are divided into 9 subdirectories by related object.

In the parent directory I ran time find . -type f -exec sh -c 'for i; do gm convert "$i" -resize 2000x2000 "${i}-resized.jpg"; done' _ {} + with the results:

real	0m54.059s

While the conversion processes ran I saw about 300-500 MB jumps in RAM usage with watch free -m with around 13-14 GB RAM free and no swap usage (i don't imagine this would be the same for a CA import, just trying to report relevant details from the test).

Rough, hopefully reasonable, calculations:

JPGs/minute: 105.44 - (60/54.059)*95 rounded down.
Estimated time for 40,000 JPGs at 105.44 per min - ~6h20m - (40,000/105.44)/60

I don't have the logs on hand but i think the last successful CA import (~40,000 files) took 4-5 days to finish. I'm not sure how the test numbers compare with CA running multiple conversions on each file. Glad to do further testing if it would help clarify things further.

I can try a variety of CA test imports and report back as well.

p.s. if relevant the 95 test JPGs rounded size distribution is below. i will try to get some better stats about the full image set soon, if that would be helpful.:

4M - 1
5M - 3
6M - 16
7M - 31
8M - 19
9M - 18
11M - 4
12M - 2
13M - 1

many thanks again

nobody

a few more tests: i ran an import of the same 95 images as above in a test instance of the current /dev/2.0 branch. I apparently misremembered and we have been using caUtils import-data not caUtils import-media. the import took 5m24s - ~3.4 sec/jpg.

to test the possible effect of the regex i deleted the imported object representations in CA, created 40,000 empty files with a .jpg extension in a directory alongside the directories containing the original images in the import folder, then ran the same import again. this bumped the import time up to 8m - ~5 sec/jpg.

to try and rule out variables in the test, i deleted all object representations again, removed the 40k empty .jpg files, and ran the import one more time. it completed in about the same time as the first test, 5m22s.

i haven't used ./caUtils --media-import before and will see if I can figure out how to use and test it as well. i hope this helps clarify things, i can do further testing or provide further details if needed.

95 JPG import:

$ time sudo -u nginx ./caUtils import-data --format=XLSX --mapping=media-import-test --source=../../import/import-media.xlsx -l /tmp -d DEBUG

CollectiveAccess 2.0 (213/GIT) Utilities
(c) 2013-2026 Whirl-i-Gig

Importing TEST7 [30.00M]                       100.0% 95/95 ETC: -. Elapsed: 05m:20s [▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩]
Imported data from source ../../import/import-media.xlsx

real	5m24.089s
user	0m0.008s
sys	0m0.007s

95 JPG import w/ 40,000 other files in import folder:

$ time sudo -u nginx ./caUtils import-data --format=XLSX --mapping=mediaimporttest --source=../../import/import-media.xlsx -l /tmp -d DEBUG

CollectiveAccess 2.0 (213/GIT) Utilities
(c) 2013-2026 Whirl-i-Gig

Importing TEST7 [36.00M]                        100.0% 95/95 ETC: -. Elapsed: 07m:51s [▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩▩]
Imported data from source ../../import/import-media.xlsx

real	8m1.014s
user	0m0.041s
sys	0m0.026s

MicahE

@nobody You mentioned these drives are external drives. When I see external drives paired with slow file listing, it makes me wonder if they're formatted using the exFAT file system, which doesn't have indexing for file names, and is therefore really slow at this kind of thing. If that's the case, you might have much better luck by keeping your import folder on another file system.

nobody

hi micah, thank you. we have formatted the drives as ext4 and xfs for imports in the past. but you're right, the drive for the tests above is exfat. will try testing again and comparing performance on the other filesystems.

a small update: we went ahead with the full import last weekend. The file count was actually 51,987 JPGs (average file size around 8 MB) and the import took about 81.8 hours (294,598s) or 5.667 s/jpg, similar to the 2nd test above.