"Image is being processed" but nothing uploads

alcoma

Hello,
Recently when trying to upload media representations we get the prompt "Image is being processed" but nothing appears to be uploaded. I can see the info for the file; name, size etc. but nothing loads when you click on the image link. We just migrated over to a new server and this is the first issue I've encountered. The configuration check shows all the necessary media apps available so I'm not sure where to look.

monica

alcoma It has probably uploaded, but in the queue to be processed.

What version are you on and did you upgrade when you migrated? Check the background processing settings in app.conf

alcoma

We're on vs. 2, schema rev. 197. Task queue is enabled. In the past sometimes just re-saving the representation would load the image up. Even clicking on the link "Image is being processed" would open the image in the viewer at least but now nothing is there though the metadata in the representation says it is ? (Our server transfer was from a vps to dedicated, same op system etc. so it was a pretty straight forward transition thankfully!)

monica

alcoma You old system may have had a cronjob running to process the task queue every minute or so and when you moved no one re-created the cronjob.

If you run caUtils process-task-queue manually and it take a little while and afterwards your images show up, then this is what is happening.

alcoma

No, we never had a cron job set up. In fact, task queue was only recently enabled because of my post (and your help) here: https://support.collectiveaccess.org/d/301881-best-pdf-renderer.

I did run the caUtils to process-task-queue and went back to the problem record and the image was loaded.
So if background processing ran before without a cron job, what could keep it from running now ? Maybe something was lost in the transition. Would any sql or php settings affect this ?

monica

what are your settings in app.conf for background tasks?

alcoma

I can't find anything in app.conf for 'background tasks' but see Task Queues ?

Task queue set up (deferred processing of uploaded media)

-----------------------------------

taskqueue_handler_plugins = <ca_lib_dir>/Plugins/TaskQueueHandlers
taskqueue_tmp_directory = <ca_app_dir>/tmp
taskqueue_max_opo_processes = 4
taskqueue_process_timeout = 3600
taskqueue_max_items_processed_per_session = 100

monica

alcoma what about these settings? https://github.com/collectiveaccess/providence/blob/424f2177f911cbb0598efc8a153da617399f2108/app/conf/app.conf#L926

alcoma

Ah, I realized our 'local' copy of app.conf was an older one that didn't have that section, so that's why I couldn't find it before.
I copied the newer default version of app.conf into the local folder. The settings for background processing are the same as in the link above.

monica

alcoma Can you double check the CA logs to see if there is anything in there about why the background processing isn't being triggered. Your system might not support the process mode, so you'd need to use socket or disable and use a cronjob instead.

BTW: Any setting that is not set in your local configuration will use that in the default configuration, so you shouldn't need to copy all that over to your local conf.

alcoma

I found 2 logs; 1 has 'warnings' (below) and the other reg log from today says:

2025-02-14 12:20:57 - ERROR --> ApplicationException: Invalid controller path
2025-02-14 12:27:11 - DEBUG --> Executing command: '/usr/bin/ffmpeg> /dev/null 2>&1'
2025-02-14 12:27:11 - DEBUG --> Executing command: '/usr/bin/mediainfo --Help > /dev/null'
2025-02-14 12:27:11 - DEBUG --> Executing command: '/usr/bin/dcraw -i 2> /dev/null'
2025-02-14 12:27:11 - DEBUG --> Executing command: '/usr/bin/gs -v 2> /dev/null'
2025-02-14 12:27:11 - DEBUG --> Executing command: '/usr/bin/gm 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/ffmpeg> /dev/null 2>&1'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/mediainfo --Help > /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/gm 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/dcraw -i 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/dcraw -i 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/gm 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/dcraw -i 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/gs -v 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/gm 2> /dev/null'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/ffmpeg> /dev/null 2>&1'
2025-02-14 12:34:13 - DEBUG --> Executing command: '/usr/bin/mediainfo --Help > /dev/null'
2025-02-14 12:34:14 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 12:34:14 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 12:34:14 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 12:34:14 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 12:34:14 - DEBUG --> Executing command: '/usr/bin/exiftool > /dev/null'
2025-02-14 12:34:15 - DEBUG --> Executing command: '/usr/bin/mediainfo --Help > /dev/null'
2025-02-14 15:45:29 - DEBUG --> Executing command: '/usr/bin/ffmpeg> /dev/null 2>&1'
2025-02-14 15:45:29 - DEBUG --> Executing command: '/usr/bin/mediainfo --Help > /dev/null'
2025-02-14 15:45:29 - DEBUG --> Executing command: '/usr/bin/dcraw -i 2> /dev/null'
2025-02-14 15:45:29 - DEBUG --> Executing command: '/usr/bin/dcraw -i 2> /dev/null'
2025-02-14 15:45:29 - DEBUG --> Executing command: '/usr/bin/gs -v 2> /dev/null'
2025-02-14 15:45:29 - DEBUG --> Executing command: '/usr/bin/gm 2> /dev/null'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/bin/ffmpeg> /dev/null 2>&1'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/bin/mediainfo --Help > /dev/null'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/local/bin/wkhtmltopdf > /dev/null 2> /dev/null'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/bin/exiftool > /dev/null'
2025-02-14 15:45:30 - DEBUG --> Executing command: '/usr/bin/mediainfo --Help > /dev/null'
2025-02-14 16:18:15 - DEBUG --> Executing command: '/usr/bin/dcraw -i '/home/b3a24e5/public_html/pawtucket/ca/uploads/_Archivist/ca-wysch-08111972_ltr.jpg' 2> /dev/null'
2025-02-14 16:18:15 - DEBUG --> Executing command: '/usr/bin/exiftool -json -a -u -g1 '/home/b3a24e5/public_html/pawtucket/ca/uploads/_Archivist/ca-wysch-08111972_ltr.jpg' 2> /dev/null'

The WARNINGS log contains:
2025-02-14 12:20:55 - WARN --> Undefined variable $filter_expr [/public_html/pawtucket/ca/app/models/ca_lists.php:603]
2025-02-14 12:20:55 - WARN --> Undefined array key "user_preferences" [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:108]
2025-02-14 12:20:55 - WARN --> Trying to access array offset on value of type null [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:108]
2025-02-14 12:20:55 - WARN --> Undefined array key "choiceList" [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:113]
2025-02-14 12:20:55 - WARN --> Undefined variable $va_profile_prefs [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:128]
2025-02-14 12:26:49 - WARN --> Undefined variable $filter_expr [/public_html/pawtucket/ca/app/models/ca_lists.php:603]
2025-02-14 12:27:07 - WARN --> Undefined variable $filter_expr [/public_html/pawtucket/ca/app/models/ca_lists.php:603]
2025-02-14 12:27:41 - WARN --> Undefined variable $elements [/public_html/pawtucket/ca/app/lib/IDNumbering/IDNumber.php:374]
2025-02-14 16:17:38 - WARN --> Undefined variable $type_id [/public_html/pawtucket/ca/app/helpers/navigationHelpers.php:1020]
2025-02-14 16:17:52 - WARN --> Undefined variable $filter_expr [/public_html/pawtucket/ca/app/models/ca_lists.php:603]
2025-02-14 16:19:04 - WARN --> Undefined array key "HEIGHT" [/public_html/pawtucket/ca/themes/default/views/editor/objects/summary_html.php:51]
2025-02-14 16:19:04 - WARN --> Undefined variable $type_id [/public_html/pawtucket/ca/app/helpers/navigationHelpers.php:1020]
2025-02-14 16:19:45 - WARN --> Undefined variable $va_return_values [/public_html/pawtucket/ca/app/lib/Search/SearchResult.php:2921]
2025-02-14 16:19:50 - WARN --> Undefined variable $type_id [/public_html/pawtucket/ca/app/helpers/navigationHelpers.php:1020]
2025-02-14 16:19:50 - WARN --> Undefined array key "list_format" [/public_html/pawtucket/ca/themes/default/views/bundles/ca_occurrences.php:111]
2025-02-14 16:19:50 - WARN --> Undefined array key "list_format" [/public_html/pawtucket/ca/themes/default/views/bundles/ca_objects.php:173]
2025-02-14 16:19:50 - WARN --> Undefined variable $filter_expr [/public_html/pawtucket/ca/app/models/ca_lists.php:603]
2025-02-14 16:26:05 - WARN --> Undefined array key "user_preferences" [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:108]
2025-02-14 16:26:05 - WARN --> Trying to access array offset on value of type null [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:108]
2025-02-14 16:26:05 - WARN --> Undefined array key "choiceList" [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:113]
2025-02-14 16:26:05 - WARN --> Undefined variable $va_profile_prefs [/public_html/pawtucket/ca/app/widgets/recentRegistrations/recentRegistrationsWidget.php:128]

monica

@seth might need to look at this one. The best I can suggest is to set up the cronjob to process the task queue every 1-5 minutes.

alcoma

How do I set one up ?

monica

alcoma
Do you have admin rights to the server it's running on?

If yes:
Place a file in /etc/cron.d/ called whatever, I used process-task-queue. In this file place the following line:
* * * * * www-data /var/www/providence/support/bin/caUtils process-task-queue

If no:
Run the command crontab -e to open a text editor. Once again, put the following line in this file
* * * * * www-data /var/www/providence/support/bin/caUtils process-task-queue and save the file.

This should run the command every minute.