Providence/Pawtucket 2.0 : problem with ElasticSearch

Franck_Theeten

Dear support,

We are working at the AfricaMuseum in Tervuren, a Museum dealing with the colonial and postcolonial history of Belgium in Central Africa. We already use CollectiveAccess for a project dealing with heritage and provenance research (Afrisurge), but would like to test it for another biological project ,made with one our partner institution, the Royal Belgium institute for Natural Sciences.

We created a test instance with version 2.0 that we would like to run with ElasticSearch, but are still struggling when binding it to ElasticSearch. We imported ca. 350 objects but the ElasticSearch index remains empty .
The test server is on :
https://naturalheritage.africamuseum.be/providence/pawtucket/
and the ElasticSearch index (version 7) is
http://193.190.223.60:9200/collectiveaccess/_search

the search.conf file of Pawtucket contains the following lines:


# Set to 2 for version 2.x, 5 for 5.x, 6 for 6.x, 7 for 7.x
elasticsearch_version = 7

# Enter the elastic search base url here (without any index names)
search_elasticsearch_base_url = http://localhost:9200/

# This is the prefix of ElasticSearch indexes used by CollectiveAccess. An index will
# be create for each indexed collectiveAcccess table, named with this prefix + the table 
# name. (Eg. collectiveacces_ca_objects). You probably don't need to change this unless 
# you're using a single ElasticSearch setup for multiple CollectiveAccess instances and/or
# other applications.
search_elasticsearch_index_name = collectiveaccess

# ElasticSearch-specific ndexing buffer size. Larger values *may* improve performance
# but will use more memory.
elasticsearch_indexing_buffer_size = 250

Is there any other options we should also configure, or should we run a background task to synchronize data with ES ?

Best regards,

Franck Theeten

Project manager
Royal Museum for Central Africa
LEUVENSESTEENWEG 13, B-3080 TERVUREN
WWW.AFRICAMUSEUM.BE

seth

Recent versions of ElasticSearch have not been tested with the current release and may be broken. We do want to update the integration, but it's not on our current roadmap.

Franck_Theeten

Hi Seth,

We could finally have Pawtucket running with ES 7 and Memcached (but not with Redis as it can't initiate user sessions with it).
It seems it doesn't recognize the parameters form app/seach.conf as it was still creating ES indices with version 5 (default value).
I had to tweak ./app/lib/Plugins/SearchEngine/ElasticSearch/Mapping.php with if (!in_array($this->version, [2, 5, 6, 7], true)) { $this->version = 7; } instead of if (!in_array($this->version, [2, 5, 6, 7], true)) { $this->version = 7; } (line 83)

In my case, I had also to modify /app/helpers/utilityHelpers.php

if (function_exists('gzuncompress'))
		{
			$ps_uncompressed_data = @gzuncompress($ps_data); 
			return unserialize($ps_uncompressed_data);
		}

instead of

if (function_exists('gzuncompress') && ($ps_uncompressed_data = @gzuncompress($ps_data))) {
			return unserialize($ps_uncompressed_data);
		}

and /app/helpers/printHelpers.php use Laminas\Stdlib\Glob as Glob2; instead of use Laminas\Stdlib\Glob as Glob; and replacing the Glob prefix by Glob2 down in the code (probably because of a conflict between Magento/Zend and Laminas\Stdlib in the Composer file).

Pawtucket works, but on big collections (> 100 000 records) we observe very slow MariaDB queries, with an extensive list of foreign key values hardcoded in the WHERE clause, eg. (the list is truncated there but reaches several thousands of values

SELECT COUNT(distinct ca_attributes.row_id) as _count, ca_attribute_values.value_longtext1, ca_attribute_values.value_decimal1, ca_attribute_values.value_longtext2, ca_attribute_values.value_integer1, ca_attribute_values.element_id FROM ca_attributes INNER JOIN ca_attribute_values ON ca_attributes.attribute_id = ca_attribute_values.attribute_id INNER JOIN ca_objects ON ca_objects.object_id = ca_attributes.row_id AND ca_attributes.table_num = 57 WHERE ca_attribute_values.element_id = '49' AND ((ca_objects.object_id IN (5022,5023,5024,5025,5026,5027,5028,5029,5030,5031,5032,5033,5034,...)) AND (ca_objects.access IN (1)) AND (ca_objects.deleted = 0)) GROUP BY value_longtext1, value_decimal1, value_longtext2, value_integer1
Is it possible to fully enable the ES search in Pawtucket, and/or optimize this query, e.g by replacing the list by a subquery or a join?