ElasticSearch Questions

sjenson

I was able to get the ElasticSearch plug in working in a test database; however, I ended up using version 2.4.6 rather 6.0, and I had to change "max_indexing_buffer_size" field in the search.conf file from 500,000 to 50,000. (I also set ElasticSearch not to upgrade so I don't end up with a broken installation). Has anyone been successful in getting ElasticSearch 6.0 to work? If so, how did you do it?

sjenson

It looks like I declared victory too soon. I created a tiny instance of CollectiveAccess and the reindexing works just fine, but the full size instance states that it cannot find any live nodes. I suspect the problem lies in running out of memory during the reindex. Any suggestions on how to get around the memory issue?

seth

The current release uses the older 2.x ElasticSearch client and is only compatible with ElasticSearch 1.x and 2.x. I committed changes last week in the GitHub development branch that provide compatibility with Elasticsearch 5.x as well. I have not tried 6.x yet. 

Where is it running out of memory? If PHP is not barfing with a fatal error then it's probably not memory. Look in the logs and let me know if you see any CURL-related errors. We just had a user with ElasticSearch failing with "no live nodes" and it ended up being a PHP 7.1-specific bug relating to CURL (which the Elastic client uses). Downgrading to PHP 7.0 resolved the issue. Not a satisfying solution, but it works.

sjenson

The error I am getting when running bin/caUtils rebuild-search-index is:  PHP Fatal error: Uncaught Elasticsearch\Common\Exceptions\BadRequest400Exception: {"error":"Incorrect HTTP method for uri [//collectiveaccess] and method [DELETE], allowed: [POST]","status":405} in /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:669Stack trace:#0 /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(640): Elasticsearch\Connections\Connection->tryDeserializeError(Array, 'Elasticsearch\C...')#1 /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(561): Elasticsearch\Connections\Connection->tryDeserialize400Error(Array)#2 /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(262): Elasticsearch\Connections\Connection->process4xxError(Array, Array, Array)#3 /var/www/html/providence/vendor/react/promise/src/FulfilledPromise.php(25): Elasticsearch\Connections\Connection->Elasticsearch\Connections{clos in /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php on line 572This computer is running Ubuntu 17.10, Elasticsearch 6.0 and PHP 7.1, so I am going to downgrade Elasticsearch and PHP to see if that solves the problem. 

seth

Definitely downgrade to 5.0. We don't do 6.0 at all yet.

sjenson

I migrated to ElasticSearch 2.4.6 and PHP 7.0, but I am getting this error again when I nearly get through the object section of the rebuilding the search index: PHP Fatal error:  Uncaught Elasticsearch\Common\Exceptions\NoNodesAvailableException: No alive nodes found in your cluster in /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ConnectionPool/StaticNoPingConnectionPool.php:51Stack trace:#0 /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php(71): Elasticsearch\ConnectionPool\StaticNoPingConnectionPool->nextConnection()#1 /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php(89): Elasticsearch\Transport->getConnection()#2 /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(229): Elasticsearch\Transport->performRequest('POST', '//all/bulk', Array, '{"index":{"ind...', Array)#3 /var/www/html/providence/vendor/react/promise/src/FulfilledPromise.php(25): Elasticsearch\Connections\Connection->Elasticsearch\Connections{closure}(Array)#4 /var/www/html/providence/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.p in /var/www/html/providence/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ConnectionPool/StaticNoPingConnectionPool.php on line 51ETA: The log from Elasticsearch reads "failed to execute bulk item (index) index {[collectiveaccess][ca_objects][39291], source[na_]}MapperParsingException[failed to parse, document is empty]"Second Edit: I forgot to include that I am running the development branch of Providence 1.7.6, Schema Revision 150

sjenson

I am now wondering if I am missing a step in the installation of Elasticsearch. Here is what I did:

Install Elasticsearch as  Deb File

Hold Elasticsearch, so it won't upgrade to 6.0

Set Elasticsearch as a service

Edit app.conf to go from SQLSearch to ElasticSearch

Edit search.conf to denote which version of Elasticsearch I am using <-ETA

Try to reindex the search

I have also in the elasticsearch.yml file, I have tried to set the network host to the IP address of this machine, localhost, etc.I have always had to go back to localhost to get the program appearing to work. When I log into Providence, the program says it's running elasticsearch, but when I reindex the search fields, I keep getting one error or another.

seth

Without seeing the actual system it's really hard to say where things are going wrong. The "no alive nodes" message is pretty generic and can be caused by a number of disparate issues.

sjenson

Ah, ok. Thanks for your help. I will keep trying.

sjenson

I carved out a little bit of time to work on this question again; one step I tried was to delete half of the database (in the experimental one) and see if the problem is simply a limitation on memory.    I started to use the binaries from the .tar files in the hopes that would help solve the problem.  When I tried to reindex the database today, I received this error:[2017-12-26 12:00:25,025][DEBUG][action.bulk              ] [Spidercide] [collectiveaccess][4] failed to execute bulk item (index) index {[collectiveaccess][ca_objects][736], source[na]}MapperParsingException[failed to parse, document is empty]    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:153)    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)    at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:584)    at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:563)    at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:211)    at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223)    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327)    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120)    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:657)    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:287)    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)    at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    at java.lang.Thread.run(Thread.java:748)I am using version 1.7.6, Schema Revision 150 of Providence Any thoughts on how I can fix this problem? Hopefully this error message is more useful than the last one I posted.

sjenson

Other potentially useful details about the system:Ubuntu 17.10Apache2 PHP 7.1MySQL 5.7java version "1.8.0_151"Java(TM) SE Runtime Environment (build 1.8.0_151-b12)Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)ETA: ElasticSearch 2.4.4

sjenson

After rereading your post, I found an old hard drive to play around with so that I could install php7.0 on an Ubuntu 16.04 machine. I am trying the binaries for ES and I am still getting problems with bulk import and now I am getting this error: [2017-12-28 09:36:24,605][DEBUG][action.admin.indices.mapping.put] [Sweetface] failed to put mappings on indices [[collectiveaccess]], type [ca_objects]MapperParsingException[No handler for type [text] declared on field [ca_objects/source_id]]    at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:307)    at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:222)    at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139)    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:118)    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99)    at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:549)    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:257)    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230)    at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:480)    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:784)    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    at java.lang.Thread.run(Thread.java:748)

sjenson

I have gotten my testing version of the database to run ElasticSearch; I must have skipped the step to increase the mmap counts (see this page).

kbecker

Hi sjenson,

I’m getting the same error you did,

CollectiveAccess 1.7.5 (147/GIT) Utilities

PHP Fatal error:  Uncaught exception 'Elasticsearch\Common\Exceptions\BadRequest400Exception' with message 'No handler found for uri [//collectiveaccess] and method [DELETE]' in /var/www/html/import/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:681

Stack trace:

#0 /var/www/html/import/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(640): Elasticsearch\Connections\Connection->tryDeserializeError(Array, 'Elasticsearch\C...')

#1 /var/www/html/import/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(561): Elasticsearch\Connections\Connection->tryDeserialize400Error(Array)

#2 /var/www/html/import/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(262): Elasticsearch\Connections\Connection->process4xxError(Array, Array, Array)

#3 /var/www/html/import/vendor/react/promise/src/FulfilledPromise.php(25): Elasticsearch\Connections\Connection->Elasticsearch\Connections{closure}(Array)

#4 /var/www/html/import/vendo in /var/www/html/import/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php on line 572

I increased the mmap counts, configured /etc/elasticsearch/elasticsearch.yml with a node and cluster name, and I’m getting a good result when calling curl -X GET 'http://localhost:9200' from commandline:

-sh-4.2$ curl -X GET 'http://localhost:9200'

{

  "name" : "CA",

  "cluster_name" : "collectiveaccess",

  "cluster_uuid" : "wXtmt5O-TNaoG5R93BChdQ",

  "version" : {

    "number" : "5.6.8",

    "build_hash" : "688ecce",

    "build_date" : "2018-02-16T16:46:30.010Z",

    "build_snapshot" : false,

    "lucene_version" : "6.6.1"

  "tagline" : "You Know, for Search"

}

It’s elastic search 5.6, PHP 5.5 and CA 1.7.5.

What elese besides increasing the mmap counts did you do to make ElasticSearch work?

Thanks for any input!

(I installed ES by adding a new yum repository and installing it with yum)

sjenson

Funny story: While trying to figure out what other steps I took, I broke something in the chain, so now I can not get it working again. Though, I am taking a break from trying to figure it out because I am rearranging the library.

kbecker

Oh no I’m sorry to hear that my asking has caused you trouble! I hope you can figure it out again, in the meantime I’ll be sticking with SqlSearch. Thanks for your effort!

seth

Sorry to be late to the party for this one. It's probably the version of Elastic you're running that's causing the problem. We've only updated the ES client to version 5 in CA 1.7.6, which is sitting in Git but not an actual release yet.

CSIPiemonte

Hi,

we have same problem.

And I noted that problem is the double backslash ( "//" ) in the url

We are using ElasticSearch 5.6.9 and if we launch this url:

http://tst-colaes01.portali.csi.it:9200/collectiveaccess/_refresh</div><div style="font-family: Arial, Verdana; font-size: 10pt; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal;">

we obtain this:

{"_shards":{"total":2,"successful":1,"failed":0}}

Instead

Instead if we launch this:

http://tst-colaes01.portali.csi.it:9200//collectiveaccess/_refresh</div><div style="font-family: Arial, Verdana; font-size: 10pt; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal;">

this is the response:

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"","index_uuid":"_na","index":""}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"","index_uuid":"na","index":""},"status":404}

So I remove one backslash from Elastic Search URL in search.conf 

# -------------------

# ElasticSearch plugin configuration

# -------------------

# Enter the elastic search base url here (without any index names)

#LM 27/11/17S PARAMETER BY PROJECT

#search_elasticsearch_base_url = http://localhost:9200/</div><div style="">search_elasticsearch_base_url = http://tst-colaes01.portali.csi.it:9200

#LM 27/11/17E

but we  have now an other error when we try to go to check configuration from browser ( http://tst-colaweb01.csi.it/cola/index.php/administrate/setup/ConfigurationCheck/DoCheck )

Updating the ElasticSearch mapping failed. This is probably because of a type conflict. Try recreating the entire search index. The original error was illegal_argument_exception: request [/collectiveaccess/ca_objects/_mapping] contains unrecognized parameter: [ignore_conflicts]

So we try to do a reindex using caUtils but we obtain this PHP Error:

[29-May-2018 17:22:08 Europe/Rome] PHP Fatal error:  Uncaught Exception: Updating the ElasticSearch mapping failed. This is probably because of a type conflict. Try recreating the entire search index. The original error was illegal_argument_exception: request [/collectiveaccess/ca_objects/_mapping] contains unrecognized parameter: [ignore_conflicts] in /data/www/html/cola/app/lib/core/Plugins/SearchEngine/ElasticSearch.php:128

Stack trace:

#0 /data/www/html/cola/app/lib/core/Plugins/SearchEngine/ElasticSearch.php(539): WLPlugSearchEngineElasticSearch->refreshMapping()

#1 /data/www/html/cola/app/lib/core/Plugins/SearchEngine/ElasticSearch.php(320): WLPlugSearchEngineElasticSearch->flushContentBuffer()

#2 [internal function]: WLPlugSearchEngineElasticSearch->__destruct()

#3 {main}

  thrown in /data/www/html/cola/app/lib/core/Plugins/SearchEngine/ElasticSearch.php on line 128

Any idea?

Thanks a lot

Luigi Montinaro

CSI Piemonte

CA - Providence 1.7.5

ElasticSearch     5.6.9

PHP 7.1.16