Search indexing quirks or misunderstandings

yelto

While trying out a new 2.0 (2.0.4) install I am getting incorrectly no results on a search by a non-intrinsic metadata "my_meta" associated with a hierarchical list. The advanced search form launches the search with a sw.word=<item_id> clause, but the indexing seems to encode the words in the labels of the list item but does not insert the numerical <item_id>, leading to no results. Logic seems to be on the side of searching by <item_id>, is this how it should work and thus the blame is on the indexing side?

Thinking the default _metadata = {} line in search_indexing.conf is not enough for proper list item indexing, I've tried adding a custom ca_attribute_my_meta line to introduce the INDEX_LIST_ANCESTORS option for the attribute. This appears to be ignored altogether, while _ca_attribute_my_meta (with a leading underscore) at least gets the attention of SearchBase.php. Is this just a typo in line 50 of search_indexing.conf or in the code?

And even with the underscore, no joy, it seems to be ignored when it come to the actual indexation. There are some old (circa 2016) messages on this board regarding similar issues with this syntax, should ca_attribute<> this work? Or maybe it requires full reindexing?

Thanks

monica

yelto I believe the ca_attribute_<element_code> is the correct way to list individual metadata elements.
Anything you change in search_indexing.conf (with exception of access_points) requires a reindexing if you want all record results to show. If you go into a record and edit the field you want to test, then this will reindex just this field.

That said, the list attribute should be indexed. Have you tried searching for the value in a basic search? as it might be the way you have implemented the advanced search that is the issue.

The INDEX_LIST_ANCESTORS i believe is to work on related list items, not list attributes, but I truly cannot be sure of that. List attributes only store the value and item_id in the database, but I'm unsure if the item_id is indexed however.

yelto

Thank you @monica, the docs do point to ca_attribute_ but even in search_indexing.conf (line 32) there is an example showing a leading underscore, and maybe more tellingly the code around l.215 in app/lib/Search/SearchBase.php also includes the underscore.

The basic search is textual and thus would not suffer from <item_id> not being indexed, but neither would it give the pinpointed result set that a search from a voc dropdown list selecting an unambiguous <item_id> on a particular metadata element is expected to provide (at least in my mind...).

yelto

Maybe the intent was to send the list_item idno (which does get indexed) instead of the item_id in the advanced search query with dropdown for a list attribute?

seth

It has a leading underscore. It's an artifact of how the indexing worked many many years ago. We should change it one of these days.

yelto

Thanks @seth. If it were to work with the underscore, I would say just change the doc. Unfortunately the indexing updates upon changes in the attribute that do happen with _metadata{} don't seem to happen with _ca_attribute_my_meta {} either. They should, right?

And alas all of this is probably unrelated to the original issue with advanced search queries...

monica

yelto adding your list metadata element to an advanced search should give you a dropdown list with the labels of each value to select. Is this not what you get?

yelto

monica Yes, that part works perfectly, the UI is fine. The problem is that when I launch the search it returns (incorrectly) no results, and when I log the queries to the db it is querying for a word '557' which suspiciously happens to be the item_id of the value of the list-based metadata element that I select in the dropdown.

SELECT swi.index_id, swi.row_id, swi.boost, swi.field_index FROM ca_sql_search_word_index swi INNER JOIN ca_sql_search_words AS sw ON sw.word_id = swi.word_id WHERE swi.table_num = 57 AND sw.word = '557' AND swi.field_table_num = '57' AND swi.field_num = 'A104';

Unfortunately the indexer does not record the <item_id> in ca_sql_search_words, only the words from all the labels corresponding to the list item (in various locales) and the idno. So no 557 in the index, thus no match. (In fact it looks like there could be incidental and incorrect matches if a label of some other item happened to include as text the same numerical value as the item_id, but I digress).

At first blush either the query is incorrectly formed from the advanced search form (and should instead, for example, send the idno of the desired list item), or the indexer should index the <item_id> of the list-based metadata.

So... are <item_id>s for list metadata elements included in the index in healthy installations with vanilla _metadata{} for ca_objects in search_indexing.conf ? Or am I just misunderstanding how all of this should work?

Thanks again

seth

What sort of advanced search form is it? Back-end or front-end?

yelto

This is an advanced search form on providence, defined with the interactive form-building tools on providence itself. System upgraded from 1.7, this appears to be a regression.

monica

This sounds like a bug. I would have assumed that advanced search would query the database directly instead of using the indexer for list items. This is so it can match on the exact list item rather than just a word which might return incorrect results if items in the same list share the same word. Eg. “cotton” and “bamboo cotton” in a fabric list.

yelto

Well like I said I'm just guessing at the intent of the code, but there are very good reasons to simulate the type of search you describe while using a uniform and efficient mechanism that can handle combined searches (text, numeric, list, etc). And when you control both sides (index and queries), you can avoid spurious matches altogether by looking for word matches in restricted fields, for instance by defining a swi.field_num = 'A104_id' where the indexer will only write <item_id>s for attribute 104, and where the queries will look for matches for list items. Since item_ids are unique and the field unambiguous, no spurious matches. Easier said than implemented and tested, obviously.

monica

yelto If the index stored the item_id along with the labels (so it can match on freetext searches too), then there is potential for the number to be in the label of the list and then match on an incorrect record (most likely rare, but could happen). Users expect Advanced searches where there is a control list, to match on that value exactly.

Does the search builder work as expected for you? I've always wondered if there is any point in keeping advanced search maintained, when there is search builder available. I do see use cases for both though unfortunately.

Be interested to hear your thoughts @seth

yelto

Before getting too deep into this, it would prudent to get the issue reproduced or (even better) works-for-me'd. This is after all core functionality, so pilot error should be ruled out first...

seth

We stopped indexing item ids. That's back now in the dev/2.0 branch. There will be a 2.0.5 release with the change later today. If you reindex your searches will work again.

yelto

Thank you Seth. Any plans for further refinements to reach the full selectivity of the voc/dropdown/item_id search? Thanks again.