Media representation import

Epieru

HelloWhat is the best method to import representation.I've tried to import in table ca_objects with link and refinery to ca_object_representations table, but merge_on_idno option results in error when media already exists, and several fields such as original_filename are not filed, which makes the media bundle in Providence incomplete or erroneous.I've tried without success to import in ca_objet_representations table without link to object, unsuccessfully as CaUtils always returns : No media was specified.What is the right method ?Is there an exhaustive list of attributes in representation refinery ? Or how to identifiy the name of the attribute that matches the name of the field  in the database ?How to manage name/preferred_labels for instance, which is in a separate table than ca_objet_representations ?Thanks for some more explanations.

stefan

As for the "No media was specified" error in your representation import: Representations can't exist without media. So you have to specify a file for the "media" field in your import, either by pointing to a file in the local file system: "/home/me/foo.tiff" or to a URL where it can pull the file (if you have pulling from URLs enabled in app.conf)In general we often times import media using the batch media importer under Import > Media, because other than things copyright and license info, representations usually don't have a lot of metadata attached. But it's possible to do it with a full-blown import, too.

Epieru

Thanks for your answer Stefan.Some media files are in the default /import directory of Providence (1.5, master-fix) installation, the same files are in a /picture subdirectory of /import that I have created, and the same files are also in a /mapping directory that collects all mappings and datafiles that I need. Media files are in the Apache server local file system.Unfortunately, none of the files is seen by CaUtils, when I try to "batch import" them in the ca_object_representations table. It looks like the ca_object_representations.media field is not accessible. In my datafile, the field is filed with the exact name of the file. I tried to add picture/ or mapping/ prefix to the name without success. In the database, the name of the file is stored in ca_object_representations.original_filename, and ca_object_representations.media contains a blob. Could it be the reason of the failure ?Alternately, if I try to import in ca_objects table, using a refinery, this works for all media directory, except that some fields in the UI representation bundle are not correctly filed during import. For instance, ca_object_representations.original_filename field remains empty, and attribute 'name' is linked to preferred_labels in the media representation UI, and to a field in ca_object_representation_labels table. Same remark for caption. It's rather complicated to identify the correct "attribute" matching with database field. Some of them do not have the same name.I do not use the batch media importer in the front end UI, because I have several representations linked to each objects, many captions in different 'locales', and naming seems highly complicated (impossible ?) in this case.My best solution is to import in ca_object_representations table with a relation to the ca_objects table. Unfortunately, the media file is not found and the record is not created of course.Do you suggest in your answer to use /home/'user' directory ? I'm using 'root' as user for my tests, and no 'user' conventional directory. In the next step, with an Internet web hoster, it's highly probable that I will not be able to access to this directory. Is there a way to indicate to Providence where to find the files ?Have you an example of working full-blown import that could lead me to the solution ?Thanks

stefan

You should point to an absolute file path for the media field and make sure the user that runs caUtils can read the file and write to the CA media directory.I don't have an example but I know we've done them before.

Epieru

Hello StefanThe CaUtils user is 'root' Debian admin, the media files are in /var/www/providence/import/picture, assuming that 'import' directory is the default root for import processes, declared in app/conf file. root is able to write, delete... in the providence/media/website directory.I have succeded in using another mapping with a refinery related to media representation, it works. Medias are created, except that fields in the database are not correctly filed (see the first part of this post, and also a simple test that I already sent to Seth and Jonathan a few weeks ago). I have used too the mediaPrefix option without problem.I have tried to use the complete absolute path from server system file root in my datafile, /var/www/providence/import/picture/media.jpg for instance, with no more success.My mapping seems correct : I've skipped all unnecessary mappings at this time. elements are in the ca_object_representations table, except preferred_labels which is named differently. But I have supposed that the syntax is identical to a mapping related to ca_objects table. Column 5 is used two times. I did the test too with a duplicated fifth column in nineth position.

Rule typeSourceCA table.elementSKIP1ca_objects.idnoSKIP2ca_objects.type_idMapping3ca_object_representations.type_idSKIP4Mapping5ca_object_representations.mediaMapping5ca_object_representations.original_filenameMapping6ca_object_representations.preferred_labelsMapping7ca_object_representations.idnoSKIP8   Rule triggersRule actionsRule   Setting nameSetting valueSettingcodeimp_representationsSettingnameImport of representationsSettinginputFormatsXLSXSettingtableca_object_representationsSettingtypefrontSettingnumInitialRowsToSkip1SettingexistingRecordPolicymerge_on_idno_with_replaceSettingerrorPolicyignoreSettingarchiveMappingyesSettingarchiveDataSetsyesSettingbasePath#XML tree#

and matches with datafile content

ca_objects.idnoca_objects.type_idtype_idlanguagemediapreferred_labelsidnoexistsITEMdocumentbackITEM_back.jpgItem backITEM BACKTRUE

I have tried several pathes too for original files, without success.Same result, with a bug (see another discussion) in display resulting in Firefox crash, with the Providence UI import menu. In this case on Debian, the user is www-data, and rights and permissions for this user are correctly setted on Providence install directory. Errors are the same one.Can you give me the path to the import process php files ? I will try to log the activity of the related functions. Perhaps could it help to find the reason of the failure.Thanks. Eric

Epieru

Am I (probably) wrong somewhere ?

Is full-blown import possible through this method ?

Should I use the representation refinery ? And try to find the good names of the attributes ?

Thanks

stefan

I just committed a fix to the master-fix branch. Object representation imports should now work as expected - hopefully. Previously it would try to insert the record without setting the media field first, causing the infamous "no media specified error". Now it'll set the media before the insert ... if you have a ca_object_representations.media mapping that is. We were going to do the v1.5.1 release this week and I think the change will make it into the release.Attached are the minimal mappings I used for testing. Both the initial import and the merge on label without reprocessing the media worked okay.I haven't had a chance to look into the representation splitter refinery but I know for a fact that we used that recently, so in theory it should work. But you never know. It's on my list ...

Epieru

Hello Stefan

Sorry for the delay, I was not at my office yesterday.Great thanks for this correction. It works.Still some missing or erroneous informations but medias are imported.

Missing: original_filename is empty, even if mapped.

Missing: I ask for a relationship object x object_media_representation, just with a line:

Rule type

SourceCA table.elementSKIP1

ca_objects

No refinery as the relationship is "null" by default. This does not work. I will try in anoher way this afternoon, using  ca_objects table with refinery, instead of ca_object_representations table

Erroneous: the media bundle in the object UI is still wrong. It does not behave as when creating a media in the object UI, the fields are not filed in the same manner (the line under the name of the representation contains informations from other fields, differs than when manualy created.

Erroneous : the existingRecordPolicy merge_on_idno_with_replace does not update the media if I change the file content, not the filename. No update is performed for the file content. The other fields are updated.

Eric

stefan

The original_filename is now set to the original file name, as you'd expect I guess. At least for representation imports. I have yet to look into the splitter.What do you mean the media bundle is "wrong"? The extracted metadata is exactly the same in my tests. See screenshot below -- [BLANK]] was uploaded by hand, "Baz" via importer. (before you ask, the Media metadata is also identical).The existingRecordPolicy is probably messed up for media fields. Since people usually don't update their media using import spreadsheets this hasn't really been a priority and I guess no one noticed until now. I'll add it to the list.

Epieru

Hello Stefan.

This Saturday, I did several tests, with an empty default profile database. In each case, I've completed the same fields. (idno, title (preferred_labels), name (original_filename), access, caption)

The details are in the attached pdf file.

I agree with you: importing media representation (without refinery) (test 3) or creating a media representation manually (test 1) give the same result for the bundle. But this method does not seem to allow import of the link with the object in the same run.

You and I know that the refinery method (test 4) is missing fields. Some fields in the bundle and database remains empty.

The bundle and fields are also different if you add an object and create a media representation using the media screen (test 2).

Regards

stefan

I just tried the object import + representation splitter version and that worked as expected too. Below are the test mapping and data.That splitter is not meant for updating existing records, they only create new or match records. It also won't create a label for you unless you tell it to (using the "preferred_labels" setting), otherwise it'll use the "idno" setting as label (see example below) or leave it empty.

stefan

Okay, now also tried out the last two things on the list1) Updating existing media (merge_on_idno) works as expected with representation mappings. You said it didn't?2) There was an issue with the missing representation type in the object splitter. I fixed that and it should now be possible to relate representations to objects from the representation side. Below are mapping and data I used for testing.