Failed database import from PastPerfect

hdace

As a museum assistant I am possibly not well suited to setup Providence but I'm doing my best. It took me a day just to get it up and running. The import mapping file and data file are not playing nice and I think it would probably take me two weeks of fiddling to figure it out.

I'm trying to export our Images Catalogue from PastPerfect so I can setup a server to make it accessible to the public on the internet. The error I'm getting is, "a valid type must be specified". I've also had, "no media was specified".

Please have a look and any advice would be a big help, thanks

PS: I did edit the app.conf file to point at the image directory

[https://docs.google.com/spreadsheets/d/1Mq40ZP3GXB8OfD_kmlzfmsIRfvY-M6AW/edit?usp=drive_link&ouid=110939141714824481342&rtpof=true&sd=true](https://)

[https://docs.google.com/spreadsheets/d/11SnRa8FX8o7VNz7hBbqucSlBoByR1xF1/edit?usp=drive_link&ouid=110939141714824481342&rtpof=true&sd=true](https://)

seth

What profile are you using? The "valid type" error may be due to the entity type in the mapping not being present in your profile. The "no media specified" error is likely due to the lack of a representation splitter. If you just map the path as is done in your mapping it has to be a valid Unix-ish path. Those backslashes won't work.

Let me know what profile is in use and I'll try to give you a working mapping for the data you have.

hdace

Thanks so much for your assistance!!!

I chose the PastPerfect profile. I don't really know what profile would be best for our purpose. I only chose that profile because I know PP, but if there is a profile that would work better, I would happily change it (except, of course, I don't know how).

Are you saying all of the backslashes should merely be replaced with regular slashes? I assume you will do that but if you want to save time, I can do it.

In the app.conf file, for root directory for media import I entered this:
batch_media_import_root_directory = /Volumes/Media Man/PPexport/data/Images/
Is this right? Also, is this right?:
allow_import_of_media_from_any_directory = 1

Thanks again!

hdace

Oh dear, I was hoping to get some assistance before Monday as I’m meeting with our director and curators to discuss the project. I can see I’m going to have to work all weekend trying to figure it out. Any more help or advice from anyone would be gratefully received 😀

monica

There are some significant errors in your mapping

You are importing into the representations table, when it should be the objects table. This is the table setting row
The setting “type” should be one of the values from your object type list. For PP profile this might be “still_image”. If you need different object types, then set a default in settings and create a mapping row to ca_objects.type_id with a new column in your source data.
The mapping row with the filename. That needs to be a splitter, like the entity, but for object representation.

Well done getting CA set up to this point. However, when meeting with your directors and curators, make sure to remind them that you are not a systems or data migration expert. If they want to make sure it’s done right, then they should invest a little to get it setup.

hdace

monica Thank you. I posted my subsequent message without having seen your message first. I still can't tell, from what you're saying, if it is actually possible to import the images simultaneously. If it is, then I will try what you say. I've been trying to use the batch importer but I keep getting an error: No import modes are configured

As for my colleagues, they asked me to try to do it precisely because there is no money for this project. But I will pass on your suggestion!

hdace

I think I've solved this problem. Got a new one but I'll start a new thread for that.

I reinstalled with a different profile, Dublincore 2018, and changed the table back to ca_objects and the type back to image. I successfully imported the entire database, which is terrific, but it did not import the images that mapped in source 6. I gather CA cannot import image media simultaneous to importing the database.

monica

hdace yes, it is possible to upload the media with a bulk import mapping. Use the representation splitter with the media attribute pointing to the source column that has the path.

If you still have trouble I can dig out an example refinery parameters once at my computer.

hdace

monica I have successfully imported the database twice now, so I'm quite pleased about that.

The second time I added:
{"attributes":{
"media": "¹",
"internal_notes": "²",
"idno": "¹"
}
}
to the cell in the Refinery Parameters column in the Source 6 row, but it didn't work. There are no images associated with objects. (BTW, I did change the backslashes to forward slashes in the database sixth column)

I'm assuming that the problem is that I don't have anything written in the Refinery column. If you are able to help with that, thanks very much!

monica

hdace you will need to use the objectRepresentationSplitter in refinery

As for your refinery parameters, you have pointed the media to source column 1, but in your example you have the filament at the end of your source data. It looks like you’ve just copied and pasted from the manual, you need to change things for your data. You will also need the representation type in the parameters.

hdace

monica I added the refinery entry and I updated the parameters, but I'm still not getting any media associated with any catalogue entry. Here are the parameters I've been using:
"{"attributes":{
"media": "⁶",
"access": "1",
"idno": “^1”,
“mimetype”: “image/jpeg”
}
}"

I also have tried to batch import the media but it won't load with the message "No import modes are configured"

Not sure what to do next...

hdace

I finally figured out that the parameters should look like this:
{
"objectRepresentationType": “front",
“attributes":{
"media": “^6”,
"idno": “^1”
}
}
and I've tried lots of different permutations. I've read many forum posts on how to use the objectRepresentationSplitter. I've checked and there are no errors in the logs.

CA just won't import the images along with the catalogue entries.

So I'm pretty close to giving up. This is the only way our museum will ever get our 26,000+ images up on the internet, because we just don't have any money to hire a data migration person or pay thousands of dollars to PastPerfect.

Anyone else with any advice?

seth

Here's is a working mapping. It is set up to run on https://demo.collectiveaccess.org, so you can try it if you like. The mapping corrects the various errors and also adds a regular expression rewrite that converts those PP backslashes into the forward slashes that are used in "conventional" file paths (Eg. everywhere but Windows).

A few notes:

In the import directory in the CA application root directory, I created a directory named rchs. Within that directory are subdirectories 025 and 056. Each contains a test image. In a real import you'd copy the entire PP media directory structure under rchs (or whatever you want to call the directory - just make sure the media prefix setting in the mapping references the correct name)
The representation and entity types, and the field codes are those used in the demo system. They may not match what is used in your profile. In particular, I mapped column 3 of your data to ca_objects.item_notes. You'll want to put it in a field appropriate for your mapping.
In the PP5 profile ind is the entity type code for individual, but front or back (not media) are the only valid representation types. Be sure to change the representation type to front if you're using the PP5 profile.
rchs-mapping-2.xlsx
13kB

You can try this out on https://demo.collectiveaccess.org. Just keep in mind that only two images are present in the import/rchs directory: 025/1971001005.JPG and 056/1971053012a.JPG

hdace

seth Thanks very much. I've downloaded your mapping and I'm looking forwards to trying it out! I'll let you know how I get on.

hdace

seth I've got everything working, although there was a big surprise I'll get to later. First and foremost, I really need to know how to make "accessible to public" all of the entries when I import the data from the spreadsheet. Eventually I am going to import over 20,000 entries all at the same time and obviously it is impractical to change each entry one at a time. I hope there is some way to add this in the mapping spreadsheet, or in the app.conf file, or somewhere so that it works. Thanks!

Initially I still couldn't get the data to load into the PastPerfect profile because the type_id, image, was wrong. It has to be still_image. I studied the profile xml file and was able to determine this and then it worked in a test. However, I found the PP profile was not as useful as the DublinCore profile. I realize that, through some customization, I would be able to improve the PP profile, but why bother when DublinCore is fine? Therefore, I left the entity type as ind.

The really big problem I had, which was that images were not importing with the catalogue entries, was caused by an anomaly which I cannot explain. It also would take several pages to explain how I worked out what the real problem was. It had nothing to do with the mapping file. I discovered that the default import folder (which I have not been able to change) is NOT ca/import/ ! The documentation, and your previous message to me, states that the import directory in the CA root directory is where image files are imported from. In my experience, this does not work. I don't know why. What DOES work is this: ca/uploads/_{administrator}/ !!!!

So, to be clear, when I put my image files in ca/uploads/_{administrator}/ the images correctly import with each catalogue entry. It also works if I put subdirectories in there, and use the subdirectory names in the data spreadsheet, for example, 026/1971053012b.JPG

Thanks to both you and Monica for your useful advice. I learned a lot from both of you. It only remains to fix the public access problem. Not to mention getting Pawtucket up and running!

--Hal

hdace

A couple of more little things...

The type_id archival also does not work with PP profile. still_image is the one that worked for me.

Also, I skipped using the mediaPrefix since I moved my test image files anyway and it didn't seem necessary to have an intermediate directory.

seth

You can batch them all to public using the batch editor. Create a set with all objects, open it in the batch editor, and make whatever changes you wish to all records in the set.

With regards to the import directory, there are actually two directories. The root import directory is site-wide and visible to all users. The other, which is the one you used, is user-specific. Either one will work, so long as you're importing as the user for which the directory is named. I don't know why the site-wide import directory didn't work for you. Two possible reasons: (1) you've changed the location of the site-wide import directory in app.conf (2) the permissions on the site-wide import directory are such that the web server user (which is what the application is running under) cannot read its contents. The second reason is the most likely.

The various type_id value changes are down to the profile you're using. The various profiles use different ids for things, and you have to adjust to suit.

hdace

I tried several times to batch edit a set I created but it failed with this message each time:

Fatal error: Uncaught TypeError: array_map(): Argument #2 ($array) must be of type array, null given in /opt/homebrew/var/www/ca/app/controllers/batch/EditorController.php:254 Stack trace: #0 /opt/homebrew/var/www/ca/app/controllers/batch/EditorController.php(254): array_map(Object(Closure), NULL) #1 /opt/homebrew/var/www/ca/app/controllers/batch/EditorController.php(63): EditorController->_initView(NULL) #2 /opt/homebrew/var/www/ca/app/lib/Controller/RequestDispatcher.php(289): EditorController->Edit(NULL) #3 /opt/homebrew/var/www/ca/app/lib/Controller/AppController.php(108): RequestDispatcher->dispatch(Array) #4 /opt/homebrew/var/www/ca/index.php(137): AppController->dispatch(true) #5 {main} thrown in /opt/homebrew/var/www/ca/app/controllers/batch/EditorController.php on line 254

seth

Please pull the 2.0.6 release (or do a pull from GitHub master or dev/2.0) to resolve this.

hdace

I've installed 2.0.6 but it's opening my old database. I want to start fresh, reimport the data & media. I used to do that by reinstalling the profile. Now CA won't let me. I vaguely remember changing a setting to get CA to let me do it but I can't remember what it was. Please let me know, thanks