Importing MARCXML records

Ammo

Hello everyone,

I'm on a rescue mission to import a ISO2709 file containing records to be imported to C/A.

Using a third-party free utility (J-ISIS) I managed to convert the ISO2709 to MARCXML. Although the XML is valid, this is a "homemade" MARC structure and converting to plain MARC (.mrc) doesn't work.

I'm struggling to import the MARCXML file to C/A more than I thought I would. Because the MARCXML is, in fact, an XML file, shouldn't it technically work with XPath selectors?

I see that to import XML it needs to be in either in FMPDORESULT or InMagicXML, but this isn't the case, it's just XML.

And to import MARC, from what I understand, it needs to be a MARC (.mrc) file.

Any way I can get around this?

Thanks,

Ammo

stefan

Hmm, the MARC data reader uses the File_MARC pear module, which should be able to parse MARCXML too, at least in theory. Maybe give that a shot? If that doesn't work, it shouldn't be a big deal to add a data reader for MARCXML.

Ammo

Looking at http://docs.collectiveaccess.org/wiki/Data_Importer#Source_syntax_for_XML_input_formats I think my syntax is correct but I have no idea what to define as inputFormat as it won't allow me to introduce XML or MARCXML.

Needless to say that trying to import the same data and mapping with inputFormat to FMPDORESULT or InMagicXML didn't work.

Attached is my current mapping regarding the MARCXML data.

EDIT: 

I was typing this update while you replied, Stefan.

Thanks for your reply, I wonder if this has ever been attempted before.

Anyhow, I should still try with MARC as inputFormat and ultimately create my own data reader for MARCXML, correct?

Thanks,

Ammo

stefan

You should definitely try the MARC reader with your MARCXML file. If that doesn't work, drop us a line and we'll add a MARCXML reader for you. It's not a big deal. Also, if you could post some example data, that'd be great!

Thanks!

Ammo

Hello stefan,

I've tried the import using MARC inputFormat, however, the following error is being logged and nothing is imported:

PHP Warning: substr() expects parameter 3 to be long, string given in /var/www/app/lib/core/Parsers/File_MARC/MARC.php on line 321

Here's the sample data i'm trying to import (changed .xml to .txt because .xml is not allowed).

Ammo

I'm checking the variables content in that file (MARC.php) on import, seems to me they're all wrong and this might have something to do with data file character encoding?

I'll try it again with different character encoding and come back.

stefan

Yeah, this is not UTF-8, even though it says so in the header. However, the PEAR library shouldn't bail because of that. 

Side note: Have you verified that you have valid MARCXML? They have a schema here: http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd</div>

Ammo

After trying so many different combinations, the error is always the same.

I wonder if I'm doing something wrong or I have to indeed create a custom dataReader for it.

Thanks,

Ammo

Hello stefan,

Thanks for coming back at me.

My data was in fact invalid, the <leader> was incorrect.

This is the XML sample I'm using now, it's valid according to freeformatter

<?xml version="1.0" encoding="UTF-8"?>

</datafield>

</datafield>

</record>

</collection>

However the error is still the same as I mentioned above.

Ammo

Stefan,

The MARC I have is "homemade" as I said early, and it turns out the <leader> tag is incorrect for the corresponding record, and looking at the source code, namely /app/lib/core/Parsers/File_MARC/MARC.php:281 and on, I think this might be where it's all going wrong.

I didn't pay much attention to this as an XPath expression would select the element no matter the parents or their values, right?

Because the <leader> is wrong, the following instructions will all point to incorrect variable values.

I'll give it a shot with a "valid" MARCXML and see where it leads.

SIDENOTE: 

The error logged is pointing to /app/lib/core/Parsers/File_MARC/MARC.php, but shouldn't it technically be using /app/lib/core/Parsers/File_MARC/MARCXML.php ?

To me, the <leader> being wrong causing all the trouble only makes sense for a pure MARC (.mrc) file, not for MARCXML.

On another note, and I should've mentioned this earlier, I'm confused with the documentation,

As this link http://docs.collectiveaccess.org/wiki/Data_Importer#Source_syntax_for_XML_input_formats clearly is referring to MARCXML, but this one http://docs.collectiveaccess.org/wiki/Cookbook_Chapter_3:_Data_Import#Importing_MARC seems to be referring to MARC (.mrc).

The only valid inputFormat for the mapping is MARC. 

Thanks.

Ammo

Hello,

Giving an update on this subject.

I went on investigating the core files for a couple more hours and even made some tries with Filemaker XML but all that went nowhere.

I thought of converting/inserting the MARCXML into Excel, but then I discovered other many issues:

- The Excel maps the XML using elements and, using MARCXML, what I really need is to map using the element attributes ('/datafield[@tag='sometag']');

- Repeatable fields create a new column for every repetition, with 50k records thats unbearable;

- There is no distinction between each record being inserted, because each line is a field, not a record as it should be.

How I solved the problem (Importing MARCXML to C/A):

Developped an XSL file and transformed the MARCXML into 'standard' simple XML, for 2 main reasons:

1) Can be imported to Excel directly with no additional effort;

2) Some field rules can be enforced not only on Excel, but on XSL too - for example, placing multiple occurences of the same field in the same element, separated by a delimiter, defining default values, etc

Then, applied the XSL transformation to the MARCXML using xmlstarlet

This is the way I ended up doing it and, to be honest, is working pretty good.

Although I'm a bit disappointed not being able to import MARCXML directly into C/A, without knowing whether it was me who did something wrong or the application is not ready for MARCXML.

Thanks,

Ammo