WARC file format

miceno

Recently one of our archivers ask to store a page from the Internet. So I had a look around potential options for it.

I finally came to WARC, the Web Archiving format used on archive.org and most libraries (see https://en.wikipedia.org/wiki/Web_ARChive)

I would like to know your experience about this. I will try to find a way to include WARC in CollectiveAccess, so I will also appreciate any feedback on it.

Regards.

seth

WARC is the defacto standard for archiving web resources. Widely used open source software such as WebRecorder (https://webrecorder.io) can create WARC files, so it's a fairly easy format to work with.

CA can accept WARC files currently as binary data, without any parsed metadata or web-based preview. This is workable, I guess, but definitely not great.

We've been discussing integrating support for WARC parsing + preview using a WebRecorder-related project for self-hosted embedding of WARC files. See https://webrecorder.org/2019/11/06/self-hosted-archival-embeds.html for more information on this. We've met the developer of this and it appears very doable to integrate it all into CA.

We agree that WARC is the way to go and it would be great to have full support for it in CA. If you want to work on this, let's coordinate efforts. Feel free to contact me directly if you want to discuss further.

seth