Tag Archives: Pronom


The PRONOM file format signature files were updated on December 17. DROID users should make sure they have the latest files.

File identification tools, part 10: Siegfried

“Do we really need another PRONOM-based file format identification tool?” That’s what Richard Lehane asked rhetorically last year on the Open Preservation Foundation blog. It was obviously rhetorical, since he’d gone ahead and done just that with a new tool called Siegfried. Siegfried recently turned up in some tweets by Ross Spencer, so it’s worth a mention here.
Continue reading

Format registry browser online

In an effort to promote interest in my format registry browser, I’ve built a Java web application out of it and put it up on Google App Engine at regbrowser.appspot.com. It lets you search PRONOM, UDFR, and the DBPedia structured summaries of format articles, by name, MIME Type, creator, and extension. It uses SPARQL Linked Data queries to obtain data.

It’s still in a rough form; the point is to show what it can do and hopefully get some interest in putting money into further development. Obvious improvements, which I may do shortly, would include checkboxes for which repositories to search and retention of text fields when returning to the search page.

UDFR times out a lot. If you get a timeout error, trying again has a good chance of working.

Format registry browser updated

I’ve posted an updated version of my file format registry browser (Zip file). It’s still very experimental, but this one makes several steps on the long path to being a useful tool.

The biggest news is that thanks to David Underdown’s input, it now talks to PRONOM. Preserv2 is probably a lost cause, since it appears no current work is being done on it and its useful results were folded back into PRONOM. This version tries to prettify results containing URIs and “@en” tags. If you don’t like that you can turn it off with a check box. The search fields have changed, and all of them now do something with all registries. The logging level can now be controlled from the config file (src/com/mcgath/regbrowser/config.properties). In some future version I’ll use a less buried config file.

Here’s my post on the first release.

Format registry browser available for download

I’ve made my experimental format registry browser available for download. It requires Java 5 or higher and a GUI environment, and Ant is required if you want to make changes. Currently it queries DBPedia and UDFR. It’s been tested on Mac OS X and Ubuntu.

Defining the file format registry problem

My previous post on format registries, which started out as a lament on the incomplete state of UDFR, resulted in an excellent discussion. Along the way I came upon Chris Rusbridge’s post pointing out that finding a solution doesn’t do much good if you don’t know what problem you’re trying to solve. This links to a post by Paul Wheatley on the same subject. Paul links back to this blog, nicely closing the circle.

So what are we trying to do? A really complete digital format registry sounds like a great idea, but what practical problem is it trying to solve? We know it’s got something to do with digital preservation. If we have a file, we need to know what format it’s in and what we can do about it. If it’s in a well-known format such as PDF or TIFF, there’s no real problem; it’s easy enough to find out all you need to know. It’s the obscure formats that need one-stop documentation. If you find a file called “importantdata.zxcv” and a simple dump doesn’t make sense of it, you need to know where to look. You need answers to questions like: “What format is it in?” “What category of information does it contain?” “How do I extract information from this file?” “How do I convert it with as little loss as possible into a better supported format?”

I have a 1994 O’Reilly book called Encyclopedia of Graphics File Formats. If old formats are a concern of yours, I seriously suggest searching for a copy. (Update: It turns out the book is available on fileformat.info!) It covers about a hundred different formats, generally in enough detail to give you a good start at implementing a reader. There are names which are still familiar: TIFF, GIF, JPEG. Many others aren’t even memories except to a few people. DKB? FaceSaver?

With some formats the authors just admit defeat in getting information. The case of Harvard Graphics (apparently no connection to Harvard University) is particularly telling. The book tells us:

Software Publishing, the originator of the Harvard Graphics format, considers this format to be proprietary. Although we wish this were not the case, we can hardly use our standard argument — that documenting and publicizing file formats make sales by seeding the aftermarket. Harvard Graphics has been the top, or one of the top, sellers in the crowded and cutthroat MS-DOS business graphics market, and has remained so despite the lack of cooperation of Software Publishing with external developers.

While we would be happy to provide information about the format if it were available, we have failed to find any during our research for this book, so it appears that Software Publishing has so far been successful in their efforts to restrict information flow from their organization.

This was once a widely used format, so if you’re handed an archive to turn into a useful form, you might get a Harvard Graphics file. How do you recognize it as one? That isn’t obvious. A little searching reveals you can still get a free viewer for older versions of Windows, but nothing is mentioned about converting it to other formats. Even knowing there’s software available isn’t helpful till you can determine that a file is Harvard Graphics.

If you have a file — it’s Harvard Graphics, but you don’t know that — what do you want from a registry? First, you want a clue about how to recognize it. An extension or a signature, perhaps. When you get that, you want to know what kind of data the file might hold: In this case, it’s presentation graphics. Then you want to know how to rescue the data. Knowing that the viewer exists would be a start. Knowing that technical information isn’t available (if that’s still true) would save fruitless searching.

Information like this is scattered and dynamic. If the Harvard Graphics spec isn’t publicly available now, it’s still possible for its proprietors to relent and publish it. The notion of one central source of wisdom on formats is an impossibility. What’s needed is a way to find the expertise, not to compile it all in one place.

We need to concentrate not on a centralized stockpile of information but a common language for talking about formats. PRONOM uses one ontology. UDFR uses another. DBPedia doesn’t have an applicable standard. What I envision is any number of local repositories of formats, all capable of delivering information in the same way. The ones from the big institutions would carry the most trust, and they’d often share each other’s information. Specialists would fill in the gaps by telling us about obscure formats like uRay and Inset PIX, or they’d provide updates about JPEG2000 and EPub more regularly than the big generalists can. The job of the big institutions is to standardize the language so we aren’t overwhelmed by heterogeneous data.

Let’s look again at those questions I mentioned, as they could apply to this scenario.

What format is it in? The common language needs a way to ask this question. Given a file extension, or the hex representation of the first four bytes of the file, you’d like a candidate format, and there might be more than one. You’d like to be able to search across a set of repositories for possible answers.

What category of information does it contain? When you get an answer about the format, it should tell you briefly what it’s for. If you got multiple answers in your first query, this might help to narrow it down.

How do I extract information? Now you want to get some amount of information, maybe just enough to tell you whether it’s worth pursuing the task or not. The registry will hopefully give you information on the technical spec or on available tools.

How do I convert it? When you decide that the file has valuable information but isn’t sufficiently accessible as it stands, you need to look for conversion software. A central registry has to be cautious about what it recommends. A plurality of voices can offer more options (and, to be sure, more risk).

This vision is what I’d like to call ODFR — the Open Digital Format Registry — even though it wouldn’t be a single registry at all.