Tag Archives: UDFR

The end of UDFR

The Unified Digital Format Registry (UDFR), created and maintained by the California Digital Library, will shut down on April 15, 2016. I don’t know whether the whole site will go away or just the ability to query the registry.

Information Standards Quarterly has an article on UDFR by Andrea Goethals. The source code repository is on GitHub.

The predecessor project, GDFR, never got to publicly usable status. The site gdfr.info still responds to pings, but apparently not to HTTP requests.

Quoting its description here, so it’s saved in at least one place if the site completely goes away:

The UDFR is a reliable, publicly accessible, and sustainable knowledge base of file format representation information for use by the digital preservation community.

A format is a set of semantic and syntactic rules governing the mapping between abstract information and its representation in digital form. While many worthwhile and necessary preservation activities can be performed on a digital asset without knowledge of its format, that is, merely as a sequence of bits, any higher-level preservation of the underlying information content must be performed in the context of the asset’s format.

The UDFR seeks to “unify” the function and holdings of two existing registries, PRONOM and GDFR (the Global Digital Format Registry), in an open source, semantically enabled, and community supported platform.

The UDFR was developed by the University of California Curation Center (UC3) at the California Digital Library (CDL), funded by the Library of Congress as part of its National Digital Information Infrastructure Preservation Program (NDIIPP). The service is implemented on top of the OntoWiki semantic wiki and Virtuoso triple store.

Format registry browser online

In an effort to promote interest in my format registry browser, I’ve built a Java web application out of it and put it up on Google App Engine at regbrowser.appspot.com. It lets you search PRONOM, UDFR, and the DBPedia structured summaries of format articles, by name, MIME Type, creator, and extension. It uses SPARQL Linked Data queries to obtain data.

It’s still in a rough form; the point is to show what it can do and hopefully get some interest in putting money into further development. Obvious improvements, which I may do shortly, would include checkboxes for which repositories to search and retention of text fields when returning to the search page.

UDFR times out a lot. If you get a timeout error, trying again has a good chance of working.

Format registry browser updated

I’ve posted an updated version of my file format registry browser (Zip file). It’s still very experimental, but this one makes several steps on the long path to being a useful tool.

The biggest news is that thanks to David Underdown’s input, it now talks to PRONOM. Preserv2 is probably a lost cause, since it appears no current work is being done on it and its useful results were folded back into PRONOM. This version tries to prettify results containing URIs and “@en” tags. If you don’t like that you can turn it off with a check box. The search fields have changed, and all of them now do something with all registries. The logging level can now be controlled from the config file (src/com/mcgath/regbrowser/config.properties). In some future version I’ll use a less buried config file.

Here’s my post on the first release.

Format registry browser available for download

I’ve made my experimental format registry browser available for download. It requires Java 5 or higher and a GUI environment, and Ant is required if you want to make changes. Currently it queries DBPedia and UDFR. It’s been tested on Mac OS X and Ubuntu.

Defining the file format registry problem

My previous post on format registries, which started out as a lament on the incomplete state of UDFR, resulted in an excellent discussion. Along the way I came upon Chris Rusbridge’s post pointing out that finding a solution doesn’t do much good if you don’t know what problem you’re trying to solve. This links to a post by Paul Wheatley on the same subject. Paul links back to this blog, nicely closing the circle.

So what are we trying to do? A really complete digital format registry sounds like a great idea, but what practical problem is it trying to solve? We know it’s got something to do with digital preservation. If we have a file, we need to know what format it’s in and what we can do about it. If it’s in a well-known format such as PDF or TIFF, there’s no real problem; it’s easy enough to find out all you need to know. It’s the obscure formats that need one-stop documentation. If you find a file called “importantdata.zxcv” and a simple dump doesn’t make sense of it, you need to know where to look. You need answers to questions like: “What format is it in?” “What category of information does it contain?” “How do I extract information from this file?” “How do I convert it with as little loss as possible into a better supported format?”

I have a 1994 O’Reilly book called Encyclopedia of Graphics File Formats. If old formats are a concern of yours, I seriously suggest searching for a copy. (Update: It turns out the book is available on fileformat.info!) It covers about a hundred different formats, generally in enough detail to give you a good start at implementing a reader. There are names which are still familiar: TIFF, GIF, JPEG. Many others aren’t even memories except to a few people. DKB? FaceSaver?

With some formats the authors just admit defeat in getting information. The case of Harvard Graphics (apparently no connection to Harvard University) is particularly telling. The book tells us:

Software Publishing, the originator of the Harvard Graphics format, considers this format to be proprietary. Although we wish this were not the case, we can hardly use our standard argument — that documenting and publicizing file formats make sales by seeding the aftermarket. Harvard Graphics has been the top, or one of the top, sellers in the crowded and cutthroat MS-DOS business graphics market, and has remained so despite the lack of cooperation of Software Publishing with external developers.

While we would be happy to provide information about the format if it were available, we have failed to find any during our research for this book, so it appears that Software Publishing has so far been successful in their efforts to restrict information flow from their organization.

This was once a widely used format, so if you’re handed an archive to turn into a useful form, you might get a Harvard Graphics file. How do you recognize it as one? That isn’t obvious. A little searching reveals you can still get a free viewer for older versions of Windows, but nothing is mentioned about converting it to other formats. Even knowing there’s software available isn’t helpful till you can determine that a file is Harvard Graphics.

If you have a file — it’s Harvard Graphics, but you don’t know that — what do you want from a registry? First, you want a clue about how to recognize it. An extension or a signature, perhaps. When you get that, you want to know what kind of data the file might hold: In this case, it’s presentation graphics. Then you want to know how to rescue the data. Knowing that the viewer exists would be a start. Knowing that technical information isn’t available (if that’s still true) would save fruitless searching.

Information like this is scattered and dynamic. If the Harvard Graphics spec isn’t publicly available now, it’s still possible for its proprietors to relent and publish it. The notion of one central source of wisdom on formats is an impossibility. What’s needed is a way to find the expertise, not to compile it all in one place.

We need to concentrate not on a centralized stockpile of information but a common language for talking about formats. PRONOM uses one ontology. UDFR uses another. DBPedia doesn’t have an applicable standard. What I envision is any number of local repositories of formats, all capable of delivering information in the same way. The ones from the big institutions would carry the most trust, and they’d often share each other’s information. Specialists would fill in the gaps by telling us about obscure formats like uRay and Inset PIX, or they’d provide updates about JPEG2000 and EPub more regularly than the big generalists can. The job of the big institutions is to standardize the language so we aren’t overwhelmed by heterogeneous data.

Let’s look again at those questions I mentioned, as they could apply to this scenario.

What format is it in? The common language needs a way to ask this question. Given a file extension, or the hex representation of the first four bytes of the file, you’d like a candidate format, and there might be more than one. You’d like to be able to search across a set of repositories for possible answers.

What category of information does it contain? When you get an answer about the format, it should tell you briefly what it’s for. If you got multiple answers in your first query, this might help to narrow it down.

How do I extract information? Now you want to get some amount of information, maybe just enough to tell you whether it’s worth pursuing the task or not. The registry will hopefully give you information on the technical spec or on available tools.

How do I convert it? When you decide that the file has valuable information but isn’t sufficiently accessible as it stands, you need to look for conversion software. A central registry has to be cautious about what it recommends. A plurality of voices can offer more options (and, to be sure, more risk).

This vision is what I’d like to call ODFR — the Open Digital Format Registry — even though it wouldn’t be a single registry at all.

The state of file format registries

Looking through UDFR is like walking through a ghost town that still shows many signs of its former promise. The UDFR Final Report (PDF) helps to explain this; it’s a very sad story of a brilliant idea that encountered tons of problems with deadlines and staffing. What’s there is hard to use and, as far as I can tell, isn’t getting used much. I don’t see any signs of recent updates.

The website is challenging for the inexperienced user, but this wouldn’t matter so much if it exposed its raw information so developers could write front ends for specific needs. Chris Prom wrote that “it is a great day for practical approaches to electronic records because all kinds of useful tools and services can and will be developed from the UDFR knowledge base.” But I just can’t see how. I wrote to Stephen Abrams a while back about problems I was encountering (including my inability to log in in Firefox — I’ve since found I can log in in Safari), and his reply gave the sense that the project team had exhausted its resources and funding just in putting the repository up on the Web.

The source code is supposed to be on GitHub, but all that I see there is four projects, three of which are forks of third-party code and the fourth just some OWL ontology files.

If it were possible to access the raw data by RESTful URLs, even that would be something. So far I haven’t found a way to do that.

In fairness, I have to admit I was part of the failure of UDFR’s predecessor GDFR. The scope of the project was too ambitious, and communication between the Harvard and OCLC developers was a problem.

The most successful format registry out there is PRONOM. Programmatic access to its data is provided with DROID. GDFR and UDFR, with “global” and “unified” in their names, both grew from a desire to have a registry that everyone could participate in. PRONOM accepts contributions, but it’s owned by the UK National Archives, and this bothers some people, but it’s the most useful registry there is. The PRONOM site itself expresses the hope that UDFR “will support the requirements of a larger digital preservation community,” and it still would be great if that could happen.

Occasionally some people have discussed the idea of an open wiki for file format information. This would allow more free-form updates than the registries, and if combined with the concept of the semantic wiki, could also be a source of formalized data. I’m inclined to believe that’s the best way to implement an open repository.

UDFR news

The Library of Congress’s digital preservation blog has an update on UDFR (Unified Digital Formats Registry). Holding a meeting of stakeholders may not be much of a “milestone,” but it shows the effort is still alive.

UDFR job openings

I’ve been informed that there are two new contract openings at the Universal Digital Format Registry (UDFR), for a project developer and a project architect. I’d be tempted myself if it didn’t mean moving to California.

The California Digital Library should have the job announcements on line shortly, though it doesn’t as I write this.