Tag Archives: software

The Java file format API graveyard

If you look for Java libraries to support specific file formats, you’ll soon come upon the gloomy graveyard of Java APIs. Sun and Oracle have a history of devising nice packages for reading and writing different kinds of files, only to abandon their maintenance. You can still find pages for them, and it takes a close look to figure out that they aren’t supported any more.

Java Advanced Imaging (JAI) was nice in its time. It still has a page on Oracle’s website, but the latest “what’s new” item is dated 2007. The page brags about customer success stories as if it were still usable code. I’ve tried working with it. It’s out of sync with the current com.sun classes, and I got only limited use out of it. In its time it was a good way to read and write image files.

Java Media Framework (JMF) runs on a 166 MHz Pentium or 160 MHz PowerPC. The downloaded jars are dated May 1, 2003. It had a nice list of supported formats.

If you’re working with audio files, javax.sound looks more encouraging. Its API is listed with Java 8. The class java.sound.sampled.AudioSystem supports reading and writing of audio files. I can’t find a list of the supported formats.

Java does reliably support some formats. Its handling of text encodings is versatile, and java.util.zip handles ZIP and GZIP.

Third-party code can come to the rescue. For reading and writing PDF, Apache PDFBox looks like the best bet. You can use Apache Tika with lots of formats, if you just need to extract metadata. Another alternative is to use ImageMagick, but it runs natively rather than under the JVM, so you have to invoke it with exec calls. im4java and JMagick can save some of the tedium. There are open source Java libraries for reading and writing specific file formats. Some may be good, some not.

If you need to deal with the guts of file formats in Java, you’ll usually have to find some good third-party code or start writing your own.

JHOVE PNG module, progress report

There’s now a JHOVE PNG module on my GitHub site. The relevant new classes are com.mcgath.jhove.module.PngModule and everything in the package com.mcgath.jhove.module.png. I could have continued from Lauri’s code as I mentioned in my previous post, but I like a more factored approach, so I continued with my own code, which has a separate class for each chunk type. Take a look at the top-level file FORKNOTES for what I’ve been doing.

It does a pretty decent job of validating files and extracting metadata now, but some chunk types are still ignored, and there are some design decisions on the extracted metadata that I’m not sure about yet. Also, JHOVE modules usually have a lot of metadata about themselves, and that’s not complete yet. If anyone wants to play with it, keeping in mind that it’s not stable code yet, please do and submit issue reports for bugs and suggestions.

JHOVE and PNG

A few days ago, I started writing a PNG module for JHOVE, partly to keep my Java skills up, partly to help me understand the PNG format. After a while I noticed there already is code for a PNG module and has been for a long time. I must have added it to SourceForge. According to a note in the code, Gian Uberto Lauri at Engineering Ingengeria Informatica S.p.a. created it in 2006. A good amount of work clearly went into it, but it won’t compile. It’s located in a non-source code directory (extramodules/it/eng/jhove/module/png/PngModule.java), so I had to copy it to src/java to try it out.
Continue reading

DROID 6.2

The UK National Archive quietly released DROID 6.2 this month. I noticed only because of some mentions on Twitter. The file dates indicate the update was released on February 16. Here’s the new portion of the changelog:
Continue reading

File fuzzing

Recently I came across the term “fuzzing” for intentionally damaging files to test the software that reads them. Most of the material I’ve found doesn’t provide a useful introduction; they assume that if you know the term, you already understand something about it. One good article is “Fuzzing — Mutation vs. Generation” on the Infosec website. According to that article, fuzzing denotes the response to file changes rather than the changes themselves, but I’m seeing the term used mostly in the latter sense.
Continue reading

fixit_tiff, a TIFF repair tool

The Sächsische Landesbibliothek – Staats- und Universitätsbibliothek Dresden (Saxon State and University Library Dresden), which somehow gets abbreviated to SLUB, has developed a tool for working with TIFF files in digital preservation. fixit_tiff is a command line utility, written in C, which can do some repairs on defective TIFF files. The focus appears to be on correcting common errors, not on repairing corrupted files. A blog post from July (in German) indicates it can do configurable validation using a simple query language.

It’s available under the same license as Libtiff. Just what is that license? The only thing I can find is a very outdated “Use and Copyright” statement, which is on a page so old it warns about patents on LZW compression. It’s available for free, anyway.

MRF for large images

NASA is using a format for online files, called MRF (Meta Raster Format), which is claimed to deliver images ten times as fast as JPEG2000 from cloud services when used with a compression algorithm called LERC. LERC is under patent by Esri, which says the technique is especially suited for geospatial applications and makes the algorithm “freely available to the geospatial and earth sciences community.” An implementation of MRF from NASA is available on GitHub under the Apache license, and an implementation of LERC is on GitHub from Esri.
Continue reading

Aside

Due to a misunderstanding of mine, there wasn’t a free preview lecture with my course on file format identification tools, even though the promotion video said there was. I’ve rectified that, and the introductory lecture is now available for viewing … Continue reading

Course on file format identification tools: Progress report

The video course which I’m developing on “File Format Identification Tools” is almost ready to submit to Udemy. I’m holding off for a little more work at the Open Preservation Foundation on JHOVE, because some user interface details are going to change from the current beta (1.12 beta). The other tools covered will include file, DROID, ExifTool, and Apache Tika. This course should be useful to both students and professionals who want to learn how to use the tools.
Continue reading

File identification tools, part 10: Siegfried

“Do we really need another PRONOM-based file format identification tool?” That’s what Richard Lehane asked rhetorically last year on the Open Preservation Foundation blog. It was obviously rhetorical, since he’d gone ahead and done just that with a new tool called Siegfried. Siegfried recently turned up in some tweets by Ross Spencer, so it’s worth a mention here.
Continue reading