Category Archives: commentary

OOXML: The good and the bad

Posted on June 27, 2014 | 1 comment

An article by Markus Feilner presents a very critical view of Microsoft’s Open Office XML as it currently stands. There are three versions of OOXML — ECMA, Transitional, and Strict. All of them use the same extensions, and there’s no easy way for the casual user to tell which variant a document is. If a Word document is created on one computer in the Strict format, then edited on another machine with an older version of Word, it may be silently downgraded to Transitional, with resulting loss of metadata or other features.

On the positive side, Microsoft has released the Open XML SDK as open source on Github. This is at least a partial answer to Feilner’s complaint that “there are no free and open source solutions that fully support OOXML.”

Incidentally, I continue to hate Microsoft’s use of the deliberately confusing term “Open XML” for OOXML.

Thanks to @willpdp for tweeting the links referenced here.

1 Comment

Posted in commentary

Tagged , ,

The uses and abuses of PDF

Posted on June 13, 2014 | 2 comments

PDF is a versatile format, but that doesn’t mean it should be used for everything. It’s a visual presentation format above all else. It lets you define a document with a specific appearance, with capabilities such as form filling and text searching. It’s not very good if you want a document that adapts to different device capabilities. If you need an editable format or a way to deliver structured data, there are much better alternatives.

When the Malaysian government released satellite data from the communications of Flight 370, which had disappeared, it delivered a PDF file. It looks very nice, but anyone who wants to extract and analyze the data has to do a lot of extra work. A spreadsheet or structured text (e.g., CSV) document is the right thing.

PDF can be used for e-books, but it’s not ideal. If you create normal sized pages, then on a phone they’ll either look tiny or require a lot of scrolling. Formats such as EPUB work better with a range of screen sizes.

Delivering text documents in PDF loses a lot of its value when the document is a scanned image rather than a text-based document. It can’t be searched, and people with visual disabilities can’t use text-to-speech. My condo association delivers its newsletters in scanned-image PDF. When I pointed out these problems at an owners’ meeting, I was told that the owners weren’t sophisticated enough to take advantage of those benefits. Our complex is a big one, and I’d be surprised if at least a few residents don’t use text-to-speech when they can. It’s not particularly hard to generate PDF files; scanning a finished document into a PDF seems like the hard way.

To maximize the usefulness of assistive technologies, you should create PDF/A if possible. It produces a slightly larger file, but it’s organized in a way that makes extraction of content easier and eliminates dependencies you might not have thought of.

Redacting PDFs is another tricky issue. If you simply black out an area, that’s the equivalent of gluing a piece of paper over it, and no harder to defeat. For advice on properly redacting documents, who better to turn to than the NSA? They may be a gang of criminals within the government, but they certainly know how to redact. It’s from 2006, though, so some of its advice could be dated.

There are lots of things you can do with PDF, but use it intelligently and where it’s appropriate.

2 Comments

Posted in commentary

Tagged

Ninjas, samurai, and artists

Posted on January 12, 2014 | Comments Off on Ninjas, samurai, and artists

Lately I haven’t been posting as much on this blog. My professional responsibilities have shifted, and much as I still love the issues of file formats, I don’t have as much time to give attention to them. There are still general programming issues that are worth blogging about, though, and I’ll occasionally address these issues here, hopefully along with occasional file format posts.

cover thumbnail, Secrets of the JavaScript NinjaThis weekend I borrowed a book from the company library called Secrets of the JavaScript Ninja. It’s a better book than I expected from the title.

In an inspired error, the cover shows not a ninja but a samurai, with colorful armor, a banner, and a long sword. A ninja is a hit-and-run assassin; he shows up out of nowhere, attacks, and vanishes. A samurai is a dedicated soldier, the Japanese equivalent of a knight, and he follows a code of honor. The software development world has too many ninjas. The samurai is a better, if not ideal, model.

From the title I was expecting a cookbook, one of the many books that provide formulas to follow but no deep understanding. Instead, Secrets of the JavaScript Ninja is about understanding the language. Such books are even rarer for JavaScript than for most programming languages. I’d always tended to think of JavaScript as a half-baked derivative of Java, one where features such as classes, packages, and inheritance were left out to cut it down to a scripting language. This book, though, shows that it’s really quite a different language, a very powerful one in its own right. I still think the language has serious problems, the biggest being lack of standardization, but reading through the book, I’m learning how to think in JavaScript’s terms and to make use of features which other languages don’t have.

It’s not the language I want to talk about here, though; it’s the approach to any language or software technology. Too many programmers don’t have any deep understanding of their craft; they just have a bag of tools that they expect to solve problems for them. The worst can’t do much more than run a Web search for the code they need or beg on Stack Overflow for a solution to their problem. Once I actually had to fix some PHP that consultants from a big-name company had pasted from a website and didn’t know how to adapt to the problem at hand — and I don’t even know PHP! Those are your ninjas.

Even the samurai isn’t a great metaphor. They were a part of Japan’s entrenched feudal culture, and their opposition to capitalism promoted the warlike mindset that culminated in Japan’s role in World War II. Our metaphors should be based on creativity, not war and violence. We should think of ourselves as architects, sculptors, artists. We learn a craft and master the tools that go into it. There’s a real psychological affinity; I know a lot more software developers who are skilled musicians than are skilled fighters.

When I see a book called Secrets of the JavaScript Artist, then I’ll be pleased.

Update: I just noticed that the book itself says: “Ninjas were chosen for their martial arts skills rather than for their social standing or education. Dressed in black and with their faces covered, they were sent on missions alone or in small groups to attack the enemy with subterfuge and stealth, using any tactics to assure success; their only code was one of secrecy.” Just what you want in a programmer, right?

Comments Off on Ninjas, samurai, and artists

Posted in commentary

Tagged , ,

In MS Word, the bullet bites back

Posted on December 27, 2013 | 1 comment

There’s nothing new about Microsoft’s ignoring standards and ruining compatibility, but knowing the details is useful. One case I just learned about, from Mark Mandel, is the way it does bullet lists. This applies to the old Word DOC format on Mac OS X.

A 2008 OpenOffice Forum discussion explains the problem. If you create a bullet list in Word and import it into OpenOffice, the bullets are turned into something odd-looking. The file doesn’t use Unicode bullets, but instead uses the Microsoft Symbol font, which has its own nonstandard encoding. This applies only to bullets generated by list styles, not to ones you type in. On Windows, OpenOffice will display the files correctly, since it has access to the needed fonts and mapping.

Apparently the issue can also be manifested when creating a DOC file with OpenOffice and importing into Word, though I’m not clear on how that happens.

The problem is that Word 97/2000/2002 isn’t fully Unicode-compatible, mapping Unicode characters to the 8-bit encodings that its fonts need. This has presumably been fixed in the more recent versions that use DOCX (Office Open XML), but DOC is still widely used as an interchange format, so it’s an important issue. It’s also an illustration of the risks of using undocumented interchange formats.

1 Comment

Posted in commentary

Tagged , ,

Tools come and go, effort must be ongoing

Posted on October 1, 2013 | Comments Off on Tools come and go, effort must be ongoing

In a comment on a JHOVE bug, I said offhandedly that it’s approaching the end of its life. This caused a certain amount of concern in Twitter discussions. Andy said that software tools are one of the best ways to “preserve specific, reproducible knowledge about processes.” I don’t think dropping support of a rather dated tool is a big concern, though, as long as the code doesn’t vanish.

A software application is good for a certain number of years before it needs to be either left as legacy code or completely rewritten. Throwing out code and starting over takes a lot of effort, but it can result in much better code. I started on JHOVE in 2003 as a contractor to the Harvard University Libraries. After a few years it became clear that some of the design decisions weren’t ideal. Its all-or-nothing approach and its tendency to give up after the first error have long been obvious problems. The PDF module is a kludge built on a crock, and that’s without even talking about its profiles. The TIFF module, on the other hand, has a fair amount of elegance.

JHOVE2 was supposed to be the successor to JHOVE. Its creators learned from JHOVE and produced a better design. What they didn’t have was enough time and money to cover all the formats that JHOVE covered. I’ve continued to work on JHOVE because I know it inside and out. Someone else could pick up the work, but it might make more sense for a newcomer to the code to join the JHOVE2 effort instead. However, Maurice noted on Twitter that there hasn’t been much activity lately on JHOVE2 issues.

Both JHOVE and JHOVE2 were funded under grants. When the grant money ended, progress slowed down. The one-time grant model is the wrong way to fund preservation software. It’s an ongoing effort; new formats arise and old ones change, and there are always bugs to fix. What I’d like to see happen is for major libraries in the US to create an ongoing consortium for preservation work, similar to the Planets project in Europe. Or better yet, a consortium bringing together libraries all over the world. It wouldn’t take a lot from any individual institution. Its job would be to maintain information, preservation tools, test suites, and so on, on an ongoing basis. Instead of rushing to create a tool and then leaving it to freelancers like (formerly) me to maintain, it would support maintenance of tools for as long as it made sense and creation of new ones when it’s appropriate.

My voice isn’t enough to call anything like this into existence, but I can hope.

Comments Off on Tools come and go, effort must be ongoing

Posted in commentary

Tagged , ,

Rescuing Macintosh Files

Posted on September 6, 2013 | Comments Off on Rescuing Macintosh Files

On Wednesday, September 4, 2013, I talked with a small gathering of the Mac Tech Group at MIT on “Rescuing Macintosh Files.” There was a good discussion, with several people contributing valuable points.

The computer presentation which I used is available as a Powerpoint or OpenOffice document. The PowerPoint one had some problems at MIT with displaying all the images, so if you have a choice the PowerPoint one may work better.

Comments Off on Rescuing Macintosh Files

Posted in commentary

Tagged , , ,

TIFF/EP vs. Exif

Posted on July 11, 2013 | Comments Off on TIFF/EP vs. Exif

I just discovered today that there are two different TIFF tags called “FocalPlaneResolutionUnit.” Tag 41488 goes by this name and is part of the Exif tag set. Accepted values for it are:

  • 1 = No absolute unit of measurement
  • 2 = Inch
  • 3 = Centimeter

Tag 37392 is a TIFF/EP (Electronic Photography) tag (working draft, final version not available online), also used in other raw formats, including DNG. Its accepted values are:

  • 1 = Inch
  • 2 = Metre
  • 3 = Centimetre
  • 4 = Millimetre
  • 5 = Micrometre

Recently I was sent a TIFF file, as a JHOVE issue, that had a tag 41488 with a value of 4. JHOVE correctly, but perhaps confusingly, reported that the fFocalPlaneResolutionUnit tag had an invalid value.

There are other tags in TIFF/EP that are equivalent, or nearly, to Exif tags. In some cases their values are identically specified, sometimes not. The Exif SubjectLocation tag is numbered 41492 and always has two shorts for its value, giving an X and Y value. The TIFF/EP counterpart is tag 37396, which can also have three shorts (specifying a circle) or four (specifying a rectangle).

I don’t know how this came about, but it’s something to watch out for in software that deals with both Exif and TIFF/EP tags. Some software may accept the EP extensions for Exif tags, but there’s no guarantee this will work.

Comments Off on TIFF/EP vs. Exif

Posted in commentary

Tagged ,

Google Reader is gone (yawn)

Posted on July 1, 2013 | Comments Off on Google Reader is gone (yawn)

feedAs of today, Google Reader is gone. (Correction: It goes away at the end of today. You still have time to export your feed list.) When its termination was announced, some writers declared it meant the end of RSS feeds. From what I’m seeing today, the attempts at panic have died away, replaced by a realization that RSS and Atom are well-understood feed formats and that lots of alternatives exist. Tristan Louis writes for Forbes:

While the death of the most popular RSS reader on the internet could have been seen as something that would represent a grave danger for RSS as a standard, for openness as a concept, and for heavy news consumption, the inverse has been true, as it only solidified RSS’ position in the world as the format for news delivery. Reader was a good product but one can hardly call it a great product and its demise will help rectify some imbalances it created in the market.

Hopefully dedicated users backed up their feed collections as an OPML file. If not, all they have to do is start collecting feed URLs again.

There’s no need to use a website at all to manage your feeds. On my iPod, I use Free RSS Reader, a simple, straightforward reader, though unfortunately it’s no longer being updated. On my main computer I use Sage, a Firefox extension.

A few columnists got a temporary boost in readership and a long-term loss in credibility by proclaiming the demise of RSS. The rest of us are still fine.

Comments Off on Google Reader is gone (yawn)

Posted in commentary

Tagged , ,

Audio and video in HTML5

Posted on June 2, 2013 | Comments Off on Audio and video in HTML5

I’ve been studying up on streaming audio and video and related issues, so lately I’ve been playing with the <audio> and <video> tags in HTML5. It’s possible to put them to good use, but there are more issues than their proponents will readily admit.

A good piece of news is that both tags do exactly the same thing except for their appearance. You can play video with the audio tag and vice versa, and they implement the same DOM model. (Of course, you won’t see anything interesting if you use <audio> for video.)

The main limitation is that these tags support only progressive streaming, which differs from “true” streaming in some important ways. Progressive streaming means downloading a file and starting to play it almost immediately, rather than after it’s finished downloading. Its disadvantages are that the bit rate can’t be adjusted while playing, you can’t keep the file from being grabbed in its entirety with a simple HTTP call, and the download continues to completion even if the user pauses the player. These aren’t always significant problems, but they mean that the new HTML5 tags aren’t the full replacement for Flash which they’re sometimes claimed to be.

There’s enough interest in true streaming that various parties have developed protocols to do it over HTTP. These include HTTP Live Streaming from Apple, HTTP Dynamic Streaming from Adobe, Smooth Streaming from Microsoft, and Dynamic Streaming Over HTTP from MPEG (which its proponents insist isn’t a protocol). There are more details on streaming on my website.

The other problem with the HTTP tags is that there’s no one encoding that all major browsers support. This problem is well known on the video side, but I was surprised to discover it’s even true for audio. The current version of Firefox doesn’t natively support MP3 in the audio tag, and the QuickTime plugin isn’t used in this case (or at least I can’t get it to work). The reason for this is software patents. There’s a good discussion of the state of MP3 with Firefox on Stack Overflow.

You can specify several <source> elements within an audio or video element, and the browser will try each one in turn till it finds one it can play. Two formats or at most three will cover all major browsers. For audio, including both an MP3 and an Ogg Vorbis version should cover all the bases; for video, MP4/H.264 and Ogg Theora should do it, though you might want to add WebM.

Specifying the type attribute as the MIME type of the file (e.g., <source src="song.mp3"
type="audio/mpeg"> helps the page to load faster, since the browser can determine without examining the file if it can play the file in principle. Make sure, however, to use only the canonical MIME types. From experimentation with various browsers, these include:

  • audio/mp4
  • audio/mpeg
  • audio/ogg
  • video/mp4
  • video/ogg
  • video/webm

If you specify application/mp3 rather than audio/mpeg for an MP3 source, the browser may decide it can’t play it even though it really can.

Another issue is the AV API for HTML5. There’s a pretty decent DOM API to go with the audio and video tags, allowing you to override the player controls and dynamically change content. Some implementations (e.g., Mozilla’s version) have added private extensions. Some people want more power, so there are third-party plugins and JavaScript libraries such as MediaElement.js that extend the API.

It’s a minefield, except that the mishaps come from the absence of an earth-shattering kaboom. Still, using the HTML5 tags is much simpler than Flash or HTTP streaming.

Comments Off on Audio and video in HTML5

Posted in commentary

Tagged , ,

JHOVE and XHTML

Posted on May 31, 2013 | 4 comments

I’m surprised I only got a complaint about this recently. Using JHOVE to validate XHTML files is often painfully slow. In fact, using anything to validate them without caching or redirection of DTDs would be painfully slow. The DOCTYPE declaration brings in the standard XHTML DTD, and it in turn brings in lots of other DTDs. These all have URLs on w3.org. As you can imagine, this is a lot of traffic converging in one place, and the response is often very slow.

JHOVE has a remedy, but it turns out not to work in this case. In the configuration file, you can declare local copies of schemas and DTDs to be loaded by the SAX entity resolver. This looks something like this:

 <module>
   <class>edu.harvard.hul.ois.jhove.module.XmlModule</class>
  <param>schema=http://www.w3.org/TR/REC-smil/SMIL10.dtd;/Users/gmcgath/schemas/SMIL10.dtd</param>
 </module>

Unfortunately, there are some problems in JHOVE 1.9. The HTML module processes XHTML files by passing them to the XML module. In this case, the module doesn’t get the parameters that the config file declared for it. In JHOVE 1.10, I’ll fix this by having the HTML module pass its own parameters to the XML module. At present, JHOVE’s processing of XHTML files makes no use of the configuration file’s instructions to the entity resolver.

There’s another complication. The XHTML DTD invokes other DTDs, and JHOVE has to get every one of those in turn. Some of them have relative URLs to other DTDs; these break when they’re redirected to local files. Even making local copies of all the files doesn’t work, as JHOVE doesn’t handle the relative URLs correctly within the file system, and making them work would require changing some existing assumptions. The best fix for the user is to get JHOVE 1.10 when it’s ready (version 1.10B2 doesn’t have the XHTML fix yet) edit all those files so that all the URLs are absolute.

This is a big chunk of work, and I haven’t tested the approach fully. Any ideas on how this might be better handled would be appreciated.

4 Comments

Posted in commentary

Tagged , , ,