New York State’s Office for Technology released a Request for Public Comment on selecting an XML-based office data format. The choices are OASIS’ ODF and Microsoft’s OOXML. Responses were due by 5 p.m. today, Dec. 28. My response is below, submitted just in time to meet the deadline. I didn’t have time to answer all the questions, so I focused on the ones I felt I could address in the greatest detail.

RESPONSE TO RFPC # 122807:

Background Information

For the past year I have been the lead programmer on AltLaw.org, a project to promote public access to federal court opinions by creating a free, full-text database of those opinions with an easy-to-use search interface. In the process of developing this web site, we have encountered many obstacles because of the way the federal courts store and publish their records. The problems we have encountered using electronic government records illustrate issues that New York State should consider in developing its electronic records policy. I will provide more details in my answers to the questions below.

Question 2

To encourage public access to State electronic records, it is important that the electronic record be the official State record rather than a draft or proxy for the “official” paper document. The greatest weakness of AltLaw.org as a legal reference is that the opinions we download from the federal courts’ web sites are not the final, official versions; those are published by and only available from the West corporation, at considerable cost.

Since the federal courts rely on West to copy-edit and correct their opinions, they are careless with respect to dates, names, and other important data. For example, we have downloaded several opinions that were decided in the year “2992″!

Furthermore, opinions published on federal court web sites lack any citation information — which is also assigned by West based on the pagination of their print volumes — making them useless for legal scholarship or court preparation.

As most professions (including the law) come increasingly to rely exclusively on electronic sources of information, it is critical that those sources become 100% reliable. To this end, electronic State records must be 1) complete, 2) accurate, 3) easily cited, and 4) acceptable for use in all official State business.

Question 3

To encourage interoperability and data sharing with citizens, business partners, and other jurisdictions, it is important that State electronic records be machine-readable. This is a more demanding requirement than simply having records in electronic form. A PDF document, for example, is electronic, but it is difficult or impossible to extract discrete data from a PDF document. This is because the PDF format is optimized toward preserving the visual appearance of a document rather than its structure.

There are two issues to consider when creating machine-readable documents. The first is “metadata.” Metadata is information about a document that may or may not be contained within the text of the document itself. For example, the metadata for a court opinion would include the name of the court, the date the opinion was released, the name of the judge writing the opinion, and the names of the parties in the case, among other data. Since most federal courts publish their opinions on their web sites in PDF format, with no metadata, AltLaw.org must rely on custom software to extract essential metadata such as titles and dates. The process is slow, difficult, and inaccurate.

It is worth noting that ODF supports metadata using the Resource Description Framework (RDF), an international standard which already forms the foundation of powerful data-analysis software. OOXML does not provide comparable metadata support.

See: http://blogs.sun.com/GullFOSS/entry/new_extensible_metadata_support_with

The second requirement for creating machine-readable documents is information about document structure. Document structure includes elements such as sections, headings, paragraphs, lists, and tables. These structures must be identifiable in the document independent of the visual formatting used to display those structures. For example, a human reader can recognize bold-face type as a section heading, but a computer program cannot. Structural information is important for automated document analysis, information retrieval (search), accessibility to physically-impaired users, and conversion to alternate formats (such as HTML). ODF provides more structural information than does OOXML.

In addition to including metadata and structural information in electronic documents, the State should implement rigorous standards to ensure that information is produced consistently. Metadata is only useful when it is stored in a known, consistent format. For example, simple information such as a date can be recorded in a dozen different ways. A date written as “02/03/04″ could be February 3, 2004 or it could be March 4, 2002. Work on AltLaw.org has shown us that there are as many ways of writing dates as there are courts to write them, and dates are by far the simplest piece of metadata to store. For New York State, if different State agencies (or, worse, individual State employees) were to record metadata in different ways, it would be almost as useless as having no metadata at all. After selecting a format for electronic records, the State must then establish formal procedures for using the metadata capabilities of that format. These procedures should be made freely available to the public for the purposes of encouraging interoperability and data sharing.

Question 4

To encourage appropriate government control of its electronic records, the State should rely wherever possible on public-key encryption technology. I am not an expert on this subject, but I urge the State to consult with computer security and encryption experts when choosing the protocols to implement. When properly used, public-key encryption can provide communications that are secure and verifiably authentic to a degree exceeding that of physical documents.

Question 5

To encourage choice and vendor neutrality when creating, maintaining, exchanging, and preserving electronic records, the State should consider the vested interests of parties promoting a particular data format. A format developed by a single corporate entity, such as Microsoft in the case of OOXML, gives that entity strong incentives to design the format to make interoperability difficult to achieve in practice, either through incomplete specification or proprietary extensions. OOXML has been criticized by others as being difficult to implement because of both of these factors. In contrast, ODF was developed by the independent, international OASIS group, with input from a variety of sources. To ensure the success of ODF, its creators have a vested interest in making interoperability as easy as possible. Thus, in the future, ODF is likely to be supported by a wider range of vendors and products than OOXML, and its adoption will promote greater competition in the markeplace.

Question 7

Regarding public access to long-term archives, the State should ensure that its archived records are available in bulk. “In bulk” means that a computer program should be able to obtain large quantities of archived records in an automated fashion, without human intervention. While developing AltLaw.org, we were often forced to write computer programs that simulate the behavior of a human user clicking through a court web site, as that was the only way to download opinions from those sites. In contrast, some court web sites make all their opinions available for download via FTP, a very simple Internet protocol designed for bulk file transfer. The latter made our job easier, and better promotes public access to archived records.

In general, State agencies should not take responsibility for providing the public with the tools to search and retrieve electronic records. Those tasks are better handled by corporate entities (such as Google) and non-profit institutions (such as AltLaw.org) that have technical expertise in those areas. The State should take responsibility for making consistent, accurate, complete data available in bulk at little or no cost to users of that data.

Question 10

Regarding the management of highly specialized data formats such as CAD, digital imaging, Geographic Information Systems, and multimedia, the State should use open, published, freely-available standards whenever possible. When open standards are not available for a particular type of data, the State should attempt to make that data available in as many competing formats as possible. For example, many database and statistical applications which use proprietary data formats can “dump” their data into simple, standardized formats such as comma-separated values (CSV). Imaging software which uses proprietary formats can usually convert files to non-proprietary standard formats as well. Wherever possible, these conversions should be “lossless,” that is, they should not lose any information in the conversion. Ideally, it should be possible to completely reconstruct the specialized data from the information contained in the simpler data formats. These practices provide insurance against future loss or corruption of data in highly-specialized formats.

Conclusion

I strongly urge the State to prefer ODF to OOXML. As one engaged in extracting data from large quantities of government records (over half a million documents, at last count), I would find ODF much more conducive to enabling my work than OOXML. I have looked at examples of the internal XML schemas used by both formats, and I find ODF much easier to read, understand, and manipulate than OOXML.

I really like LaTeX. So much so that I bought a used copy of the original LaTeX “blue book” just so I could write a class file to print my freshman English papers in MLA format, which requires breaking most of the typesetting rules that make LaTeX output look so professional to begin with.

But there’s no question about the ugliness of LaTeX (or plain TeX) source. At times it borders on incomprehensible. LyX helps, but only if you like LyX’s editor.

When I was an office temp I saw dozens of people struggle to typset 100+ page documents in Microsoft Word. Word is a pretty powerful tool, but it’s just not up to the task. Small, maddening inconsistencies appear that are difficult to correct. Large documents require a lot of memory and cause crashes. And when you have multiple people working on the same document, only one of whom understands how to use styles properly, it becomes nightmarish.

The trouble with WYSIWYG is that what you see is all you get. Why should an author — who should only be thinking about the content, not the presentation — be faced with decisions about line and page breaks while working on a first draft?

So here’s what I want: an editor that looks and feels like Microsoft Word, but that only permits structural editing — section headings, emphasis, etc. Then use TeX or a similar typesetting system to generate printed documents. To make this useful to a general audience, one would also need a “style editor” to modify the behavior of the typesetter.

The problem with this is that the idea of “structural editing” seems quite alien to a lot of computer users. People like WYSIWYG. They want to work on screen with something that looks like a finished document. I believe this is actually a very inefficient way to work, since one is distracted by formatting concerns from the actual writing, but I don’t know how to convince anyone else of this. But if the “structural editor” could be styled to match the user’s expectations of what the final document will look like, we could have the best of both worlds. Writers will be able to work in an environment they already know and feel comfortable with, and editors and publishers will be spared the frustration of fixing formatting inconsistencies introduced by writers.

When I was a kid, I took piano lessons. Every day, after I couldn’t put it off any longer, I would sit down and practice a piece of music. Whenever I made a mistake, I would stop, go back to a point just before the mistake, and start again. The problem with this technique was that I never learned to play anything all the way through without stopping. Even if I didn’t make a mistake, I would automatically stop and repeat the difficult parts several times before continuing. This became an ingrained habit that took ages to break.

A few years later, my family bought a PC, which I used to type my school assignments. Like everyone else in the world, I quickly discovered how easy it was to edit on the screen. No messing about with white-out or correction tape, just “backspace” and start again. I learned to type on a computer, so I never learned to type without the crutch of a backspace key. For a long time after that, I automatically hit the backspace key after certain words I typically mistyped, even if I had typed them correctly.

Now I have a modern word processing application which will check my spelling, grammar, capitalization, and even my formatting while I type. One would think this would leave me free to concentrate better on the writing itself, but I find the reverse is true. All those little red squiggles distract me from the words I’ve written. I don’t care if my words are misspelled until I’m finished writing and into the editing phase.

I think today’s word processors are too focused on the editing phase, and their “helpful” distractions make it more difficult to write coherent prose that flows logically from one sentence to the next. I find myself falling into a stacatto, disjointed pattern of short sentences and disconnected paragraphs, more like an outline than prose.

Gary King wrote recently, “It’s up to us to ensure that the tools we use are optimizing the important tasks, not the trivial ones.” That’s why when I want to get any serious writing done, I launch Emacs full-screen.

This poster hangs in my cubicle. The caption reads, “If a pretty poster and a cute saying are all it takes to motivate you, you probably have a very easy job, the kind robots will be doing soon.”

Besides saying “nyah nyah” to the superiors who never stay in my cubicle long enough to read the caption, I more or less believe what the poster says. My job is very easy. In a few years, a robot will be able to do it.

One of my primary job functions is to be a human interface to technology, so that the important people don’t have to be bothered to learn how to use their computers. With better A.I. — not even strong A.I., just better language handling and learning algorithms — a computer could learn to interpret vague requests and do the right thing. The important people wouldn’t even lose the ability to blame their mistakes on their underlings — they could just blame their computers, and people nowadays are more understanding of computer errors than human error. And with handwriting and voice recognition, no one would ever need a human typist.

The second function of my job is searching for, sorting, and organizing information. Computers are great at that already, and soon software will bridge the gap from today’s web search, e.g. “Seth Greenberg San Francisco lawyer,” to “Find the phone number of the lawyer I talked to at the conference in San Francisco last month.” Pattern recognition systems are already widely used in areas such as finance and medicine. A.I. — again, not strong A.I., just better — will give us general-purpose pattern recognition systems that can be applied to any type of data. Internet research is already very useful, and “search agent” software that can learn to recognize items of interest will be eventually be more thorough than a human researcher. Currently, the public web lacks sufficient accurate information for serious research. The really good sources are locked up in proprietary databases with their own restricted search interfaces. Those interfaces will need to be opened up to third-party search systems — which does not preclude charging for the content in the database itself — to remain competitive in a software agent-driven research market.

The third major function of my job is basically minor physical labor: printing, photocopying, stuffing envelopes, mailing, and generally trasporting physical documents from one location to another. Once we more fully embrace digital communications, this task will dwindle down to nothing. This embrace will depend on a few things: 1) computer displays that are equal or superior to printed text for reading long documents, 2) a legal framework that accepts electronic “signatures” as valid, and 3) a generational change of attitude.

So I will, within a couple of decades, be replaced by a robot. There will not be an android sitting in my cubicle, but a combination of better hardware, smarter software, and cultural changes will leave me with nothing to do. Hopefully by then I’ll be doing something more interesting.

I think nearly all computer users can be divided into three broad categories based on way they think about computers.

The vast majority of computer users are application-oriented. They have training and experience exclusively with commercial software. They understand concepts peculiar to computers such as files, folders, saving, and deleting. They live in a WYSIWYG world; although they may be aware that what they see on the screen is not an entirely accurate representation of what the computer is actually doing, they are not interested in understanding hidden implementations. They have learned how to map their thinking onto the capabilities of the applications they regularly use, and they accept whatever limitations that thinking may impose. They are pragmatic, learning as much as they need to get their work done. A new problem requires a new piece of software. Spreadsheet “programmers” fall into this category, as may some programmers who work primarily with application scripting tools such as Microsoft Visual Basic for Applications. Nearly all commercial software is targeted at this group of users, who can be considered “computer-literate.”

The second largest group consists of goal-oriented users. These users focus exclusively on the goals they want to accomplish and neither understand nor care about the software they use to accomplish those goals. This could be called the “I just want to type a letter” group. They only see the final product. They do not care about, for example, the difference between a word processor document and a PDF image of that same document, so they do not understand why they can make textual edits to one and not the other. They could be described as “computer-illiterate,” even if they work with computers on a regular basis. Many very intelligent people, scientists and scholars, fall into this category. They are frustrated by the limitations of the software they use because they do not understand the reasons for those limitations. Though they may use common terms such as “files,” they typically map those concepts onto their real-world metaphorical analogues, resulting in confusion. (“Why can’t I keep this picture in my email file?”)

The third and smallest group of computer users — ironically, the original computer users — is comprised of hackers. Hackers are computer-oriented computer users. They have learned how to think like a computer, to understand the processes the computer goes through. They favor small tools (e.g. the command line, shell scripts) over large applications because they want to be in precise control of what the computer is doing at all times. They comfortably work with data in “raw” formats such as text files. This does not necessarily mean they are tied down with minutiae of implementation; often they can work at much higher levels of abstraction than other users. Hackers tend to seek out the abstract patterns inherent in whatever end result they are working towards, then implement those patterns in the computer. A new problem rarely requires new tools, merely a new application of existing tools. They will create whatever new tools are needed to bring the computer up to the level of the problem, rather than trying to adapt the problem to fit the computer. On the other hand, their solutions tend to be brittle, with a lot of exposed complexity that makes them unsuitable for non-hacker users.

Nearly all commercial software is application-oriented, while most open-source software is hacker-oriented. Very little software, in my experience, is truly goal-oriented.

In looking around the offices where I have worked, I see innumerable places where existing technology could be leveraged to expedite, simplify, or otherwise enhance day-to-day workflow. Things could happen faster, cheaper, with fewer staff. So why isn’t this done? I think there are two fundamental problems:

The first is the disconnect between the potential of the technology and the knowledge of those who use it. There’s an awful lot of software out there. I expect almost no one understands all the capabilities of even standard office suites. I have been called a “Word specialist” by temporary employers, but I certainly won’t claim to know every feature of Microsoft Word. I list Excel on my resume, but Excel has hundreds of features I know nothing about. It is difficult, even in this day and age, to hire quality technical personnel with sufficiently broad knowledge to combine the capabilities of different software packages in useful ways.

The second disconnect is between those who know how to use the technology and those who direct how it is to be used, i.e. management. Dilbert’s pointy-haired boss notwithstanding, I don’t really believe managers deserve the blame for misuse of technology. They lack training and experience, but they also lack technical personnel who are sufficiently skilled at realizing and applying the potential of technology. This problem is only made worse when managers learn something of technology’s potential without learning the complexity of the steps required to implement it.

So we have not one but two vast gulfs of understanding. No wonder everyone keeps muddling along with half-baked solutions. I don’t know how to fix this. Yet another piece of software is obviously not the solution. I think nothing short of a paradigm shift in the way software is designed and used would suffice.