When governments and other public bodies adopt open standards, everyone wins. Consider, for example, the Open311 standard. When a city implements an Open311 endpoint, its citizens suddenly have the option of using any software which has been developed to support the Open311 standard. There’s no need for civic hackers in Chicago to develop one iPhone app, only for another group of civic hackers in New York to implement a substantially similar app, just because the two cities use different APIs.

For those developers, the use of open standards acts as a force multiplier: they don’t have to know anything about the cities where their apps are being used, because they all adhere to the same standards. Civic software is, for the most part, the domain of non-profits and individuals working on their own time. Development resources are far from unlimited, and, simply put, we have neither the time nor money to spend on what might be characterized as niche applications which only have use in a limited geographic area, or with one government’s proprietary API.

Closer to home, I’ve watched over the past few years as developers have expended needless effort building transit apps for the Washington, D.C. metropolitan area, simply to accommodate local transit authorities’ refusal to publish clean, high-quality data in standard formats.

Arlington County’s Mobility Lab Transit Tech initiative has developed two applications which rely on data from transit authorities. One is a package for driving real-time transit signs, and the other is Transit Near Me, a mobile webapp for mapping transit options.

I want to emphasize that I don’t mean to minimize the work of the Mobility Lab developers—in the end, they did what they needed to in order to be able to ship a working product, given the data they had access to.

Having said that, though, these applications are not all that different from similar transit apps which have already been built. The Mobility Lab’s real-time sign, for example, is (in terms of basic design concepts) not all that different from the OneBusAway sign mode.

Granted, the Mobility Lab’s real-time sign looks more polished, and includes support for transit modes like bike sharing, but imagine if instead of building a new piece of software from the ground up, the Mobility Lab developers had worked to polish the OneBusAway sign mode and add support for other transit modes?

Had they done so, every city which uses OneBusAway would have been able to benefit immediately from the improvements.

But, there’s a problem. OneBusAway consumes real-time transit information in the GTFS-realtime and SIRI VM formats. Out of the agencies in the region, only Montgomery County Ride On and VRE provide GTFS-realtime data. The other agencies which provide real-time data use proprietary formats which are incompatible with GTFS-realtime. Without detouring too deeply into technical territory, WMATA’s proprietary API actually provides all of the information that would be necessary to construct a GTFS-realtime feed for Metrobus, were it not for the fact that the API uses route, stop, and trip identifiers which are completely different from those in the static GTFS schedule.

The same goes for Transit Near Me; it is, in essence, a mobile version of the OpenTripPlanner system map. Cities around the world have adopted OpenTripPlanner; wouldn’t they also benefit from an interactive system map optimized for mobile devices?

OpenTripPlanner is designed to consume clean, well-constructed GTFS feeds, while instead Transit Near Me must include various work-arounds for idiosyncrasies in WMATA’s data: bad shapes which must be replaced with data from shapefiles, stop IDs which are only available in the API and not the GTFS feed, etc.

I should emphasize again that this isn’t just about the Mobility Lab; their work happens to highlight the problem particularly well, but they’re not the only developers to get caught up in this maelstrom:

@kurtraschke @fixwmata @grafxnerd Oh man I had to wrestle so many bears in their API to make that app work. SO MANY.

— Kyle Hill (@kylehill) November 5, 2012

What’s the solution? Civic hackers need to stand together with each other, and stand up for good software engineering principles. I doubt that any one developer alone will be able to convince WMATA to get their data in order (goodness knows I’ve tried). But if we stand together and recognize that reinventing the wheel over and over again is not a productive use of our time, we may be able to convince data providers to embrace open standards. When we do, it will have benefits not just locally, but for people around the world who benefit from the work of civic hackers.

More recently, I wanted to set up the D2Plug in a new application, for which I intended to upgrade to Debian unstable. It was then that I began to understand what was really going on with the D2Plug. As I discovered, support for the D2Plug’s hardware simply never made it into the mainline kernel.

As such, you’re stuck with an old kernel version, and dependence on Globalscale’s source tree.

I would not characterize the D2Plug as an expensive paperweight just yet. There is a kernel fork which aims to provide a modern kernel for the CuBox, a platform similar to the D2Plug, so there’s still some hope.

However, without a modern kernel, the D2Plug will indeed become an expensive paperweight soon. As distributions move to the 3.x kernel series (as many have already), it will become harder and harder to run modern software on the D2Plug. It is for this reason that I was ultimately unable to upgrade the D2Plug to Debian unstable.

Though there is a considerable amount of hype surrounding the Raspberry Pi (a similar, though considerably lower-specification ARM board), you may be surprised to find that it, too, lacks mainline kernel support as of this writing. Granted, the Raspberry Pi comes with a suitable OS image, and other distributions, like Raspbian, are now available, but it’s nothing like the x86 world where you can download a generic installer and have it work out-of-the-box.

Of course, some plug computers and other ARM devices do have mainline kernel support; this isn’t a condemnation of every ARM device out there. It’s also possible that the D2Plug will eventually get mainline kernel support, although that seems unlikely; the last attempt appears to have been in May 2011 when Greg Kroah-Hartman asked “What’s with the reliance on the 2.6.32 kernel tree? That is quite old now, has anyone done any forward porting of the code yet?”

Considering the Raspberry Pi’s popularity, it seems much more likely that it will get mainline kernel support, and I look forward to being able to run stock ARM builds of Linux distributions on the Pi, in which case I’d be much more likely to buy one. So long as it is dependent on custom builds, the risk is that it, too, could end up stuck on old distributions like the D2Plug is.

Unfortunately, it’s not a success story: the app has never been used in production. As the article explains, “Muni hopes to put the app to good use some day, but the agency is $29 million over budget and cannot afford to buy the iPads required to run the software…[nor] is the city willing to invest $100,000 to run a pilot program.”

The costs involved here—a few hundred dollars each for some iPads, perhaps a few thousand dollars to fund a stipend for a civic coder, even $100,000 for a pilot—pale in comparison to the costs associated with the big-name IT consulting firms that governments are used to dealing with.

In addition, startups, teams of civic coders, and open source projects can often deliver a working prototype or even a completed project much faster than conventional development teams. As the New York Times describes, “a small team of volunteers took just 10 days last summer to create [the app].”

Unfortunately, the City of San Francisco is out of touch with the realities of technology: “‘Start-ups fail at a high rate,’ said Jay Nath, chief innovation officer of San Francisco. ‘As stewards of taxpayer dollars, we need to be thoughtful of using that money wisely and not absorbing too much risk.’” Nath is right about one thing: start-ups do fail at an alarming rate. But that’s not the risk you might think it is, because startups aren’t like conventional development projects.

Unlike conventional projects, startups fail fast. Instead of wasting years and millions of dollars, when a startup has an idea that isn’t going anywhere, it winds up quickly. Maybe it was a bad (or even outright infeasible) idea to begin with, or the startup had the wrong team, or they tried to do too much at once. Maybe their idea’s been superseded by a newer, even better technology. Whatever the reason may be, the startup doesn’t just grind away for years, running up a million-dollar bill. Instead, they admit that they can’t deliver, and get out gracefully.

Consider, for example, the FBI’s Virtual Case File, a five-year, $170-million development effort that never actually delivered any working software. Imagine if the VCF project had failed after three or six months, not five years. Imagine if it had spent less than a million dollars before failing, not $170 million. Of course, the project still wouldn’t be done—but we’d have known that something was wrong up front, instead of finding out five years later, after millions of taxpayer dollars had been wasted on a doomed development effort.

More importantly, startups do have the agility necessary to keep up with the ever-changing technology marketplace. A development effort that takes five or ten years is bound to deliver a product that is obsolete as soon as it arrives, unless major changes are made along the way.

The conventional development practices used by many government agencies and their contractors don’t incorporate that kind of agility. Specifications and requirements are written early in the project’s life, perhaps even before a development team has been selected (if the project must be put out for bids). Even if the requirements are found to be lacking—or flat-out wrong—development marches on. In the end, the team will deliver a product that meets the requirements (thus satisfying the bean-counters) but which is already out-of-date and which doesn’t actually do what users need it to do.

I alluded to these problems in my recent coverage of WMATA’s initiative to install real-time information displays at bus stops. By only considering bids from vendors with “standard, proven products” and “successful existing and fully operational implementations, in multiple transit agencies”, they potentially shut out innovative startups (or even teams of civic coders, like the Mobility Lab).

It’s entirely possible that the first team to tackle a thorny problem may fail—but rather than casting them as “failures that burn holes in the city’s budget”, we’ve got to communicate to governments and taxpayers alike that not all failures are the same. There’s a big difference between a project that runs for years, spends millions of dollars, and has nothing to show for it in the end, and a project that fails after just a few months, has spent well less than a million dollars, and can identify what went wrong, so the next project will be more successful.

When it comes to technology, the best way for governments to be good ‘stewards of taxpayer dollars’ is to adopt successful development practices: small, agile, competent teams, that build inexpensive, flexible products, and fail quickly if they can’t get the job done. The old way—forking over millions and millions to high-priced contractors until they finally declare defeat, then taking it up in a years-long legal battle—just doesn’t look like good stewardship anymore. Sure, established companies may have a long track record that startups don’t, but what’s it a record of? We don’t need any more million-dollar failures. We need smart civic coders developing next-generation solutions like SMART Muni, and we need governments to accept, embrace, and support them.

You can see the issue on any individual post page, such as this one from the demo site:

Basic Maths post page rendering in Chrome 17.0.932.0 on Mac OS X 10.6.8

Basic Maths post page rendering in Firefox 5.0.1 on Mac OS X 10.6.8

Notice how the borders at the top and bottom of the meta box overrun the first sidebar column in Chrome.

Not being a CSS wizard, I’m not in a good position to derive a minimal test case from the theme’s CSS, and I can’t say precisely when the issue first appeared in Chrome, either (although I will say that I run the Chrome dev version). I know this makes for a bad bug report, but I figure it’s better than nothing to document the issue publicly. The issue doesn’t seem to be specific to certain Basic Maths installations, either; both this site and the demo site, as well as others I’ve come across online, are similarly affected in Chrome.

However, if you look at the files more closely, you’ll soon see why they’re structured the way they are: they are, simply put, massive. The problem is in the stop_times.txt file; the largest, for Brooklyn, is nearly 700 megabytes. Concatenate them together, and you get a 2 gigabyte file containing more than 30 million records. (This is a result of how the feeds are constructed, as dissected in this thread on the developer mailing list)

Most tools designed for working with GTFS feeds simply can’t handle anything that large (or they choke badly). Yet, at the same time, many tools also assume that there will be a single feed per agency, so the per-borough feeds (which have some degree of overlap) can be something of a pain to work with.

This leads to a conundrum: you can work with the feeds one borough at a time (although even then, with some difficulty, as even the individual borough feeds are rather large), but there’s no good way to see the whole city’s bus service at once.

It turns out that with some ingenuity, this problem can be solved, although doing so takes some time and CPU resources. The basic strategy is to first naively merge the feeds together, and then refactor the merged feed, to reduce the number of stop times. The refactoring is described in this post by Brian Ferris.

Actually merging the feeds together isn’t that hard; the agency.txt, calendar.txt, calendar_dates.txt, routes.txt, and shapes.txt files are identical across the five feeds. The stops.txt file has to be merged and then deduplicated, but this can be done with simple command-line tools. For the trips.txt and stop_times.txt files, there’s no other option than to concatenate them together. This does result in a massive stop_times.txt file, but it’s only temporary.

After producing the naively concatenated feed, apply the previously-mentioned OneBusAway GTFS transformer (described in more detail here) to the feed.

The transformer will need about 8 GB of memory to run (so launch the JVM with -Xmx10G, or thereabouts), and on an EC2 large instance, it’ll take about 10 minutes. When it’s done, you’ll have a stop_times.txt file which contains around 6 million records, which isn’t quite so bad (considering that the entire merged and refactored feed for the five boroughs ends up being about the same size as the unmodified feed for Brooklyn alone, it’s actually almost good).

As an aside, here’s how I constructed the merged feed; I’m always a fan of solutions which make use of basic Unix tools.

mkdir nyct_bus_merged
cd nyct_bus_merged
cp ../google_transit_manhattan/{agency.txt,calendar.txt,calendar_dates.txt,routes.txt,shapes.txt} .
 
for file in ../google_transit_{manhattan,bronx,brooklyn,queens,staten_island}/stops.txt; do
	tail -n +2 $file >> stops_unmerged.txt
done;

head -n 1 ../google_transit_manhattan/stops.txt > stops.txt
cat stops_unmerged.txt | sort | uniq >> stops.txt
rm stops_unmerged.txt


head -n 1 ../google_transit_manhattan/trips.txt > trips.txt
for file in ../google_transit_{manhattan,bronx,brooklyn,queens,staten_island}/trips.txt; do
	tail -n +2 $file >> trips.txt
done;

head -n 1 ../google_transit_manhattan/stop_times.txt > stop_times.txt
for file in ../google_transit_{manhattan,bronx,brooklyn,queens,staten_island}/stop_times.txt; do
	tail -n +2 $file >> stop_times.txt
done;
#then zip the feed and apply the GTFS transformer

(Finally, a disclaimer: I haven’t extensively tested the feed which is the result of the process described in this post. It’s possible that this process has unintended consequences which could affect its integrity or usefulness for certain applications.)

At a bare minimum, posting the BibTeX file online somewhere would fulfill my desire to get the information out there. But not everyone out there who might benefit from the bibliography uses BibTeX. For many people, I fear a .bib file would be nothing more than unintelligible gibberish; outside of academic circles (and even then, outside of the hard sciences), TeX is not particularly well-known.

The next alternative would be to post the bibliography online as a PDF or HTML file. This alternative is considerably more accessible to non-BibTeX users, but actually makes life harder for people who would like to be able to copy references (as BibTeX source) to use in their own BibTeX files (common practice in communities of TeX users). Merely rendering the entire contents of the file also loses some of the metadata—the comments associated with entries, the groups and keywords, etc.

There are also specialized tools (like bibtex2html) for converting a BibTeX file to HTML. But there, still, the results fall short; the output is mostly static text. I wanted a tool that would make good use of the keywords entered in BibDesk, and which would provide links between publications and authors. I also wanted a tool which would be equally useful for BibTeX users, who would be helped by having access to the BibTeX source for each entry, and non-BibTeX users, who would be helped by having formatted bibliography entries. I therefore set out to built a tool that would meet my needs; the result is htmlbib.

One of the items of concern for me was that the bibliography entries be formatted properly; after having taken care to make sure that the information was added to BibDesk so that it would be rendered well, I did not want to have some generic template used to create HTML for each entry. So, I ended up cobbling together an arrangement that actually uses BibTeX and tex4ht to produce HTML for each entry using the desired BibTeX style (in my case, IEEEtran), so that the entries look the same in the preview as they would in an actual publication. This is slow, but the preview results are cached, so subsequent runs are faster.

As for parsing the BibTeX file, since I’m already familiar with scripting BibDesk, I decided to use appscript to call BibDesk from Python. The result is therefore not portable from OS X, but it suits my needs. There are BibTeX parsing libraries for Python, so porting to another platform would only require substituting one of those libraries of the calls to BibDesk; the rest is pure Python, with the exception of lxml, and the aforementioned preview code, which expects a functioning TeX installation on the system.

The HTML is produced using Jinja2 templates, which for now are stored in the application egg. The default, built-in template is built on Blueprint CSS and jQuery along with jQuery Tools. It wouldn’t be too hard to provide an option for using user-specified templates instead of the built-in template.

I’ve uploaded some sample output to demonstrate what htmlbib does.

Browser automation tools, like Selenium, and libraries like mechanize help fill the gap somewhat, but they’re far from providing the same rich environment that AppleScript does. To give a concrete example, I was recently working on a spreadsheet listing Twitter accounts for the top 50 transit agencies in the US (more on that project here). In the spreadsheet, I’d listed agencies’ accounts by username (that is, @username). But what I really wanted was a link to each account on Twitter (that is, http://twitter.com/username). I could have entered the links manually, but that would have required needless manual work. If I were using a conventional spreadsheet application on the desktop, I could have used whatever macro or scripting facility it provided, or I could have exported the file to CSV and used sed and awk to get the job done. But I was working in the cloud; I knew there had to be a better way.

Enter Google Apps Script. Google Apps Script provides for Google’s cloud-based applications the same scriptability that AppleScript provides for desktop applications on the Mac. In only a few minutes, after studying the documentation, I was able to produce a script which achieved the desired effect.

Buoyed by my quick success, I decided to try going a step further: what if I could use the Twitter API to automatically set each cell’s comment to the most recent Tweet? Doing so would give viewers a quick preview of the Twitter account’s content, without leaving the spreadsheet. Working off of some sample code from Google, I quickly wrote another script to do the job. I ran into trouble for a while until I found that the “Callback URL” in the Twitter application settings must be set to https://spreadsheets.google.com/macros; once that was done, everything worked perfectly. (Incidentally, the error message given in that case, “unexpected error”, is completely useless, and gives no clue as to the actual problem.) From there, all I had to do was set up a time-based trigger to run the script automatically so the Tweets would update periodically, and I was done.

For me, the real point—and the power of Google Apps Script—is how quickly and easily I was able to not only automate otherwise-tedious processes, but draw in data from disparate sources and display it automatically. I’ve only scratched the surface of what can be done with Google Apps Script; the technology can be made to do a lot more.

This isn’t meant solely as a criticism of the LASR site; I’ve noticed this trend on a lot of ‘design’ sites. Lots of flash (even if they’re not built with Flash), and not a lot of usability. These sites break the Web in a variety of ways; one of the more egregious ones is not using real hyperlinks. You can’t link to the individual pages within these ‘sites’, and they break the Back button, too. Broadly speaking, these are practices which break the Web. Unfortunately, it’s started to infect even mainstream Web sites, like the new Gawker and Twitter. This is, needless to say, a bad idea.

Fancy transitions between pages are neat, but there is no need for a static Web site to require JavaScript. Doing so goes completely against the notion of progressive enhancement. And while it’s (somewhat) reasonable for a Web application like Twitter or (especially) Gmail to require JavaScript, a static site—whether it’s the news or the results of a design competition—should not depend on JavaScript, and it should have real hyperlinks that actually work. That’s how the Web works. Fail to get that right, and you’ve broken the Web.

The answer is the shortDOI service, which transforms DOIs into shortcuts that are a lot easier for your readers to use. Every shortDOI generated is itself a DOI, so the conventional risk of a URL-shortener shutting down and taking the shortcuts along with it isn’t a problem. As long as the DOI system is functioning, shortDOIs will be resolvable.

For example, the DOI 10.1109/JRPROC.1929.221679 can be dereferenced by using the URL http://dx.doi.org/10.1109/JRPROC.1929.221679. When this is shortened with shortDOI, the result is the DOI 10/bpc. This can be dereferenced with the URL http://dx.doi.org/10/bpc (note that that’s no different than any other DOI), but, more importantly, it can also be dereferenced with the URL http://doi.org/bpc. It’s this last URL that is important for our purposes, as it’s the shortest.

Now, shortDOIs can be manually generated, but why bother, if the process can be automated? I use BibDesk for managing references, and BibDesk is a scriptable application, so an AppleScript was the easiest solution to the problem. I’ve posted the script on GitHub; you can find it here. For every publication in a BibDesk document which has a DOI entered and which does not have a shortDOI shortcut, it will retrieve the shortDOI shortcut for the publication’s DOI, and store it in the URL field.