Did I never blog this? Eejit! Here's the LAARC API

I have a feeling I never blogged about the existence of the LAARC API. Currently this is only simple search and we've not as yet "eaten our own dog food" i.e. rebuilt the LAARC catalogue site itself on top of this - that's waiting for the advanced search API, though I couldn't guess at a delivery date for that...
I think I'd meant to do a big explanatory post and since I didn't get round to that just never mentioned it. So for now I'll just get the word out and say, this is the work of Julia Fernee who re-engineered the whole back-end of the LAARC system to make it work sweetly, fixed stuff broken by a change to Mimsy XG, got digital downloads behaving again and ironed out various other un-noticed bugs. Ultimately it's all still based on the work of Sarah Jones and later Mia Ridge, but Julia's work now puts the database in a place where we can build from. Play with the API and let us know what you think.

Museum of London websites down

...and will be for a bit. Our just-appointed Head of ICT, Adam Monnery, is doing his bit. With any luck things will be running by the weekend but don't hold your breath.

Shakespeare's first theatre

Here's the BBC story and video about a bit of news we'll probably see in a few other places soo, the discovery of what's thought to be the first theatre to show the work of William Shakespeare (and where he also worked as an actor). Though Museum of London Archaeology is going through a tough time at the moment they are, as ever, involved in some very exciting work. Maybe a little less romantic that Romeo and Juliet itself, but thrilling enough.

Museum of London API enhancements

I've been extending the MOLA publications database lately so that it now includes archaeological sites to which the publications relate. For pretty much all of these we have decent geographical point data (as we should, having dug most of them) and of course this is great for playing around with the data. So I've got a KML output going for that web API, which returns placemarks for sites corresponding to publications returned by your query. In the placemark are links to all the publications related to the site. You can also go the other way round: either query by the publication's ID to get its details and (in KML mode) related sites; or query by site, which returns simple XML for each related publication or, if you choose KML, returns all points related to books linked to that site (potentially lots of points, not just the one in your query). What it all means is that you can see your search results in Google Earth or on a map if you like by pasting in the URL for the KML e.g. http://www.museumoflondon.org.uk/museumoflondon/food/rest.aspx?source=pubs&mode=kml&period=roman
So far I've not put this into the public interface because I think to do so requires some consideration, but this will come. I'm pleased in part because I can start to use this API in my own behind-the-scenes integration. For example I've also just done a load of work on the site summaries that we publish for all the work that MOLA does. The KML for these is cleaned up and the old ASP/XSLT thing I did to search across these (and other reports) by borough has been refreshed. It now lets you search by site code, and now that I can get at publications via site code it's a pretty small step to get it also to return related publications, which should come very soon (perhaps before going-home time). Because they're in very different data sources (XML files and SQL Server) it wasn't so straight forward before, now it is. The publications API is mainly for internal consumption like this (it also runs the user-facing publications page itself, at a lower level). I'd be really pleased, though, if others find a use for this, and any thoughts about the rights and wrongs of how I've done it would be gratefully received.

Clarification

Following my off-the-cuff post about the Museum of London's strike (in which I did not participate, not being a member of a union), it's become apparent that there is some confusion as to the status of this blog. It's important, therefore, that I make it very clear that the contents of The Doofer Call are a personal expression only. This is a research diary for my PhD that I've chosen to make public in order to engage in debate and help people find things that interest me; my profile states this. However, because I talk regularly about things I'm doing at the Museum I can see that casual readers might mistakenly believe that I am writing in an official capacity, or with official sanction, and any confusion is my fault and for me to address. Hence this post.

Friday's missive may turn out to be factually incorrect (one might argue that the pay at MoL is not rubbish, or that the pay rise was not over a year late) or inaccurate in its representation of the reasons for the disgruntlement that led to the unions striking, but that's not really the point. What's important is that it's clear that what I wrote was not an official press release, but the completely unofficial, independent expression of someone who is also employee, written at home, out of office hours (like this).

So for clarity, I am posting a disclaimer that can also be read as a declaration of interests. Please read it here.

My apologies to anyone who previously mistook this blog to be anything other than my thoughts in my words, I hope this has made things clearer.

Note: all of the organisations with which I have a formal relationship are included as tags on this post in the hope that people searching for the organisations and stumbling across this blog will also see this clarification.

Victorian photographers API now also alphatastic

Didn't manage (didn't dare) to upload this before going-home time on Friday. There is now another string to the bow of the nascent REST alpharama here. You can query the people in the database of 19th Century photographic London (pardon the contraction) at the PhotoLondon website we launched recently. Right now, it only sends you back a list of people matching your criteria (100 records per page) with no details. I will soon produce a detailed machine-friendly record page for each person (an example of a human-friendly one is here)

The querystring takes most of parameters you can see on the form including surname, forename, search text (but not multiple words like the form), gender, year of birth/death, "alive in..." year, place of origin, photographic occupation, non-photographic occupation, and presence of attached images. Here's an example search so you can see what you'll get back right now. Here's a wider one.

There are bugs, no doubt, and it will be of more use when I can send you the person's details, as well as a list of countries and occupations to build queries from. I realise I also need to write some proper documentation. Still, it's a start. Please tell me if it might be useful to you, and what you'd like to see it doing/do with it.

Small API update

A couple of small advances on the API front (again, see here)

• fixed a bug on the geo thing. For some reason an imbecilic code error wasn't breaking the script on my machine, but did on the web server. Now fixed.


• a CDWALite-lite output for individual object records (example). There's more to add, glitches to fix, and ideally a better solution to the URL, but it's a start. Next thing is a search interface but that depends upon agreement within the Museum. A good solution may be to combine CDWALite and OpenSearch-style RSS, with the records enabling users to find the data end-point, as well as the HTML rendering. In due course I'll probably add tags to HTML record pages to point at data like this, or I may do it with some POSH.


• the photoLondon website data now has a basic API, which I'll put on the live site next week. It returns basic person details and search parameters include: surname, forename, keyword, birth year, death year, "alive in" year, gender, country of origin, photographic occupation and non-photographic occupation. I'll work on the search result format soon, as well as the person details.

MoL APIs live (but very alpha)

Well there's more to do, but an alpha version of three services is now there if you want to play. All of them currently put out only XML, no JSON or raw text, GEDCOM or whatever else, but this may change and new services may be added. Have a read here. There is an events database, publications from our Archaeology Service, and a sort of geocoding tool. I've rejigged the code so that adding a different (XML) output format to these involves just writing XSLT and putting in a new value for the "mode" parameter, rather than fiddling around with C#, recompiling and all that. All feedback most welcome. But don't bug me about a collections API!

To start you off, here are links to one request from each service.

• events API. This won't work forever as the events will expire. Uses the format that Upcoming outputs with xCal, DC and geo extensions
• publications API
• the geothingy converter whatsit

Made an API

No, not a collections database interface, Mia's onto that one. But I'm scheduled to work on the events database about now, and with the mashup day (now full up, I think) coming it seemed a good idea to grab a day and finish off this job. For ages we've had an RSS feed which gets a lot of use, including consumption by sites like dockland.co.uk, and of course search functionality onto the events web pages, but the feed is (a) a bit basic and (b) just shows everything for the next 14 days.
I wanted to turn the static feed into a properly searchable REST interface, as well as load it with more good data. So I plugged a hole whereby only a start date could be specified, put in filters for audience, event type, and keyword (like the web pages), and neatened up the separation into XML. Now it can be churned out in raw straight-from-the-database XML or in RSS 2.0 with extra hCalendar. I've got a couple of tweaks to make (adding in addresses and geo data) and then will invite the world. The plan is that you can set the format in the query string, so RSS2 will be one option but so will xCal (as used in Upcoming's XML) and whatever else - it'll just pick up a stylesheet and apply it on the fly. There are more filters I could apply, and documentation to do. Then thinking about pagination with OpenSearch extensions... and then who knows. Most of it will wait, so I'll post soon with the details on the where and how.

The slightly better shape we're in

Well one thing I didn't mention yesterday was the good news, which is that Friday saw the official launch internally of a new set of guidelines for "digital programmes". This is something that Pete had been working on for some time, doing exactly what I've been arguing for from the point of view of planning, not just for current usefulness, but for longevity. That is, he makes explicit links between the strategic and business aims of the organisation, and our digital activities. From there he turns this into a set of principles or activities that must be followed/carried out by "any member of staff considering the creation of a digitally based resource either as an item in its own right or in support of an exhibition, publicity campaign or event ". It's a very thoughtful document and a really big step for us. In fact it's pretty big full stop - 25 pages big.

Another part of the document outlines the planned ICT committee (heavy on the web), and gives a dozen strategic considerations that underlie the document such as centralised programme assessment, interoperability, technical fit, and some interesting ones like flexibility, which relates to the ability of potential partners to respond to our needs.

There's plenty more in there. There are still holes to fill, but I'm hopeful that this document will make a difference to how commissioning takes place from now on, with fewer nasty surprises for us and better allocation of resources for all.

The shape we're in (with a musical digression)

Today I'm still making the most of the quiet to dig into my vinyl and I'm working to the sounds of the german underground, mainly circa 1970-75. Right now, though, Phew (1981). This just so blows my mind. Can fans must have it, and they can get it on CD now (perhaps I will too, it's so hard to get at my LPs most of the time). Given that I've been writing about what our organisation needs to do to get its digital act together perhaps its good that the previous listening has been pretty calming: Alpha Centauri, Tone Float and Outside the Dream Syndicate (not that mellow, actually, the last one). Phew has lovely peaceful moments (like Dream, now), but a driving, anxious motion to some tracks too (Signal). The legendary Roger of defunct Revolver Records in Bristol sold me my copy, I don't think I've thanked him enough and probably never will, but thanks anyway, Rog.

I've been thinking about the holes in our policies (notably concerning the preservation of UGC - we may not want to bother, but we have to at least formulate a policy towards it), and I've been looking back over the 6 years I've been there and thinking about how our responsibilities have developed. We need to work up a business case for a larger team, and perhaps one of a different shape. Frankly it's a no-brainer, but the case needs to be made and it's a good exercise to outline those changes. We also need to work out with the Godot-like web committee (I'm sure it will be along any day now, with purposeful stride and a keen sense of what needs to be done) quite what role we want the web folk at MoL to have. There is a wide range of things we could be doing, and right now we try to do most of them, with some key exceptions like web design and rich media work (Flash, video), which we always farm out. But we do too much and can't always do it properly, and I feel the need right now to consolidate, to work on our infrastructure. If we can decide on exactly what we should be done in-house, and what can be effectively handled out of house, then we can get and give a clear idea of just what resources we need. Right now we have our content manager Bilkis (the one net gain since 2002), Mia, who does a lot of web work as well as various databases, and me, essentially just web.

Personally I reckon we need more people in the team, but there are alternatives, even if they're idiotic: the Museum could decide to draw in its horns and use our expertise just to commission, integrate and manage third party work, as well as content. We certainly do plenty of advising on this at the moment, and most of the vendors we've used recently have done great work for us (please drop me a line if you want to know whose work was not up to scratch), but I'm one of those who think the core work of a museum web team must also include taking care of the plumbing and probably building/running the core CMS too. Perhaps when I've finished my musings I'll post a proper list here of what functions I identify, what I think we should take into our remit, and what I think we should hand out. Best get on with it, then.

OSGB-lat/long web service for GIS

We have loads of geographical information. Trouble is, it's almost all in OSGB grid reference form, which is no good for feeding to apps like Google Maps. Worse, much of it uses old-style 100km squares (mainly TQ, which covers London). We've taken a couple of approaches to this - hand-making a few maps in Google (for example, our Olympics work is mapped here), and using batch processing scripts from ESRI or others to manipulate the data in the geographical fields of our archaeology database, creating latitude and longitude values to accompany the OS grid references. However there is still a good set of data that needs another approach - for example, our site summaries for the last decade and more are available online. These XML-driven pages contain only OSGB data. I plot them very crudely onto our own ESRI-based map application, but would much rather have KML to work with.

So to the point. I came across a great script on the Hairy Spider site that also runs a web service. I wanted to take this further so that (a) I could pass in lots of values, not just one at a time (b) it could handle the TQ-style syntax (c) not keep on hitting friendly Mr Spider's server. The code available is for the convertion from OS eastings and northings to lat/long only and I've not tried to reproduce his "proper" web service, but I do now have something that will work for my needs. I can pass in a querystring like

gr=tq709098,SW465987,tl123456,51232456,512300245600

and get back something like:



50.8618352280453
0.431144625126249
tq709098


50.7318874223712
-5.59642485933937
SW465987


52.0968934973257
-0.358719120254802
tl123456


52.0968934973257
-0.358719120254802
51232456


52.0968934973257
-0.358719120254802
512300245600



[note that the last three values in the query were all for the same grid reference but in different formats, producing the same lat/long]
For me this is pretty useful. I may well extend this to take more parameters and pass out KML, but the main thing is having a means to convert the data on the fly over HTTP.

Many thanks to Hairy Spider for doing all the hard work on this. I've tested the outputs and they're very close to the OS's own tool, so good work HS!

Next thing is to use this. I'll let you know when I do, and we might also be open about making the service publicly available if that would be of assistance to anyone that might be reading this.

Standing back for a moment [cross-post from mymuseumoflondon.org.uk]

[cross-post from mymuseumoflondon.org.uk]

Hi, it’s the web-monkey again. Things have been pretty intense lately, due in large part to the end of the financial year and the need to wrap up all sorts of budgets.

My part in the various projects I work on ranges from major to peripheral - sometimes some serious programming, sometimes offering advice on commissioning, sometimes just doing a little tweaking ready for integrating someone else’s work. All the same I’ll flag up a couple of things I’ve been involved in lately, at least of those that have now launched, even if I didn’t do that much myself - after all, where else do we sing about some of this stuff? Too often it ends up sort of dribbling out because we’re all too busy or exhausted to make a song and dance about it. So, here we go:

• The Great Fire of London website, orientated at children of Key Stage 1 age (5-7) and their teachers. This is the result of a partnership between the Museum of London, National Portrait Gallery, The National Archives, London Metropolitan Archives, and London Fire Brigade Museum. It’s cool. Thanks to ON101 for building the game and designing the site, and our own Mariruth Leftwich for shepherding the whole thing. Also via Mariruth comes a game to complement our Digging Up the Romans learning resource.


• At last we have sort of launched “The Database of 19th Century Photographers and Allied Trades in London: 1841-1901“. This is the electronic representation of the amazing work done by David Webb in cataloguing thousands of people in that industry in Victorian times. I built the database, hmm, several years ago for another partnership we’re in, but it was never launched for reasons that even now seem obscure. Anyway, it’s now live and, though it needs an overhaul even now, it’s great to think it may at last start being useful. I want to open the data up for mash-ups….when I get some time.


• The Sainsbury Archive, a fantastic resource at Museum in Docklands, has a new site through the efforts of archivist Clare Wood


• I can’t tell you about the work I’ve been doing on republishing an archaeological reference text, because it’s not ready yet. If you can find the test URL, well, you’re very sneaky.


• Any day now we’ll see the launch of the “Family Favourites” pages on the Museum in Docklands website. Go and seek it out, there’s a fun game and an introduction to various highlights of the galleries there.


• It’s just a promo site until the exhibition itself happens, but have a look at the Jack the Ripper pages. That’s gonna be well worth a visit - get yourself some tickets!


• Geek stuff: some time ago I made a machine-friendly interface to look at the database of publications our archaeology service (MoLAS) produces. Whilst working towards the launch of http://www.museumoflondonarchaeology.org.uk/ I decided I wanted to change the architecture of the publications application, which for one thing makes it easy to drop little nuggets of info about our publications around the site, all fed from a database. The solution I went for also works for machine access by anyone, and I hope it will be just a start: we’d like to make our events available like this, and in time our collections. For the record, it’s basically REST/XML, drop us a line if you want to use it (though I imagine that it will be the collections and events that will have wider appeal - note that events already have an RSS feed, which is used on sites like docklands.co.uk).


• And check out our events programme, I’ve just uploaded the May to August programme.

Now, what have I forgotten to mention?

Of course, there’s more in the pipeline, keep your eyes on all our sites!

What's new

Stuff seen:

SearchMe beta, a search engine which shows a visual results (images of the web pages) categorised (as e.g. museum, art, shopping, fishing). Quite nice. Silverlight I think. It's a bit SW (in its results clustering, for example), though how it goes about doing this I don't know, but other "semantic" search stuff has shown up lately. TextWise (small "sw", I guess) has just been reviewed by TechCrunch, which was doubtless part of the point of offering a $1m prize for suggesting uses for its technology. Hakia is another such.

Stuff I've been doing the last week or two:

• the Great Fire of London site for Key Stage 1 kids finally soft-launched.



• working on templates for the Londinium site - the bulk of my time right now



• preparing the digital republication of an out of print handbook for identifying roman pottery fabrics. I probably mentioned it before, it involved the export of Quark to PDF, the export of PDF to XML, translation via several XSLT steps and manual clean-up to TEI-Lite, and finally modification of some XSLT to display this as an HTML page. Most of this was a while ago; right now I'm getting ready for the images which will need to be embedded once all the scanned thin sections are ready.



• testing out and integrating Flash interactives with our CMS. Several are pretty much ready for launch, including two from the London Sugar and Slavery exhibition, and two games



• advising as best I can on the development of the replacement map interface for the LSS gallery



• fretting over the re-branding exercise the MoL group is engaged in. how much work is it worth doing right now to fix issues on the sites if we'll be overhauling the whole thing in the autumn?


• testing new search engine SearchMe (see above). Didn't get good results for "roman london" yet, but it's only indexed a billion pages or so....
  

Shock discovery: poor communication wastes resources

Yesterday I found out about some work that's being done on our behalf by the company that advertises our jobs. At the request of our HR people, they've almost finished building some custom pages to mimic the look of our group portal pages. Trouble is, the web techs (in this case, me) were never part of the discussions and I can't help but feel that we'd probably have got things done very differently if we had been. Certainly I have some concerns about what we'll get and I would have liked to explore other options. What are my concerns? They can mostly be expressed adequately with the word "sustainability":

• Content maintenance. It mimics the look of our CMS pages, but the content isn't integrated with our CMS. Changes to site structure won't be reflected in the menus, nor would updated content.
• Visual maintenance. The look of this site will change (we dearly hope) and I can't change their pages
• Google. I don't know how they look upon sites that look like copies of existing sites and point at their pages. I suspect it might look like spamming and I wouldn't want to be blacklisted.
• Site stats. We can't (readily) integrate the job site's stats with ours (if we get them at all). Not a huge deal to me but a factor.
• Cost. I don't know what this will have cost, but five minutes after getting hold of an RSS feed from their site I had integrated it into our own, replicating the most important part of what they'd done. I suspect we could have done it cheaper, in short!

I don't know if, had we talked about this properly, we would have ended up doing something different. It depends upon what the important parts of the "site" are, and on what the job site can offer beyond a pretty sparse RSS feed, but I think we could have negated the need for at least some of what they did. There's meant to be a process we follow for every single new media project, so that it passes by the right eyes to let us make any recommendations. This time that broke. HR did speak to people on our team but the plans didn't reach me (at least, not that I recall), and I think this has something to do with a communication failure about just what the plans involved. I may not have made it clear the sort of things we are able to do in terms of integrating third party sites (or at least what we'd be up for trying), or perhaps the scope of HR's plans weren't really clear to start with. Or perhaps I just need to make it clear that every single piece of third party work, no matter how small it seems, must go through me. One way or another, our communications have been lacking and it looks like we've ended up doing things the wrong way.

Small new app?

Probable reuse of events RSS feed or perhaps more directly from the data access layer: a screen in the Museum of London foyer to show the day's forthcoming events. We have most of what's needed to do this at present but need to (a) tweak things to show just the (remainder of) the current day and (b) to resolve questions of how to make it look nice, given that people aren't attaching images to events in the database at present, though they can.