About Me

My photo
Web person at the Imperial War Museum, just completed PhD about digital sustainability in museums (the original motivation for this blog was as my research diary). Posting occasionally, and usually museum tech stuff but prone to stray. I welcome comments if you want to take anything further. These are my opinions and should not be attributed to my employer or anyone else (unless they thought of them too). Twitter: @jottevanger

Thursday, December 22, 2011

Three years with Spinal Muscular Atrophy III: one parent’s take

Since people reading this post are probably not regular readers of ths blog (who is?), a quick word of warning: I don't do concise. I could have split it into multiple posts but that's just cheating, so this is what you get.

This post could alternatively be entitled 4½ years with SMA type III, or even 6 years, but last month our third and youngest child turned 6 and shortly before that was the 3rd anniversary of our getting the diagnosis of his condition, which had first become noticeable (but unrecognised) when he was about 18 months old. It feels like a good time to talk a bit about what has happened in the interim, not least because I know that as a parent receiving a diagnosis of SMA type III it was difficult to get a grip on what it would mean for our child and for our family as a whole (we have two older children, aged 8 and 10 as I write). Perhaps by putting this out there it might come in the way of some other mother or father anxiously googling the diagnosis they have just had for their beloved child, and maybe reading about our experience will, if not console them, then thin out the terrifying cloud of the unknown that seemed to suddenly appear before us. No two cases of SMA are the same – it is a spectrum disorder (type III is at the mild end), and you cannot take one experience and expect to see the same pattern elsewhere. All the same, hearing about a single case is better than hearing about none. I also wanted simply to celebrate our kid, whose disability is merely a strand of his life and whose character and spirit, experiences and growth are unique, beautiful, special – just like those of our other two children. SMA is not the definition of his life but an aspect of reality for him (and to a lesser extent for all of us). So this is going to be a few snapshots of his and our experiences over the last three years, including a few pictures and videos, to generally illustrate that you too will cope and your kid will thrive in their own way.
To kick off, and coz I love all three of them, here's a snap of our lot a year ago:

What is SMA?
I need to take short step back first for some explanation of SMA. I would recommend that you do NOT pay this much mind if it is important to you, because I am not a doctor and am writing from memory and personal experience, not from a book (coz I’m on the train). There are also very detailed resources online but it can be hard to know from these what to actually expect for your child, or when, so talk to your specialist. A good starting point, though, is the Jennifer Trust for SMA website
For those who have not come across Spinal Muscular Atrophy (like my wife, Fiona, and I before we met the consultant for the first time), it is a degenerative neuromuscular disorder. We discovered that there are a lot of these. Some such disorders, such as the various muscular dystrophies, affect the muscles directly. Some affect the nerve fibre, like Multiple Sclerosis. SMA also affects the nerves directly, but by disrupting the junction in the spinal cord to the motor neurons leading to the limbs and thorax. A muscle is made up of bundles of fibres each with their own nerve fibre, and those without a properly firing nerve obviously don’t work and that fibre will atrophy, leaving the muscles themselves weaker. The pattern tends to be of it affecting the upper part of the legs, arms, and respiratory system (in that order). At present there is no known cure for SMA. As I say, the condition presents in a spectrum of severity but is classified into three types on the basis of the timing of its onset, as well as “adult onset SMA” which appears to have somewhat different causes and effects. At the mild end is type III, in which children start to show symptoms around 18 months. They are likely to be walking prior to onset and it is the deterioration of their mobility together with lots of falls and difficulty in getting into a standing position that are the likely signs that will have parents looking for answers. The weakness may affect the arms or breathing too, although probably later and not as severely as in type II. In type II symptoms become evident around 6 months, I believe, and the child is unlikely ever to walk. As it progresses the arms will likely also be affected and later on possibly the respiratory system too, shortening life expectancy. Type I is evident from birth and tragically is generally fatal in early infancy. I feel quite callous writing that, because I know that there might be parents who read this who face that world-shattering prospect and I do not. If this is you, I hope you will accept my sincerest sympathy and most heartfelt wishes for happy times with your child.
SMA has a genetic basis with (AFAIK) no significant established environmental factors that influence its onset or its progression. It is recessive, meaning in essence that both parents must be carriers of the mutation. Perhaps surprisingly the mutation is estimated to be present in around 1 in 40 of the population. The maths of this means that on average 1 in 1600 couples will both possess the gene. Assuming that they are both carriers rather than having 2 copies of the faulty gene (in which case they would themselves be affected by SMA), there is a 1 in 4 chance that any given child of theirs will have 2 copies of it. Which is how it came about that Fiona and I have two children without SMA and one with. The result is that around 1 child in 6500 is born with SMA – about 100 kids a year in the UK. Not a trivial number, but not so many that you are likely to have come across many of them.
Enough background, then. To our story.

There’s a problem
To start with we just noticed that Kid3’s feet appeared to have turned in and his gait was changing, with his feet moving further apart. This was at around 1½ years old, and on the advice of Fiona’s father (a very experienced GP) we had him checked out. In short he ended up with splints to support his feet. These did appear to help somewhat, but it became clear that his legs were still not doing what they should. He would fall over frequently, often with his legs folding underneath him and his head hitting the ground. He’d get to his feet by walking his hands up his legs (the “Gower’s sign” typical of SMA). He still couldn’t climb stairs by an age where children can usually do this. He also had a hand tremor, sometimes visible, sometimes just felt, sometimes not evident at all, and we associated this with emotional moments – excitement, upset. My father-in-law recommended that we arrange to see a paediatric neurologist, and now I know what his fears were.

We went with Kid3 and Fiona’s father to Addenbrookes Hospital in Cambridge for a consultation, during which the names of various conditions came up, mainly SMA and muscular dystrophies of various sorts, with Duchennes the only type that really fitted the symptoms. I’d read and heard a little about the MD but with its many forms I hadn’t quite registered what the implications of Duchennes might be, but it was clearly a strong candidate. It was clear now that our little boy’s condition was not one of slack tendons or anything other than a serious neuromuscular issue, which I’d not really wanted to countenance before. Fiona was more aware than I of what it meant, and it showed in her reaction. A blood test for a protein would be a first step to confirm whether Duchennes was likely, along with genetic tests for that and for SMA, and we went straight off that day for the blood sample to be taken – in itself a painful sight. We understood that it would take several weeks for the genetic test to be done and left expecting a wait. In something of a daze we went home incapable of doing or thinking much. We had my parents-in-law to support us and gently talk us through what it all might mean, but I think I still didn’t really get it properly.
The next day at work I spent the morning looking up Duchennes and the horror grew and grew in me. This is a condition that, I learnt, might see our son immobile before his teens and dead by 20. The world was falling apart and I was being hollowed out. But not long after midday, Fiona rang and told me that the protein test had already been done and ruled out Duchennes MD, and that the consultant had wanted to let us know as soon as possible so that we would not go through the agony of thinking that was the likely diagnosis for longer than necessary. He later told us that he really had thought it the most likely candidate. I don’t need to spell out for you the feeling of relief we shared. I didn’t really know what SMA might mean, though it was now the obvious candidate, but I did know things looked markedly less bleak than they had. A colleague came round at about that point to talk about some work thing or other and I just cried with relief on his shoulder. And that was the end of easily the worst few hours I have ever experienced, which ended effectively with a diagnosis no-one would wish for but with, at the same time, an insight into what might have been, and into the feelings of a parent being told their child’s life would be cut short. In some way this has stayed with me whenever things have been difficult. It’s not enough always to say “ah well, it could be worse” when you are brought face-to-face with a new consequence or worsening of your kid’s condition: that still hurts like hell; and yet, I thank my stars for what we and he have.

Facing it
There were then electrode tests to understand the strength of the nervous impulses in our son’s arms and legs. The results fitted with the SMA diagnosis, which was confirmed by the genetic test. At our next meeting with the consultant we talked through what type III SMA would actually mean, what research was underway, and what our next steps might be. Perhaps the hardest thing, in terms of understanding the implications of the diagnosis for Kid3, was (and is) dealing with the unknown, because the “progress” (degeneration of the nervous connections) varies so much and you’re desperate to know what to expect and when. Scratch that, actually. The hardest thing? Powerlessness. You want to be able to do something, anything, to make a difference. To be entirely dependent upon other people (never mind the fact that they can’t do very much, at least in terms of providing a cure) is difficult, but I suppose it was also being able to get to the bottom of the question: is there actually anything I can be doing? Am I missing any opportunities because I haven’t asked the right questions – is there a clinical trial, a scheme, that we’ve missed? I cannot speak highly enough of Dr Verity, the consultant we had, and his advice and the efforts of my father-in-law to discover everything possible about SMA, to interpret it for us and reassure us that we were not missing anything, these were so valuable. Still, we had a long journey to accommodate the new reality. I won’t attempt to describe that journey, and Fiona and I certainly handled it differently at times. I will say that the support of family and friends was immeasurably important, but in the end the only way to move onto a new phase was for us to gradually absorb the idea that this is simply our “normal”, not an acute situation that could be addressed and put behind us. Instead, what our son would regard as the normal state of affairs from his earliest memories needed to become that for us: something where the practical implications and needs of the situation are handled calmly and, without a feeling of crisis or drama, just dealt with. Usually that works.
One of the other things that we just have to feel our way through is how much help to offer. We don't want to make him overly dependent or to interfere when for his own self-esteem and confidence he should be left to, say, get himself upright or try something new that he may well fail at, but equally things are more difficult or tiring for him than for most other children. But with any kid one might push them to walk further than they've done before, but never as far as you could walk, and when your kid has reduced mobility it's a matter of calibrating differently.

What has happened in the intervening time? Well Kid3’s walking isn’t noticeably different: he falls over a little less, perhaps, because he knows better how to support himself on things, so there’s no way you’d expect him to go more than a few yards without a hand to hold. Stairs are usually climbed on hands and knees, although he has handrails on both sides (of very narrow stairs – I never thought that would actually be helpful!) and occasionally uses these to help him up or down. What he does have over other kids is impressive upper body strength, which comes from using his arms to do so much stuff his legs would otherwise do. He’ll climb to his brother’s top bunk essentially using just his arms and with help he’ll take on trees, monkey bars and climbing frames too, and he throws a mean punch. Hold his hand and you get a steel grip you’d not expect from a some sweaty little kid’s mitt. He likes to do acrobatics that use this strength and that other kids can’t do (still looking for a video for that). He'll do a simulated parachute drop too - that was cool!

He’s been at school a year now, settling in like any kid, and it’s safe to say his very outgoing character and huge energy have put him in good stead.
When he was 4½ Kid3 got a wheelchair, which he just loves, and he mastered it the moment he first sat in it.

The wheelchair was one of those things where as a parent you are perhaps a little reluctant to accept that it might be necessary. Indeed I was a little worried that it might see him exercising his legs less, which is important because it will help him to make the most of the muscle fibres that are receiving signals and keep his legs as strong as they can be. But of course it’s been a wholly good thing, giving him the freedom to go where he pleases and at speed and to feel like he’s on equal terms with other kids, plus it’s something he can do and other people can’t. We got a massive off-roader beast at the same time from Delichon (bloody expensive but so worth it). Check this out:

This baby has done extreme stuff and it’s beautifully engineered and good for kids up to mid-teens.

Having perhaps 20-25% of the usual amount of strength in his legs (just a guess) doesn’t mean he can’t do anything with them, of course. He has swimming lessons as well as hydrotherapy, and he loves a game of football, although this one definitely involves a lot of support from an adult (possibly swinging him in the air like a croquet mallet). We tried a balance bike, thinking that having no pedals it might work for him, but the act of balancing itself takes both control over the legs and a degree of core stability – another area weakened by SMA. So for his 6th birthday we got him an Ezy Roller, which is like a little go-kart powered by his arms wiggling a handlebar from side to side. Here's a clip of the boys hunting me down on Ezy-roller and scooter:

So he can now join in as his siblings skateboard, scoot and roller-blade – and we also tried a Micro Scooter, which has two wheels at the front so has enough stability for him to ride, a narrow rear to keep wheels out of he way of his trailing foot, a really low deck so he can reach the floor without having to bend his knee (an action that is sure to cause him to collapse), and steering that is not based on twisting the handlebars but tilting them - all of which seem to help. It was just brilliant to see him ride this. He had a lot of falls (but he’s made an art of that) but I didn’t really expect we’d see him scooting. Both of these came directly from Micro-Scooters’ base on Mersea Island, which is near to us, and I have to say a big thank you to them for letting us try stuff to our hearts’ content. Brilliant company. Below is a clip of him scooting. It's not exciting but it shows what's possible with this style of scooter. It also shows me doing what I talked about earlier and muddling my way through when to help and when not to, for better or worse.

My wobbly legs
One of our early worries was for how our son would handle it when he realised that he couldn’t do things that others could. We could see how his determination and vivacity helped him take on all sorts of things without complaint, but knew that at some point he was sure to think, “why is it just me that can’t do the things my friends can?” Will there come a point where he feels this and despairs, or turns to rebellion or anger? I guess we’ll see.
He is aware of his condition, of course, but treats it in a very matter-of-fact way. Sometimes it’s heart-breaking to see how matter-of-fact he can be, in fact. One thinks, “this shouldn’t have to seem normal to him”. He sometimes calmly sits out a game knowing that it’s not something he can do, though he may well recruit someone to help him do things his way. Only once do I remember him making a rather sad remark. I was carrying him down the stairs at a friend’s house where he saw his reflection in a big mirror, with his legs swinging freely as they tend to do, and he told me “I hate to see my wobbly legs when I’m walking”.
We’ve had several open discussions with the other children about what SMA is and means (normally on long car journeys), with him listening in. It’s hard to know if that’s the right thing to do, but that’s our approach: be open, don’t make a drama of it, don’t make him feel abnormal whilst being clear that he is naturally going to need help. I don’t think we have a strategy, we just have to feel our way and take our cues from him.

Help from outside
Beyond our friends and families, whose support has been so precious, we’ve had a lot of help from outside agencies and it’s well worth knowing where you can look. Actually the first to mention (again) is the Jennifer Trust, who provided us with a list of the various agencies we could turn to and some of the steps we could expect. The Muscular Dystrophy Campaign is a much larger charity than JTSMA with a wider focus but including SMA in its remit. With a bigger constituency it has quite busy forums where you might find a place to seek advice or share experiences.
Various services come through the NHS, from physiotherapy and occupational health, to wheelchair services, to consultations with paediatric neurologists. Our paediatrician helped in many ways other than the clinical, putting us in touch with people and advising on form filling (there’s a bit of that). As far as keeping in touch with research goes, aside from asking your paediatric neurologist you keep you informed there is a research network, Treat-NMD (it started as a European initiative but has gone global), that is worth looking into. It runs patient registries and a whole lot more besides. Like us you may not feel ready to face thinking about it immediately but do take a look.
A child that needs as much help moving around as one with SMA is entitled to (and needs) certain benefits, and you should get this underway. Check out the Motability scheme too (even if it uses ColdFusion for its website), which has enabled us to get a vehicle big enough for wheelchairs and buggies and also means that Fiona can get to all the many appointments.
The primary school and county council have worked together to provide support in class for Kid3 and I recommend pushing for this, especially if the school is an older building where the physical environment may be tricky for kids with “wobbly legs” or wheelchairs and where adaptations may be hard to implement. Getting around the classroom, participating in the playground and in PE all need our son to have help, and he gets that from a dedicated assistant. It may seem overkill but I assure you that it’s not, and because he’s a bright kid who doesn’t need extra help when sitting down at his work it means that the class also gets an extra adult to pitch in and help anyone, which is good for everyone. So whilst there’s less money to go round and it may be more difficult now to persuade your council to provide classroom assistance, don’t be afraid to try.
I mentioned swimming lessons, and again this is an experiment that has worked. Having taught our two older children the swimming teachers have been happy to take on the challenge of finding techniques that work for our youngest, giving him genuine swimming skill, confidence in the water, and noticeably improved muscle tone too. It’s exhausting for him, but that’s a good thing.
I’m sure there have been other sources of help and there are certainly many people I’m grateful to but I want this to focus on things that could be useful to others and I think this is a good start. I hope it’s helpful.

There is no conclusion. We carry on, we have a happy life with our kids, there are annoyances and things that we all miss out on or that Kid3 in particular misses out on, but really he's just like any other 6-year-old mentalist. He has no super-powers but he has a big smile, smart brain and lots and lots of attitude. We don’t know what’s ahead of us, but that’s just life. We count our blessings, and then they shout at us, kiss us, draw a picture, run over our feet with a wheelchair. They may even stand there three feet high in a Darth Maul costume, rabbit-punch us and then fall over backwards with the kick-back... And so the blessings pile up, only a little black and blue.

So if you've just found out that you are in our situation, with a child that is going to have difficulties you never foresaw, please don't despair. I hope you've seen that there are ways round a lot of things and always, always a lot of fun to be had. All my best to you.

Postscript. As I got to the end of writing this, I became aware of a blog started recently by the parents of Estrella, a little girl who lost her life to SMA but a month ago at the age of 8 months. Her story, their story, is very different from ours and humbles me. I read the words of people in the midst of grief who are yet trying their very best to do something positive; and now I feel more strongly than ever the sentiment in the title of that blog: Smash SMA. Amen to that.

Monday, November 07, 2011

New IWM websites pt.III: the, um, website

So, as on the evening before we switch our new website over from beta to fully live status I finally get round to the website part of this series of blog posts. In part 1 we did brand, e-commerce and hosting. In part 2, collections and licensing. Here, we'll look in too much detail at building the core website itself. Sorry, it's a long 'un.

Why a new website?
For the last 8 (I think) years, IWM has used BoxUK’s Amaxus CMS to run its websites. Naturally the sites were getting creaky and the CMS itself has been superseded, IWM itself has changed, and so has how the web works – both technically and in terms of the behaviour of web users and the language of interaction that they understand. A clean sweep was in order, which means a variety of strands of work. This much was clear when Carolyn Royston (our head) and Wendy Orr (our Digital Projects Manager, and so lead on the website project) outlined their ambitions to me when I started at IWM in May 2010.

Choosing the platform
Research for the new sites had began some time before that May but planning really kicked off in June. For me, the first key deliverable was the selection of a technical solution, but although I had a fair idea which way I would go I wanted to know a variety of other things first. How can you choose a CMS without knowing the functional specification, and how can you really know that without settling the information architecture to some degree, and the ways that people will find content and interact with the site? Decisions on whether we’d be supporting a separate mobile site, for instance (we don’t, at least not for now), and our plans for legacy sites all could have an impact. But of course you can only work out so much of this beforehand, and most questions seem to lead to others in a Gordian knot, so in the end you have to assess the situation as best you can, put together your own set of technical priorities, and make your selection as something of a leap of faith. I had the benefit of advice from various knowledgeable people in the sector who told us of their experiences with various CMSs, in particular IMA’s Drupal mage Rob Stein and the V&A’s Richard Morgan and Rich Barrett-Small, and we also had demos of a couple of commercial CMSs. Most importantly, though, we had Monique Szpak, whose role in this project (and my learning process at IWM) really needs a blog post of its own. Her experience with various open source products including Drupal was key, and after we identified that as our preferred solution she built us a proof-of-concept late last year to confirm that Drupal was likely to be able to do what we needed, and to assess the likelihood that Drupal 7, which at that point was still in alpha, would be ready when we needed it. With this information we took an informed gamble that it would be, and the choice was made.

As I already said, we started development work even before settling finally on Drupal, as a piloting project, and this continued whilst we were developing our plans for content, IA and design. There were a number of things we knew we wanted, even if the functionality was still hazy – with Monique’s help we’ve instituted agile practices which positively encourage trial, error, testing and improvement. This change, in fact, together with the development environment we’ve gradually (& painfully) pieced together and the implementation of tools like Jira and Subversion, has been fundamental to making this project work, and it would have been impossible without Monique. Whilst she worked on prototyping more functionality, I did some groundwork on indexing shop and external sites. Then in the spring Toby Bettridge joined us, fresh from working on the Drupal part of the V&A’s new site. He and Monique worked very closely (with the help of Skype) and long before the design work was complete we had basic versions of the taxonomy, events, multi-index search and collections functionality done, amongst other things.
Although I’ve been paying attention to what they do, my hands-on involvement in Drupal development has been pretty much nil and I still understand the CMS far less than I’d like, so anything I say about Drupal development here needs to be read with that in mind! I do get, though, that one picks modules carefully, develops new ones with reserve, and never hacks core. We started developing with Drupal 7 before it was released, and even when it was there were (and remain) quite a lot of modules that weren’t ready to use. We thought the gamble was worthwhile, though, and forged ahead. In time we did incorporate some of them, although unfortunately we still don’t have some of the things promised by e.g. Workbench. Along the way Monique and Toby also did some vital module development of their own, notably a custom collections search module (using Search Api Solr Search), media embedding for authors using IWM’s oEmbed service, entity lists (old-style Drupal nodes had lists, but new-style entities didn’t), and some administrative tools.
My role in development? I’ve often felt somewhat awkward, if I’m honest, about my fit, because having elected to go with an entire technology stack and various development practices that were new to me, I often found I couldn’t really contribute practically, even where I understood some things well. For instance, although I have plenty of experience with Solr, my practical contribution to integrating it with Drupal was negligible; likewise if I knew what was required to fix some HTML/JS/CSS at the front end, I could not implement this in an unfamiliar environment for fear of messing up Drupal or making some Subversion faux pas. I think I’ve made but one single (successful) check-in of Drupal code. I concentrated instead on sorting out development and live hosting, working on getting the collections data right, filling the holes in the spec as we noticed them, and so on. I spent a good while working out how the media streaming worked and how to embed that in our pages, using the DAMS’ web service to build a light-weight SOAP-free alternative (an oEmbed service) that could both serve our websites and potentially 3rd parties. When everything calms down, though, I need to properly get to grips with the codebase.

Information architecture, discovery & URLs
Working out the logic for a site that’s going to function for several years is not easy. One can change that logic if necessary, but you really need to know how likely that is to happen in order to give your designers some parameters to work within – how flexible do menus have to be? How directed will the user be, and how much should any one piece of content be located in a specific part of the site? As I said earlier, the brand structure and the 5 IWM branches were a big factor in how we had to organise content, since we needed to make things readily discoverable whatever the user’s journey, but without making them context-free and confusing. Another pair of conflicting priorities were the wish to avoid having too many top-level menu items and the wish to keep the site fairly flat without obliging too many clicks to find content.
Sites like the V&A, which relaunched earlier this year, have taken adventurous routes to delivering masses of content to users (or users to content) – in the V&A’s case, centring around search and introducing a sort of machine-learning to categorise content and indeed to identify what categories might exist. Brave stuff, and a great solution to the huge volume of content they have there.
At IWM we played for a while with the idea of a taxonomy driven site, wondering if we could use a set of taxonomies as facets onto different aspects of the site that would let users cut across a traditional hierarchical organisation of content. We’ve kind of gone with a watered-down version of that, wherein the structure of the content is fairly obvious and on the whole quite flat but we’ve used controlled terms and free tagging to help make things more discoverable to users coming from other angles. This is pretty conventional and at the moment of limited power, but in due course we will make greater efforts to align our taxonomies (in particular our history taxonomy) with the controlled terminology used in our collections. This was too much to do in this phase, but when that happens we should be able to make ever-better connections between our collections and pages like our “Collections In Context” history pages, learning resources, galleries and perhaps events. A learning-focused vocabulary will do the same, but right now our e-learning resources are pretty much non-existent.
Perhaps more important than taxonomy at the moment is search, which has been a key way of integrating content that lies outside our main site. We’ve elected to run 4 separate Solr indexes for this, and to keep them separate owing to the distinctive nature of their content. We have the Drupal index itself; collections data; an index of products extracted from our Cybertill e-shop; and a crawl (using Nutch) of a number of IWM sites that are outside of Drupal, such as blogs and the “Their Past, Your Future” learning resource. The last one needs a lot more work but as a quick-and-dirty way of ensuring that those legacy sites weren’t left out in the cold it works. And yes, a Google custom search engine would have been an alternative but then it would not have worked in the same way as the other searches, with deep integration into Drupal and the ability to treat the results as entities and reuse them elsewhere.
One obvious change with the new site is that, well, it’s one site. Previously we used a morass of subdomains for somewhat independent branch sites and even for the collections-pages-that-weren’t-collections-search (collections search had its own domain, no less). I for one found it pretty confusing. With the rebrand making the “IWM-ness” of all of our branches more prominent we were able to do the same on the website. I had been through a similar exercise at the Museum of London, and though it was not an identical situation some conundrums and dilemmas were shared by both. How to make it easy to access non-branch specific content and information to all users in the same place as branch-specific content, and how to make sure that people are well aware that the latter pertains only to one physical site? How to cross-promote? Like MoL we had no specific digital brand, nor a mother brand to distinguish particular projects or sites from cross-organisational activities. I hope we found a solution that works for our users and not just for IWM itself, but time (and more user-testing) will tell.
The expectation that we’d move content around and that the organisation of material on the site, as seen by users in menus etc, would not be forever, prompted me to seek a URL structure that was a little more abstract. I didn’t want URL components necessarily to be the same as top-level menu items, which might disappear, but to relate to more stable concepts of what IWM does and offers whilst remaining meaningful. That doesn’t mean permanent URLs but hopefully relatively long-lasting and predictable ones. In one area – collections – we do aim for the URLs to be “permanent”, though (whatever that means). What I tried to do was put what I imagined to be the most stable aspects towards the left hand end of the URL path, things like “corporate” and “visits”, because I envisaged these as being more stable than even branch names (we might get more branches, or rename them again). I also wanted to be able to put non-branch content under these. The result is that we don’t have branch names at the top of a hierarchy but reappearing in a few places – visist/iwm-london as well as events/iwm-london and others. It may seem messy but I hope it’s reasonably predictable all the same, and it means we never need a catch-all URL to cope with the miscellany that we hadn’t foreseen would ever exist outside branches.

We appointed the Bureau for Visual Affairs, who were responsible for the National Maritime Museum’s new website’s design, to do the same for us. Judge for yourself how they’ve done, although good or bad the credit or blame are not all theirs, even when it comes to aesthetics. Design and content go hand in hand, and in some places we’re still working to improve the latter to make the best of the former. Under the covers, too, the HTML that’s spat onto the page is the result of BVA’s HTML coders’ fine work at one end, Drupal at the other, and the best efforts of our devs to bridge the gap. And sometimes the gap was pretty big.
The theming process was one area where our plans went somewhat awry. We had two experienced Drupal developers on our team, but as there was plenty for them to do in back-end development we were planning on the theming being handled by whoever we appointed as designers. BVA, however, are not a Drupal house but their design was what got us all excited, so we reached an arrangement with a third company to subcontract to BVA to do this part of the work. Having done it once, this is not something I would recommend - at least not unless you can make it very clear who answers to whom and where the line lies between development work, theming, and HTML development (and who pays who for what). We ended up some weeks behind but got back on track with the help of Ed Conolly of http://www.inetdigital.co.uk, who moonlighted as a themer for a few weeks and helped put a spring back in everyone’s step. Bravo Ed!

Early in our content planning we decided what we’d migrate from the old sites (not a lot), what we’d need to keep going (a small set of microsites) and, broadly speaking, what we’d want to add to the new site. Killing off content doesn’t usually sit too well with me, who’s a conservationist and archivist by inclination. My instinct is that it’s sure to be useful to someone to have pretty much everything we’ve ever done remain available, but that’s nonsense really and far from helping people could end up confusing them, not to mention sucking up resources for maintenance that would be much better spent on creating new content of real worth. We did have an awful lot of pages that related to old exhibitions and so on, and were very keen to disentangle ourselves as fully as possible from our old content management system, Amaxus 3. In the end we have kept three or four microsites from that. Other content needed substantial alterations to bring it up to date and suit it to the new site structure.
However, beyond the core, practical information about visits etc., we wanted to do something that would directly serve the core purpose of the IWM: to tell the stories of conflict through the material we hold; and we wanted to do it in a rich, immersive way. BVA came up with a solution that looked lovely, although we went through a few iterations in order to make it easier to create the content and to draw parts of it from the collections middleware. We wanted HTML that could be generated almost automatically, which opens up other potential uses for the template. This took away some of the visual sophistication with which BVA won our hearts, and I suspect that they were a little unhappy to see this go, but this is a site that we want to add to frequently and without having to use HTML developers to do it, so I think we found a happy medium. Our “Collections in Context” (or simply, “history”) section contains over 100 articles at present, using images, audio and video to tell stories spanning from the First World War to the present conflicts in which the UK is involved. They were written by one of IWM’s historians and put carefully worked into the CMS by our team in a close collaboration that we hope to turn into a rolling programme of content creation, perhaps reflecting current events or notable anniversaries. I hope in due course we can extend the use of the format to other parts of the site and other voices, perhaps enabling its use as a tool for our website visitors. The people who deserve a shout-out for writing, editing, and/or inputting the hundreds of content pages that make up the new site are New Media’s Jesse Alter and Janice Phillips together with Maggie Hills, who has joined us for a busy few months.
BVA brought a couple of other bits of bling to the site, with the aim of a more engrossing, immersive experience. First amongst these is the “visual browse”, a slideshow mechanism that underlies many of our pages and is brought to the fore by clicking a tab at the top left. We can make any number of these and surface them on the pages where they are relevant – for instance, each branch has its own visual browse.

Is it any good?
When I stand back from whatever details might be preoccupying me on a given day I’m really pleased with the overall effect of what we’ve done, but of course I am not a typical user and what will count will be the feedback we get from our users. But for me, I’m especially pleased with the history pages and the way that our collections are now used there and in the search pages. I am also pretty pleased with the balance we’ve found between the individual branches (essentially, the needs of the physical visitor) and the cross-branch/non-branch activities and content, but because this is necessarily a compromise I expect that it will not work for everyone.
I have reservations too. I think the lack of a mother brand is a problem, and I think we need to make the home page work harder to offer a powerful message of what IWM as a whole is. The lack of fly-out menus is galling to me, although the ones for branches work well. It means more of a leap into the unknown and more clicks to find what you’re after. Our lovely, lovely history content is hard to find. Mobile performance is not that great – the whole site is too wide to load full-width with the text legible, and a ton of stuff loads onto the page that is not important for the mobile user. It functions OK, but it’s far from an optimised experience.
So, my opinions aside, there’s plenty to do over the coming months. But it will feel mighty good to have this milestone out of the way: November 8th – switchover day.

Wednesday, November 02, 2011

New IWM websites pt.II: Collections and licensing

This is the second post about our "big bang" and the things we launched on October 4th. In the first post I talked about the new brand, e-commerce, and hosting. Here I'll talk a bit about collections and the closely related issue of licensing.

The collections
So, now we’re getting more into the area I can talk about more knowledgeably. The opportunity to help reimagine how IWM’s collections are brought to the public through digital media was one of the key attractions that brought me here 18 months ago (though there were several other extremely compelling reasons. I think some of them are still in post). I felt I could bring something useful from my experiences at the Museum of London, having been part of the team that brought an ambitious new system there just before I left.
I hope I’ll be able write it all up properly soon, but I’ll keep it brief here. The collections online project at IWM had two objectives: firstly to build the foundational infrastructure for all future data-driven collections applications; and secondly to build the first public interface onto that infrastructure with the collections search pages in the new website. Only the bare essentials of the infrastructure were to be built in this phase: purely what was necessary to deliver to the web application and to be provide enough of an architecture for us to plug in the planned extra features later on. We have plenty in line for phase 2, but we’ll tie all that stuff in with specific front-end requirements.
Simon Chambers came on board with us to project manage this one and he did an incredible job of marshalling the requirements, prioritising them, working with a number of departments and strong-willed people and getting things as quickly as possible to the point where we could deliver the baseline of what the website needed. We ultimately decided to work with Knowledge Integration, who built the CIIM for us at MoL and who could bring us an existing application that fitted our needs very well.
Essentially the CIIM, at least the part we’ve implemented so far, pulls data from the collections management system (in our case Adlib) and remodels it to serve the needs of discovery and delivery as opposed to data management. These are very different things, and that a CollMS may do the latter very well doesn’t make it ideal for the former. This is as much about how the database is used across the organisation’s varied collections as it is about the technical qualities of, say, Adlib specifically, because this architecture allows us to intervene in the data between its source and front-end applications that use it – to remodel and align it, to integrate it with other data sources or enrich it, to prepare associated media, and to optimise it for full text searching, for instance. The big job since February, when things kicked off in earnest, has been modelling the data correctly. I have to admit I seriously underestimated the complexity of getting this right, and we had a series of problems to do with the API it was extracting from and the readiness of some of the data, but in a way this illustrates why it’s a good thing to be able to do all this work away from the front-end.
The result is a first-pass at a Solr index that, along with 3 others, lies at the heart of discovery on our new website. Try out the search engine here, or here are some good searches to get you going. Watch a video. Listen to interviews or momentous radio broadcasts (Czech alert). Oh and if you find an object like this you'll see that it's part of a collection, and can see a complete listing of that collection. Our priorities mean that we’ve deferred implementation of some of the features we want in the longer term but we know we can leap into action soon. In fact, Tom Grinsted, our multimedia manager, has put together a project with UCL and K-Int that gives us a focus for some of this functionality and is just getting underway now, which is rather exciting. Luke Smith and Giv Parvaneh are also busy planning various projects for the next few years as part of the centenary of the First World War that will also draw on and feed into the system. So watch this space.

All that e-commerce work around collections has also meant reviewing the way we licence our material. Recent developments at national and European level – notably the creation of the Open Government Licence by the National Archive (TNA) – and the steps that some of our peers have made in offering their assets for creative reuse to the benefit of all, have also had an impact. IWM has now launched its User Licence (essentially the OGL), which frees up almost 200,000 images, audio recordings and films, for non-commerical use. Regular "fair dealing" restictions apply to others, like this nice Ronald Searle picture. I'm afraid we've not yet got a filter in collections search for items with this licence, but try right-clicking on an image on an item page to see if you can download or embed it.
The licence applies to IWM-generated content and data too, so although we don’t yet have a public API to our collections the data around them is up for grabs. Hopefully I'll have more to report on this before too long.

In the next post I'll talk through the website itself. Stay awake at the back!

Tuesday, October 25, 2011

On [that] day, 4th October 2011... pt 1

Tuesday October 4th was a big one for the Imperial War Museum. We in the New Media department had been working towards it for well over a year, but several other parts of IWM also had first-night nerves as the presented something shiny and new to the public. All in all we launched a new brand, the beta of the all-new IWM website, a re-skinned image licensing site, overhauled e-shop, redesigned print sales website, and phase 1 of a new system for delivering collections information to our public digital interfaces. We also took the covers off a new licence – the IWM User licence – which applies to lots of data and content and a large number of the images, audio and film (yes indeed) on the new site. The common theme, you’ll note, is that the web is part of all of these, so it’s fair to say we were probably the nerviest of all departments!
The timing was no accident, of course: Carolyn, our head of department, was not far into her plotting before it became apparent that we’d be launching our new website along with the new brand, and that we’d be tying in wherever possible with a 3-year e-commerce programme. The brand re-launch in turn was timed to fit with the opening of a major new exhibition coming to IWM London, "Shaped by War", and the e-commerce programme entailed some important implications for licensing too, so the dependencies were complex.
The brand
I can’t report much about the brand except to say that timing was critical, and that what we were most interested in was whether or not the brand structure would have any implications for information architecture and site navigation. We needed to know how we’d be expected to deal with what could be considered informally to be sub-brands (whether that be our branches, e-commerce sites, exhibitions, partnerships etc). To remain uncontroversial I will limit myself to saying that we were not given a recipe-book for how to apply the brand in a digital environment and that evolving its digital expression has been quite a challenging process. Personally I quite like the wedge device, although as lapsed geologist it reminds me of a horst. Or perhaps a graben, I always mixed them up. I think it’s fair to say that reactions have been polarised but the commenters on branding and design websites, at least, appear to dig it.
E-commerce sites
Over the last couple of years IWM has been using the excellent services of consultant Alice Grant, who has led a comprehensive e-commerce programme with literally dozens of projects within it. In particular she has reviewed the way we conduct business around our collections assets, with one vital development being the establishment of a commercial unit. She’s also been co-ordinating a number of activities to try to ensure that when we relaunched the website our B2B and B2C e-commerce offers made sense and worked well.
Over on the site at http://www.iwmcollections.org.uk/ that you can still see for a couple more weeks, we have split out the image sales and licensing functionality and re-skinned that, and put it onto isl.iwmcollections.org.uk. The ISL aspect of the old site will persist for a while yet because its functionality will remain part of our business processes for the foreseeable future, whereas the straightforward collections search part will shortly be retired, having been replaced by the corresponding part of our new site (of which more later). The ISL site again sports the new brand, was designed by Christian and implemented by Andrew Stephens in our ICT department.
I should mention the film sales site, which has been bubbling under for the last year but not really been promoted. We’ve now got a strengthened infrastructure and rely on this to deliver media through the core sites too, so I think it’s worth shouting a bit about film.iwmcollections.org.uk/ too. This B2B-facing site is again a Statham reskinning of the basic site built by CIS, whose DAMS we use (Imagen). Looks like we need to get the new logo in there though, Christian...
Our consumer-facing print sales site run by Cabinet UK at http://www.iwmprints.org.uk/ also had a complete facelift and its range multiplied about 10-fold, I believe. Not only that, it links back to our main site’s collection records pages. I reckon it looks pretty slick.
Finally, earlier this year we launched a heavily re-skinned e-shop in the knowledge that we might be obliged to revisit the whole process when the branding process was complete or when Cybertill launched their long-anticipated new software. The old site was haggard but, working within the constraints of the existing templating system, we were limited in what we could do. Nevertheless, with some imaginative design courtesy of our very own Christian Statham and some creative coding courtesy of Garry Taylor at the Bouncing Ball we ended up with a really good solution. Actually, that doesn’t do it justice at all. Under the skin, you’ll find quite a lot remains of the minging HTML that the system churns out and which cannot be changed at present, but which Garry’s CSS and jQuery skills have made look 10 years newer than their real age. For the beta website launch we made only small changes, mainly to tie in with the new brand.
All of these sites have another new dimension which is not necessarily visible on the sites themselves: they are integrated with the new website. We have built an index for our e-shop catalogue and integrated it into the site, returning results in the global search and enabling us to build carousels or products for promotion and use them anywhere; and images or films for which you can buy a licence or hard copy offer you this opportunity when you look at them in our collections pages.
It was time to revisit how we host, to give us more control and at the same time slash costs. From an arrangement where our main sites were hosted on 3 dedicated servers at Rackspace we have moved to a hybrid solution with a beefy dedicated server for databases, Solr and some media, and a variable number of cloud servers to run web applications, including our core Drupal installation. We’ve also virtualised the old CMS to run a few legacy bits of that site. Whilst previously our hosted sites were isolated, we are now in a position to integrate them securely with services running on our own network, for instance with an oEmbed service that we use to tie the DAMS in with Drupal, or for replicating data or connecting to mail servers (things that could be done other ways but which are easier and more secure with the VPN we have in place). Parts of our web offer are delivered directly from our DMZ – our streaming media, the ISL site and our blogs – but again we’re now well placed to change this, if we wish, without huge investment, as the cloud part of our solution makes rapid expansion or scaling straightforward.
Having come from a background of essentially Windows/IIS/ASP+ ASP.Net/SQL Server and made the decision that we should go with the full LAMP stack, this was one of several steep learning curves I’ve been on (am still on). I’ve never been a server admin although I’ve picked up a certain amount of knowledge over the last 12 years, but it was pretty much like starting from scratch when I was faced with a bare-bones RHEL server and a Putty window. Fortunately I’ve got several experienced colleagues to put me on the right path or do various thing on my behalf. Together with some help from Rackspace themselves we’ve got there. I still have a long way to go but now that we’ve launched the beta at least it’s looking a bit less intimidating.

In Part 2 I'll say a bit about collections, licensing changes, and the main website itself. Then if I get that far I might do a post to say how things have been going in the first couple of weeks.

Friday, October 21, 2011

SOAPhar, so good

All of a sudden video and sound on our new beta site, streamed from our DAMS, stopped working this week. We embed these using a (currently) private oEmbed service which gets its data from a SOAP service on the DAMS to construct the streaming URL for a given media item, with which it then assembles the embedding code. The SOAP call was timing out and giving the following error:

Fatal error: Uncaught SoapFault exception:

[WSDL] SOAP-ERROR: Parsing Schema: can't import schema from 'http://schemas.xmlsoap.org/soap/encoding/' in C:\[myfile]:20
Stack trace: #0 C:\[myfile](20): SoapClient->SoapClient('http://[DAMS WSDL URL]', Array)

That schema file is being imported at a namespace declaration in the DAMS SOAP service's WSDL file. It could be accessed just fine with a browser but the PHP soapclient call which loads the WSDL file had no such luck and would just time out.
I have no idea why a service that had been running fine for 8 months or so suddenly went freaky, but I did suspect another unheralded proxy change, since these tend to play havoc with our dev environment and at the moment the oEmbed service is running there. Other people have experienced problems with this encoding schema at xmlsoap.org, but since it's going to be present in all proper SOAP calls that doesn't really tell us whether it's indicative of a problem with the URL and xmlsoap.org, or of network issues. All the same, what worked for this guy worked for me. Well actually, whilst he simply removed the namespace import element, whereas, I saved the WSDL for the SOAP service that my oEmbed script called and edited it to point at a local copy of the encoding schema at xmlsoap.org. Happily my SOAP call seemed to work just as well with a local copy of the WSDL and everything started working again. Hooray! I had wondered whether I would be scuppered because I couldn't edit the original WSDL file, but since the WSDL doesn't actually do anything, it just lays out the instructions for a SOAP consumer to use, it was aboslutely fine to edit and run it locally.
Perhaps this will help others suffering the same grief.

Sunday, October 16, 2011

Flash 10.2 on Android phones with ARM: partial success

Imagine my disappointment and annoyance and slight feeling of miss-selling* when I discovered that my otherwise lovely new Android phone (a Samsung Galaxy Ace / S5830) couldn't install Adobe FlashPlayer. OK, it's a low-end device but that's lame. Word is that the ARM chip is less powerful than Adobe would like. However rumour also has it that it's perfectly powerful enough for the job. And without Flash, there's no iPlayer and no embedded YouTube (I can use the YT app that's pre-installed, but not watch videos embedded with the Flash player).

So when I tried to install Flash from the Android marketplace and was told I couldn't I did a little looking round and found that some very useful people had hacked the 10.2 version of the official player to make it installable on devices with slower chips. I was a little nervous, of course, about putting an unsourced app on, but although I might have unwittingly installed some well-disguised malware so far there's no sign of that. My success is only partial, however, because iPlayer is still unhappy and wants me to go to the marketplace to download again. I understood it only needed Flash 9 but perhaps it's using some other way to tell that it's not happy with my phone (presumably based on what plugin the browser is reporting). Embedded YouTube clips work now, but videos from our own collections at IWM don't work :-(

So I'm still rather peeved that the limitations of the phone were not made clear, and also that as users we are aren't even offered the choice of installing the official FlashPlayer, performance reservations notwithstanding. After all, I can see that the hacked version does work perfectly well on those sites that don't whinge about it, so chip speed evidently isn't a deal-breaker.

In any case, if you want to have a go with it, check it out here:


I could have linked straight to the APK but that seems a little rude given that you're getting something for nowt so you do have to go through a couple of screens of adverts; keep going!

*to be honest, not just a feeling of miss-selling but of stupidity, having forgotten caveat emptor and done sod all research before going into the 3 shop. I leapt at the chance to upgrade considerably for no extra cash, and I wouldn't have paid much more anyway.

Friday, September 23, 2011

Europeana's Data Exchange Agreement: more than just a licensing change

The announcement yesterday that the Europeana Foundation had officially adopted the new Europeana Data Exchange Agreement may have passed you by, but it seems to me to be a vital moment in Europeana's history, or if you prefer, its future. This agreement is key to its sustainability – by which I mean, not that it is a guarantee of long-term viability, but that it couples the resourcing and the value proposition of the service more transparently than ever before. Let me expand on this.

For Europeana to be more than just a portal website it must be part of the data landscape, an ever-growing circle on that famous Linked Data diagram. I don’t think anyone concerned with the project could conceive of it as anything else nowadays. The “Distribute” strand of the 2011-15 Strategic Plan (PDF) pivots on the re-use of content away from the Europeana site, via the API or Linked Data channels. This is the Europeana that I am talking about sustaining, and this is what the EDEA makes possible. Without it, the offer could only be the ghost of that: a portal site. With it, it could become pervasive. It’s the difference between a drinks machine and mains water supply. Coca Cola like to slap on a label saying “Buxton” and sell water for 10,000 what it costs from the tap to those people that come to the drinks machine. Thames Water, by contrast, put it in pipes and get it everywhere – unbranded, plentiful, pervasive. The EDEA brings into focus a similar choice for content providers: do they want their water in the mains, or would they rather put it in bottles and try to find buyers on their own? Europeana has staked its future - I do mean that - on the mains supply approach.

The two most significant resources that Europeana relies upon are finance and content*. You might also add users, or you might prefer to count them on the “benefit” side of a cost/benefit equation. Crudely put, finance comes largely from the European Commission and content largely comes from cultural heritage content partners (either direct data providers or, more commonly, via aggregators). These two stakeholder groups, providing the grist to Europeana’s mill, are looking for different things, although their interests overlap greatly. To sustain a social enterprise, which is at heart concerned with generating public good rather than a net profit, requires whoever is providing the resources for it to perceive that the value thus generated is worth the cost. If this value proposition is not strong enough to persuade them to supply the necessary resources, well, it won’t be resourced and will fail.

Europeana has now laid out its value proposition to content providers, along with what it requires in return. Given the vision of the Strategic Plan's “Distribute” strand, this requirement could never really be less than the CC0 license the EDEA requires, but equally the “mains supply” model of cultural content is the foundation of the value proposition that the EDEA offers to museums, libraries and archives that sign the new agreement: you will find your content hosed across the internet, heliping you to deliver your public mandate but also carrying new traffic back to your site and opening up commercial opportunities. The word “exchange” in the title of the Agreement also highlights the other part of the value proposition: that the enrichments Europeana makes to the raw metadata it receives will also be CC0 licensed and will be given back to the providers. Note also that whilst the EDEA is not a contract with the funders, it again reflects the value proposition to them: culture on tap through openly licenced, reusable data.

So the EDEA is not just a new licensing agreement. If you like, it is the sustainability plan in super-brief form: give us your content and let us set it free, and here’s what you get in return - oh and funders, this is why we're worth the €€€. Since the Strategic Plan laid out a vision of open-licensed, distributed and open data, this was really the inescapable conclusion. If it finds sufficient favour, that vision of a mains supply of culture may have a long life, and that bubble on the Linked Data diagram may keep growing. If it doesn’t, well, it’s either back to the drinks machine or back to nothing. It's worth the gamble, I reckon.

* Oh, and political support. This is a big one, and is the source of powerful backing for the open data train to which Europeana has hitched its wagon. If Commissioner Neelie Kroes has anything to do with it this train is going to gather speed rapidly – see her speech to the OpenEurope Summit this week for starters.

Saturday, July 09, 2011

Back to the start

I've been reading a hell of a lot of documents around Europeana lately as I write the case study for my thesis. And I mean a hell of a lot, many scores of them, including all sorts of work plans, funding agreements, visions, strategies and business plans, meeting minutes, white papers, conference presentations, functional specifications and all the rest. Perhaps the most revealing for me, though, has been to dig back into the prehistory of the service, to look at the documents that reveal its origins and relationship to predecessors such as The European Library and EDLproject (not the same as EDLnet, which itself turned into Europeana). I've had some fascinating discussions and insights from various key parties over the years about the real history, but only now have I actually got around to ferreting out some of the relevant documents published by the European Commission that show not merely the degree of its committment to the project but the sophistication of its ambition from the very start.
In particular it's interesting to note how much of the current vision genuinely was present at the very start. I think people have sometimes assumed that EDL/Europeana were really conceived as a web portal first and a hub for content reuse second, and probably for some motivations that would make the nose of a John Birchian twitch frenetically. But from its very inception it was truly about encouraging interoperability in support of reuse, of stimulating the creative economy, and squeezing more value from the continent's heritage by allowing it to flow to wherever it's needed - at least that's how I read it, and my recent reading matter reinforces that. I won't labour the point any more, I will simply quote extensively from the eContentPlus Work Plan for 2005 (which was actually a plan for work to be done/money spent from 2006, the year before EDLnet kicked off)

Cultural and scholarly institutions, including archives, libraries and museums, are developing or creating digital collections, either by digitising or by acquiring digital resources. Though often described on institutional web-sites, these resources lack visibility at European and global level, because there is insufficient interoperability between existing networks, across types of cultural organisation and collection, and across different types of content. This is aggravated by the range of different legacy systems and practices for describing cultural and scientific resources.

Effective access and re-use requires an infrastructure which can support a range of functions, including: discovery of collections and of individual items; disclosing conditions for and authenticating use; and integrating tools, such as thesauri and ontologies, to enable multilingual/multicultural access and use. Re-use (aggregating and creatively adding to this content) also requires enriched digital objects which can eventually bedelivered through these services, supporting new economic and business models and user communities.

eContentplus aims at leveraging the multilingual availability of significant assets of digital cultural, scientific and scholarly content by supporting the development of interoperable collections and objects - on which multilingual and cross-border services can be built - and by supporting solutions to facilitate exposure, discovery and retrieval of these resources. Actions should increase the opportunities and scope for accessing these resources, tackle multilingual issues, and support the emergence of enriched cultural content. There are two complementary objectives for the work in this area: promoting an enabling infrastructure in support of access and use; and stimulating content enrichment.

[quoted from eContentPlus work programme 2005 p.10]

Saturday, June 25, 2011

The LOD-LAM star system....

....is not in a galaxy far far away, although I believe it came out of San Francisco so that may not be too much of a stretch. The recent LOD-LAM workshop there on the question of Linked Open Data in Libraries Archives and Museums seems to have been a very lively and stimulating event and resulted in, amongst other things, a star rating system for linked open cultural metadata.
Mia yesterday posted a question to the MCG list asking for reactions to the scheme, which addresses in particular the issue of rights and rights statements for metadata - both the nature of the licence, which must reach a minimum level of openness, and the publication of that licence/waiver. Specifically she asked whether the fact that even the minimum one-star rating required data to be available for both non-commercial and commercial use was a problem for institutions.

My reply was that I felt it essential, in order for it to count as linked data (so I'm very pleased to see it required for the most basic level of conformance). But here I'd like to expand on that a bit and also start to tease out a distinction that I think has been somewhat ignored: between the use of data to infer, reason, search, analyse, and the re-publication of data.

First, the commercial/non-commercial question. I suppose one could consider that as long as the data isn't behind a paywall or password or some other barrier then it's open, but that's not my view: I think that if it's restricted to a certain group of users then it's not open. Placing requirements on those users (e.g. attribution) is another matter; that's a limitation (and a pain, perhaps) but it's not closing the data off per se, whereas making it NC only is. Since the 4 different star levels in the LOD-LAM scheme all seem to reflect the same belief that's cool with me.

The commercial use question is a problem that has bedevilled Europeana in recent months, and so it is a very live issue in my mind. The need to restrict the use of the metadata to non-commercial contexts absolutely cripples the API's utility and undermines efforts to create a more powerful, usable, sustainable resource for all, and indeed to drive the creative economy in the way that the Europeana Commission originally envisaged. With a bit of luck and imagination this won't stay a problem for long, because a new data provider agreement will encourage much more permissive licences for the data, and in the meantime a subset of data with open licences (over 3M objects) has been partitioned off and was released this very week as Linked Open Data. Hurrah!

This brings us to the question of how LOD is used and whether we need a more precise understanding of how this might relate to the restrictions (e.g. non-commercial only) and requirements (e.g. giving attribution) that could be attached to data. I see two basic types of usage of someone else's metadata/content: publication e.g. displaying some facts from a 3rd party LOD source in your application; and reasoning with the data, whereby you may use data from 3rd party A to reach data from 3rd party B, but not necessarily republish any of the data from A.

If LOD sources used for reasoning have to be treated in the same way as those used for publication you potentially have a lot more complexity to deal with*, because every node involved in a chain of reasoning needs to be checked for conformance with whatever restrictions might apply to the consuming system. When a data source might contain data with a mixture of licences, so you have to check each piece of data, this is pretty onerous and will make developers think twice about following any links to that resource, so it's really important that aggregators like Culture Grid and Europeana can apply a single licence to a set of data.

If, on the other hand, licences can be designed that apply only to republication, not to reasoning, then client systems can use LOD without having to check that commercial use is permitted for every step along the way, and without having to give attribution to each source regardless of whether it’s published or not. I'm not sure that Creative Commons licences are really set up to allow for this distinction, although ODC-ODbL might be. Besides, if data is never published to a user interface, who could check whether it had been used in the reasoning process along the way? If my application finds a National Gallery record that references Pieter de Hooch’s ULAN record (just so that we’re all sure we’re talking about the same de Hooch), and I then use that identifier to query, say, the Amsterdam Museum dataset, does ULAN need crediting? Here ULAN is used only to ensure co-reference, of course. What if I used the ULAN record’s statement that he was active in Amsterdam between 1661-1684 to query DBPedia and find out what else happened in Amsterdam in the years that he was active there? I still don’t republish any ULAN data, but I use it for reasoning to find the data I do actually publish. At what point am I doing something that requires me to give attribution, or to be bound by restrictions on commercial use? Does the use of ULAN identifiers for co-reference bind a consuming system to the terms of use of ULAN? I guess not, but between this and republishing the ULAN record there’s a spectrum of possible uses.

Here's an analogy: when writing a book (or a thesis!), if one quotes from someone else's work they must be credited - and if it's a big enough chunk you may have to pay them. But if someone's work has merely informed your thinking, perhaps tangentially, and you don't quote them; or if perhaps you started by reading a review paper and end up citing only one of the papers it directed you to, then there's not the same requirement to either seek their permission to use their work, nor to credit them in the reference list. There's perhaps a good reason to try to do so, because it gives your own work more authority and credibility if you reference sources, but there's not a requirement - in fact it's sometime hard to find a way to give the credit you wish to someone who's informed your thinking! As with quotations and references, so with licensing data: attributing the source of data you republish is different to giving attribution to something that helped you to get somewhere else; nevertheless, it does your own credibility good to show how you reached your conclusions.

Another analogy: search engines already adopt a practical approach to the question of rights, reasoning and attribution. "Disallow: /" in a robots.txt file amounts to an instruction not to index and search (reason) and therefore not to display content. If this isn't there, then they may crawl your pages, reason with the data they gather, and of course display (publish) it in search results pages. Whilst the content they show there is covered by "fair use" laws in some countries, in others that’s not the case so there has occasionally been controversy about the "publication" part of what they do, and it has been known for some companies to get shirty with Google for listing their content (step forward, Agence France, for this exemplary foot-shooting). As far as attribution goes, one could argue that this happens through the simple act of linking to the source site. When it comes to the reasoning part of what search engines do, though, there's been no kerfuffle concerning giving attribution for that. No one minds not being credited for their part in the page rank score of a site they linked to – who pays it any mind at all? – and yet this is absolutely essential to how Google and co. work. To me, this seems akin to the hidden role that linked data sources can play in-between one another.

Of course, the “reasoning” problem has quite a different flavour depending upon whether you’re reasoning across distributed data sources or ingesting data into a single system and reasoning there. As Mia noted, the former is not what we tend to see at the moment. All of the good examples I know of digital heritage employing LOD actually use it by ingesting the data and integrating it into the local index, whether that's Dan Pett's nimble PAS work or Europeana's behemoth. But that doesn't mean that it's a good idea for us to build a model that assumes this will always be the case. Right now we're in the earliest stages of the LOD/semweb project really gathering pace - which I believe it finally is. People will do more ambitious things as the data grows, and the current pragmatic paradigm of identifying a data source that could be good for enriching your data and ingesting it into your own store where you can index it and make it actually scale may not stay the predominant one. It makes it hard to go beyond a couple of steps of inference because you can't blindly follow all the links you find in the LOD you ingest and ingest them too – you could end up ingesting the whole of the web of data. As the technology permits and the idea of making more agile steps across the semantic graph beds in I expect we'll see more solutions appear where reasoning is done according to what is found in various linked data sources, not according to what a system designer has pre-selected. As the chains of inference grow longer, the issue of attribution becomes keener, and so in the longer term there will be no escaping the need to be able to reason without giving attribution.

This is the detail we could do with ironing out in licencing LOD, and I’d be pleased to see it discussed in relation to the LOD-LAM star scheme.

Thursday, June 09, 2011

Hack4Europe London, by your oEmbedded reporter

I spent today at the London edition of Hack4Europe, held at the British Library. It was co-hosted by the BL, Culture Grid/Collections Trust and Europeana, and it one of 4 such hackdays around Europe in a single week all aiming to give the Europeana APIs a good work-out and to uncover new ideas for how they can be used. It was a really fun and interesting day, although for me it ended in gentle humiliation as I couldn't get my laptop to output to the projector. Compared to my previous attempts at days like this it had all gone swimmingly up till then, so to fall at the last hurdle was a bummer! There were lots of very creative, clever (and funny) ideas on going around and you should keep your eyes open because some may come to fruition in due course, but right now I'm going to indulge myself and talk mainly about what I attempted, because my presentation was a total #fail. So this is not really much of a report at all. Better luck next time.

oEmbed for Europeana
I took with me a few things I'd worked on already and some ideas of what I wanted to expand. One that I'd got underway involved oEmbed.
If you haven't come across it before, oEmbed is a protocol and lightweight format for accessing metadata about media. I've been playing with it recently, weighing it up against MediaRSS, and it really has its merits. The idea is that you can send a URL of a regular HTML page to an oEmbed endpoint and it will send you back all you need to know to embed the main media item that's on that page. Flickr, YouTube and various other sites offer it, and I'd been playing with it as a means of distributing media from our websites at IWM. Its main advantages are that it's lightweight, usually available as JSON (ideally with callbacks, to avoid cross-domain issues), and most importantly of all, that media from many different sites are presented in the same form. This makes it
easier to mix them up. MediaRSS is also cool, holds multiple objects (unlike oEmbed), and is quite widespread.
I've made a javascript library that lets you treat MediaRSS and oEmbed the same so you can mix media from lots of sources as generic media objects, which seemed like a good starting point for taking Europeana content (or for that matter IWM content) and contextualising it with media from elsewhere. The main thing missing was an oEmbed service for Europeana. What you have instead is an OpenSearch feed (available as JSON, but without the ability to return a specific item, and without callbacks) and a richer SRW record for individual items. This is XML only. Neither option is easily mapped to common media attributes, at least not to the casual developer, so before the hackday I knocked together a simple oEmbed service. You send it the URL of the item you like on Europeana, it sends back a JSON representation of the media object (with callback, if specified), and you're done.(Incidentally I also made a richer representation using Yahoo! Pipes, which meant that the SRW was available as JSON too.)

Using the oEmbed
With a simple way of dealing with just the most core data in the Europeana record, I was then in a position to grab it "client-side" with the magic of jQuery. I'm still in n00b status with this but getting better, so I tried a few things out.
Inline embedding
First, I put simple links to regular Europeana records onto an HTML page, gave them a class name to indicate what they were, and then used jQuery to gather these and get the oEmbed. This was used to populate a carousel (too ugly to link to). An alternative also worked fine: adding a class and "title" tag to other elements. Kind of microformatty. Putting YouTube and Flickr links on the same page then results in a carousel that mixes of all of them up.
Delicious collecting
Then I bookmarked a bunch of Europeana records into Delicious and tagged them with a common tag (in my case, europeanaRecord). I also added my own note to each bookmark so I could say why I picked it. With another basic HTML page (no server-side nonsense for this) I put jQuery to work again to:

  1. grab the feed for my tag as JSON

  2. submit each link in the feed to my oEmbed service

  3. add to the resulting javascript object (representing a media object) a property to hold the note I put with my bookmark

  4. put all of these onto another pig-ugly page*, and optionally assemble them into a carousel (click the link at the top). When you get half a dozen records or more this is worthwhile. This even uglier experiment shows the note I added in Delicious attached to the item from Europeana, on the fly in your browser.
I could have held the bookmarks elsewhere, of course - say in a GoogleDocs spreadsheet, or maybe Zotero - but Delicious is my day-to-day bookmarking application and it's very convenient to collect stuff from a button in my browser toolbar. Adding new tags to put together new collections is easy too.

I suppose what I was doing was test driving use-cases for two enhancements to Europeana's APIs. The broader was about the things one could do if and when there is a My Europeana API. My Europeana is the user and community part of the service, and at some point one would hope that things that people collect, annotate, tag, upload etc will be accessible through a read/write API for reuse in other contexts. Whilst waiting for a UGC API, though, I showed myself that one can use something as simple as Delicious to do the collecting and add some basic UGC to it (tags and note). The narrower enhancement would be an oEmbed service, and oddly I think it's this narrower one that came out stronger, because it's so easy to see how it can help even duffer coders like me in mixing up content from multiple sources.

I didn't
What I didn't manage to do, which I'd hoped to try, was hook up bookmarking somehow with the Mashificator, which would complete the circle quite nicely, or get round to using the enriched metadata that Europeana has now made available including good period and date terms, lots of geo data, and multilingual annotations. These would be great for turning a set of Delicious bookmarked records into a timeline, a map, a word-cloud etc. Perhaps that's next. And finally, it would be pretty trivial to create oEmbed services for various other museum APIs I know and to make mixing up their collection on your page as easy as this, with just a bit of jQuery and Delicious.

Working with Jan
Earlier in the day I spent some time working with Jan Molendijk, Europeana's Technical Director, working on some improvements to a mechanism he's built for curating search results and outputting static HTML pages. It's primarily a tool for Europeana's own staff but I think we improved the experience of assembling/curating a set, and again I got to strech my legs a little with jQUery, learning all the time. He decided to use Delicious too to hold searches, which themselves can be grouped by tags and assembled into super-sets of sets. It was a pleasure and a privilege to work with the driving force behind Europeana's technical team; who better to sit by than the guy responsible for the API?

*actually this one uses the Yahoo! Pipe coz the file using the oEmbed is a bit of a mess still but it does the same thing

Friday, April 29, 2011

What are they building in there?

I started this weeks ago and have decided I should just get it out there rather than finish it! That's the main lesson I've mis-learned from agile, although I've mis-learned lots more besides.

The work of the New Media department at the IWM is very complex at the moment, as it has been since I started almost a year ago. We're now a pretty large department and I'm not deeply involved in all aspects of what we do, but we're also a tight team and I think we all keep an eye on what's happening in other areas - which is great and necessary, but sometimes it all gets pretty confusing! Part of our team needs to focus on keeping our existing sites going a little longer, but they too are also working on the new developments. So with so much going on I thought I'd step back for a moment and try to lay it out for my own benefit as much as anyone's - it's very easy to get lost in the many strands of activity.

We're working on a number of overlapping projects under our direct control, but we also have crucial dependencies upon projects that belong to other departments. For us the flagship project - the one where the whole organisation is watching us - is the redevelopment of our core websites, which will launch later this year. This is also the mothership - the project that most other activities rely on or feed into. These include

  • a new mechanism (and UI) for aggregating, preparing and delivering collections content
  • the ongoing effort to improve our collections metadata (not our work, of course)
  • a wide-ranging e-commerce programme (which runs for well over a year after we relaunch the sites)
  • a realignment of IWM's identity
  • a refresh of multimedia throughout our branches

  • overhauling our web hosting

  • and finally, sorting out our development environment and re-thinking how we work (not a project but a big effort)

Some other departments have had to make considerable efforts so that the road-maps for the projects they own align with our own, and thanks to those efforts I think we're going to be able to deliver most of what we hoped to and maybe some extras when we launch, but of course that will just be the start of a rolling programme of improvement. We will start, for instance, with temporary solutions to some of our e-commerce provision. It's an intricate dance where we try to avoid doing things twice or, still worse, do things badly, but we also try to avoid being so dependent upon doing something perfectly (or that's out of our control) that we risk failing to deliver anything. For this reason we're limiting our ambitions concerning e-commerce to improving integration at a superficial level but postponing deeper integration such as unified shopping baskets and the like.

For me there are quite a few learning curves involved in all of this. I've got thin experience with the LAMP stack (we're using most or all of this, at least in our development environment - live O/S TBC), and nil with Drupal (we're using D7 for the core sites). My experience of working as part of a team of developers is also limited and has taken place in purely Microsoft environments, whereas we're adopting agile techniques and using tools like Jira/Greenhopper and SVN (no more Visual Source Safe) to help in this. The infrastructure at IWM is quite different to my old haunt, too, and finding my way round that, finding out even what I needed to know and how to get it done has taken pretty much until now. We do now have a development environment we can work with, though, both internally and with third parties, and in due course I hope to understand it! Besides that, there's the ongoing activity of researching and testing new solutions and practices, which is always fascinating, sometimes frustrating, and generally time-consuming.

So the new website will be in Drupal 7, although we may continue to use WordPress for blogs (WP3 is another new one for me). We were treading a fine line by choosing to use version 7, a decision we made a few weeks before the betas and, finally, RC1 came out, but we're feeling vindicated. Because some modules still have quite a few wrinkles there are some blockages (Media Module anyone?), but on the whole we feel it was a risk worth taking - there are major benefits in this version, and D6 will be retired well before D7. So when bugs have come up, Monique and Toby (the two Drupal devs we're so lucky to have working on this) have been balancing their efforts in working around problems, contributing fixes, and putting things on hold until patches are available. They've undertaken a fair amount of custom work where essential and we have a lot of functionality in place ready for theming. Design and theme development are taking place out-of-house, though, so the next few weeks should be interesting!
[June note: and so it has proved]

Lots of our thinking is around integration, of course, both to do now and in readiness for future developments. There's a lot around discovery and whilst I've not been anywhere near as involved with Drupal development as I'd like, I've been doing some of the work on search. We'll be using separate Solr cores to index Drupal content, e-shop products, collections metadata, stock photography, and a variety of non-Drupal/legacy IWM sites (indexed with Nutch). This will support an imperfect, loosely-coupled sort of integration, one that we can improve bit by bit.

Our strategy for delivering collections-related content builds on the software that we commissioned from Knowledge Integration when I was at the Museum of London. That conceptual approach and architecture (which I'll save for another post) was one idea I was really eager to bring with me to IWM, but I have to admit I found my advocacy skills seriously lacking when it came to explaining the case for it. All the same, we got there in the end and I believe that there's a growing understanding of the benefits of the approach, both immediately and in the long term - and now that we're making real progress with implementation confidence is growing that we can do what we promised. Phase 1 is strictly limited in any case, so that we have what we need for the website launch and that's it; Phase 2 will have the fun stuff - the enrichment, contextualisation, and nice things for developers - and we'll start building other things on top of the mechanism too, because of course the website is but one front-end.
[June note: K-Int have packaged up the software into a product which they launched at the recent OpenCulture exhibition. Very well worth checking out.]

That's where I got to. Will add more soon. Suffice to say that things remain quite busy but that we do at least now have a newly-skinned e-shop which teething problems aside looks a damn sight better than before, thanks to the efforts of Christian, Wendy, Kieran and Garry "The Bouncing Ball" Taylor.