Saturday, March 03, 2012

Installing Windows 8 Consumer Preview as a virtual machine

Windows 8 Consumer Preview is out. By all accounts the new OS is a radical departure from the Windows UX paradigm of the last 2 decades and it's quite possibly going to be a very important OS, not least because of convergence it apparently shows between desktop and mobile, with the interface rethought with a focus on touch. I've not played with it yet because I've just completed the installation, but I thought I'd put up a couple of words about that process. So far it's evident that the look is very different indeed.
Most of these instructions are explained in more detail and with screenshots here so you might want to consult that too, but that was written for an earlier release and I found a couple of gotchas fow which I've gleaned answers so I thought it worth putting them here. I've used VMWare Player, which is free. There are alternatives, just pick your version carefully as not all will work with Win8. So here's a quick step-by-step.

Step 1
Grab the 32 bit ISO from here
Step 2
Install VMWare Player 4. Version 3.x doesn't work and will give you a "HAL_INITIALIZATION_FAILED" error (I know, I tried). VMWare Workstation 8 also works apparently, as do some versions of other virtualisation software.
Step 3
Create a new VM. Just point at the ISO, wherever you saved it, and VMWare will do the rest. Pick "Windows" and then "Windows 7", don't put in a licence key. Let it go.
If you get an error saying "Windows cannot read the setting from the unattended answer file", disable the floppy drive in your VM - it's an icon at the bottom right (see here).
Step 4
When you get to the licence key screen, use the following: DNJXJ-7XBW8-2378T-X22TX-BKG7J (see here)
Step 5
Set up your account. I'm not sure how optional this was but I did it anyway. They ask for a lot of compulsory info but it does open up SkyDrive and other interesting aspects of the new OS so it's worth doing, I think. You can always bullshit.

You're done. Then find your way around. Hint: the Windows key is useful.

Wednesday, January 25, 2012

Solr to Google Earth

This is a basic how-to you can probably find done better elsewhere, but since I didn't find all the bits in one place myself I thought I may as well put this up.

The task: show query results from Solr on a map or in Google Earth using the latitude/longitude data in there. Make the results update as you move around, because there may be too many to bring back all at once.
The technology: Solr, XSLT, KML, PHP, Apache web server

This is a pretty common sccenario and for basic needs and people that aren't already into full-blown mapping solutions this is going to be a better choice. That said, for the latter there are plenty of options and you may want to investigate, for instance, OSGeo/OSGeo4W.

I had a small amount of time to evaluate out some possibilities for a future project, so I needed something quick and familiar. I'd done some Solr-based mapping a couple of years back so I had some code to nick. So, having turned some OSGB36 data into WGS84 lat/longs (thanks for the help, @portableant!) I got it into a Solr index, which I'm not going to go into here except to say that I used the "tdouble" datatype because a trie field seems like a good idea for efficient searching, and you need something that can cope with all those floating points. I believe there are proper geo data types but I'm ashamed to say I've not even bothered looking at them, I think with them you could do fancier proximity search and the like but basic is fine for me. So here's are the relevant bits from the schema.xml:

<types>
...
<fieldtype name="tdouble" omitnorms="true" class="solr.TrieDoubleField" positionincrementgap="0" precisionstep="8">
...
</types>
<fields>
...
<field name="latitude" indexed="true" type="tdouble" stored="true">
<field name="longitude" indexed="true" type="tdouble" stored="true">
...
</fields>

With the index done and queries working it was a matter of getting some KML out. Solr will transform XML on the fly so converting its XML output into KML is really not hard. Plus I already had a transform to cannibalise. It goes something like this:

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" space="preserve">
<xsl:output type="text/xml; charset=UTF-8">
<xsl:preserve-space elements="*">
<xsl:template match="/">
<kml xmlns="http://www.opengis.net/kml/2.2">
<document>
<name>Results</name>
<xsl:apply-templates select="//doc[double/@name='longitude']">
</document>
</kml>
</xsl:template>
<xsl:template match="doc">
<placemark id="{str[@name='id']}">
<description>
<xsl:text escaping="yes">
<![CDATA[ <![CDATA[ ]]></xsl:text>
<p>
<xsl:value-of select="str[@name='title']">
</p>
<xsl:value-of select="']]>'" disable-output-escaping="yes" />
</description>
<name><xsl:value-of select="str[@name='title']"></name>
<point>
<coordinates><xsl:value-of select="double[@name='longitude']">,<xsl:value-of select="double[@name='latitude']">,0</coordinates>
</point>
</placemark>
</xsl:template>
</xsl:stylesheet>

Put your own preferred fields in here, of course. Some notes: the CDATA bits are because you need to put HTML in your "description" element into CDATA, but outputting this with XSLT takes a little lateral thinking because it needs to see your CDATA declaration as....CDATA. Hence this structure (and the bit for closing the section with "]]>"). Secondly, note that I start by selecting the "doc" elements that Solr returns but filtered for the presence of "longitude", since we only want to show things that have a point. Actually you may do the filtering in the Solr query instead (or a filter query). As we'll get to later on, in fact. Final note: your fields will obviously be different and you'll want to put something interesting into the "description" element, which is what pops up in balloons on Google Maps and the like.
Getting KML out like this is just fine for showing this stuff on Google Maps, but this wasn't working for me on Google Earth. The reason is that GE wants the correct content type, whereas GMaps, OpenLayers etc don't care. GE isn't bothered by the file extension AFAIK, but headers? Yes. The other thing I wanted to do was create a network link, which is a means by which a request for KML can be updated to restrict it to a geographical area (a bounding box). With a network link you can specify how the north, east, south and west limits are expressed, which makes it pretty easy to slot those values into a Solr URL. However because of the content type issue this wasn't going to work.
So, here's a PHP file that proxies the Solr query and spits it out with the right content type:

<?php
/*
Google Earth wants text/plain or application/vnd.google-earth.kml+xml so this script is a proxy that will pass on all parameters to solr and return the results with the right headers
Set up a bunch of defaults like the number of points you want and the default bounding box, which is basically the whole world here (I think)
*/
$rows=100;
$start=0;

if(is_numeric($_GET["start"])){
$start=$_GET["start"];
}
if(is_numeric($_GET["rows"])){
$rows=$_GET["rows"];
}
$lat0=-90;
if(isset($_GET["lat0"])){
$lat0=$_GET["lat0"];
}
$lat1=90;
if(isset($_GET["lat1"])){
$lat1=$_GET["lat1"];
}
$lon0=-180;
if(isset($_GET["lon0"])){
$lon0=$_GET["lon0"];
}
$lon1=180;
if(isset($_GET["lon1"])){
$lon1=$_GET["lon1"];
}
if(isset($_GET["q"])){
$text="text:".$_GET["q"]."+AND+";
}

$solrbaseurl = "http://localhost:8080/solr/myindex/select/?";
$url=$solrbaseurl."q=".$text."longitude:[$lon0+TO+$lon1]+AND+latitude:[$lat0+TO+$lat1]&wt=xslt&tr=kml.xsl&start=".$start."&rows=".$rows;
$s = file_get_contents($url);
header('Content-Type: application/vnd.google-earth.kml+xml');
echo $s;
?>

[CAUTION: see below for a note on web server configuration for another necessary step.]
This script was written with a network link in mind, so I decided to pass in latitide and longitude start and finish values using the preferred parameters for this, but it's up to you how you do it (see the docs here but they don't make it clear that you can pick your own format for the bounding bpox parameters). I take them (lat0, lat1, lon0, lon1) and put them into the Solr query so that you end up with something like:
q=text:church+AND+longitude:[-1.5+TO+1.5]+AND+latitude:[48.5+TO+51.5]
(the "text:church+AND+" part is only put in if a query term was also specified)
So that gets you a set of results within a bounding box. So if you can update this with a new bounding box every time the user's viewport changes it's pretty useful. So the next thing is the network link itself. It's another KML file which again I needed to make on the fly so I could call it from a form, and, of course, I had to put it out with the right headers so here's another PHP script:

<?php
//this script is a proxy to create a KML file for a network link
header('Content-Type: application/vnd.google-earth.kml+xml');
print("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
?>
<kml xmlns="http://www.opengis.net/kml/2.2">
<NetworkLink>
<name/>
<visibility>0</visibility>
<open>0</open>
<description>A network link to some results</description>
<refreshVisibility>0</refreshVisibility>
<flyToView>0</flyToView>
<Link>
<href>http://localhost/myapp/solrKmlLatlongs.php?q=<?php echo $_GET["q"];?></href>
<refreshInterval>2</refreshInterval>
<viewRefreshMode>onStop</viewRefreshMode>
<viewRefreshTime<1</viewRefreshTime>
<viewFormat<lat0=[bboxSouth]&lat1=[bboxNorth]&lon0=[bboxWest]&lon1=[bboxEast]</viewFormat>
</Link>
</NetworkLink>
</kml>

The "Link" element in that KML points at the other file, proxying Solr (in this case located at http://localhost/myapp/solrKmlLatlongs.php)
"viewFormat" is the part where you can specify how the bounding box parameters are to be passed into your KML-emitting script. There's other stuff you can look up for yourself.
Basically, if you call this script with your solr query it will chuck out the KML with the network link, which you can view in GE (it will be switched off by default). Then whenever you zoom in or move around the query will refresh with a new bounding box. For me, with a pretty big dataset that you can't really load all at once, I may start with a text query that covers the whole of the UK (no bounding box parameters) but, being limited to, say, 100 results, these may be scattered all over the map. As you zoom in and shift round, it will load more and more results according to your current view.
The final thing to note is that it's not probably enough to set the content type in PHP. In my case I needed to add a couple of lines to my Apache config (httpd.conf) so it knew what to do:

AddType application/vnd.google-earth.kml+xml .kml
AddType application/vnd.google-earth.kmz .kmz

I hope this helps

Thursday, December 22, 2011

Three years with Spinal Muscular Atrophy III: one parent’s take

Since people reading this post are probably not regular readers of ths blog (who is?), a quick word of warning: I don't do concise. I could have split it into multiple posts but that's just cheating, so this is what you get.

This post could alternatively be entitled 4½ years with SMA type III, or even 6 years, but last month our third and youngest child turned 6 and shortly before that was the 3rd anniversary of our getting the diagnosis of his condition, which had first become noticeable (but unrecognised) when he was about 18 months old. It feels like a good time to talk a bit about what has happened in the interim, not least because I know that as a parent receiving a diagnosis of SMA type III it was difficult to get a grip on what it would mean for our child and for our family as a whole (we have two older children, aged 8 and 10 as I write). Perhaps by putting this out there it might come in the way of some other mother or father anxiously googling the diagnosis they have just had for their beloved child, and maybe reading about our experience will, if not console them, then thin out the terrifying cloud of the unknown that seemed to suddenly appear before us. No two cases of SMA are the same – it is a spectrum disorder (type III is at the mild end), and you cannot take one experience and expect to see the same pattern elsewhere. All the same, hearing about a single case is better than hearing about none. I also wanted simply to celebrate our kid, whose disability is merely a strand of his life and whose character and spirit, experiences and growth are unique, beautiful, special – just like those of our other two children. SMA is not the definition of his life but an aspect of reality for him (and to a lesser extent for all of us). So this is going to be a few snapshots of his and our experiences over the last three years, including a few pictures and videos, to generally illustrate that you too will cope and your kid will thrive in their own way.
To kick off, and coz I love all three of them, here's a snap of our lot a year ago:

IMG_6272
What is SMA?
I need to take short step back first for some explanation of SMA. I would recommend that you do NOT pay this much mind if it is important to you, because I am not a doctor and am writing from memory and personal experience, not from a book (coz I’m on the train). There are also very detailed resources online but it can be hard to know from these what to actually expect for your child, or when, so talk to your specialist. A good starting point, though, is the Jennifer Trust for SMA website
For those who have not come across Spinal Muscular Atrophy (like my wife, Fiona, and I before we met the consultant for the first time), it is a degenerative neuromuscular disorder. We discovered that there are a lot of these. Some such disorders, such as the various muscular dystrophies, affect the muscles directly. Some affect the nerve fibre, like Multiple Sclerosis. SMA also affects the nerves directly, but by disrupting the junction in the spinal cord to the motor neurons leading to the limbs and thorax. A muscle is made up of bundles of fibres each with their own nerve fibre, and those without a properly firing nerve obviously don’t work and that fibre will atrophy, leaving the muscles themselves weaker. The pattern tends to be of it affecting the upper part of the legs, arms, and respiratory system (in that order). At present there is no known cure for SMA. As I say, the condition presents in a spectrum of severity but is classified into three types on the basis of the timing of its onset, as well as “adult onset SMA” which appears to have somewhat different causes and effects. At the mild end is type III, in which children start to show symptoms around 18 months. They are likely to be walking prior to onset and it is the deterioration of their mobility together with lots of falls and difficulty in getting into a standing position that are the likely signs that will have parents looking for answers. The weakness may affect the arms or breathing too, although probably later and not as severely as in type II. In type II symptoms become evident around 6 months, I believe, and the child is unlikely ever to walk. As it progresses the arms will likely also be affected and later on possibly the respiratory system too, shortening life expectancy. Type I is evident from birth and tragically is generally fatal in early infancy. I feel quite callous writing that, because I know that there might be parents who read this who face that world-shattering prospect and I do not. If this is you, I hope you will accept my sincerest sympathy and most heartfelt wishes for happy times with your child.
SMA has a genetic basis with (AFAIK) no significant established environmental factors that influence its onset or its progression. It is recessive, meaning in essence that both parents must be carriers of the mutation. Perhaps surprisingly the mutation is estimated to be present in around 1 in 40 of the population. The maths of this means that on average 1 in 1600 couples will both possess the gene. Assuming that they are both carriers rather than having 2 copies of the faulty gene (in which case they would themselves be affected by SMA), there is a 1 in 4 chance that any given child of theirs will have 2 copies of it. Which is how it came about that Fiona and I have two children without SMA and one with. The result is that around 1 child in 6500 is born with SMA – about 100 kids a year in the UK. Not a trivial number, but not so many that you are likely to have come across many of them.
Enough background, then. To our story.

There’s a problem
To start with we just noticed that Kid3’s feet appeared to have turned in and his gait was changing, with his feet moving further apart. This was at around 1½ years old, and on the advice of Fiona’s father (a very experienced GP) we had him checked out. In short he ended up with splints to support his feet. These did appear to help somewhat, but it became clear that his legs were still not doing what they should. He would fall over frequently, often with his legs folding underneath him and his head hitting the ground. He’d get to his feet by walking his hands up his legs (the “Gower’s sign” typical of SMA). He still couldn’t climb stairs by an age where children can usually do this. He also had a hand tremor, sometimes visible, sometimes just felt, sometimes not evident at all, and we associated this with emotional moments – excitement, upset. My father-in-law recommended that we arrange to see a paediatric neurologist, and now I know what his fears were.

Diagnosis
We went with Kid3 and Fiona’s father to Addenbrookes Hospital in Cambridge for a consultation, during which the names of various conditions came up, mainly SMA and muscular dystrophies of various sorts, with Duchennes the only type that really fitted the symptoms. I’d read and heard a little about the MD but with its many forms I hadn’t quite registered what the implications of Duchennes might be, but it was clearly a strong candidate. It was clear now that our little boy’s condition was not one of slack tendons or anything other than a serious neuromuscular issue, which I’d not really wanted to countenance before. Fiona was more aware than I of what it meant, and it showed in her reaction. A blood test for a protein would be a first step to confirm whether Duchennes was likely, along with genetic tests for that and for SMA, and we went straight off that day for the blood sample to be taken – in itself a painful sight. We understood that it would take several weeks for the genetic test to be done and left expecting a wait. In something of a daze we went home incapable of doing or thinking much. We had my parents-in-law to support us and gently talk us through what it all might mean, but I think I still didn’t really get it properly.
The next day at work I spent the morning looking up Duchennes and the horror grew and grew in me. This is a condition that, I learnt, might see our son immobile before his teens and dead by 20. The world was falling apart and I was being hollowed out. But not long after midday, Fiona rang and told me that the protein test had already been done and ruled out Duchennes MD, and that the consultant had wanted to let us know as soon as possible so that we would not go through the agony of thinking that was the likely diagnosis for longer than necessary. He later told us that he really had thought it the most likely candidate. I don’t need to spell out for you the feeling of relief we shared. I didn’t really know what SMA might mean, though it was now the obvious candidate, but I did know things looked markedly less bleak than they had. A colleague came round at about that point to talk about some work thing or other and I just cried with relief on his shoulder. And that was the end of easily the worst few hours I have ever experienced, which ended effectively with a diagnosis no-one would wish for but with, at the same time, an insight into what might have been, and into the feelings of a parent being told their child’s life would be cut short. In some way this has stayed with me whenever things have been difficult. It’s not enough always to say “ah well, it could be worse” when you are brought face-to-face with a new consequence or worsening of your kid’s condition: that still hurts like hell; and yet, I thank my stars for what we and he have.

Facing it
There were then electrode tests to understand the strength of the nervous impulses in our son’s arms and legs. The results fitted with the SMA diagnosis, which was confirmed by the genetic test. At our next meeting with the consultant we talked through what type III SMA would actually mean, what research was underway, and what our next steps might be. Perhaps the hardest thing, in terms of understanding the implications of the diagnosis for Kid3, was (and is) dealing with the unknown, because the “progress” (degeneration of the nervous connections) varies so much and you’re desperate to know what to expect and when. Scratch that, actually. The hardest thing? Powerlessness. You want to be able to do something, anything, to make a difference. To be entirely dependent upon other people (never mind the fact that they can’t do very much, at least in terms of providing a cure) is difficult, but I suppose it was also being able to get to the bottom of the question: is there actually anything I can be doing? Am I missing any opportunities because I haven’t asked the right questions – is there a clinical trial, a scheme, that we’ve missed? I cannot speak highly enough of Dr Verity, the consultant we had, and his advice and the efforts of my father-in-law to discover everything possible about SMA, to interpret it for us and reassure us that we were not missing anything, these were so valuable. Still, we had a long journey to accommodate the new reality. I won’t attempt to describe that journey, and Fiona and I certainly handled it differently at times. I will say that the support of family and friends was immeasurably important, but in the end the only way to move onto a new phase was for us to gradually absorb the idea that this is simply our “normal”, not an acute situation that could be addressed and put behind us. Instead, what our son would regard as the normal state of affairs from his earliest memories needed to become that for us: something where the practical implications and needs of the situation are handled calmly and, without a feeling of crisis or drama, just dealt with. Usually that works.
One of the other things that we just have to feel our way through is how much help to offer. We don't want to make him overly dependent or to interfere when for his own self-esteem and confidence he should be left to, say, get himself upright or try something new that he may well fail at, but equally things are more difficult or tiring for him than for most other children. But with any kid one might push them to walk further than they've done before, but never as far as you could walk, and when your kid has reduced mobility it's a matter of calibrating differently.

Developments
What has happened in the intervening time? Well Kid3’s walking isn’t noticeably different: he falls over a little less, perhaps, because he knows better how to support himself on things, so there’s no way you’d expect him to go more than a few yards without a hand to hold. Stairs are usually climbed on hands and knees, although he has handrails on both sides (of very narrow stairs – I never thought that would actually be helpful!) and occasionally uses these to help him up or down. What he does have over other kids is impressive upper body strength, which comes from using his arms to do so much stuff his legs would otherwise do. He’ll climb to his brother’s top bunk essentially using just his arms and with help he’ll take on trees, monkey bars and climbing frames too, and he throws a mean punch. Hold his hand and you get a steel grip you’d not expect from a some sweaty little kid’s mitt. He likes to do acrobatics that use this strength and that other kids can’t do (still looking for a video for that). He'll do a simulated parachute drop too - that was cool!


He’s been at school a year now, settling in like any kid, and it’s safe to say his very outgoing character and huge energy have put him in good stead.
When he was 4½ Kid3 got a wheelchair, which he just loves, and he mastered it the moment he first sat in it.


The wheelchair was one of those things where as a parent you are perhaps a little reluctant to accept that it might be necessary. Indeed I was a little worried that it might see him exercising his legs less, which is important because it will help him to make the most of the muscle fibres that are receiving signals and keep his legs as strong as they can be. But of course it’s been a wholly good thing, giving him the freedom to go where he pleases and at speed and to feel like he’s on equal terms with other kids, plus it’s something he can do and other people can’t. We got a massive off-roader beast at the same time from Delichon (bloody expensive but so worth it). Check this out:

IMG_5716
This baby has done extreme stuff and it’s beautifully engineered and good for kids up to mid-teens.

Having perhaps 20-25% of the usual amount of strength in his legs (just a guess) doesn’t mean he can’t do anything with them, of course. He has swimming lessons as well as hydrotherapy, and he loves a game of football, although this one definitely involves a lot of support from an adult (possibly swinging him in the air like a croquet mallet). We tried a balance bike, thinking that having no pedals it might work for him, but the act of balancing itself takes both control over the legs and a degree of core stability – another area weakened by SMA. So for his 6th birthday we got him an Ezy Roller, which is like a little go-kart powered by his arms wiggling a handlebar from side to side. Here's a clip of the boys hunting me down on Ezy-roller and scooter:


So he can now join in as his siblings skateboard, scoot and roller-blade – and we also tried a Micro Scooter, which has two wheels at the front so has enough stability for him to ride, a narrow rear to keep wheels out of he way of his trailing foot, a really low deck so he can reach the floor without having to bend his knee (an action that is sure to cause him to collapse), and steering that is not based on twisting the handlebars but tilting them - all of which seem to help. It was just brilliant to see him ride this. He had a lot of falls (but he’s made an art of that) but I didn’t really expect we’d see him scooting. Both of these came directly from Micro-Scooters’ base on Mersea Island, which is near to us, and I have to say a big thank you to them for letting us try stuff to our hearts’ content. Brilliant company. Below is a clip of him scooting. It's not exciting but it shows what's possible with this style of scooter. It also shows me doing what I talked about earlier and muddling my way through when to help and when not to, for better or worse.



My wobbly legs
One of our early worries was for how our son would handle it when he realised that he couldn’t do things that others could. We could see how his determination and vivacity helped him take on all sorts of things without complaint, but knew that at some point he was sure to think, “why is it just me that can’t do the things my friends can?” Will there come a point where he feels this and despairs, or turns to rebellion or anger? I guess we’ll see.
He is aware of his condition, of course, but treats it in a very matter-of-fact way. Sometimes it’s heart-breaking to see how matter-of-fact he can be, in fact. One thinks, “this shouldn’t have to seem normal to him”. He sometimes calmly sits out a game knowing that it’s not something he can do, though he may well recruit someone to help him do things his way. Only once do I remember him making a rather sad remark. I was carrying him down the stairs at a friend’s house where he saw his reflection in a big mirror, with his legs swinging freely as they tend to do, and he told me “I hate to see my wobbly legs when I’m walking”.
We’ve had several open discussions with the other children about what SMA is and means (normally on long car journeys), with him listening in. It’s hard to know if that’s the right thing to do, but that’s our approach: be open, don’t make a drama of it, don’t make him feel abnormal whilst being clear that he is naturally going to need help. I don’t think we have a strategy, we just have to feel our way and take our cues from him.


Help from outside
Beyond our friends and families, whose support has been so precious, we’ve had a lot of help from outside agencies and it’s well worth knowing where you can look. Actually the first to mention (again) is the Jennifer Trust, who provided us with a list of the various agencies we could turn to and some of the steps we could expect. The Muscular Dystrophy Campaign is a much larger charity than JTSMA with a wider focus but including SMA in its remit. With a bigger constituency it has quite busy forums where you might find a place to seek advice or share experiences.
Various services come through the NHS, from physiotherapy and occupational health, to wheelchair services, to consultations with paediatric neurologists. Our paediatrician helped in many ways other than the clinical, putting us in touch with people and advising on form filling (there’s a bit of that). As far as keeping in touch with research goes, aside from asking your paediatric neurologist you keep you informed there is a research network, Treat-NMD (it started as a European initiative but has gone global), that is worth looking into. It runs patient registries and a whole lot more besides. Like us you may not feel ready to face thinking about it immediately but do take a look.
A child that needs as much help moving around as one with SMA is entitled to (and needs) certain benefits, and you should get this underway. Check out the Motability scheme too (even if it uses ColdFusion for its website), which has enabled us to get a vehicle big enough for wheelchairs and buggies and also means that Fiona can get to all the many appointments.
The primary school and county council have worked together to provide support in class for Kid3 and I recommend pushing for this, especially if the school is an older building where the physical environment may be tricky for kids with “wobbly legs” or wheelchairs and where adaptations may be hard to implement. Getting around the classroom, participating in the playground and in PE all need our son to have help, and he gets that from a dedicated assistant. It may seem overkill but I assure you that it’s not, and because he’s a bright kid who doesn’t need extra help when sitting down at his work it means that the class also gets an extra adult to pitch in and help anyone, which is good for everyone. So whilst there’s less money to go round and it may be more difficult now to persuade your council to provide classroom assistance, don’t be afraid to try.
I mentioned swimming lessons, and again this is an experiment that has worked. Having taught our two older children the swimming teachers have been happy to take on the challenge of finding techniques that work for our youngest, giving him genuine swimming skill, confidence in the water, and noticeably improved muscle tone too. It’s exhausting for him, but that’s a good thing.
I’m sure there have been other sources of help and there are certainly many people I’m grateful to but I want this to focus on things that could be useful to others and I think this is a good start. I hope it’s helpful.


Conclusion
There is no conclusion. We carry on, we have a happy life with our kids, there are annoyances and things that we all miss out on or that Kid3 in particular misses out on, but really he's just like any other 6-year-old mentalist. He has no super-powers but he has a big smile, smart brain and lots and lots of attitude. We don’t know what’s ahead of us, but that’s just life. We count our blessings, and then they shout at us, kiss us, draw a picture, run over our feet with a wheelchair. They may even stand there three feet high in a Darth Maul costume, rabbit-punch us and then fall over backwards with the kick-back... And so the blessings pile up, only a little black and blue.

So if you've just found out that you are in our situation, with a child that is going to have difficulties you never foresaw, please don't despair. I hope you've seen that there are ways round a lot of things and always, always a lot of fun to be had. All my best to you.

Postscript. As I got to the end of writing this, I became aware of a blog started recently by the parents of Estrella, a little girl who lost her life to SMA but a month ago at the age of 8 months. Her story, their story, is very different from ours and humbles me. I read the words of people in the midst of grief who are yet trying their very best to do something positive; and now I feel more strongly than ever the sentiment in the title of that blog: Smash SMA. Amen to that.

Monday, November 07, 2011

New IWM websites pt.III: the, um, website

So, as on the evening before we switch our new website over from beta to fully live status I finally get round to the website part of this series of blog posts. In part 1 we did brand, e-commerce and hosting. In part 2, collections and licensing. Here, we'll look in too much detail at building the core website itself. Sorry, it's a long 'un.





Why a new website?
For the last 8 (I think) years, IWM has used BoxUK’s Amaxus CMS to run its websites. Naturally the sites were getting creaky and the CMS itself has been superseded, IWM itself has changed, and so has how the web works – both technically and in terms of the behaviour of web users and the language of interaction that they understand. A clean sweep was in order, which means a variety of strands of work. This much was clear when Carolyn Royston (our head) and Wendy Orr (our Digital Projects Manager, and so lead on the website project) outlined their ambitions to me when I started at IWM in May 2010.

Choosing the platform
Research for the new sites had began some time before that May but planning really kicked off in June. For me, the first key deliverable was the selection of a technical solution, but although I had a fair idea which way I would go I wanted to know a variety of other things first. How can you choose a CMS without knowing the functional specification, and how can you really know that without settling the information architecture to some degree, and the ways that people will find content and interact with the site? Decisions on whether we’d be supporting a separate mobile site, for instance (we don’t, at least not for now), and our plans for legacy sites all could have an impact. But of course you can only work out so much of this beforehand, and most questions seem to lead to others in a Gordian knot, so in the end you have to assess the situation as best you can, put together your own set of technical priorities, and make your selection as something of a leap of faith. I had the benefit of advice from various knowledgeable people in the sector who told us of their experiences with various CMSs, in particular IMA’s Drupal mage Rob Stein and the V&A’s Richard Morgan and Rich Barrett-Small, and we also had demos of a couple of commercial CMSs. Most importantly, though, we had Monique Szpak, whose role in this project (and my learning process at IWM) really needs a blog post of its own. Her experience with various open source products including Drupal was key, and after we identified that as our preferred solution she built us a proof-of-concept late last year to confirm that Drupal was likely to be able to do what we needed, and to assess the likelihood that Drupal 7, which at that point was still in alpha, would be ready when we needed it. With this information we took an informed gamble that it would be, and the choice was made.

Development
As I already said, we started development work even before settling finally on Drupal, as a piloting project, and this continued whilst we were developing our plans for content, IA and design. There were a number of things we knew we wanted, even if the functionality was still hazy – with Monique’s help we’ve instituted agile practices which positively encourage trial, error, testing and improvement. This change, in fact, together with the development environment we’ve gradually (& painfully) pieced together and the implementation of tools like Jira and Subversion, has been fundamental to making this project work, and it would have been impossible without Monique. Whilst she worked on prototyping more functionality, I did some groundwork on indexing shop and external sites. Then in the spring Toby Bettridge joined us, fresh from working on the Drupal part of the V&A’s new site. He and Monique worked very closely (with the help of Skype) and long before the design work was complete we had basic versions of the taxonomy, events, multi-index search and collections functionality done, amongst other things.
Although I’ve been paying attention to what they do, my hands-on involvement in Drupal development has been pretty much nil and I still understand the CMS far less than I’d like, so anything I say about Drupal development here needs to be read with that in mind! I do get, though, that one picks modules carefully, develops new ones with reserve, and never hacks core. We started developing with Drupal 7 before it was released, and even when it was there were (and remain) quite a lot of modules that weren’t ready to use. We thought the gamble was worthwhile, though, and forged ahead. In time we did incorporate some of them, although unfortunately we still don’t have some of the things promised by e.g. Workbench. Along the way Monique and Toby also did some vital module development of their own, notably a custom collections search module (using Search Api Solr Search), media embedding for authors using IWM’s oEmbed service, entity lists (old-style Drupal nodes had lists, but new-style entities didn’t), and some administrative tools.
My role in development? I’ve often felt somewhat awkward, if I’m honest, about my fit, because having elected to go with an entire technology stack and various development practices that were new to me, I often found I couldn’t really contribute practically, even where I understood some things well. For instance, although I have plenty of experience with Solr, my practical contribution to integrating it with Drupal was negligible; likewise if I knew what was required to fix some HTML/JS/CSS at the front end, I could not implement this in an unfamiliar environment for fear of messing up Drupal or making some Subversion faux pas. I think I’ve made but one single (successful) check-in of Drupal code. I concentrated instead on sorting out development and live hosting, working on getting the collections data right, filling the holes in the spec as we noticed them, and so on. I spent a good while working out how the media streaming worked and how to embed that in our pages, using the DAMS’ web service to build a light-weight SOAP-free alternative (an oEmbed service) that could both serve our websites and potentially 3rd parties. When everything calms down, though, I need to properly get to grips with the codebase.

Information architecture, discovery & URLs
Working out the logic for a site that’s going to function for several years is not easy. One can change that logic if necessary, but you really need to know how likely that is to happen in order to give your designers some parameters to work within – how flexible do menus have to be? How directed will the user be, and how much should any one piece of content be located in a specific part of the site? As I said earlier, the brand structure and the 5 IWM branches were a big factor in how we had to organise content, since we needed to make things readily discoverable whatever the user’s journey, but without making them context-free and confusing. Another pair of conflicting priorities were the wish to avoid having too many top-level menu items and the wish to keep the site fairly flat without obliging too many clicks to find content.
Sites like the V&A, which relaunched earlier this year, have taken adventurous routes to delivering masses of content to users (or users to content) – in the V&A’s case, centring around search and introducing a sort of machine-learning to categorise content and indeed to identify what categories might exist. Brave stuff, and a great solution to the huge volume of content they have there.
At IWM we played for a while with the idea of a taxonomy driven site, wondering if we could use a set of taxonomies as facets onto different aspects of the site that would let users cut across a traditional hierarchical organisation of content. We’ve kind of gone with a watered-down version of that, wherein the structure of the content is fairly obvious and on the whole quite flat but we’ve used controlled terms and free tagging to help make things more discoverable to users coming from other angles. This is pretty conventional and at the moment of limited power, but in due course we will make greater efforts to align our taxonomies (in particular our history taxonomy) with the controlled terminology used in our collections. This was too much to do in this phase, but when that happens we should be able to make ever-better connections between our collections and pages like our “Collections In Context” history pages, learning resources, galleries and perhaps events. A learning-focused vocabulary will do the same, but right now our e-learning resources are pretty much non-existent.
Perhaps more important than taxonomy at the moment is search, which has been a key way of integrating content that lies outside our main site. We’ve elected to run 4 separate Solr indexes for this, and to keep them separate owing to the distinctive nature of their content. We have the Drupal index itself; collections data; an index of products extracted from our Cybertill e-shop; and a crawl (using Nutch) of a number of IWM sites that are outside of Drupal, such as blogs and the “Their Past, Your Future” learning resource. The last one needs a lot more work but as a quick-and-dirty way of ensuring that those legacy sites weren’t left out in the cold it works. And yes, a Google custom search engine would have been an alternative but then it would not have worked in the same way as the other searches, with deep integration into Drupal and the ability to treat the results as entities and reuse them elsewhere.
One obvious change with the new site is that, well, it’s one site. Previously we used a morass of subdomains for somewhat independent branch sites and even for the collections-pages-that-weren’t-collections-search (collections search had its own domain, no less). I for one found it pretty confusing. With the rebrand making the “IWM-ness” of all of our branches more prominent we were able to do the same on the website. I had been through a similar exercise at the Museum of London, and though it was not an identical situation some conundrums and dilemmas were shared by both. How to make it easy to access non-branch specific content and information to all users in the same place as branch-specific content, and how to make sure that people are well aware that the latter pertains only to one physical site? How to cross-promote? Like MoL we had no specific digital brand, nor a mother brand to distinguish particular projects or sites from cross-organisational activities. I hope we found a solution that works for our users and not just for IWM itself, but time (and more user-testing) will tell.
The expectation that we’d move content around and that the organisation of material on the site, as seen by users in menus etc, would not be forever, prompted me to seek a URL structure that was a little more abstract. I didn’t want URL components necessarily to be the same as top-level menu items, which might disappear, but to relate to more stable concepts of what IWM does and offers whilst remaining meaningful. That doesn’t mean permanent URLs but hopefully relatively long-lasting and predictable ones. In one area – collections – we do aim for the URLs to be “permanent”, though (whatever that means). What I tried to do was put what I imagined to be the most stable aspects towards the left hand end of the URL path, things like “corporate” and “visits”, because I envisaged these as being more stable than even branch names (we might get more branches, or rename them again). I also wanted to be able to put non-branch content under these. The result is that we don’t have branch names at the top of a hierarchy but reappearing in a few places – visist/iwm-london as well as events/iwm-london and others. It may seem messy but I hope it’s reasonably predictable all the same, and it means we never need a catch-all URL to cope with the miscellany that we hadn’t foreseen would ever exist outside branches.

Design
We appointed the Bureau for Visual Affairs, who were responsible for the National Maritime Museum’s new website’s design, to do the same for us. Judge for yourself how they’ve done, although good or bad the credit or blame are not all theirs, even when it comes to aesthetics. Design and content go hand in hand, and in some places we’re still working to improve the latter to make the best of the former. Under the covers, too, the HTML that’s spat onto the page is the result of BVA’s HTML coders’ fine work at one end, Drupal at the other, and the best efforts of our devs to bridge the gap. And sometimes the gap was pretty big.
The theming process was one area where our plans went somewhat awry. We had two experienced Drupal developers on our team, but as there was plenty for them to do in back-end development we were planning on the theming being handled by whoever we appointed as designers. BVA, however, are not a Drupal house but their design was what got us all excited, so we reached an arrangement with a third company to subcontract to BVA to do this part of the work. Having done it once, this is not something I would recommend - at least not unless you can make it very clear who answers to whom and where the line lies between development work, theming, and HTML development (and who pays who for what). We ended up some weeks behind but got back on track with the help of Ed Conolly of http://www.inetdigital.co.uk, who moonlighted as a themer for a few weeks and helped put a spring back in everyone’s step. Bravo Ed!

Content
Early in our content planning we decided what we’d migrate from the old sites (not a lot), what we’d need to keep going (a small set of microsites) and, broadly speaking, what we’d want to add to the new site. Killing off content doesn’t usually sit too well with me, who’s a conservationist and archivist by inclination. My instinct is that it’s sure to be useful to someone to have pretty much everything we’ve ever done remain available, but that’s nonsense really and far from helping people could end up confusing them, not to mention sucking up resources for maintenance that would be much better spent on creating new content of real worth. We did have an awful lot of pages that related to old exhibitions and so on, and were very keen to disentangle ourselves as fully as possible from our old content management system, Amaxus 3. In the end we have kept three or four microsites from that. Other content needed substantial alterations to bring it up to date and suit it to the new site structure.
However, beyond the core, practical information about visits etc., we wanted to do something that would directly serve the core purpose of the IWM: to tell the stories of conflict through the material we hold; and we wanted to do it in a rich, immersive way. BVA came up with a solution that looked lovely, although we went through a few iterations in order to make it easier to create the content and to draw parts of it from the collections middleware. We wanted HTML that could be generated almost automatically, which opens up other potential uses for the template. This took away some of the visual sophistication with which BVA won our hearts, and I suspect that they were a little unhappy to see this go, but this is a site that we want to add to frequently and without having to use HTML developers to do it, so I think we found a happy medium. Our “Collections in Context” (or simply, “history”) section contains over 100 articles at present, using images, audio and video to tell stories spanning from the First World War to the present conflicts in which the UK is involved. They were written by one of IWM’s historians and put carefully worked into the CMS by our team in a close collaboration that we hope to turn into a rolling programme of content creation, perhaps reflecting current events or notable anniversaries. I hope in due course we can extend the use of the format to other parts of the site and other voices, perhaps enabling its use as a tool for our website visitors. The people who deserve a shout-out for writing, editing, and/or inputting the hundreds of content pages that make up the new site are New Media’s Jesse Alter and Janice Phillips together with Maggie Hills, who has joined us for a busy few months.
BVA brought a couple of other bits of bling to the site, with the aim of a more engrossing, immersive experience. First amongst these is the “visual browse”, a slideshow mechanism that underlies many of our pages and is brought to the fore by clicking a tab at the top left. We can make any number of these and surface them on the pages where they are relevant – for instance, each branch has its own visual browse.

Is it any good?
When I stand back from whatever details might be preoccupying me on a given day I’m really pleased with the overall effect of what we’ve done, but of course I am not a typical user and what will count will be the feedback we get from our users. But for me, I’m especially pleased with the history pages and the way that our collections are now used there and in the search pages. I am also pretty pleased with the balance we’ve found between the individual branches (essentially, the needs of the physical visitor) and the cross-branch/non-branch activities and content, but because this is necessarily a compromise I expect that it will not work for everyone.
I have reservations too. I think the lack of a mother brand is a problem, and I think we need to make the home page work harder to offer a powerful message of what IWM as a whole is. The lack of fly-out menus is galling to me, although the ones for branches work well. It means more of a leap into the unknown and more clicks to find what you’re after. Our lovely, lovely history content is hard to find. Mobile performance is not that great – the whole site is too wide to load full-width with the text legible, and a ton of stuff loads onto the page that is not important for the mobile user. It functions OK, but it’s far from an optimised experience.
So, my opinions aside, there’s plenty to do over the coming months. But it will feel mighty good to have this milestone out of the way: November 8th – switchover day.

Wednesday, November 02, 2011

New IWM websites pt.II: Collections and licensing

This is the second post about our "big bang" and the things we launched on October 4th. In the first post I talked about the new brand, e-commerce, and hosting. Here I'll talk a bit about collections and the closely related issue of licensing.


The collections
So, now we’re getting more into the area I can talk about more knowledgeably. The opportunity to help reimagine how IWM’s collections are brought to the public through digital media was one of the key attractions that brought me here 18 months ago (though there were several other extremely compelling reasons. I think some of them are still in post). I felt I could bring something useful from my experiences at the Museum of London, having been part of the team that brought an ambitious new system there just before I left.
I hope I’ll be able write it all up properly soon, but I’ll keep it brief here. The collections online project at IWM had two objectives: firstly to build the foundational infrastructure for all future data-driven collections applications; and secondly to build the first public interface onto that infrastructure with the collections search pages in the new website. Only the bare essentials of the infrastructure were to be built in this phase: purely what was necessary to deliver to the web application and to be provide enough of an architecture for us to plug in the planned extra features later on. We have plenty in line for phase 2, but we’ll tie all that stuff in with specific front-end requirements.
Simon Chambers came on board with us to project manage this one and he did an incredible job of marshalling the requirements, prioritising them, working with a number of departments and strong-willed people and getting things as quickly as possible to the point where we could deliver the baseline of what the website needed. We ultimately decided to work with Knowledge Integration, who built the CIIM for us at MoL and who could bring us an existing application that fitted our needs very well.
Essentially the CIIM, at least the part we’ve implemented so far, pulls data from the collections management system (in our case Adlib) and remodels it to serve the needs of discovery and delivery as opposed to data management. These are very different things, and that a CollMS may do the latter very well doesn’t make it ideal for the former. This is as much about how the database is used across the organisation’s varied collections as it is about the technical qualities of, say, Adlib specifically, because this architecture allows us to intervene in the data between its source and front-end applications that use it – to remodel and align it, to integrate it with other data sources or enrich it, to prepare associated media, and to optimise it for full text searching, for instance. The big job since February, when things kicked off in earnest, has been modelling the data correctly. I have to admit I seriously underestimated the complexity of getting this right, and we had a series of problems to do with the API it was extracting from and the readiness of some of the data, but in a way this illustrates why it’s a good thing to be able to do all this work away from the front-end.
The result is a first-pass at a Solr index that, along with 3 others, lies at the heart of discovery on our new website. Try out the search engine here, or here are some good searches to get you going. Watch a video. Listen to interviews or momentous radio broadcasts (Czech alert). Oh and if you find an object like this you'll see that it's part of a collection, and can see a complete listing of that collection. Our priorities mean that we’ve deferred implementation of some of the features we want in the longer term but we know we can leap into action soon. In fact, Tom Grinsted, our multimedia manager, has put together a project with UCL and K-Int that gives us a focus for some of this functionality and is just getting underway now, which is rather exciting. Luke Smith and Giv Parvaneh are also busy planning various projects for the next few years as part of the centenary of the First World War that will also draw on and feed into the system. So watch this space.

Licensing
All that e-commerce work around collections has also meant reviewing the way we licence our material. Recent developments at national and European level – notably the creation of the Open Government Licence by the National Archive (TNA) – and the steps that some of our peers have made in offering their assets for creative reuse to the benefit of all, have also had an impact. IWM has now launched its User Licence (essentially the OGL), which frees up almost 200,000 images, audio recordings and films, for non-commerical use. Regular "fair dealing" restictions apply to others, like this nice Ronald Searle picture. I'm afraid we've not yet got a filter in collections search for items with this licence, but try right-clicking on an image on an item page to see if you can download or embed it.
The licence applies to IWM-generated content and data too, so although we don’t yet have a public API to our collections the data around them is up for grabs. Hopefully I'll have more to report on this before too long.


In the next post I'll talk through the website itself. Stay awake at the back!

Tuesday, October 25, 2011

On [that] day, 4th October 2011... pt 1

Tuesday October 4th was a big one for the Imperial War Museum. We in the New Media department had been working towards it for well over a year, but several other parts of IWM also had first-night nerves as the presented something shiny and new to the public. All in all we launched a new brand, the beta of the all-new IWM website, a re-skinned image licensing site, overhauled e-shop, redesigned print sales website, and phase 1 of a new system for delivering collections information to our public digital interfaces. We also took the covers off a new licence – the IWM User licence – which applies to lots of data and content and a large number of the images, audio and film (yes indeed) on the new site. The common theme, you’ll note, is that the web is part of all of these, so it’s fair to say we were probably the nerviest of all departments!
The timing was no accident, of course: Carolyn, our head of department, was not far into her plotting before it became apparent that we’d be launching our new website along with the new brand, and that we’d be tying in wherever possible with a 3-year e-commerce programme. The brand re-launch in turn was timed to fit with the opening of a major new exhibition coming to IWM London, "Shaped by War", and the e-commerce programme entailed some important implications for licensing too, so the dependencies were complex.
The brand
I can’t report much about the brand except to say that timing was critical, and that what we were most interested in was whether or not the brand structure would have any implications for information architecture and site navigation. We needed to know how we’d be expected to deal with what could be considered informally to be sub-brands (whether that be our branches, e-commerce sites, exhibitions, partnerships etc). To remain uncontroversial I will limit myself to saying that we were not given a recipe-book for how to apply the brand in a digital environment and that evolving its digital expression has been quite a challenging process. Personally I quite like the wedge device, although as lapsed geologist it reminds me of a horst. Or perhaps a graben, I always mixed them up. I think it’s fair to say that reactions have been polarised but the commenters on branding and design websites, at least, appear to dig it.
E-commerce sites
Over the last couple of years IWM has been using the excellent services of consultant Alice Grant, who has led a comprehensive e-commerce programme with literally dozens of projects within it. In particular she has reviewed the way we conduct business around our collections assets, with one vital development being the establishment of a commercial unit. She’s also been co-ordinating a number of activities to try to ensure that when we relaunched the website our B2B and B2C e-commerce offers made sense and worked well.
Over on the site at http://www.iwmcollections.org.uk/ that you can still see for a couple more weeks, we have split out the image sales and licensing functionality and re-skinned that, and put it onto isl.iwmcollections.org.uk. The ISL aspect of the old site will persist for a while yet because its functionality will remain part of our business processes for the foreseeable future, whereas the straightforward collections search part will shortly be retired, having been replaced by the corresponding part of our new site (of which more later). The ISL site again sports the new brand, was designed by Christian and implemented by Andrew Stephens in our ICT department.
I should mention the film sales site, which has been bubbling under for the last year but not really been promoted. We’ve now got a strengthened infrastructure and rely on this to deliver media through the core sites too, so I think it’s worth shouting a bit about film.iwmcollections.org.uk/ too. This B2B-facing site is again a Statham reskinning of the basic site built by CIS, whose DAMS we use (Imagen). Looks like we need to get the new logo in there though, Christian...
Our consumer-facing print sales site run by Cabinet UK at http://www.iwmprints.org.uk/ also had a complete facelift and its range multiplied about 10-fold, I believe. Not only that, it links back to our main site’s collection records pages. I reckon it looks pretty slick.
Finally, earlier this year we launched a heavily re-skinned e-shop in the knowledge that we might be obliged to revisit the whole process when the branding process was complete or when Cybertill launched their long-anticipated new software. The old site was haggard but, working within the constraints of the existing templating system, we were limited in what we could do. Nevertheless, with some imaginative design courtesy of our very own Christian Statham and some creative coding courtesy of Garry Taylor at the Bouncing Ball we ended up with a really good solution. Actually, that doesn’t do it justice at all. Under the skin, you’ll find quite a lot remains of the minging HTML that the system churns out and which cannot be changed at present, but which Garry’s CSS and jQuery skills have made look 10 years newer than their real age. For the beta website launch we made only small changes, mainly to tie in with the new brand.
All of these sites have another new dimension which is not necessarily visible on the sites themselves: they are integrated with the new website. We have built an index for our e-shop catalogue and integrated it into the site, returning results in the global search and enabling us to build carousels or products for promotion and use them anywhere; and images or films for which you can buy a licence or hard copy offer you this opportunity when you look at them in our collections pages.
Hosting
It was time to revisit how we host, to give us more control and at the same time slash costs. From an arrangement where our main sites were hosted on 3 dedicated servers at Rackspace we have moved to a hybrid solution with a beefy dedicated server for databases, Solr and some media, and a variable number of cloud servers to run web applications, including our core Drupal installation. We’ve also virtualised the old CMS to run a few legacy bits of that site. Whilst previously our hosted sites were isolated, we are now in a position to integrate them securely with services running on our own network, for instance with an oEmbed service that we use to tie the DAMS in with Drupal, or for replicating data or connecting to mail servers (things that could be done other ways but which are easier and more secure with the VPN we have in place). Parts of our web offer are delivered directly from our DMZ – our streaming media, the ISL site and our blogs – but again we’re now well placed to change this, if we wish, without huge investment, as the cloud part of our solution makes rapid expansion or scaling straightforward.
Having come from a background of essentially Windows/IIS/ASP+ ASP.Net/SQL Server and made the decision that we should go with the full LAMP stack, this was one of several steep learning curves I’ve been on (am still on). I’ve never been a server admin although I’ve picked up a certain amount of knowledge over the last 12 years, but it was pretty much like starting from scratch when I was faced with a bare-bones RHEL server and a Putty window. Fortunately I’ve got several experienced colleagues to put me on the right path or do various thing on my behalf. Together with some help from Rackspace themselves we’ve got there. I still have a long way to go but now that we’ve launched the beta at least it’s looking a bit less intimidating.

In Part 2 I'll say a bit about collections, licensing changes, and the main website itself. Then if I get that far I might do a post to say how things have been going in the first couple of weeks.

Friday, October 21, 2011

SOAPhar, so good

All of a sudden video and sound on our new beta site, streamed from our DAMS, stopped working this week. We embed these using a (currently) private oEmbed service which gets its data from a SOAP service on the DAMS to construct the streaming URL for a given media item, with which it then assembles the embedding code. The SOAP call was timing out and giving the following error:

Fatal error: Uncaught SoapFault exception:

[WSDL] SOAP-ERROR: Parsing Schema: can't import schema from 'http://schemas.xmlsoap.org/soap/encoding/' in C:\[myfile]:20
Stack trace: #0 C:\[myfile](20): SoapClient->SoapClient('http://[DAMS WSDL URL]', Array)
...

That schema file is being imported at a namespace declaration in the DAMS SOAP service's WSDL file. It could be accessed just fine with a browser but the PHP soapclient call which loads the WSDL file had no such luck and would just time out.
I have no idea why a service that had been running fine for 8 months or so suddenly went freaky, but I did suspect another unheralded proxy change, since these tend to play havoc with our dev environment and at the moment the oEmbed service is running there. Other people have experienced problems with this encoding schema at xmlsoap.org, but since it's going to be present in all proper SOAP calls that doesn't really tell us whether it's indicative of a problem with the URL and xmlsoap.org, or of network issues. All the same, what worked for this guy worked for me. Well actually, whilst he simply removed the namespace import element, whereas, I saved the WSDL for the SOAP service that my oEmbed script called and edited it to point at a local copy of the encoding schema at xmlsoap.org. Happily my SOAP call seemed to work just as well with a local copy of the WSDL and everything started working again. Hooray! I had wondered whether I would be scuppered because I couldn't edit the original WSDL file, but since the WSDL doesn't actually do anything, it just lays out the instructions for a SOAP consumer to use, it was aboslutely fine to edit and run it locally.
Perhaps this will help others suffering the same grief.