The announcement yesterday that the Europeana Foundation had officially adopted the new Europeana Data Exchange Agreement may have passed you by, but it seems to me to be a vital moment in Europeana's history, or if you prefer, its future. This agreement is key to its sustainability – by which I mean, not that it is a guarantee of long-term viability, but that it couples the resourcing and the value proposition of the service more transparently than ever before. Let me expand on this.
For Europeana to be more than just a portal website it must be part of the data landscape, an ever-growing circle on that famous Linked Data diagram. I don’t think anyone concerned with the project could conceive of it as anything else nowadays. The “Distribute” strand of the 2011-15 Strategic Plan (PDF) pivots on the re-use of content away from the Europeana site, via the API or Linked Data channels. This is the Europeana that I am talking about sustaining, and this is what the EDEA makes possible. Without it, the offer could only be the ghost of that: a portal site. With it, it could become pervasive. It’s the difference between a drinks machine and mains water supply. Coca Cola like to slap on a label saying “Buxton” and sell water for 10,000 what it costs from the tap to those people that come to the drinks machine. Thames Water, by contrast, put it in pipes and get it everywhere – unbranded, plentiful, pervasive. The EDEA brings into focus a similar choice for content providers: do they want their water in the mains, or would they rather put it in bottles and try to find buyers on their own? Europeana has staked its future - I do mean that - on the mains supply approach.
The two most significant resources that Europeana relies upon are finance and content*. You might also add users, or you might prefer to count them on the “benefit” side of a cost/benefit equation. Crudely put, finance comes largely from the European Commission and content largely comes from cultural heritage content partners (either direct data providers or, more commonly, via aggregators). These two stakeholder groups, providing the grist to Europeana’s mill, are looking for different things, although their interests overlap greatly. To sustain a social enterprise, which is at heart concerned with generating public good rather than a net profit, requires whoever is providing the resources for it to perceive that the value thus generated is worth the cost. If this value proposition is not strong enough to persuade them to supply the necessary resources, well, it won’t be resourced and will fail.
Europeana has now laid out its value proposition to content providers, along with what it requires in return. Given the vision of the Strategic Plan's “Distribute” strand, this requirement could never really be less than the CC0 license the EDEA requires, but equally the “mains supply” model of cultural content is the foundation of the value proposition that the EDEA offers to museums, libraries and archives that sign the new agreement: you will find your content hosed across the internet, heliping you to deliver your public mandate but also carrying new traffic back to your site and opening up commercial opportunities. The word “exchange” in the title of the Agreement also highlights the other part of the value proposition: that the enrichments Europeana makes to the raw metadata it receives will also be CC0 licensed and will be given back to the providers. Note also that whilst the EDEA is not a contract with the funders, it again reflects the value proposition to them: culture on tap through openly licenced, reusable data.
So the EDEA is not just a new licensing agreement. If you like, it is the sustainability plan in super-brief form: give us your content and let us set it free, and here’s what you get in return - oh and funders, this is why we're worth the €€€. Since the Strategic Plan laid out a vision of open-licensed, distributed and open data, this was really the inescapable conclusion. If it finds sufficient favour, that vision of a mains supply of culture may have a long life, and that bubble on the Linked Data diagram may keep growing. If it doesn’t, well, it’s either back to the drinks machine or back to nothing. It's worth the gamble, I reckon.
* Oh, and political support. This is a big one, and is the source of powerful backing for the open data train to which Europeana has hitched its wagon. If Commissioner Neelie Kroes has anything to do with it this train is going to gather speed rapidly – see her speech to the OpenEurope Summit this week for starters.
About Me
- Jeremy
- Web person at the Imperial War Museum, just completed PhD about digital sustainability in museums (the original motivation for this blog was as my research diary). Posting occasionally, and usually museum tech stuff but prone to stray. I welcome comments if you want to take anything further. These are my opinions and should not be attributed to my employer or anyone else (unless they thought of them too). Twitter: @jottevanger
Showing posts with label value. Show all posts
Showing posts with label value. Show all posts
Friday, September 23, 2011
Saturday, August 21, 2010
No decisions without value/s
I was getting hot under the collar the other day listening to an interview with some religious type who was upset by the decision to prevent a Catholic charity from excluding same-sex couples from using their adoption service. The Charities Commission ruled that they could either change their policy, or stop acting as an adoption agency, and as far as they were concerned the former was not an option so the latter was the only possible outcome. Now, nothing is likely to bring me round to their antediluvian* point of view re parenthood and sexuality, but it did help calm me down to remember that, whilst the discussion may often hover around "facts" (because they're easier for everyone to get a grip on, even when flaky), it's actually not about something that has a right or wrong answer: it's about a decision to support one side or another of the argument, and decisions are fundamentally dependent upon values: these are the only thing that give the "facts" any weight for the decision maker.
"Facts", in principle, are established by gathering objective evidence. Decisions, on the other hand, are the reverse: they always involve value. Value, whilst it can be a purely emotional response (a desire for something, or a positive feeling towards it), can also be transformed into values (not the same thing), which in a sense provide a template for valuing a given phenomenon. In other words, value is the outcome of an assessment, whilst values are a tool that for making that assessment.
Further, values (the tool/template) may be the internal values of an individual, perhaps embedded sub-consciously in a decision or even somatic ("instinctual", if you like, though that's a debate to avoid); or they may the externalised values embodied as a set of rules guiding a decision. This ostensibly makes the decision objective, but only by removing the value judgement into an earlier stage of the decision-making: the framing stage, where the rules for the decision are composed. These rules might even be ossified into an automated process, where rules are coded as machine instructions. They may also have been distorted along the way, so that in fact the rules represent no values held by a person at all. Whichever is the case, the values of a person or a group of people have been injected at some stage in order to make a decision possible.
The push and pull of emotion are a tide that can move our internal values incessantly, and this is one reason why it is often helpful to turn values into a set of explicit rules. Another is to externalise them so that they can be discussed and referred to as law, policy, doctrine or maybe something more informal. Of course, there's a psychological cost if this then results in dissonance between the codified values and the internalised values.
So in making decisions, or trying to influence the decisions made by other people or systems, or merely trying to understand the perspective of someone who argues in a direction that you just don't get, should we focus on the facts or on the values? Should we try to gather more evidence to demonstrate that the facts are as we see them? Should we examine the values held by another party and attempt to present the facts consonantly with them? Should we even try to change the values that are so critical in turning facts into value into decisions? Well, that's a tough one. But the one thing you can't do is address value directly: value is the product of the perceived "facts" of a scenario and the process or template for assessing the value of those facts. You can address the facts, and you can address the process of assessing but it's pointless to simply state that the value is something other than the assessment itself.
Ultimately, values themselves can be shaken by enough well-aimed evidence, but these may be of a quite different category to those facts that are put together with established values in order to reach a decision. For example, the evidence or experience required to make me a Catholic and subscribe to the values of the person I heard interviewed I cannot even imagine, even though ultimately it must be possible (perhaps on the road to Damascus). On the other hand, there are some facts through which I might be persuaded that it was wrong to prevent that particular charity from handling adoptions as it chose - for example, if it were the only agency in the country and the result of this decision was that there would be no adoption agencies at all, I might (might) feel, for purely pragmatic reasons, that it would be better to let them carry on even if that meant living with their discriminatory policy. Thankfully that's not the situation!
I don't know if this framework I've outlined is useful or right. I'm not a psychologist, a philosopher or a logician. Perhaps there are people out there who have very different ways of making decisions and this model of splitting out values from assessments of value doesn't work for them. I myself can see complications in examining the adoption agency question, which is nothing if not a clash of values. For my research into the role of decision making in digital sustainability, though, I think it may be a useful distinction to make, extending the insights given by Lessig's modalities of regulation (PDF) (this is typically looked at as a legal theorem but it's really about the "regulation" of decisions).
*yes, I know
"Facts", in principle, are established by gathering objective evidence. Decisions, on the other hand, are the reverse: they always involve value. Value, whilst it can be a purely emotional response (a desire for something, or a positive feeling towards it), can also be transformed into values (not the same thing), which in a sense provide a template for valuing a given phenomenon. In other words, value is the outcome of an assessment, whilst values are a tool that for making that assessment.
Further, values (the tool/template) may be the internal values of an individual, perhaps embedded sub-consciously in a decision or even somatic ("instinctual", if you like, though that's a debate to avoid); or they may the externalised values embodied as a set of rules guiding a decision. This ostensibly makes the decision objective, but only by removing the value judgement into an earlier stage of the decision-making: the framing stage, where the rules for the decision are composed. These rules might even be ossified into an automated process, where rules are coded as machine instructions. They may also have been distorted along the way, so that in fact the rules represent no values held by a person at all. Whichever is the case, the values of a person or a group of people have been injected at some stage in order to make a decision possible.
The push and pull of emotion are a tide that can move our internal values incessantly, and this is one reason why it is often helpful to turn values into a set of explicit rules. Another is to externalise them so that they can be discussed and referred to as law, policy, doctrine or maybe something more informal. Of course, there's a psychological cost if this then results in dissonance between the codified values and the internalised values.
So in making decisions, or trying to influence the decisions made by other people or systems, or merely trying to understand the perspective of someone who argues in a direction that you just don't get, should we focus on the facts or on the values? Should we try to gather more evidence to demonstrate that the facts are as we see them? Should we examine the values held by another party and attempt to present the facts consonantly with them? Should we even try to change the values that are so critical in turning facts into value into decisions? Well, that's a tough one. But the one thing you can't do is address value directly: value is the product of the perceived "facts" of a scenario and the process or template for assessing the value of those facts. You can address the facts, and you can address the process of assessing but it's pointless to simply state that the value is something other than the assessment itself.
Ultimately, values themselves can be shaken by enough well-aimed evidence, but these may be of a quite different category to those facts that are put together with established values in order to reach a decision. For example, the evidence or experience required to make me a Catholic and subscribe to the values of the person I heard interviewed I cannot even imagine, even though ultimately it must be possible (perhaps on the road to Damascus). On the other hand, there are some facts through which I might be persuaded that it was wrong to prevent that particular charity from handling adoptions as it chose - for example, if it were the only agency in the country and the result of this decision was that there would be no adoption agencies at all, I might (might) feel, for purely pragmatic reasons, that it would be better to let them carry on even if that meant living with their discriminatory policy. Thankfully that's not the situation!
I don't know if this framework I've outlined is useful or right. I'm not a psychologist, a philosopher or a logician. Perhaps there are people out there who have very different ways of making decisions and this model of splitting out values from assessments of value doesn't work for them. I myself can see complications in examining the adoption agency question, which is nothing if not a clash of values. For my research into the role of decision making in digital sustainability, though, I think it may be a useful distinction to make, extending the insights given by Lessig's modalities of regulation (PDF) (this is typically looked at as a legal theorem but it's really about the "regulation" of decisions).
*yes, I know
Tuesday, July 21, 2009
Ithaka report ith a catastrophe (for me)
I am tho thcrewed.
Ithaka S+R has produced a report for JISC, the SCA, NEH and NSF entitled "Sustaining digital resources: an on-the-ground view of projects today". It's a great report, outlining a sensible approach to what sustainability actually means, strategies to achieve it, and how a number of current projects put these strategies into practice. It's a follow-up to "Sustainability and revenue models for online academic resources" (2008).
The problem, for me, is that much of the originality that I hoped my PhD work would offer has vanished overnight. Their definition of sustainability; their explicit linking of financial sustainability to the value offer; their arguments for leadership that offers clarity of purpose and evidence of success; their clear-eyed distinctions between the financial versus mission-based value on which non-profits must base their measures of success and arguments for their future: all of these could have been plucked from my private writings and debates with my supervisor over the last 3 years. Yet of course they haven't been; they're the result of parallel thinking, but these guys have gathered the evidence that I'm still at the early stages of assembling.
It's not that my thoughts were especially profound, but I've never previously found any authors linking sustainability and the value proposition so clearly to the digital work of cultural heritage institutions. Now that niche is no longer empty. I'm really pleased to see that there are others of like mind out there, and of course a report like this gives me a great reference in support of my work, but the problem for me is not just one of dented pride, because I have to demonstrate considerable originality in my work and without that, no PhD. It's a pickle.
On the upside, there are some differences. The Ithaka report emphasises how important it is to be clear on the sources of value (and cost) in order to make the case to funders for continued support. I will too, but I hope to investigate more deeply the influence of decision-making processes and the sources of "friction" that may cause a discrepancy between what a resource is "worth" and what people are prepared to invest in it (complicated, of course, by questions of opportunity cost and uncertainty). Ithaka mainly looked at digitisation projects, in the sense of those offering sets of digitised material such as images, maps, papers or databases. I'm just as interested in the value of learning objects, games, mobile phone tours, exhibition websites (though admittedly my core case studies are less diverse than this), and only in a cultural heritage context, not education. I'm interested in certain "modalities of constraint" (c.f. Lessig) that aren't addressed by Ithaka, and in questions of risk management. I am looking at three varied partnerships and the effects that collaboration has on decision-making, value and funding, and partnership is an area that the Ithaka report, by its own admission, examines only briefly. So perhaps all is not lost, but whilst I genuinely think that this report is excellent and provides a lot for people in the digital humanities to chew over, it's given me quite a tricky challenge. I'm not about to give up my PhD, but with everything else that's happened over the last year I have to confess it's taken me pretty bloody close.
P.S. Ah bollocks, now Nick's at it too (para 6). Shows how un-original I was in the first place.
*Nancy L Maron, K Kirby Smith and Matthew Loy, 2009. "Sustaining digital resources: an on-the-ground view of projects today. Ithaka case studies in sustainability" http://www.ithaka.org/ithaka-s-r/strategy/ithaka-case-studies-in-sustainability
Ithaka S+R has produced a report for JISC, the SCA, NEH and NSF entitled "Sustaining digital resources: an on-the-ground view of projects today". It's a great report, outlining a sensible approach to what sustainability actually means, strategies to achieve it, and how a number of current projects put these strategies into practice. It's a follow-up to "Sustainability and revenue models for online academic resources" (2008).
The problem, for me, is that much of the originality that I hoped my PhD work would offer has vanished overnight. Their definition of sustainability; their explicit linking of financial sustainability to the value offer; their arguments for leadership that offers clarity of purpose and evidence of success; their clear-eyed distinctions between the financial versus mission-based value on which non-profits must base their measures of success and arguments for their future: all of these could have been plucked from my private writings and debates with my supervisor over the last 3 years. Yet of course they haven't been; they're the result of parallel thinking, but these guys have gathered the evidence that I'm still at the early stages of assembling.
It's not that my thoughts were especially profound, but I've never previously found any authors linking sustainability and the value proposition so clearly to the digital work of cultural heritage institutions. Now that niche is no longer empty. I'm really pleased to see that there are others of like mind out there, and of course a report like this gives me a great reference in support of my work, but the problem for me is not just one of dented pride, because I have to demonstrate considerable originality in my work and without that, no PhD. It's a pickle.
On the upside, there are some differences. The Ithaka report emphasises how important it is to be clear on the sources of value (and cost) in order to make the case to funders for continued support. I will too, but I hope to investigate more deeply the influence of decision-making processes and the sources of "friction" that may cause a discrepancy between what a resource is "worth" and what people are prepared to invest in it (complicated, of course, by questions of opportunity cost and uncertainty). Ithaka mainly looked at digitisation projects, in the sense of those offering sets of digitised material such as images, maps, papers or databases. I'm just as interested in the value of learning objects, games, mobile phone tours, exhibition websites (though admittedly my core case studies are less diverse than this), and only in a cultural heritage context, not education. I'm interested in certain "modalities of constraint" (c.f. Lessig) that aren't addressed by Ithaka, and in questions of risk management. I am looking at three varied partnerships and the effects that collaboration has on decision-making, value and funding, and partnership is an area that the Ithaka report, by its own admission, examines only briefly. So perhaps all is not lost, but whilst I genuinely think that this report is excellent and provides a lot for people in the digital humanities to chew over, it's given me quite a tricky challenge. I'm not about to give up my PhD, but with everything else that's happened over the last year I have to confess it's taken me pretty bloody close.
P.S. Ah bollocks, now Nick's at it too (para 6). Shows how un-original I was in the first place.
*Nancy L Maron, K Kirby Smith and Matthew Loy, 2009. "Sustaining digital resources: an on-the-ground view of projects today. Ithaka case studies in sustainability" http://www.ithaka.org/ithaka-s-r/strategy/ithaka-case-studies-in-sustainability
Monday, May 04, 2009
A dawning realisation?
[third in a recent series of observations and unfinished semi-coherent thoughts I just need to get out of the way]
Nowadays everyone I talk to questions the metrics they use. More than that, people seem keener to dig into what they may mean in terms of value. Seb Chan is amongst those in our sector that's exploring how to make better, and better use of, measurements, and closer to home, Dylan Edgar's work with the London Hub dug into similar issues.
Last week in a catch-up with the director of my division we touched on his own objective of "improving the website". In itself it's encouraging that the objective is there, as part of the reorganisation we are currently experiencing, but "improving the website" is a pretty broad ambition. I think it's a subject that we'll revisit in more depth soon, but it was clear that our director was as aware as we web types were that when you lift up that rock you'll find a tangled mess of questions. Before you talk about "improving" you need to identify what you consider to be valuable, and to disentangle theoretical "improvements" from impact, preparedness, experimentation etc. Obviously a set of measurements that to some degree reflect these valued qualities are a sine qua non for managing their realisation, and so here's a reference to provoke a little more thought on the subject that I won't dig into here, but has had me rethinking my own attitudes web stats and the whole evaluation problem: Douglas W Hubbard, 2007, How to measure anything : finding the value of "intangibles" in business. *
In any case I find it encouraging that in this discussion and others with senior colleagues there seems to be a dawning awareness that we have a complex, multidimensional environment to deal with, wherein the varieties of "success" may be as varied as between all the departments within a museum. I'm not sure that it would always have been the case that the higher echelons were aware of the perils of trying to evaluate our digital programmes, although perhaps any senior manager worth their salt will have long ago twigged that a website is not "improved" merely by adding pages, Flash splashes and video - evaluating the more familiar physical museum is no easier, after all, and nor is improving it. We do need to have that conversation about what we mean by "website" with senior management, though. Is it only geeks that see this as only a part of our digital presence?
When it comes to the use of web stats of various sorts, there have always been lots of complaints about them, but I suspect that in this discussion too we are seeing greater recognition that it's not about visitors versus hits. Maybe it's not even enough to focus on "impact" since the heart of the matter arguably lies a level deeper than that: the first step is figuring out what impact itself means in the context of the museum's mission, and in this networked environment in the mission of the meta-museum that we must realise we are a part of.
Rhetorical question for the day, then: Is there a mission for the meta-museum, and do we measure up to it?
*I hope to post about this book properly, eventually, but don't wait for that, try to check out the book which, for all its flaws of repetition, is full of useful ideas and tools.
Nowadays everyone I talk to questions the metrics they use. More than that, people seem keener to dig into what they may mean in terms of value. Seb Chan is amongst those in our sector that's exploring how to make better, and better use of, measurements, and closer to home, Dylan Edgar's work with the London Hub dug into similar issues.
Last week in a catch-up with the director of my division we touched on his own objective of "improving the website". In itself it's encouraging that the objective is there, as part of the reorganisation we are currently experiencing, but "improving the website" is a pretty broad ambition. I think it's a subject that we'll revisit in more depth soon, but it was clear that our director was as aware as we web types were that when you lift up that rock you'll find a tangled mess of questions. Before you talk about "improving" you need to identify what you consider to be valuable, and to disentangle theoretical "improvements" from impact, preparedness, experimentation etc. Obviously a set of measurements that to some degree reflect these valued qualities are a sine qua non for managing their realisation, and so here's a reference to provoke a little more thought on the subject that I won't dig into here, but has had me rethinking my own attitudes web stats and the whole evaluation problem: Douglas W Hubbard, 2007, How to measure anything : finding the value of "intangibles" in business. *
In any case I find it encouraging that in this discussion and others with senior colleagues there seems to be a dawning awareness that we have a complex, multidimensional environment to deal with, wherein the varieties of "success" may be as varied as between all the departments within a museum. I'm not sure that it would always have been the case that the higher echelons were aware of the perils of trying to evaluate our digital programmes, although perhaps any senior manager worth their salt will have long ago twigged that a website is not "improved" merely by adding pages, Flash splashes and video - evaluating the more familiar physical museum is no easier, after all, and nor is improving it. We do need to have that conversation about what we mean by "website" with senior management, though. Is it only geeks that see this as only a part of our digital presence?
When it comes to the use of web stats of various sorts, there have always been lots of complaints about them, but I suspect that in this discussion too we are seeing greater recognition that it's not about visitors versus hits. Maybe it's not even enough to focus on "impact" since the heart of the matter arguably lies a level deeper than that: the first step is figuring out what impact itself means in the context of the museum's mission, and in this networked environment in the mission of the meta-museum that we must realise we are a part of.
Rhetorical question for the day, then: Is there a mission for the meta-museum, and do we measure up to it?
*I hope to post about this book properly, eventually, but don't wait for that, try to check out the book which, for all its flaws of repetition, is full of useful ideas and tools.
Saturday, November 29, 2008
NEDCC conference
"Persistence of Memory:Sustaining Digital Collections" is a conference being run shortly in Chicago by the NEDCC (December 9-10), and I should have stuck up a note about it yonks ago but it's mainly strictly "digital preservation" stuff as opposed to sustainability in the way I treat it. However there are a couple of papers that look closer to my research interests, notably Simon Tanner's (of KCL), entitled "Making Digital Preservation Affordable: Values and Business Models": the emphasis on valuation is key. Katherine Skinner's "Collaborative Adventures in Digital Preservation: Creating and Sustaining External Partnerships" may be relevant too, although it looks as though it's in essence about developing networks of partners for preservation activity. Having spent the first part of this week hanging out with various very interesting people from KCL, it's clear the place is stuffed with people I should be pestering for insights in my research area. If I whisper, perhaps they won't hear me coming....
Thursday, November 27, 2008
Building communities
<<<<<BIG WARNING>>>>>
Major pontification with no real conclusions follows. Look away now.
[UPDATE: I forgot to include the Big Idea I mention below, which I've now written about separately]
I got back late [last] Wednesday from a meeting/seminar/workshop/conference thing (papers now here) in at the e-Science Institute in Edinburgh which I'm still digesting [a week after I started writing this...] but which gave me plenty to think about and introduced me to many interesting people from the digital arts & humanities world (DAH*). Weird though it seems when written down, this world intersects all too rarely with that of museums, libraries and archives(MLAs). That said, one aim of the organisers seemed to be to develop ideas for filling the void left by the Arts and Humanities Data Service, which was wound up this year and did have some relationships with MLAs, not least my own via the LAARC.
Anyway to the point. The object of the exercise was to explore community-building in DAH. What conditions favour this, what sorts of communities may fare best, what structures within the sector can or would help? I must confess that I frequently felt confused about whether we were speaking about the AHDS-related problem or a wider question, and whether the communities in question were partnerships and networks, or audiences for some product/service that the former might provide. Nevertheless there were several interesting presentations on day 1 (Tuesday), and I think fruitful discussion on day 2, when we worked in breakout groups to brainstorm a few questions around the topic., with at least one really stimulating idea emerging, courtesy of an antipodean mind (now why isn't that surprising?) I'll pick out a few of the things that grabbed me.
Themes
- demonstrating value. Value to funders, if you're going to get their support for your network/partnership/wha'evah. Value to those you want as partners. Value to the people you want to use whatever it is you're promoting. It wasn't the idea that was interesting but the increasing recognition of the need not just to have a good idea, but to sell it. There's an ever-widening discussion of what makes a good indicator of value in various contexts - I know this from the museum and cultural world, but of course the humanities/academic world is grappling with the same hydra. Perhaps for my circle it's a debate over the appropriate usage of web stats, whilst for many at the meeting it was about citations. Same problem.
- marketing. Plenty of cross-over with the above, in fact. As well as the obvious sales pitch aspect, if you're trying to build a community, there's the flipside of marketing: learning your market's needs in order to create the product that it wants.
- plumbing. There was a lot of talk about infrastructure, which means a great variety of things: in a way what is the facilitated at one level becomes the facilitator at another (i.e. infrastructure), much as data and metadata can seem to be the same thing seen from different angles. There was a kind of infrastructure that cropped up more than once, though I'm hard pushed to characterise it other than by negatives. It's not generally physical: no cabling, not necessarily servers. It's not the provision of some fundamental service. It's where an agent/network/partnership helps to hook up or facilitate services and, umm, servicees (sorry). A kind of metaservice, if you like. So we had Bamboo, CLARIN, DARIAH and tge-ADONIS, TextGrid all considering or planning to act as intermediaries of one sort or another between DAH services or data and consumers (machine or human). I was struck by the parallels with some of the work I already knew about in the cultural heritage area. Collections Trust has exciting plans about where it will position its Culture Online venture, and Lexara [read my disclaimer] have realigned their Magic Studio to make it the "plumbing" between content providers (including services like Flickr) and other services or end-users/creators. In the commercial world, Gnip is also putting itself in this space, between social apps and those built upon them.
- Roles and responsibilities. Really I suppose I just mean that a need for new roles is becoming apparent without us yet knowing who in our communities (or outside, perhaps) should be responsible for fulfilling them. We look at the rapid changes in the red-in-tooth-and-claw commercial sector, where networks of networks and a swirling and intermixing of content and services constantly throws up new ways of doing things, and we start to see the opportunities for new services to accomplish our parochial aims, recognise that we can't provide them, or that they're better provided collectively, or that our peers would benefit from them too; but we can be held back from turning this collective need/benefit into collective action in part because we haven't got conventions for assigning/allocating/assuming the new roles implied by these possibilities.
- Building on what we have. Starting a new community initiative from scratch, including the community, may be unwise for a number of reasons. Often it may be better to look at existing organisations and see how their role could be adapted or expanded, whether by the creation of a special interest group, a regional su-committee, a project or whatever. This way you start with a community and you have an established brand to build from, with both the name and the objectives/values pre-existing to an extent in the minds of (potential) stakeholders. I had in mind the likes of the MCG, MCN, CNI: organisations that could provide the foundations for specialist sub-groups or initiatives.
As you can tell, I guess, a week later I still haven't reached any conclusions. I was hoping that reading my notes and writing this would bring some revelations or clarity, but no. The Edinburgh meeting was more like the exercise stirred up a lot of mud that I'd always known was there and I still don't know how to deal with it, but I think at least I now have a slightly better idea of what it's made. I guess, with my scant experience of big collaborative projects, I am poorly equipped to interpret what I heard from the point of view of the question at hand: community building. All the same I did hear about lots of cool stuff and met fascinating people, and attending was well worthwhile for me - I hope I gave something back. So thank you, Seth Denbo (source of my invitation) and all of the organisers of the seminar. I hope it gave you some clues as to where to take DARIAH, and I hope we can find new ways to knit the digital heritage and digital arts and humanities closer together.
* Apologies. I tried to write this avoiding initials for this but it proved too much work. I doubt it will catch on.
Thursday, November 20, 2008
Securing endowment
No, not endowments. Decision-making literature* addresses the "endowment effect". In a nutshell, I mean that phenomenon whereby people value what they have more than what they might have, although they be the same. This is tied to the observation that loss of utility is felt more keenly than a gain of the same utility. If I am given a mug (to follow Beach and Connolly's example) I may well require a larger payment to give it up that I would be prepared to pay for it.
This is probably a useful tool to wield when trying to get support for some resourcing decisions in an organisation, particularly those that can be expressed as relating to assets the organisation already possesses, rather than new ones it might acquire. In a scenario where I am seeking funds to support old digital resources, careful preparation of the ground to develop an endowment effect amongst the executive before taking the proposal to the board may be fruitful (this is obvious to the point of being banal, right?) If I just tell then what we have, what it could do in the future, and how we plan to maintain it, it's like saying "here's a mug that will hold your coffee and make you happy, it will cost you a fiver, gimme!" If instead I encourage a sense of pre-existing ownership - an emotional committment - a feeling that this thing is already part of the museum and its loss reduces us, then the costs I propose that we bear are more like the purchase offer for the mug that is turned down: in other words the board may be more likely to turn down the opportunity to avoid those costs. It's like saying "you've got a mug, you love it, if you want to keep the cash then I'm afraid you'll lose the mug. Bummer, but that's life. So cough up!"
How you actually cultivate within the executive that sense of ownership is another question. You could also argue that this is little different to just identifying and drawing out the value of an asset to the decision-makers, but it's a more emotional proposition than simply making them aware of an assets existence and potential future utility, and it may be worth exploring how one can play the psychological game and encourage emotional engagement and attachment to pre-existing digital resources when fighting for the funds to suppport them. It may actually be as important as the rational explication of value, however much we might like to believe decisions would be rational.
Nina Simon amongst others has talked extensively and profoundly on Museum 2.0 about developing engagement amongst our audiences, and I'm starting to realise how this could be equally important within the organisation: if we want to persuade people of the rightness of what we're arguing for, there can be more to it than pure reason, Kant there? (apologies for the shameless mis-referencing here of something I clearly don't understand. I only understand puns.) It's internal marketing, I suppose, and nothing new.
* OK, let's be honest, all I know about is the book I'm reading right now and blogged about previously
This is probably a useful tool to wield when trying to get support for some resourcing decisions in an organisation, particularly those that can be expressed as relating to assets the organisation already possesses, rather than new ones it might acquire. In a scenario where I am seeking funds to support old digital resources, careful preparation of the ground to develop an endowment effect amongst the executive before taking the proposal to the board may be fruitful (this is obvious to the point of being banal, right?) If I just tell then what we have, what it could do in the future, and how we plan to maintain it, it's like saying "here's a mug that will hold your coffee and make you happy, it will cost you a fiver, gimme!" If instead I encourage a sense of pre-existing ownership - an emotional committment - a feeling that this thing is already part of the museum and its loss reduces us, then the costs I propose that we bear are more like the purchase offer for the mug that is turned down: in other words the board may be more likely to turn down the opportunity to avoid those costs. It's like saying "you've got a mug, you love it, if you want to keep the cash then I'm afraid you'll lose the mug. Bummer, but that's life. So cough up!"
How you actually cultivate within the executive that sense of ownership is another question. You could also argue that this is little different to just identifying and drawing out the value of an asset to the decision-makers, but it's a more emotional proposition than simply making them aware of an assets existence and potential future utility, and it may be worth exploring how one can play the psychological game and encourage emotional engagement and attachment to pre-existing digital resources when fighting for the funds to suppport them. It may actually be as important as the rational explication of value, however much we might like to believe decisions would be rational.
Nina Simon amongst others has talked extensively and profoundly on Museum 2.0 about developing engagement amongst our audiences, and I'm starting to realise how this could be equally important within the organisation: if we want to persuade people of the rightness of what we're arguing for, there can be more to it than pure reason, Kant there? (apologies for the shameless mis-referencing here of something I clearly don't understand. I only understand puns.) It's internal marketing, I suppose, and nothing new.
* OK, let's be honest, all I know about is the book I'm reading right now and blogged about previously
Tuesday, July 15, 2008
Why Gnip caught my eye: a bit more depth (just a bit)
Eric Marcoullier commented on my last post on Gnip and I wrote him the following e-mail because, as I say, it's about time I worked through a little bit the reason why his baby caught my attention (not that it's a particularly worked through working through, but hey, it's a start). It was a bit much for a comment but enough for a post, so here you go.
*******************************************************************
Many thanks for taking the time to look at my brief notes, you must be a busy man so I really appreciate it. It's definitely time I tried to put some flesh on the bones because it's true, I've barely sketched the link between Gnip and my own preoccupations.
My research is looking at how museums keep their digital stuff useful; in other words, how and when we keep on trying to squeeze value out of the digital stuff we've invested in. I'm trying to put a particularly museum-y spin on it because it would be all too easy to look, for example, at general questions related to digital preservation (yawn). Hence I'm exploring the specific conditions and challenges that museums have to face, as well as the way in which they value what they hold - as a "memory institution" with a remit to preserve and to serve the public, a museum has potentially got a slightly different way of valuing what it holds, though arguably this won't really apply to digital material except in special cases (like digital art). So that's the basic thread of my research: looking at how museums can and do decide a strategy for maximising value from their digital assets, and for planning new ones.
Of course, no museum is an island (that's kind of the point of the 'net, right?) and I'm inevitably thinking a lot about the relationships between museums and other parties that might provide or use services and data to/from them - this is key to extracting value, but it's also a dependency for which we need to understand the risks. In the museum community, a lot of the talk (for a couple of decades or more, now) is about how we share our most obvious USP: our collections data. Loads of work has been done on this and yet we still seem to be a long way from the dream of a way of effectively integrating the collections of more than a few institutions. This is why I've been working with the European Digital Library/Europeana project. The reason that Gnip caught my eye was because it suggests another model for data interchange. It may be not be appropriate for the scenario of sharing collections data, and one could argue that in some ways other museum initiatives share some of its characteristics (federated search, metadata harvesters etc.), but I was interested in whether we could learn from the model of a neutral mediating agent as rather than a central pool of data. We're not short of standards but we are short of co-ordinating mechanisms that we can all trust and feel we leave us with some control over "our" data.
The actual purpose of Gnip as an exchange for social data was probably of secondary interest to me, but of interest all the same - it's just an area I don't know much about. I think that on the whole museums won't need to concerns themselves directly about how whatever it is they do will relate to Gnip: I presume that if they incorporate a third party service in their site, or perhaps have an installation of WordPress, then a lot of the mechanics may be dealt with already (or will be in due course). But concern about interoperability and data portability may well be a reason why many museums (my own included) haven't yet done an awful lot with social software - although there are some notable exceptions. If Gnip helps to address these concerns then all that will still be lacking is our imagination!
One other possibility is that museum applications could indeed work with Gnip to integrate individuals' public information with their own services - say, by drawing links between a person's list of interests or music preferences, and what's in a museum's (or a library's)collection; or by suggesting events to attend based on user location, age and interests. I don't understand Gnip well enough to know if this is plausible, though, but it's an intriguing prospect.
*******************************************************************
Thanks again to Eric for taking the time to contact me, I think it speaks well of new ventures like this (OpenCalais was another) when the key people go out out of their way to make contact with the people that are talking about them.
*******************************************************************
Many thanks for taking the time to look at my brief notes, you must be a busy man so I really appreciate it. It's definitely time I tried to put some flesh on the bones because it's true, I've barely sketched the link between Gnip and my own preoccupations.
My research is looking at how museums keep their digital stuff useful; in other words, how and when we keep on trying to squeeze value out of the digital stuff we've invested in. I'm trying to put a particularly museum-y spin on it because it would be all too easy to look, for example, at general questions related to digital preservation (yawn). Hence I'm exploring the specific conditions and challenges that museums have to face, as well as the way in which they value what they hold - as a "memory institution" with a remit to preserve and to serve the public, a museum has potentially got a slightly different way of valuing what it holds, though arguably this won't really apply to digital material except in special cases (like digital art). So that's the basic thread of my research: looking at how museums can and do decide a strategy for maximising value from their digital assets, and for planning new ones.
Of course, no museum is an island (that's kind of the point of the 'net, right?) and I'm inevitably thinking a lot about the relationships between museums and other parties that might provide or use services and data to/from them - this is key to extracting value, but it's also a dependency for which we need to understand the risks. In the museum community, a lot of the talk (for a couple of decades or more, now) is about how we share our most obvious USP: our collections data. Loads of work has been done on this and yet we still seem to be a long way from the dream of a way of effectively integrating the collections of more than a few institutions. This is why I've been working with the European Digital Library/Europeana project. The reason that Gnip caught my eye was because it suggests another model for data interchange. It may be not be appropriate for the scenario of sharing collections data, and one could argue that in some ways other museum initiatives share some of its characteristics (federated search, metadata harvesters etc.), but I was interested in whether we could learn from the model of a neutral mediating agent as rather than a central pool of data. We're not short of standards but we are short of co-ordinating mechanisms that we can all trust and feel we leave us with some control over "our" data.
The actual purpose of Gnip as an exchange for social data was probably of secondary interest to me, but of interest all the same - it's just an area I don't know much about. I think that on the whole museums won't need to concerns themselves directly about how whatever it is they do will relate to Gnip: I presume that if they incorporate a third party service in their site, or perhaps have an installation of WordPress, then a lot of the mechanics may be dealt with already (or will be in due course). But concern about interoperability and data portability may well be a reason why many museums (my own included) haven't yet done an awful lot with social software - although there are some notable exceptions. If Gnip helps to address these concerns then all that will still be lacking is our imagination!
One other possibility is that museum applications could indeed work with Gnip to integrate individuals' public information with their own services - say, by drawing links between a person's list of interests or music preferences, and what's in a museum's (or a library's)collection; or by suggesting events to attend based on user location, age and interests. I don't understand Gnip well enough to know if this is plausible, though, but it's an intriguing prospect.
*******************************************************************
Thanks again to Eric for taking the time to contact me, I think it speaks well of new ventures like this (OpenCalais was another) when the key people go out out of their way to make contact with the people that are talking about them.
Thursday, July 10, 2008
Delayed post #1: aspects of [web] preservation
[[this post never got finished but I'm having a clear-out of my drafts and they're getting published or deleted, ready or not]]
Brian Kelly just blogged [[hmm, well, back in July, I think....]] on the JISC-PoWR site about "three key aspects of web preservation": experience, information, and access. I have a fourth, but I've been using it in the context of "sustainability" (the subject of my thesis), and so first I want to say a couple of words about this vs preservation, since although I've been writing papers for Ross for a couple of years arguing about the distinction between the concepts, I've not really rehearsed this in public before.
For the last couple of years I've been arguing that the problem of sustainability (S) is distinct from that of archive-style preservation (P). I won't go into the details of the distinction here but in essence I see P as concerning the persistence of a state, and S as the persistence of a process or activity. Recent work by Chris Rusbridge and others has been blurring the boundary ever more, although in a useful way: by questioning the purpose of preservation and weighing up what are the important aspects ("significant properties") of resources, they have been starting to argue for an approach to preservation that looks a lot more like what I was previously describing as sustaining. I still sort of believe that it is useful to distinnguish between the two concepts, but there's a lot of overlap.
The aspect that I think is especially pertinent to sustaining is "purpose". I don't think this is the same as the "experience" that Brian cites Kevin Ashley on (although experience and information may feed into purpose). It's focused on the objectives of the resource, which may be attainable through radically different experiences; for example (in the case of the environment in which KA operates), the learning objectives that an educational resource was created to serve. For a museum, perhaps a resource was prepared for use in a temporary exhibition, with the objective of enriching the experience visitors to that physical space by illustrating relationships betweeen objects, and providing media resources to bring them to life. When that exhibition closes, the original objective is partially voided - there is no physical visit to enrich - but aspects of it may still be viable - the objects probably still exist and the tales about them are still worth telling, perhaps more so than ever since we've stuck them back in the store-room out of easy access. Brian was talking about web preservation, of course, and I've taken a non-web resource as an example, but my interest in the question of sustainability extends beyond the web and the point applies regardless.
In any given digital resource that we're talking about preserving/sustaining, experience and information at least (perhaps access, too, sometimes) will have contributed to the original purpose to varying degrees - sometimes the experience is the whole purpose, sometimes it's an unimportant side effect of providing access to information. And sometimes it's important to the preserver regardless of its significance to the original purpose.
So if the original purpose is no longer served by a resource, what then for our "preservation" plans? There are still often reasons to preserve (freeze) or sustain (keep alive) an application, or aspects of it - in other words, some sources of value, ranging from historical interest to new uses, which may let a resource adapt and survive. The "significant properties" idea fits in to this. For me, if you are trying to maintain some quality of the original it's more of a preservation activity; whereas if you are more fundamentally interested in continuing to extract value through maintaining utility of whatever sort, we're talking sustaining.
Brian Kelly just blogged [[hmm, well, back in July, I think....]] on the JISC-PoWR site about "three key aspects of web preservation": experience, information, and access. I have a fourth, but I've been using it in the context of "sustainability" (the subject of my thesis), and so first I want to say a couple of words about this vs preservation, since although I've been writing papers for Ross for a couple of years arguing about the distinction between the concepts, I've not really rehearsed this in public before.
For the last couple of years I've been arguing that the problem of sustainability (S) is distinct from that of archive-style preservation (P). I won't go into the details of the distinction here but in essence I see P as concerning the persistence of a state, and S as the persistence of a process or activity. Recent work by Chris Rusbridge and others has been blurring the boundary ever more, although in a useful way: by questioning the purpose of preservation and weighing up what are the important aspects ("significant properties") of resources, they have been starting to argue for an approach to preservation that looks a lot more like what I was previously describing as sustaining. I still sort of believe that it is useful to distinnguish between the two concepts, but there's a lot of overlap.
The aspect that I think is especially pertinent to sustaining is "purpose". I don't think this is the same as the "experience" that Brian cites Kevin Ashley on (although experience and information may feed into purpose). It's focused on the objectives of the resource, which may be attainable through radically different experiences; for example (in the case of the environment in which KA operates), the learning objectives that an educational resource was created to serve. For a museum, perhaps a resource was prepared for use in a temporary exhibition, with the objective of enriching the experience visitors to that physical space by illustrating relationships betweeen objects, and providing media resources to bring them to life. When that exhibition closes, the original objective is partially voided - there is no physical visit to enrich - but aspects of it may still be viable - the objects probably still exist and the tales about them are still worth telling, perhaps more so than ever since we've stuck them back in the store-room out of easy access. Brian was talking about web preservation, of course, and I've taken a non-web resource as an example, but my interest in the question of sustainability extends beyond the web and the point applies regardless.
In any given digital resource that we're talking about preserving/sustaining, experience and information at least (perhaps access, too, sometimes) will have contributed to the original purpose to varying degrees - sometimes the experience is the whole purpose, sometimes it's an unimportant side effect of providing access to information. And sometimes it's important to the preserver regardless of its significance to the original purpose.
So if the original purpose is no longer served by a resource, what then for our "preservation" plans? There are still often reasons to preserve (freeze) or sustain (keep alive) an application, or aspects of it - in other words, some sources of value, ranging from historical interest to new uses, which may let a resource adapt and survive. The "significant properties" idea fits in to this. For me, if you are trying to maintain some quality of the original it's more of a preservation activity; whereas if you are more fundamentally interested in continuing to extract value through maintaining utility of whatever sort, we're talking sustaining.
Monday, May 12, 2008
Slicing the market
To follow up my recent posts on the 21st century museum and the National Collections Online feasibility study, here's a little more about the question of how funds are allocated by bodies like HLF, and how come we've tended to see them requiring digitisation projects to run from end to end, from digitisation to web publication.
I don't really know enough about the programmes that were run in the past like, for example, NOF Digitise and the HLF (still funding today). However from the projects that I do know that drew on these funding streams, it seems clear enough that concentrating on digitisation per se was never enough - a user-facing product was expected. These did have a sort of aggregated interface (http://www.enrichuk.net/, now pointing at MICHAEL UK) but not really what I've got in mind.
It's quite understandable that this practice persists, indeed there is a good case for it. For funders, they want to see direct user impacts, and a website that can generate stats is nice for that. For museums, well they have a two-fold role - to hold and to document their collections; and to research and interpret them for various audiences. Naturally, then, there will be forces within them urging them to do something with the stuff they digitise, and do it now. The downside is that this has led to fragmentation of the materials digitised for these programmes.
My argument is that for modest sized projects it's not good to spread yourself too thinly trying to do everything, but instead to concentrate on part of the value chain. It's a straighforward architectural thing, and essentially I've realised that I'm just asking for a shift in the assumptions of funding bodies from demanding vertical slices to demanding (contributions to) horizontal slices - in short, to encourage the building of a strong sector-wide layered architecture. Vertical businesses deal in multiple (if not all) links in the chain from production to consumption. This is not really suited to much of what we do. Indeed even behemoths like Amazon don't yet do this (though they a strategically building out from their core business) - they don't actually write books or commission, edit and publish them, print them, or market them, or at the other end deliver them. They flog 'em and dispatch them, and act as an agent for others to do the same.
It's crazy to ask, as a precondition for funding, that a museum or partnership does the whole chain from digitisation and record enhancement, to search engine development, perhaps aggregating records, through to building the user experience as a website, a game etc. Things get done again and again and again, effects are diluted, money wasted. Lots of good stuff will still be made, of course. I don't know what this horizontal architecture would be like, though just possibly we can see parts of it falling into place with PNDS and the Integrated Architecture, and I don't know to what degree it would play a part in many projects in which UK museums engage, but I'm just suggesting that instead of HLF, for one, getting hung up on projects doing the whole end-to-end thing they should support us in coming up with a model to slice the market differently, working on one step at a time, and recognising that the very process of creating the digitised content is a valuable step that is somewhat independent of the creation of an audience-facing instance of it. Vertical slices make silos.
For more on slicing and dicing see here. Shalom.
I don't really know enough about the programmes that were run in the past like, for example, NOF Digitise and the HLF (still funding today). However from the projects that I do know that drew on these funding streams, it seems clear enough that concentrating on digitisation per se was never enough - a user-facing product was expected. These did have a sort of aggregated interface (http://www.enrichuk.net/, now pointing at MICHAEL UK) but not really what I've got in mind.
It's quite understandable that this practice persists, indeed there is a good case for it. For funders, they want to see direct user impacts, and a website that can generate stats is nice for that. For museums, well they have a two-fold role - to hold and to document their collections; and to research and interpret them for various audiences. Naturally, then, there will be forces within them urging them to do something with the stuff they digitise, and do it now. The downside is that this has led to fragmentation of the materials digitised for these programmes.
My argument is that for modest sized projects it's not good to spread yourself too thinly trying to do everything, but instead to concentrate on part of the value chain. It's a straighforward architectural thing, and essentially I've realised that I'm just asking for a shift in the assumptions of funding bodies from demanding vertical slices to demanding (contributions to) horizontal slices - in short, to encourage the building of a strong sector-wide layered architecture. Vertical businesses deal in multiple (if not all) links in the chain from production to consumption. This is not really suited to much of what we do. Indeed even behemoths like Amazon don't yet do this (though they a strategically building out from their core business) - they don't actually write books or commission, edit and publish them, print them, or market them, or at the other end deliver them. They flog 'em and dispatch them, and act as an agent for others to do the same.
It's crazy to ask, as a precondition for funding, that a museum or partnership does the whole chain from digitisation and record enhancement, to search engine development, perhaps aggregating records, through to building the user experience as a website, a game etc. Things get done again and again and again, effects are diluted, money wasted. Lots of good stuff will still be made, of course. I don't know what this horizontal architecture would be like, though just possibly we can see parts of it falling into place with PNDS and the Integrated Architecture, and I don't know to what degree it would play a part in many projects in which UK museums engage, but I'm just suggesting that instead of HLF, for one, getting hung up on projects doing the whole end-to-end thing they should support us in coming up with a model to slice the market differently, working on one step at a time, and recognising that the very process of creating the digitised content is a valuable step that is somewhat independent of the creation of an audience-facing instance of it. Vertical slices make silos.
For more on slicing and dicing see here. Shalom.
Tuesday, April 08, 2008
Significant properties workshop - report
DCC/JISC significant properties workshop (British Library, 7/4/2008)
I'm not going to write up in detail all that was presented on Monday, but highlight a few things that seemed important to me, and work out a couple of thoughts/responses of my own. I haven't yet had a chance to read the papers that were sometimes referred to at the workshop (links to them here, some are huge!) so my questions may be answered there.
I'm not going to write up in detail all that was presented on Monday, but highlight a few things that seemed important to me, and work out a couple of thoughts/responses of my own. I haven't yet had a chance to read the papers that were sometimes referred to at the workshop (links to them here, some are huge!) so my questions may be answered there.
- JISC’s INSPECT project, run by CeRch at KCL, has set a framework for identifying and assessing the value of significant properties (SPs), and the success of their preservation; and initiated several case studies looking at SPs in the context of sets of similar file formats (still images, moving images etc) and categories of digital object (including e-learning objects and software).
- 5 broad SP “classes” (behaviour, appearance/rendering, content, context and structure) are identified by INSPECT. These don’t seem to include space to describe the “purpose” of a digital object (DO), unless this is somehow the combined result of all other SPs. But an objective such as “fun” or “communicates a KS2 concept effectively to the target audience” needs to be represented, especially for complex, service-level resources. Preserving behaviour or content but somehow failing to achieve the purpose would be to miss the point.
- Something I’m still unclear on: is it that a range of SPs are identified that can be given a value of significance for a given “medium” or format? Or is it that a set of SPs is identified for a format, and the value given according to each instance (or set of instances) submitted for presentation? In other words, it a judgement made of the significance of a property for a format/medium, or for a given preservation target?
- Once identified, SPs provide a means for measuring the success of preservation of a file format (whether the preservation activities entail migration to or from that format, or emulation of systems that support it).
- The two classes of object explored in the workshop (software and e-learning objects) are typically compound, and are much more variable than file formats. They will inherit some (potential) SPs from their components, but others (many behaviours, for example) may be implicit in the whole assemblage.
- Andrew Wilson (keynote speaker, NAA) raised the importance of authenticity. His archivists’ point of view of this concept is not identical with that in museums, or that which I'm using in my research, but it’s useful nonetheless. I have, however, already discarded it as a significant property for most museum digital resources, with the exception of the special case of DRs held as either evidence, or accessioned into collections. Archivists’ focus on informational value and “evidence” as the core measure of (and motivation for) authenticity isn’t always useful for DRs, but it is nice and clear-cut.
- The software study drew out the differences between preservation for preservation’s sake – the museum collecting approach – and preservation for use, where the outputs are the ultimate measure of success. The SPs for these scenarios differ.This paper was very interesting, and perhaps (along with the Learning Objects paper) came closest to my own concerns, but the huge variety of material under the banner of “software” clearly makes it very difficult to characterise SPs. The result is that many of those identified look more like preservation challenges than SPs in themselves. Specifically, dependencies of various sorts might count as a significant property in a “pure preservation” scenario; but in most cases they are, more likely, simply a challenge to address to maintain significant properties of other sorts, such as functionality, rendering, and the accuracy of the outputs.
- I suggested in Q&As that my reason for being interested in SPs probably differed from that of a DO-preserving project or organisation, although they have plenty in common. Andrew Wilson said that he saw the sort of preservation (sustaining value) that I was talking about as being the same as preserving in the archiving sense. I disagree, in part at least, because:
- He made the case for authenticity. This doesn’t apply when one is using SPs to help planning for good management, where we just want to make sure that we’re making best use of our resources.
- For me, SPs could prove an important approach for planning new resources, whilst for archives they are primarily for analysing what they’ve received and need to preserve (although they could in theory feed into future formats, or software purchasing decisions)
- Whilst for preservation purposes it may often be necessary to decide at a batch or format level what SPs are highly valued and hence what efforts will be invested in their maintenance, for questions of managing complex resources for active use, case-by-case decisions (based on idiosyncratic SPs?) may be the norm.
- For preservation, the “designated community” is essentially a presumptive audience, whose needs should be considered. For museums looking to maximise value from their resources, the SPs will reflect the needs of the museum itself (its business objectives and strategic aims), although ultimately various other audiences are the targets of these objectives. Perhaps there’s not so much difference here.
- Fundamental to all these differences is the fact that for archives etc, the preservation operation in which they are engaged is the core activity of the organisation. In other situations, like planning for sustainability, it is not preservation of a digital object, but its continued utility in some form (any form), i.e. the continued release of value, that counts.
These differences are largely of degree, but to me there is still a worthwhile distinction between preservation and sustainability. In a sense, preservation is the action and sustainability the continued ability to perform that action, so SPs are a way of reconciling preservation with the need for it to be sustainable. Perhaps the lack of a category that outlines the objectives, rather than the behaviour, of a digital object reflects this difference between preserving and sustaining.
Wednesday, January 16, 2008
Alex Iskold on "The Danger of Free"
The Danger of Free - ReadWriteWeb
The question we've perhaps ignored rather. It's true, we're all perhaps a bit too keen on getting stuff for free. We see stuff delivered through the screen as intangible, but that doesn't always equate to us thinking it has no value or should be free, so why is this so with web-delivered services and content? Is it just a habit, and one we can back out of if we see that it's going to mean poorer quality goods, or is it just in our nature to go for the free or to feel suspicious of the value we'll get from stuff on the web? If we have confidence that it will be trustworthy and value for money are we more likely to pay?
The question we've perhaps ignored rather. It's true, we're all perhaps a bit too keen on getting stuff for free. We see stuff delivered through the screen as intangible, but that doesn't always equate to us thinking it has no value or should be free, so why is this so with web-delivered services and content? Is it just a habit, and one we can back out of if we see that it's going to mean poorer quality goods, or is it just in our nature to go for the free or to feel suspicious of the value we'll get from stuff on the web? If we have confidence that it will be trustworthy and value for money are we more likely to pay?
Tuesday, August 14, 2007
Moaning about stats, coz I hate them, basically
We have been having a very stimulating debate on the whole "what's the point of (automated)web stats?" question (by which I mean to exclude data gathered through surveys, focus groups etc.) . I guess I should acknowledge from the off the my problem is that I'm lazy, so since doing anything useful with web stats involves some work and I'd rather be doing something else, I don't do enough with them. Consequently I subconsciously look for a reason not to use them that doesn't require acknowledging my lazy-arseness, and such reasons are abundant. However in truth I know that there is utility to be found in server logs or client-based methods or whatever you choose to employ, as long as you look deeply enough. The main issue I have is with using simplistic figures as KPIs that are meant somehow to make results comparable across projects and institutions. I still think that's stupid.
Of course, if I get what I wish for, there's going to be a whole load more work in the evaluation process than churning out some stats, which isn't ideal for a lazy sod, but it's not really the effort that bothers me, it's the pointlessness. Worse than that, it's the potential to distract you from what really matters, taking you in the direction of what doesn't, simply because that's what's measured. Happily we're in a sector where at least people don't die if you do this sort of idiot target-chasing thing, unlike health, but still, it's not good.
So below are e-mails relating to a forthcoming project to evaluate web stats methods in the London Hub: their use at present and recommendations for future practice. I wrote a response to the whole idea, which is being shepherded by the estimable Dylan Edgar (formerly of the Scottish Museums Council, and now ICT co-ordinator of the London Hub). Dylan wrote back with a comprehensive response, for which I'm most grateful.
[I wrote:]
If you stand in the sea in a pair of wellies waiting to see how quickly they fill up, in the hope that this will tell you how quickly sea levels are changing in response to global warming, you might learn something. However it will probably be more to do with what the tide is doing, how big the waves are, whether any large boats have passed by or whether you're generally unlucky and keep falling over. If your friend is doing the same, you can compare your results better if your wellies are the same height, but perhaps they're standing on a rock or by chance got hit by a large piece of chop.
Web stats tell you a similar story about how successful your web offering is in achieving your mission-relevant goals. If we have the same software as our peers, it's like having the same wellies - a small improvement but still measuring a proxy that is quite some way off representing what we're really interested in. I know we all recognise this, and I know too that we all have to, firstly, give our funders what they ask for in terms of KPIs, and secondly, be able to lean on some kind of indicator to see how well we're doing. But I think it would be much more interesting and much more useful if we were to explore means other than site stats as indicators of our impact and success relative to mission. Of course it would be good in a pedantic, mechanical way if we used a common platform to record and "analyse" these obligatory KPIs, but it's like polishing turnips (sic). I'd love to see us making an argument to MLA (and above that to the DCMS) to develop KPIs that actually mean something - and these may be different from institution to institution and project to project. They may be amenable. You'll know the debates going on about cultural/social value at the moment (Tessa Jowell, John Holden, Peter Cannon-Brookes, Jim Collins and others), so perhaps this is a cue to start prodding the funders about finding better ways of evaluating success - ones that might not resolve into easy figures but that keeps us focussing on what matters.
Having said all that, this is clearly an area that is thought by the Hub and its partners to be important and I certainly don't know all the ways in which people use our stats and may find them useful. Right now, though, what we are collecting is (as far as I can tell) chiefly used for reporting. We in MST look at them sometimes to give us a clue about what parts of the site are doing well, what search terms seem to drive traffic to us, who's linking to us etc., but we look through sceptical lenses and interpret everything we see with a lot of knowledge that simply isn't embodied in those stats, and isn't available to funders. In that sense they can be a useful tool for making improvements, but not really for demonstrating success except in the most clumsy, questionable way. Similarly, if the cows are lying down you might want to think about taking a mac with you on a walk, but if you were making a weather report you would only say "it rained today" if you saw it raining.
I should just expand on why stats are becoming ever more deficient PIs. Our digital resources aren't only used by our website visitors. Our data are taken from feeds and reused on other websites (for which we cannot access the stats), our images are taken and put into personal sites and blogs, we are bookmarked in de.icio.us and events taken from our pages and added to personal Google Calendars. As the semanticised web inches forward, more and more of our stuff moves into places where we aren't measuring it (and can't). The OAI data handed over to the PNDS is part of a Hub (and MoL) job of work, but if they were to decide to open up their repositories for other parties to build applications with, how do we measure the impact of our data? Not through measuring browser-based visits to our websites. We have started pilot work with Simulacra that encourages teachers to build interactives from our content, but whilst we will get stats from Simulacra we can't assess how good these interactives are, how much of their content is "ours" (they can combine all sorts of material), how they're used in the classroom once they're downloaded, and so on. And if we decide to put some curatorial time into creating great content on Wikipedia, how do we know if it's "successful" in the sense of contributing to our strategic goals? Not from web stats. But if we can't measure success like that, should we accept that the whole idea is wrong? If I make a KML file and upload it to Google, or even host it on our server, how do I know how much use someone has had from it by downloading it once and viewing it perhaps many times in their local installation of Google Earth? If we can't know, or can't prove it with hits and visits data, should we not do it? I'd say yes, we should do it, but we need some new ways of guesstimating how successful it is. I want to be able to do things that can't be demonstrated to have had a great impact through the current KPIs (or perhaps through any means). I think this means moving beyond crude web stats, as well as thinking more about what those stats mean.
I don't think the project is wrong, but I do think it's perhaps putting a lot of effort into improving something that is never going to be that useful, whereas it could be about making the case for useful KPIs and developing new techniques for assessing them. I note that there are some references to other tools in the brief (surveys etc.), which is good. Maybe we can slant it more in this direction, but keeping in mind the reason: not to come up with new ways of measuring usage, but assessing impact relative to mission.
See also Seb Chan, talking about how to keep an eye on what people are saying about you (citing our very own Mia Ridge, too): http://www.powerhousemuseum.com/dmsblog/index.php/2007/08/09/social-media-measurement-brand-awareness-and-trust-in-the-cultural-sector/
[and Dylan replied:]
Thanks for getting back to me on this. Of course you're right - web stats are always going to be a rather blunt instrument, and I'm not suggesting that they provide a complete means of establishing the impact or otherwise of our web sites. Throw in Web 2, 3 and the rest of it and it gets even messier of course! I've always been an advocate of taking a more rounded approach to evaluating the impact of online delivery - in fact I commissioned some research back in 2004 which resulted in guidance supporting museums to start doing exactly the kind of thing you're suggesting. This was subsequently embedded into government funding streams for ICT (although not as a KPI): http://www.scottishmuseums.org.uk/areas_of_work/ICT/digitisation.asp
One of the most important recommendations from this work was that museums shouldn't rely on web stats in isolation as a measure of impact. However, they can make a useful contribution to establishing impact if used in conjunction with other methodologies.
Some other work has been done in this area. The EU ran a conference last year which brought experts together from around Europe and set out to look at ways in which cultural heritage bodies can evaluate the impact that their digitised resources are having on people. I was invited along to talk on the subject, and there was some useful discussion on the various different approaches that could be taken as well as looking at definitions of 'impact': http://www.nba.fi/nrg
...however, we didn't get anywhere near to coming up with a consistent measure that could be applied across the board. Incidentally, MLA and DCMS were involved in this conference, so don't be too hard on them - they are well aware of the issues here!
The Hub partners are already doing a lot of this of course, but it's still not formalised as a KPI. I agree that in an ideal world we'd be working towards a more meaningful indicator for establishing the impact of web delivery. However, in reality this is still a very very long way off.
A recent review of a selection of Renaissance KPIs by Morris Hargreaves McIntyre recommended that the one measure relating to web use (visits) remain as is for the foreseeable future, but MLA should "assess need for additional web performance indicators". There's a very good reason for taking such a cautious approach when looking at new indicators, and that is museums' capacity to deliver. We have already found that the current web KPI is collected inconsistently across the country, and even within the regional Hubs themselves. This is mainly because of loose definitions and the range of different systems that museums are using to collect the data.
If we tried to go straight from this situation to working up a more complex set of National indicators that really do set out to measure impact in the round, I don't think that museums would be able to provide meaningful data consistently throughout the sector (at least without a great deal of investment in systems, training, staff etc.). This would be counterproductive because the resulting KPI would probably fail, and confidence in the whole idea of measuring the value of web delivery would be lost, both within government and the wider museum community. This is why I think we need to take a more gradual, staged approach.
Despite their limitations, I do believe that web stats are important. Firstly, they are important politically. For years we have been arguing that the web is an essential delivery route for museums. This is only now being acknowledged by government, who are (quite rightly in my opinion) asking us to justify their investment and tell them just how important it is through this KPI, admittedly in a rather primitive way. It's naive to think that we can continue to sermonise about the importance of the web if we aren't then prepared to stand up and be counted on it when we are asked to. So it's an important opportunity for us to engage and show willing, rather than dismiss out of hand.
Reporting web stats in this way is also important because it shows that it can be done! As you rightly point out, establishing the real impact of online delivery is complex and is by no means an exact science. However by getting this relatively simple measure right, we can demonstrate how museums can deliver consistent data on web use that can be built on towards more meaningful indicators for the future. If we can't or won't do it for a simple KPI, how can we expect to do the more fancy stuff?
As you say though, the stats are limited in what they can tell us in isolation. However analysis is becoming more sophisticated, and if used correctly stats can provide us with a usable indication of how much the different aspects of our sites are being used, and even something of how they are being used and who is using them. So I don't think this is worthless information by any means, as long as we accept that there are limits to what it can tell us and there's always going to be a certain level of inaccuracy involved (that everyone is going to be subject to of course, which helps). But then what data of this kind is ever 100% accurate? I would say that web stats are on a par with gallery visitor numbers, which are often collated automatically and don't tell us anything about visitor experience or the museum's ultimate impact on the people coming through the door. However, as a sector we've been happy to allow this to become accepted currency and have supplied the data for years now in return for government funding. So why not web stats as well? If anything, they tell us more about our users than automated gallery counters.
In this project we want to look at how the London Hub collects data for the existing KPI. We also want to build on this by establishing a more detailed set of quantitative measures that will provide more detail on the extent to which the sites are being used, and that a small group of four museums can collect in a consistent and meaningful way. This will be a challenge in itself. More importantly in my view, we also want to look at how the Hub can be using this information, and ultimately why we're collecting it. As you say much of the KPI data goes into a black hole, which doesn't help because museums don't necessarily see the relevance of the data that they have collected. So by producing practical recommendations on how the Hub can be actively sharing and using this information in the future, we make it more relevant and it becomes a more helpful resource.
This project isn't going on in isolation though. It's being supplemented by more qualitative work this financial year, profiling the Hub's online audiences, understanding their requirements and expectations, and establishing the extent to which these are currently being met. Importantly, this work will also provide a shared methodology for the Hub to continue this process in the coming years. This combination of quantitative and qualitative work will, I believe, gradually build up a better overview of the impact that the Hub's collective online product is having on users.
My own view is that we need to take it slowly, and better methodologies for establishing impact will emerge as we become more comfortable with the idea and as we start implementing this kind of initiative. The whole concept is still in its infancy, and we need to be careful to bring museums with us rather than imposing something that will be unworkable in the short-term and jeopardise the work that's already happened.
[There wasn't much need for further debate in a sense, since we both had pretty clear positions on the utility of stats and Dylan's case is strong for pursuing them in the context of this project, but I my final thoughts were:]
Thanks for the comprehensive response. I know I was very negative about web stats when they're not in fact the devil's spawn, and used wisely can tell us useful things. You've made a lot of very valid points (not least about visitor figures), and you clearly have a strategy for moving towards a situation where we have to rely on stats less whilst working with the political reality. Unfortunately the sophisticated analysis that can make these stats less blunt and more insightful isn't performed before they are given to the funders that use them as KPIs. Nor could it be, because such analyses are case-specific and context-dependent, so we keep on working to deliver targets that at worst might seriously divert us from doing what is really important, simply because we aren't allowed to demonstrate excellence in any other way. So whilst there's more that can be done with stats, we aren't allowed to do so where it counts, so they remain most useful internally (where we can try to use all the knowledge that can be squeezed out) rather than externally, where they cherry-pick the most dessicated fruit.
If MLA, DCMS et al really are interested in working towards better measures, and I'm sure you're right that they are, it would be nice to think that this project could in some small way start to explore that shift and demonstrate to them that we as museums are also keen to look at what really matters. Having said that, I accept what you say, that to move to a whole new national system of KPIs would be a huge step and not one to take in one go. But I wish we could, at least, start discussing with funders the possibility of experimenting with more interesting measures on a project-by-project basis.
[and then Dylan responded with more clarifications and some thoughts on the direction of the London Hub. Some of it is not really for public consumption but here's a bit]
Funding organisations like HLF, MLA etc have always expected museums to have clear plans for how they will evaluate the success of their projects. It's the job of the applicant to make the case for how they will do this - it's part of the application process and a good project needs to have good evaluation built in from the outset. Having said that though, funding organisations do have a role to play in providing guidance to applicants on the kind of thing they are expecting.
The way we approached this in Scotland was to come up with high-level guidelines for museums on how to start evaluating the impact of web resources, beyond using stats alone (see previous email). The guidance was quite generic, but designed to get people thinking about how they can establish the overall success or otherwise of their projects. This broke down into two areas - process and outcomes (i.e. impact).
One of the main reasons for doing this was to integrate it into our funding streams, which was starting to pay dividends by the time I moved down here. The way this worked in practice was if a museum came to us with an application for ICT funding they were expected to show us how they would integrate the recommendations in practice into their project, and they would be held to account if they didn't end up doing what they said they would...
The challenge is making the link clear to museums who don't necessarily see the value of evaluation, and just view it as another hoop to jump through.
Demonstrate the impact of web site = providing clear case to funders = more funding further down the line...
Many thanks to Dylan for the debate and for permission to publish that which isn't libellous! :-)
Of course, if I get what I wish for, there's going to be a whole load more work in the evaluation process than churning out some stats, which isn't ideal for a lazy sod, but it's not really the effort that bothers me, it's the pointlessness. Worse than that, it's the potential to distract you from what really matters, taking you in the direction of what doesn't, simply because that's what's measured. Happily we're in a sector where at least people don't die if you do this sort of idiot target-chasing thing, unlike health, but still, it's not good.
So below are e-mails relating to a forthcoming project to evaluate web stats methods in the London Hub: their use at present and recommendations for future practice. I wrote a response to the whole idea, which is being shepherded by the estimable Dylan Edgar (formerly of the Scottish Museums Council, and now ICT co-ordinator of the London Hub). Dylan wrote back with a comprehensive response, for which I'm most grateful.
[I wrote:]
If you stand in the sea in a pair of wellies waiting to see how quickly they fill up, in the hope that this will tell you how quickly sea levels are changing in response to global warming, you might learn something. However it will probably be more to do with what the tide is doing, how big the waves are, whether any large boats have passed by or whether you're generally unlucky and keep falling over. If your friend is doing the same, you can compare your results better if your wellies are the same height, but perhaps they're standing on a rock or by chance got hit by a large piece of chop.
Web stats tell you a similar story about how successful your web offering is in achieving your mission-relevant goals. If we have the same software as our peers, it's like having the same wellies - a small improvement but still measuring a proxy that is quite some way off representing what we're really interested in. I know we all recognise this, and I know too that we all have to, firstly, give our funders what they ask for in terms of KPIs, and secondly, be able to lean on some kind of indicator to see how well we're doing. But I think it would be much more interesting and much more useful if we were to explore means other than site stats as indicators of our impact and success relative to mission. Of course it would be good in a pedantic, mechanical way if we used a common platform to record and "analyse" these obligatory KPIs, but it's like polishing turnips (sic). I'd love to see us making an argument to MLA (and above that to the DCMS) to develop KPIs that actually mean something - and these may be different from institution to institution and project to project. They may be amenable. You'll know the debates going on about cultural/social value at the moment (Tessa Jowell, John Holden, Peter Cannon-Brookes, Jim Collins and others), so perhaps this is a cue to start prodding the funders about finding better ways of evaluating success - ones that might not resolve into easy figures but that keeps us focussing on what matters.
Having said all that, this is clearly an area that is thought by the Hub and its partners to be important and I certainly don't know all the ways in which people use our stats and may find them useful. Right now, though, what we are collecting is (as far as I can tell) chiefly used for reporting. We in MST look at them sometimes to give us a clue about what parts of the site are doing well, what search terms seem to drive traffic to us, who's linking to us etc., but we look through sceptical lenses and interpret everything we see with a lot of knowledge that simply isn't embodied in those stats, and isn't available to funders. In that sense they can be a useful tool for making improvements, but not really for demonstrating success except in the most clumsy, questionable way. Similarly, if the cows are lying down you might want to think about taking a mac with you on a walk, but if you were making a weather report you would only say "it rained today" if you saw it raining.
I should just expand on why stats are becoming ever more deficient PIs. Our digital resources aren't only used by our website visitors. Our data are taken from feeds and reused on other websites (for which we cannot access the stats), our images are taken and put into personal sites and blogs, we are bookmarked in de.icio.us and events taken from our pages and added to personal Google Calendars. As the semanticised web inches forward, more and more of our stuff moves into places where we aren't measuring it (and can't). The OAI data handed over to the PNDS is part of a Hub (and MoL) job of work, but if they were to decide to open up their repositories for other parties to build applications with, how do we measure the impact of our data? Not through measuring browser-based visits to our websites. We have started pilot work with Simulacra that encourages teachers to build interactives from our content, but whilst we will get stats from Simulacra we can't assess how good these interactives are, how much of their content is "ours" (they can combine all sorts of material), how they're used in the classroom once they're downloaded, and so on. And if we decide to put some curatorial time into creating great content on Wikipedia, how do we know if it's "successful" in the sense of contributing to our strategic goals? Not from web stats. But if we can't measure success like that, should we accept that the whole idea is wrong? If I make a KML file and upload it to Google, or even host it on our server, how do I know how much use someone has had from it by downloading it once and viewing it perhaps many times in their local installation of Google Earth? If we can't know, or can't prove it with hits and visits data, should we not do it? I'd say yes, we should do it, but we need some new ways of guesstimating how successful it is. I want to be able to do things that can't be demonstrated to have had a great impact through the current KPIs (or perhaps through any means). I think this means moving beyond crude web stats, as well as thinking more about what those stats mean.
I don't think the project is wrong, but I do think it's perhaps putting a lot of effort into improving something that is never going to be that useful, whereas it could be about making the case for useful KPIs and developing new techniques for assessing them. I note that there are some references to other tools in the brief (surveys etc.), which is good. Maybe we can slant it more in this direction, but keeping in mind the reason: not to come up with new ways of measuring usage, but assessing impact relative to mission.
See also Seb Chan, talking about how to keep an eye on what people are saying about you (citing our very own Mia Ridge, too): http://www.powerhousemuseum.com/dmsblog/index.php/2007/08/09/social-media-measurement-brand-awareness-and-trust-in-the-cultural-sector/
[and Dylan replied:]
Thanks for getting back to me on this. Of course you're right - web stats are always going to be a rather blunt instrument, and I'm not suggesting that they provide a complete means of establishing the impact or otherwise of our web sites. Throw in Web 2, 3 and the rest of it and it gets even messier of course! I've always been an advocate of taking a more rounded approach to evaluating the impact of online delivery - in fact I commissioned some research back in 2004 which resulted in guidance supporting museums to start doing exactly the kind of thing you're suggesting. This was subsequently embedded into government funding streams for ICT (although not as a KPI): http://www.scottishmuseums.org.uk/areas_of_work/ICT/digitisation.asp
One of the most important recommendations from this work was that museums shouldn't rely on web stats in isolation as a measure of impact. However, they can make a useful contribution to establishing impact if used in conjunction with other methodologies.
Some other work has been done in this area. The EU ran a conference last year which brought experts together from around Europe and set out to look at ways in which cultural heritage bodies can evaluate the impact that their digitised resources are having on people. I was invited along to talk on the subject, and there was some useful discussion on the various different approaches that could be taken as well as looking at definitions of 'impact': http://www.nba.fi/nrg
...however, we didn't get anywhere near to coming up with a consistent measure that could be applied across the board. Incidentally, MLA and DCMS were involved in this conference, so don't be too hard on them - they are well aware of the issues here!
The Hub partners are already doing a lot of this of course, but it's still not formalised as a KPI. I agree that in an ideal world we'd be working towards a more meaningful indicator for establishing the impact of web delivery. However, in reality this is still a very very long way off.
A recent review of a selection of Renaissance KPIs by Morris Hargreaves McIntyre recommended that the one measure relating to web use (visits) remain as is for the foreseeable future, but MLA should "assess need for additional web performance indicators". There's a very good reason for taking such a cautious approach when looking at new indicators, and that is museums' capacity to deliver. We have already found that the current web KPI is collected inconsistently across the country, and even within the regional Hubs themselves. This is mainly because of loose definitions and the range of different systems that museums are using to collect the data.
If we tried to go straight from this situation to working up a more complex set of National indicators that really do set out to measure impact in the round, I don't think that museums would be able to provide meaningful data consistently throughout the sector (at least without a great deal of investment in systems, training, staff etc.). This would be counterproductive because the resulting KPI would probably fail, and confidence in the whole idea of measuring the value of web delivery would be lost, both within government and the wider museum community. This is why I think we need to take a more gradual, staged approach.
Despite their limitations, I do believe that web stats are important. Firstly, they are important politically. For years we have been arguing that the web is an essential delivery route for museums. This is only now being acknowledged by government, who are (quite rightly in my opinion) asking us to justify their investment and tell them just how important it is through this KPI, admittedly in a rather primitive way. It's naive to think that we can continue to sermonise about the importance of the web if we aren't then prepared to stand up and be counted on it when we are asked to. So it's an important opportunity for us to engage and show willing, rather than dismiss out of hand.
Reporting web stats in this way is also important because it shows that it can be done! As you rightly point out, establishing the real impact of online delivery is complex and is by no means an exact science. However by getting this relatively simple measure right, we can demonstrate how museums can deliver consistent data on web use that can be built on towards more meaningful indicators for the future. If we can't or won't do it for a simple KPI, how can we expect to do the more fancy stuff?
As you say though, the stats are limited in what they can tell us in isolation. However analysis is becoming more sophisticated, and if used correctly stats can provide us with a usable indication of how much the different aspects of our sites are being used, and even something of how they are being used and who is using them. So I don't think this is worthless information by any means, as long as we accept that there are limits to what it can tell us and there's always going to be a certain level of inaccuracy involved (that everyone is going to be subject to of course, which helps). But then what data of this kind is ever 100% accurate? I would say that web stats are on a par with gallery visitor numbers, which are often collated automatically and don't tell us anything about visitor experience or the museum's ultimate impact on the people coming through the door. However, as a sector we've been happy to allow this to become accepted currency and have supplied the data for years now in return for government funding. So why not web stats as well? If anything, they tell us more about our users than automated gallery counters.
In this project we want to look at how the London Hub collects data for the existing KPI. We also want to build on this by establishing a more detailed set of quantitative measures that will provide more detail on the extent to which the sites are being used, and that a small group of four museums can collect in a consistent and meaningful way. This will be a challenge in itself. More importantly in my view, we also want to look at how the Hub can be using this information, and ultimately why we're collecting it. As you say much of the KPI data goes into a black hole, which doesn't help because museums don't necessarily see the relevance of the data that they have collected. So by producing practical recommendations on how the Hub can be actively sharing and using this information in the future, we make it more relevant and it becomes a more helpful resource.
This project isn't going on in isolation though. It's being supplemented by more qualitative work this financial year, profiling the Hub's online audiences, understanding their requirements and expectations, and establishing the extent to which these are currently being met. Importantly, this work will also provide a shared methodology for the Hub to continue this process in the coming years. This combination of quantitative and qualitative work will, I believe, gradually build up a better overview of the impact that the Hub's collective online product is having on users.
My own view is that we need to take it slowly, and better methodologies for establishing impact will emerge as we become more comfortable with the idea and as we start implementing this kind of initiative. The whole concept is still in its infancy, and we need to be careful to bring museums with us rather than imposing something that will be unworkable in the short-term and jeopardise the work that's already happened.
[There wasn't much need for further debate in a sense, since we both had pretty clear positions on the utility of stats and Dylan's case is strong for pursuing them in the context of this project, but I my final thoughts were:]
Thanks for the comprehensive response. I know I was very negative about web stats when they're not in fact the devil's spawn, and used wisely can tell us useful things. You've made a lot of very valid points (not least about visitor figures), and you clearly have a strategy for moving towards a situation where we have to rely on stats less whilst working with the political reality. Unfortunately the sophisticated analysis that can make these stats less blunt and more insightful isn't performed before they are given to the funders that use them as KPIs. Nor could it be, because such analyses are case-specific and context-dependent, so we keep on working to deliver targets that at worst might seriously divert us from doing what is really important, simply because we aren't allowed to demonstrate excellence in any other way. So whilst there's more that can be done with stats, we aren't allowed to do so where it counts, so they remain most useful internally (where we can try to use all the knowledge that can be squeezed out) rather than externally, where they cherry-pick the most dessicated fruit.
If MLA, DCMS et al really are interested in working towards better measures, and I'm sure you're right that they are, it would be nice to think that this project could in some small way start to explore that shift and demonstrate to them that we as museums are also keen to look at what really matters. Having said that, I accept what you say, that to move to a whole new national system of KPIs would be a huge step and not one to take in one go. But I wish we could, at least, start discussing with funders the possibility of experimenting with more interesting measures on a project-by-project basis.
[and then Dylan responded with more clarifications and some thoughts on the direction of the London Hub. Some of it is not really for public consumption but here's a bit]
Funding organisations like HLF, MLA etc have always expected museums to have clear plans for how they will evaluate the success of their projects. It's the job of the applicant to make the case for how they will do this - it's part of the application process and a good project needs to have good evaluation built in from the outset. Having said that though, funding organisations do have a role to play in providing guidance to applicants on the kind of thing they are expecting.
The way we approached this in Scotland was to come up with high-level guidelines for museums on how to start evaluating the impact of web resources, beyond using stats alone (see previous email). The guidance was quite generic, but designed to get people thinking about how they can establish the overall success or otherwise of their projects. This broke down into two areas - process and outcomes (i.e. impact).
One of the main reasons for doing this was to integrate it into our funding streams, which was starting to pay dividends by the time I moved down here. The way this worked in practice was if a museum came to us with an application for ICT funding they were expected to show us how they would integrate the recommendations in practice into their project, and they would be held to account if they didn't end up doing what they said they would...
The challenge is making the link clear to museums who don't necessarily see the value of evaluation, and just view it as another hoop to jump through.
Demonstrate the impact of web site = providing clear case to funders = more funding further down the line...
Many thanks to Dylan for the debate and for permission to publish that which isn't libellous! :-)
Thursday, May 10, 2007
Value of heritage sites
This series of workshops looks interesting, at least to keep an eye on (there's only one left anyway) as a locus of discussions on how to identify value in real-world heritage.
http://www.le.ac.uk/ms/contactus/valhistenvir.html
http://www.le.ac.uk/ms/contactus/valhistenvir.html
Thursday, May 03, 2007
Cross-posted comment: on real museums and digital value
Cross-posted comment to Holly Witchey's post on musematic (since it's what I had on my mind today anyway):
Hi Holly, you won’t remember me but we shared breakfast in Pasadena…. I do understand how you can sometimes get to feel like this. I don’t have much to add to your thoughts on communication breakdown – you’re right, communication takes both effective transmission and reception. Your remarks on the value of our whole digital enterprise, though, chimed very serendipitously with my own musings on the way to work this morning. I was thinking: some museums that hold a preponderance of “real” objects, others contain more in the way of dioramas, reconstructions, replicas, interactives and experiences; indeed some have nothing “real” at all. Does this lead them to have different attitudes to their digital holdings or place different value upon them? Despite the AAM’s Code of Ethics, not all museums (in the broad definition that the AAM also holds) have collections per se. In fact, the section you quote includes not just collections but “exhibition materials”, and maybe that’s where we can salve our consciences a little: exhibition materials could well include digital resources. In the end it’s true, most of the time in most cases it’s the collections that really count and they must take priority, and museums always have to balance, to choose, and they do. As well as going on building and maintaining collections they have to use them in all sorts of ways to get value now as well as in the future. That’s where we come in – only rarely are we creating works of art; mostly we’re making stuff that brings art, history, science, ideas to people. It would be foolish to spend too much on that, like it would be foolish to spend all the money on gallery refurbs and none on building and caring for collections, but still it’s valuable work. Occasionally we might even make something that could become digital heritage (i.e. a digital thing worth keeping) as opposed to digitized heritage, and in this time of flux and exciting experiments we are surely seeing some genuinely valuable bodies of knowledge and experience that perhaps we’ll want to “preserve for posterity”. I’m really interested to find out if and when our digital stuff is anywhere near as precious as our real stuff (it’s my research area, as it happens), but anyway don’t be too down! If people needed museums in the past, not just collections, then they need us now, in the same way
Hi Holly, you won’t remember me but we shared breakfast in Pasadena…. I do understand how you can sometimes get to feel like this. I don’t have much to add to your thoughts on communication breakdown – you’re right, communication takes both effective transmission and reception. Your remarks on the value of our whole digital enterprise, though, chimed very serendipitously with my own musings on the way to work this morning. I was thinking: some museums that hold a preponderance of “real” objects, others contain more in the way of dioramas, reconstructions, replicas, interactives and experiences; indeed some have nothing “real” at all. Does this lead them to have different attitudes to their digital holdings or place different value upon them? Despite the AAM’s Code of Ethics, not all museums (in the broad definition that the AAM also holds) have collections per se. In fact, the section you quote includes not just collections but “exhibition materials”, and maybe that’s where we can salve our consciences a little: exhibition materials could well include digital resources. In the end it’s true, most of the time in most cases it’s the collections that really count and they must take priority, and museums always have to balance, to choose, and they do. As well as going on building and maintaining collections they have to use them in all sorts of ways to get value now as well as in the future. That’s where we come in – only rarely are we creating works of art; mostly we’re making stuff that brings art, history, science, ideas to people. It would be foolish to spend too much on that, like it would be foolish to spend all the money on gallery refurbs and none on building and caring for collections, but still it’s valuable work. Occasionally we might even make something that could become digital heritage (i.e. a digital thing worth keeping) as opposed to digitized heritage, and in this time of flux and exciting experiments we are surely seeing some genuinely valuable bodies of knowledge and experience that perhaps we’ll want to “preserve for posterity”. I’m really interested to find out if and when our digital stuff is anywhere near as precious as our real stuff (it’s my research area, as it happens), but anyway don’t be too down! If people needed museums in the past, not just collections, then they need us now, in the same way
Subscribe to:
Posts (Atom)