Living Labs: Evaluation Results

Technology Concept validation in Living Labs Environment

Further to a Living Lab Open Call and to a structured activity of identification of consistent and compatible “triplets” (one Living Lab, one museum with a tight relationship with the Living Lab and one FP7 C.H. technological projects ready to make available a prototype product or solution to be experimented at the museum) three concept validation activities were executed in Amsterdam, Paris and Sofia in the months of October and November 2014: Deliverable D.2.2.A provides details these Living Lab validation experiments.

  • Amsterdam Smart City (ASC), The Netherlands:  experiment based on CHESS C.H. FP7 project at the Stedelijk Museum – Amsterdam
  • Integrative Usage Lab (IUL-LUTIN), France: experiment based on TAG CLOUD C.H. FP7 project at the Cité des Sciences et de l'Industrie - Paris
  • Digital Spaces Living Lab (DSLL), Bulgaria: experiment based on meSch C.H. FP7 project at the National Museum of History – Sofia

The following diagram summarizes the overall process that has been defined and implemented :

What is a Living Lab

Living Labs are user-driven Open Innovation ecosystems and, according to ENoLL (the European Network of Living Labs, a community with a sustainable strategy for enhancing innovation on a systematic basis). A Living Lab employs four main activities: Co-Creation (co-design by users and producers), Exploration (discovering emerging usages, behaviours and market opportunities), Experimentation (implementing live scenarios within communities of users), Evaluation (assessment of concepts, products and services according to socio-ergonomic, socio-cognitive and socio-economic criteria).

Concept validation in action at the Stedelijk Museum  in Amsterdam

Within a carefully designed testing environment, Antenna staff handed out the iPads and did research into the user experience. Visitors were offered free tours on the CHESS-programmed iPads in order to get enough results.

In order to construct a control group, Antenna programmed half of the iPads with a “random” tour in the same look-and-feel. Visitors did not know if they were following a personalized route or a “random” route. After the tour, Antenna staff undertook short interview with the visitors about their visit, with questions mainly related to the experience.

The experiment ran from November 21 to December 15 and generated 63 responses.


Concept validation in action at the Cité des Sciences et de l'Industrie  in Paris

Participants were wearing eye-tracking glasses while visiting the museum, searching for QR-code to flash in order to get the content. The eye-tracking device allows understanding the process of using Cooltura: (Phase 1) searching for Q-R Code in the environment, (Phase 2) flashing, (Phase 3) getting the content in the content of the museum object

Concept validation in action at project at the National Museum of History in Sofia

The Loupe trials was carried out in October in Hall 2 of the museum, where the Ancient Thracian culture and art exhibits are hosted (Thrace during the Ancient times – 6th – 1st century BC – 6th C AD).


Name of Exhibit

Story Screens Related to This Exhibit


Thracian Helmet


Thracian type helmet

4 c. BC

It has a forward-pointing peak and resembles the shape of the leather cap worn by the Thracian people.

Success in war was often symbolised by the winged goddes of victory Nike.


Overall lessons learned from the 3 experiences in Living Labs.

An overview of the three concept validation outcomes coming from the experiments run in Amsterdam, Paris and in Sofia shows how there are significant margins of improvement in the take up of technologies for Cultural Heritage in European museums. The pervasivity in the use of mobile devices, smartphones and tablets, is a driving factor but it must be supported by proper training plans for professional operators working at museums (and within related environments with a specific attention to school teachers, to students and young people in general) in order to make effective the presence of apps with cultural storytelling on mobile devices. The experiments were executed at three big museums in European cities and it is possible that some lessons learned do not have a general validity and that some points cannot apply to museums of more limited dimensions: running new experiments at different size museums would be an advisable opportunity and the contacts (at project level and at personal level) started between the various people and organisations involved in the experiments have to be continued.

The objective of giving users (visitors) a satisfactory experience must evolve into creating conditions for repeating the visit and this is to give clear benefits to the whole cultural and touristic ecosystem: social networks are fundamental in this respect and availability of multimedia contents is a key point. Image/object recognition libraries start to be an essential feature of museum libraries and this should represent the infrastructure for building up new technological tools (products, solutions, apps, platforms) to meet more and more visitors’ needs. The need for personalisation (from customizing the contents to assigning specific visit paths) is a main issue too and it has to be carefully considered avoiding situations in which the visitor can feel himself driven away from part of the exhibits he wanted, on the contrary, look with more detail or with an enhanced presence of additional audio / image / video contents.

Gamification (and at which extent it has to be applied within cultural contexts, specifically at museums) remains an aspect to be furtherly clarified. The importance of ergonomy, mostly when highly technological are addressed to young or elder people, also has to be fully recognized.

Ergonomy, gamification, solution scalability (big vs small museums as already pointed out, but also e.g. open-air vs traditional museums) are some of the issues that would deserve the launch of new initiatives (within the frame of contexts similar to eCultValue and to the Open Call for Living Labs “Technologies for Cultural Heritage - Concept Validation at European Museums”): these new initiatives can exploit the Best Practice of process definition and execution for Living Lab involvement in a museum validation as described in the present report. It can be also reasonably assumed (considered the complex ecosystem and multi-actor context where the experiments occurred in a successful way) that this Best Practice is likely to find applications even outside the Cultural Heritage domain in order to give opportunities to the European community of Living Labs to enforce their role of innovation drivers and co-creation catalysts addressed at sustainable ecosystem models as the one demonstrated as feasible  by the three eCultValue Living Lab experiments.

Workshop “Design Innovation and the Emerging Cultural Heritage”

The results from the experiences where discussed during a dedicated workshop “Design Innovation and the Emerging Cultural Heritage” in the framework of ESoCe Net  conference:

“PEOPLE DRIVEN SOCIAL INNOVATION WELLBEING, ENERGY AND CULTURAL HERITAGE” on Dec 1st 2014, in Rome. Power point presentations are available at  

ESoCE-Net Industry Forum - 2014 Proceedings

The workshop program is reported below.

Design Innovation and the Emerging Cultural Heritage, Introduction by Federico Mussano – ESoCE-Net

Panel of Living Labbers, innovation designers and Cultural Heritage innovators

Concept validation in Living Labs for adoption of new technological solutions in adaptive storytelling and augmented reality for Cultural Heritage:

Augmented Reality Experimentation at Cité des Sciences et de l'Industrie (CSI) - Paris, Silvia de los Rios Perez – TAG CLOUD & Charles Tijus – IUL-LUTIN &  Elhadi Djebbari – CSI

Adaptive Storytelling Experimentation at Stedelijk Museum - Amsterdam, CHESS & Niels de Jong – Amsterdam Smart City & Erna Bomers – Stedelijk

Object Mediated Augmented Reality Experimentation at National Museum of History - Sofia, Massimo Zancanaro – meSch & Stavri Nikolov – Digital Spaces Living Lab

At the ESoCE-Net iForum in Rome (December 1st, 2014), representatives from the Stedelijk Museum and ASC/Antenna presented the project to a broad range of professionals in the European cultural sector