Get Inside Their Heads: Learning from visitors

Unknown Speaker 00:00
Welcome, thank you for being here. I am Silvina Fernandez do gay, hi friendly faces. I work at the US Holocaust Memorial Museum in Washington DC. And this presentation is get inside their heads learning from visitors. So it's a little flippant title is my sort of shorthand of making sure that we are talking to visitors and understanding what they are experiencing in the museum and with our particular subject matter. So just curious how many people in the room do visitor research as part of your work? Okay, so any one of you could be giving this talk? I come at it. Oh, boy. Oh, no, I'm having serious technical difficulties today. Okay. Um, okay, so I'm gonna have to wing it today. So I work, it's okay. I my technical skills are not up to par today. But I work in a group called future projects. And we are all about prototyping and, and testing and asking questions and looking at new technologies and new techniques and trying to figure out, what does it mean in a museum setting? What does it mean with our particular content. So we're a small team. And, you know, prototyping is embedded in the work that we do. So part of our mandate also is to kind of spread this culture throughout our institution. This image just shows a picture of a project that we did with Parsons School of Design and some of their graduate students, which was an amazing collaboration. And an opportunity came up at the museum to have a space that we could use specifically for visitor testing. And we branded it, we call it studio three, we actually did some title testing for what we should call it, we had been using internal language museum experience lab. So we've tested that out, we learned anything with lab at the Holocaust Museum is not a good idea. So we gave it this name studio three. And it sort of had double duty, it was both a space where we could go in there and pop up exhibitions or little small testing sites, and at the same time, and it's downtime, it would also be kind of allowance where we people could sit and charge their phones and relax. So it's, we've it's been up for about a year. And we've done a few different, we've done lots and lots of different things in that space. And I'm going to share two of the projects that we we did with not just our team, but also teams throughout the institution. So this kind of work really is more of formative evaluation, right? It's the stuff that you do at the beginning of a design process, you really want to make sure that you are understanding how things are landing with your visitors as you're developing things. So this particular project was based was was the project team was taking an existing temporary exhibition called Americans a holocaust, which is open right now. And designing a traveling version, kind of a panel exhibition that could be popped up in libraries around the country. And they wanted to understand sort of what, how particular pieces of the exhibition were being understood. So in this, there were a bunch of things that they tested, this is just one of them. And, in particular, this graphic, which in the exhibition is a room size installation infographic, this is not a great picture. But on the left hand side, there's a wall that has two pieces of information, don't really want to get into too many details about the specifics of the content. But the top half shows the number of the quota for the visas that they were that were available in the US for immigrants coming from Europe. And then the bottom half shows the number of people on the waiting list. So it's it's room scale. So it's showing a lot of information. And this was this graph here is kind of the first attempt at trying to capture all of that information. So they took this to the floor of the museum. They went to studio three, and they asked visitors a bunch of questions. And some of the interview questions that they asked were, in your own words, can you describe what you think this is about? So having open ended questions, is really important in this kind of work because you're just you just want to get people to talk to you and tell you what they're thinking.

Unknown Speaker 04:57
Another type of question is having them right Think things. So in this case, you know, there were specific elements that they were looking at. And they had them rank them from most, to least interesting. And then you probe why, again, you're trying to like, get people to talk to you asking if they found anything confusing, or difficult to understand, and then closing with kind of an open ended question of, you know, what else? What else do you want to share with me, so all of that, you know, the data that you're collecting are really people's words. And they're thinking through thinking through things. So one of the, you know, what made this particular test successful was a few things. In this particular case, the project team was working with an outside designer, so and they were also they had an established exhibition that was already complete, and they had the learning goals and the objectives already established. So they built the testing, they knew specifically what they were looking for. And they built testing into the schedule with a designer, which was not something that this company had previously done. But they embraced it. And they, they, you know, took the results of the testing, to heart and they built it into their, into their schedule, the curator already articulated key messages for each section. So that was what the team was measured, measuring against when they were doing the testing. Also, all of the team members were involved. So everybody understood why they were asking certain questions, everybody had a chance to talk to visitors and understand kind of the responses that they were getting. And everybody could defend what the results were of the of the test. And they took the time to plan, execute and reflect on the results, which is this, this type of work takes a lot of time. And you really have to, you know, build it into to the schedule, especially, you know, the analysis that you on the back end, it's when you do this kind of qualitative testing, you're not going to get obvious answers, and you have to take the time to sort of think things through. So this was the change that happened over the course of several, several iterations. And this is the next project that I want to talk about, which is more about kind of an exploratory evaluation. So in this case, we were looking at AR and we didn't really have specific goals or objectives in mind that we were looking at, we were just kind of exploring this new technology and trying to understand, could we use it to enhance some poster sets. So we have, we have some poster sets that are adapted from temporary exhibitions that are available electronically, and people print them out. And they can put them on display in different situations, teachers in a classroom and things like that, and there, but there's a lot of extra content that goes along with it. So there's some videos when we thought, well, is AR a good way to kind of deploy some of this

Unknown Speaker 07:51
some of this extra content, how would how would we use that? How would you know, visitors kind of experience the AR component. So in this case, we just wanted to kind of get some baseline data about how people were experiencing AR it with the poster set. So we weren't necessarily measuring against particular goals. We and so in that case, because it was so open ended, we did some observations, as well as asking people some interview questions. And we asked both about the functionality of the technology and AR as well as the kind of experience and the meaning that people were having in this in this experience. So we set up eight posters, I think the full poster set was like 20, some of the posters had an AR component, some didn't, you know, we wanted to know, would people just go to the ones that had the AR? Would they look at all of them? You know, would they watch all of the the AR content that popped up? Or, you know, videos of historical film footage or or history testimonies? Would people watch those things all the way through? So we had a whole bunch of questions. And here's some of the inner sample interview questions that we asked again, some open ended questions. What did you think? What were your impressions? Did anything stand out for you? Those are some of my favorite questions to sort of just get people talking and trying to understand how they're thinking and have them walk you through their, their experience? How did it feel to use the phone? Would you be interested in something like this? You know, preference questions can be kind of hard. Sometimes people just want to say what they think you want to hear. You know, people could say, Yes, that sounds great. And then when you actually build it and put it out there, they don't use it. So those kinds of questions can be a little tricky. I also really like Likert scales. So on a scale of one to seven, you know, how would you you know, how would you rate this experience? And then you ask a probing question why and you know, those two data points can be really helpful to try to understand, you know, how people are experiencing something. We also this is a sample of the observation mazing person on earth They have created this great checklist. So the combination of what people were doing with how they were thinking it through, kind of gave us this pile of, of qualitative data to sort through. So okay, so just some sort of practical tips about, you know, when you're doing this kind of qualitative work and working with your visitors, is to ask for consent, ensure confidentiality, you know, the script that you have, when you you know, do your, your ask is really important. You know, you want to make sure that you make people comfortable that they understand that this is anonymous. And, you know, that, that takes practice to kind of, you know, figure out how to how to introduce yourself and how to explain what you're doing. Also talk to a range of visitors, you know, don't just go for the people that look nice and make you feel comfortable. When you're doing this kind of thing. You want to make sure your sample is random, you don't have to do a statistical number of people. But it's important to, to make sure you're talking to a range of people. also test your instrument, test your questions out with a small number of people before you go out and and do you know your full scale evaluation, make sure that you're asking the question the right way, it's not too confusing. So just you just need a few people to test that out. open ended questions are great, and really listen and be respectful and take it as an opportunity. And a privilege that you are getting to hear people's stories and have them share with you what they're thinking. Google Forms and Sheets are actually a really great way to collect the data. I like to take notes by hand when I'm talking to people, and then I transcribe all of that, and putting it into a Google Form standardizes it, so you can send it to anybody on the team that that did the interview. And then all of that gets exported into a spreadsheet that you then you can sort through, take photos and document everything, you're gonna forget how you set something up, or what you were thinking, when you are looking at your date on you're interpreting it, every time you come back to people's comments, you're gonna have different insights. So if you code something a certain way, you know, you're you're understanding somebody's answer in a certain way, you know, document why you decided to, you know, categorize it in a certain way. So when you come back, you know, what you were thinking. And just, when you're doing this kind of work, be curious, be self reflective,

Unknown Speaker 12:30
critique how you're doing things and continually improve, the way that you are doing this takes a lot of practice, you're going to do some bad tests, on your way to doing really good ones. And there isn't a magic instrument that has like the perfect questions to ask each situation. And each scenario is a little bit different. So you want to make sure that you're, you're doing this work and getting better at it. So just sort of some concluding thoughts, doing this kind of qualitative work really builds capacity and knowledge among staff. So we really tried to build a culture of, of talking to visitors doing this kind of work so that our that the projects that we're working on the products that we're building are better, but also so that we can share this kind of information, just across the institution. It's always a good reminder, when you're talking to visitors that you are not the target audience, you know, you can ask your colleagues, you know, Hey, check this out, look at this, what do you think, but at the end of the day, you're serving a different audience. So just it's when you talk to visitors, it's just a continual reminder of you know, that they see things differently than we do on the inside of staff. Visitors are really generous, at least our visitors are really generous, even when they're rejecting you, and they're telling, you know, they're really nice about it. So don't let that stop you from from talking to your visitor visitors. I am an Off The Charts introvert, I find this work really, really hard. But I do it. Because every time I do I have amazing conversations with people. And it's always valuable. And it's always validating the work that we're doing. So I encourage you, even though it's hard to do it. And and it takes time takes a lot of electric a lot of time it takes practice, but you will have a better product at the end for it. So thank you. Okay, I think even with the tricky start, I left some time for questions. speaking into the mic.

Unknown Speaker 14:45
Thank you, Serena, could you maybe give one or two examples of how doing these visitor surveys has really helped you or has has caused you to change some of the exhibitions that you were that you were planning and I'm particularly interested in also when When you found out about the use of AR, but that's two separate things.

Unknown Speaker 15:04
So specific examples of how we change

Unknown Speaker 15:07
based on what you discover something, just evaluations,

Unknown Speaker 15:11
so I didn't put this particular one up. We are. So we're working on an exhibition right now on the Rohingya Muslim community in Burma. And we were trying to figure out, I don't have any, so we try and figure out which image to use as an attract for, for the entrance to the exhibition. So people would see it coming out of the permanent exhibition across the hall. And, you know, they might see a big photo mural and maybe a title. So what, which image basically, our question was, which image should we use to get people intrigued and come walk down the hall and come into this new exhibition? And so we all had our favorite photos that we loved, and we're like, oh, let's definitely use this one. This one's great. And, but we knew we had to test it. And the ones that we loved flopped, like, there was an image of a man, you know, looking out into the distance over the refugee camp. And, you know, we just thought that this was, you know, so poignant. And, you know, beyond the refugee camp, or the mountains of Burma, and he's, you know, in a refugee camp in Bangladesh. And without context, you know, people said things like, oh, he looks like a guard at the camp, or, you know, just other things, we were like, Oh, my God, I never would have thought of that. And we realized, Okay, we have to kill this darling. And use the one that you know, and it's still a picture, that's great, and that we love, but it wasn't one of our favorites. And so, that was again, like, you know, we had to put things in front of people to understand what they were thinking and how they were reacting. So that's kind of one example. And we can talk about AR

Unknown Speaker 17:09
I'm wondering if staff headings idea about sharing, like works in progress, or things that are in prototype phase? And if how, if you have any advice for

Unknown Speaker 17:18
getting past that, yeah. So there are a lot of perfectionists. And I think you just have to, you just have to sort of embrace it and get comfortable with it. And, you know, I think once you once you just commit and decide to do the work, the results that you get are so valuable that you that it'll make up for any sort of anxiety, you feel about putting something out that's kind of have to bake, and it really does make your work better, and it doesn't have to look sloppy. And you know, to be fair, some of these, the first one that I showed you, with the Americans in the Holocaust, those prints were huge, and they actually cost a lot of money to print. But we didn't have to do it that way. And, you know, sometimes just printing something out large scale is enough. And you know, when they were testing actually the text for that, they realized, okay, the text in this caption isn't working, they went to the printer, they printed out a new thing, and they just like pasted it up. People are actually really, visitors are really, really forgiving. You know, if you introduce it, as you know, we're testing something out and you're helping us out. It doesn't need to look perfect. You're there all the time, you're not really walking away from it. So there's always context for for anything that seems a little half baked when you're doing it.

Unknown Speaker 18:48
Hi, I just had a question about the staff you use to approach and interview visitors? Were you using existing floor stuff? Or did you have to hire out and train different people?

Unknown Speaker 18:59
So it was kind of a mix, actually. The the sort of the project was defined by the staff that the project team, but then we trained people, if we needed extra hands, we just trained people, we explained kind of the context of the project, you know how to ask the questions. So it can be a little tricky sometimes because you know, there's sort of a lot of nuance, especially with the follow up questions and how you probe. So sometimes people just do sort of more of a checklist like I've asked this question, I've written down their answer, and I asked the next question. So I think how you script things and how you train people is important. But if you need extra hands, you know, absolutely, definitely you just have to give some context to what is it's not like here's a survey, go administer this. It's a little more nuanced and that

Unknown Speaker 19:59
I apologize if I'm As this I'm curious about the inception of the room itself specifically like was that, like my institution? We're even though we're we have a pretty big campus, every square foot of kind of spaces is demanded for and I'm curious how you even made the case to say, hey, let's carve off 1500 squarespace just based off that I don't even quite know what to tell you, you're gonna happen. Yeah, thank you for awesome

Unknown Speaker 20:26
for asking that, actually. So this, the space that we used was in a gallery that has that's usually has temporary exhibitions in it. So it's kind of a corridor for rooms with a hallway running all the way through it. And it's sort of it's one of the sort of exits out of the museum. So we come out of the exhibition, and there's a couple of ways you can sort of leave. And so sometimes people walk through this hallway because there's exhibitions in this space. So one of those exhibitions was closing down, and the rooms are became available. And we had sort of been pushing for a space where we could do this kind of work. And so they said, Yes. So it was a 800 square foot room. And, you know, that was a little bit limited. We didn't, but basically, it was an empty box. And we bought some furniture that we could move around, like little poufs and benches and things. And so it was actually fascinating to see what people did with the movable furniture because you walk away and people actually like, did their own little, you know, nukes and things. And one kid came in, he was like, Oh, my God, I want to build a fort and like wanted to pile them all up. So. So yeah, it was definitely a luxury. It's going away in February, probably because an exhibition is going to be going in there. But But yeah, it was it was but you don't need a space. You don't need a space to do this kind of stuff. We just got really lucky.

Unknown Speaker 22:07
This is only slightly related. But I really need to know what your results are from AR poster testing.

Unknown Speaker 22:15
I don't I don't have those off the top of my head but we let's we can talk. Okay, so I think we're at time. Thank you all. I will be here for a little bit. Do you have any more questions? I think there's coffee. Enjoy the conference. Thank you