Using your website to support the visit experience, before, during, and after

Unknown Speaker 00:00
Hello, everybody. And without I think we'll get started. Thanks for coming to this talk mouth fully titled, using your website to support the visit experience before, during, and after three case studies in a post AP landscape. I don't know if that gets a woo, but I'll take it. My name is Colin Brooks. I'm the senior developer at the Whitney Museum of American Art. And I am pleased to be joined by Dan Michaelson partner at the design and technology firm linked by air and Boris Shea who's Senior Manager of user experience at the mat. And these presentations, we're going to be talking about three pretty different experiences at the Met the shed and the Whitney respectively. But they all do have in common and focus on the visit experience and explaining both the art and the institutions themselves. And with that, I'm going to hand it over to Bora. Great.

Unknown Speaker 00:51
Thank you. Okay, so we're going to talk at the Met about the primer, which is a new digital tool that helps people prepare for their visit. And we took this approach of relating to people before they come to a museum and why should someone care? And what should they care about? Museums are oftentimes overwhelming. And we know we know that, because people often come and don't come with takeaways. So what can we do to prepare people for their visit in a meaningful way in an accessible way? How can we be inclusive instead of exclusive. And as we know, stories are something that everyone connects with desire for storytelling is hardwired in our DNA, of storytelling as a design tool that can help people understand themselves and understand art. So we took this approach of storytelling, to spark emotion, make make sense of things, and hopefully have people learn something before they come visit. We know that storytelling increases audience engagement and loyalty. So we wanted this digital experience to enhance people's visit before they come. But hopefully, after they visit, they can also engage with the content. And as content consumption becomes more sticky. And as we spend most of our times on our phones, how can we create something that's meaningful? And why should people care? Why should they come to this website before their visit? Why is this important for users. And we came up with a hypothesis of reinventing the digital storytelling experience with a new framework that can support support and multiple content verticals, we wanted to create a tool for producers and editors for this primer that could be used independently of a product development team. So ideally, creating solution and immersive solution add a toolbox of content modules that they can reuse. And these would be user friendly, obviously, but they can be the visitors primary source of preparing for their visit. So before doing anything, obviously, we looked at the comparative landscape, what are other institutions doing? We're not the first ones to take this approach. I'm sure storytelling is like the word that everyone's using. And how do we contextualize that, but also, where can we look outside of the museum space? What do people use on their daily life and what other digital tools do we also use that we can replicate behaviors, we looked at the fine arts museum where they do storytelling in a similar way. And these are beautifully designed, holistic looks at their exhibitions called the insights. We looked at the Van Gogh Museum, or here, it's another storytelling tool where they're learning about Van Gogh and his brother. And we also looked in the media space. So this is an interesting example from the Huffington Post's where they're using storytelling and parallax to showcase big moments. So they use large images, and long format text to tell the story. We also looked at spot as something like Spotify, where they have this experience of choose your own adventure. Choose Your Own Adventure user experience is sort of becoming the thing with Netflix and Netflix and the show Bandersnatch. And I was going to say, where users can choose their own path and understand and learn at their own pace. So how can we combine all of these experiences into a product that makes sense for our users? And we took the initial approach with content that was approachable to first time visitors, we wanted to create flexible technology, where the goal was to a pure appeal to younger audiences. The Med med visitor is the Met member is ranges to an older one, but the med visitor on the website leans younger. So how can we appeal to those people to actually come to the museum? And how can we give people at least three main takeaways before they visit by creating three tiers of engagement based on their time investment? So we came up with some product goals, which was prepare prospective audiences for their experience of the Met in advance of their visit, attract and engage visit planners And again invite users to choose their own adventure. And then we looked at our audiences, and obviously our audiences leans from the curious to the informed, we have audience that as a casual browser, we have the art enthusiasts and the professional. And we focus on where can we have the most impact? Obviously, the informed will know a lot before visiting. So our largest impact would be with a curious visitor. So how can we create content that can have the largest impact for those who casual browsers, and

Unknown Speaker 05:33
it doesn't mean that we're excluding the informed of the professional user, but we're including everyone in this circle. And in creating three tiers of engagement, we took this approach of skim swim and dive in the skim experienced, you'd have a simple scroll through where you can just quickly for 10 seconds, we know that the average user spends eight seconds on a website. So how can we get people to get three takeaways for in 10 seconds, what can we do? I've forgot to mention that the primary goal of this product would be a mobile device, since most people are checking out what's happening at a museum the day before, when they're walking, they're usually on their phone. So we wanted the optimal experience to be on a mobile device. So we focus on that 10 Second, scroll through most likely, like we do on Instagram, we wanted in the content shot to do to get the top level highlights and three main takeaways. And then in the 30 seconds, if you have 20 more seconds to spare for that art enthusiasts that maybe got this in an email users engaged with other modules that are presented in this primer, and the additional layer of information on the topics. And then if you're a committed met member, you have two minutes to spare or more, you can really deep dive into the articles. And these are a numbers that, you know, we are aiming for. But as we know, from eight seconds to the average, two minutes and something seconds of people spending on our site, we figured that we can tackle these three groups of users. So we came up with this initial prototype. This was just a hypothesis we this is not real content, the prototype was just meant to give an idea is oftentimes when you create these concepts, it's hard for people to gauge what they're talking about. So we took one of our exhibits, and created a prototype, which I will play here. And in this prototype, this is meant to showcase what the primer would be like and we want it to be as close as possible as the real thing. So users can get an idea. This just shows the prototype in action. But it was basically an envision prototype that we got to test with real people. And here it shows different levels of engagements what it would feel like if you wanted to interact with this, how does it give the feeling of the storytelling combined with Choose Your Own Adventure UX?

Unknown Speaker 08:03
And we also user tested the name so we came up with a lot of names, there was a lot of opinions choices, what do we call this thing? Why should it be called a primers because it Prime's you for the experience? What's the first look? What's a millennial thing to call it? Firstly, what's that was my favorite, but it didn't pass? What are these things that we're trying to tell people and we went through multiple rounds of user testing. And we're trying to get the most contexts not preferential study. And I'm gonna play some a clip of user testing up. But you'll you'll know, in user testing, well, we really focus on now go back here. In looking through preference versus people understanding context, a lot of people liked the word the essential or people thought, hey, I love to look to call this first look, we were on looking at that we're looking at context of how people understood with the word primer, no one had any confusion. So we went in that direction, because that was the path of least resistance. And it had the greatest context. So in really understanding how people understood the Word, and the content they were presented, we chose primer also because it applied to this priming you for your visit. And we thought it was applicable to that. And so so far, we've launched three of them. We have launched one for plate, an exhibit for playing loud, which was instruments of rock and roll, one for an exhibition that's been going on for two years in praise of painting. And the latest one is the last night which is Maximilian for emperor, we took three different approaches to these. The first one was our sort of MVP, the minimum viable product where it was simple, we the content itself was lovely because there was a lot of rock stars and faces and beautiful videos. So we've got a lot of interactive components and IT people engaged with it. So we looked at The data and we thought, how can we improve this? And how can we make some hypotheses on what people really want to learn. And we created this. The second one in praise of painting where we had additional content already, we had a lot of audio guide content. And we had a lot of beautiful images from the Dutch masterpieces. So we made the hypothesis that people really wanted to do some close looking. So we created these videos where you listened, and you looked closely, and we thought that these would be beautiful to share and people would engage with. And our assumption was sort of proved proven wrong, because not a lot of people engage with it, people most likely don't play sound. And we saw that even though it had captions, most people skip through the experience. So for each primer, as we looked at it, we saw how people interacted through one through qualitative user testing. One through looking at scroll maps and seeing how far people scroll through for play it loud is relatively short, and for him praise of painting was much longer, and we saw that people abandoned. So for the last night, then we took a different approach. And we streamlined the the content. So in for creating these three primers with the similar approach. And then for every single one learning what what people were interacting with, and pushing through to create experiences that were meaningful to people. We had this assumption then in Swim, swim and dive, we'll get three solid groups of people. But we saw that most people were in the swim area. Obviously, everybody's skimmed. But most people weren't interacting with a dive components in and what I mean by a dive component were either long format, text or videos, long videos, and then play it loud, we had about six essays. And in praise of painting, we start, we only did three. But again, we saw that people weren't interacting. So for Maximilian, the emperor, then we just did one deep dive. And we really focus on that skim experience, because that's where people are. And that's where they're spending most of their time. So I'll show here some features that we use here, I was talking about the video player, and this is a module that can be used in different ways. So it can be used for videos and playing videos for content that we already had, or for that close looking. This is showing for the flexibility of the modules, but also the ability to create content, not based on the module, but based on the intention of the producer. This is one of our most successful modules, which is called the flip card, similar to when you're on Spotify. And they used to have this UX where you were looking at a song and you could flip and you'd learn more about the song. And it proved super successful people like to interact. So what we're finding through our data is that our most engagement comes from users having something to interact with, they want sort of a stop sign where they can have the ability to click and learn something.

Unknown Speaker 13:04
And then this is another one of our those interaction spots, and it's called the artwork hotspot, Here users can click through and look at the hotspots. And each one of those hotspots zooms into the image. And you can learn more details about it. In this particular example, this is life deaths side by side. So I discovered it talks about the themes of life and death and how they're represented in these paintings for Dutch masterpieces. This module is again, we use it for the last night primer and it proves to be that this is where people actually interact, even on mobile devices. They get to play around with this. So how are we doing? We've got about 93.5k unique users on the primer so far, and almost 250,000, unique pageviews we are attempting to do better. I wouldn't say that we're doing great, but I think we want to reach people where they are people come bless them to the Met website and look at it. And I find that very humbling. I think it's great that people are actually accessing and. Okay, and here. So something that I wanted to call out, I think this was really interesting. We had our hypothesis that 62% came from mobile devices. And this continues to be true. In fact, for the last night primer, it went up to 68%. With 42% coming directly from the Met website. 83% of users come from Google, followed by Facebook with 9.5%. So we see the traffic is coming from Google, but I think it also solidifies that a hypothesis that for younger audiences, which group Facebook and Instagram, that's where our users are coming from. So what are users doing what they're there for play it loud. We saw the most engagement with us Good to experience. So once you saw those flip cards and the videos, that's where people were for improvement in praise of painting, we saw low engagement with the audio content but consistent focus on the interactive features. So for the last night, we saw higher engagement for the swim modules, as well as the dive experience. This was very interesting, because the skim experience was super short. But once people got to the end, because it was short, and because it was interesting, they decided to go and read that long form content. And for what we did was once we did a lot of user testing and understanding how people were engaging with the content, and we streamlined and created a more compact experience, we also started to focus on the voice, we took a more playful approach into explaining the concept of Maximilian the first and propaganda that it had a point of view, something that maybe you'd read on, let's say, New York Magazine, and that kind of tone, showed higher engagement. So once we we compared, time spent in the on our primers, the first primer was one minute and 57 seconds. So people came there and stayed for that long. The second one was three minutes and 14 seconds, even though and I'd like to say the Dutch masterpiece, one was the longest. But the last night one, which was the shortest people are spending three minutes and 47 seconds. So for shorter experience they spending the most time and our hope is that they're not only engaging, but taking those main takeaways. And here are some initial user testing big loud and exciting. So hopefully people, people are giving us positive feedback. But I think most most important is that they're understanding the context of this content being presented to them. Thank you.

Unknown Speaker 17:05
So I'm going to talk about a feature of the sheds website called at the shed. The Shed is a new cultural organization, also in New York. And it's it's a kind of a hybrid organization. It's both a contemporary arts museum, with several exhibition galleries. And it's also a significant Performing Art Space in New York with several theaters of different sizes. So at the shed is a feature of the sheds main website, our role was to design and develop the sheds website. And on that site, there are two tabs on the homepage. Basically, I'm not at the shed, which is the default. And I'm at the shed, where you can indicate you can kind of self identify as as being out the shed. And so it's that second tab that I'm going to talk about today. The moral of the story or the hypothesis that I want to focus on, is that when we really make an effort to work, cross functionally, to integrate not only the website, but also the way that visitors experience signage, ticketing, emails, in other words, pushing to really integrate marketing and visitor services at a museum or at a cultural organization. Then visitors and also prospective visitors feel welcome, they feel confident and they are engaged.

Unknown Speaker 18:34
Imagine building that drives. So the shed famously is like a large box with an equally large canopy that can roll out and it is mostly rolled out and that forms a theater. The reason why that's important for this talk is that you don't necessarily know where your entrance is. It literally can move around. And different programs, different different theatrical performances or shows can involve different ways of entering different ways of queuing. These are questions that we hear if you in the lobby basically I just walked in what can I do here? So the shed is sort of on Hudson Yards, which is a huge new tourist area in Manhattan. And it is in part a museum. But unlike some other museums in New York, you need a ticket for every separate exhibition. So there's a really sort of wide range of programming strategies at the shed from you know, seeing a play or a pop concert which obviously you need a ticket for two contemporary arts visual exhibitions, which you also need a ticket for. And some of those tickets are pretty affordable but you do generally need a specific ticket. So helping people with That kind of basic confusion, I have a ticket, but I can't find it. So the shed is, in theory, all digital, theoretically, they have no box office and you buy a ticket on your on your phone, when you're there, or from home on your computer. In practice, some people do really expect a will call. And so of course, there is a way to have someone help you with that. And so trying to sort of explain all those subtleties, is a challenge for visitors, services staff for signage. You know, when we started working with the shed, one of the first things that we did was talked to a really wide range of possible audiences, and is really important to the shed to be inclusive, and to reach out to audiences who might not normally go to contemporary arts museums at all, or go to the theater, let's say. So, this is just one example. But one thing that we heard from like all kinds of demographics is people wanting to know if there will be people like me there when I get there. And it's, you know, it's like practical questions or quasi practical questions like, What should I wear? You? will I will I, how will I be comfortable both socially and physically? But also, just how does it work? How, you know, how long is the show? What's the intermission? What are kind of what happens if I bring a stroller? Where do I check my coat? And so we delivered on that in part by showing lots of pictures like this, quote implies, but also in more practical ways, which I'll show you in a second. So I guess sort of the question that we asked and sort of indirectly answer that person and others like them, is whether the sheds website is its primary website can model a journey to the shed. Thing, pretty sure this one's going to work. So here we see, starting from the default homepage, I'm not at the show, and the site is kind of personalized. So if you have tickets to something then notifies you about that, and kind of conversational, sort of concierge voice. The the nod of the shed homepage has sort of cards for each event. But if you have a ticket to that event, then that card is represented as your ticket. So the QR code just sort of appears on it shows a really great open call program for emerging artists. So you here we've tapped I'm at the shed. So let's say gone. And in this scenario, I have a ticket. This was actually this, I did actually have this ticket. And so it welcomes you, you have a ticket wallet here. It kind of prompts you to see the digital program, which gives you more information, sort of like a playable on your phone. And then for everybody, whether you have a ticket or not. There's sort of basic information about amenities. We also show the directory at the bottom, so everything that's happening today, and that directory is personalized. So if you have tickets for a particular thing, it indicates that and sometimes that's a complicated story. So you may get tickets for one event, and that might come with free tickets for some other exhibition that's also happening or you remember at a particular level, and that gains you access to three of the five things that are happening. And so that's indicated here as well.

Unknown Speaker 23:38
So just sort of taking a look at some snapshots, as you kind of use this site. And this all show mobile, but all these features are actually also available on desktop. So this is sort of the state if, if you don't have a ticket, then it kind of explains to you how that situation works in the lobby, and kind of importantly, prompts you to log in because you may have a ticket and we just don't know it. If you if you purchased it on this phone, we would know it but if you purchased it on a different device than then you would need to log in. So this is digital signage in the lobby. And I think one important point I want to make is that partnership and collaboration is really key. So we didn't design or develop this signage. But the the website, content management system does drive the content on this signage. And we worked really closely with the brand designers other means and with the signage, people or volume to create an experience that it feels very cohesive. So that same directory that you encounter in signs is also very consistent with the directory that you hold in your hand but this time it's personalized. So one of the things that happens on the app, the shed homepage is it features what the shed calls in the works content, which are sort of little mini videos or posts about stuff that's happening behind the scenes that the shed right now in the shed, Commission's newly commissioned almost everything that it shows. So it's pretty interesting to see what they're working on. And so we can tie that to the membership, for example. So use this as a call to action to become a member, or thank you for being a member and helping to produce this kind of content. And then that same in the works content, you may also encounter on screens around the space. So, you know, I think that by holding this content in your hand, there's also a sense of really kind of being a part of the shed or being a part of the building. Even beyond the sort of literal calls to action. Something that we're working on right now is for an exhibition that's opening a couple of weeks called manual override. And that's it's an it's a visual arts exhibition organized by a critic named Nora Khan. And she's really interested in sort of exploring the bounds of like privacy, and also in delivering images and documents to visitors while they're at the exhibition. And so that is something that this platform is sort of pretty well set up to do is to create not only this kind of general concierge voice that guides you to the institution, as Colin sort of, nicely introduced it as, but also at times the more specific experience, that could be a sort of a hybrid, digital physical experience in a particular show. So here, you see someone who is logged in as a member. So they're saying their membership card, we can thank them for kind of tie that membership to sponsorship of these free documents that everybody can have. And then also sort of promote the exhibition on the second floor. And then when you are there, there's this kind of fun, or not fun, quasi AirDrop that an Archon will sort of deliver a PDF to which you can enjoy

Unknown Speaker 27:16
whether you like it or not. So what were our goals in building this feature, generally,

Unknown Speaker 27:25
from the beginning, actually, originally, there were three tabs, I'm not at the shed, I'm going to the shed, and I am at the shed. And these sort of actually map pretty nicely to, to the discussion at the keynote this morning of a framework of, of have can connection, empathy, and an empowerment. So when you're not at the shed, we kind of want to extend a hand and ask you to come and make it really clear and easy how to get tickets, actually, we've intentionally tried to make this website feel more commercial than, than your average Museum website. And we argued that that was a way to actually broaden our reach to do additional demographics and audiences, and was by sort of falling into a pattern that that people are familiar with. And so by saying, we like we want your business, we want you to come and here's how to, here's the button, you click to do it. We then want to make it really clear how to navigate sort of the complexity of getting into the shed and getting to your show. And then once you arrive, we would like to potentially, we'd like you to become a member, for example, to engage in that way. We'd also like you to find out about other shows, and the one that you want is that you were intending to come to, or to, for example, watch in the works content. So this sort of gets deeper into how we sort of adapted those brand goals of like just being welcoming, most fundamentally and also being innovative digitally, with sort of more functional goals of engagement and even revenue. That could sort of what started as kind of a brand idea, then becomes a way to create functional outcomes. How's it going? So one thing that we kind of imagined from the start in, which I may be hinted at is that lots of users would visit at the shed while they're actually at the shed, but lots of other users would cheat and take a sneak peek from home. And that's why we have such a simple choice at the beginning. Actually, it's not location. We don't ask for location permission or something like that. But we want you to be able to use this feature to understand what the experience is like when you arrive and to really sort of walk through the steps And that is approximately how it's borne out. So about 7% of all website users use at the shed ATS as it's abbreviated here. At the shed users are more often in New York than then users who don't try out the shed, they're more often on Wi Fi. So far, they don't generate more revenue in terms of membership or tickets than people who never press the watershed button. But they do convert on other sort of engagement goals more frequently, like signing up for email or watching videos. So you know, something that I would still like to work more on. Or an interesting point is that it's pretty clear that most users as of now still show their ticket using Apple wallet or email, rather than using the mobile website features that I just showed you. It would be great to keep on developing strategies to make sure that visitors know that they can use the website to show their ticket. And some do, obviously, because if they are doing that, then we can talk to them about other content that they could engage with. On the other hand, of course, we want to meet visitors wherever they are. And if email is most convenient, obviously, we want to support it. In Florida survey visitors find at the shed useful staff and this is I think it's sort of a general caution from vendor to museums or from vice versa. Staff find and sort of why that the future is maybe sort of slightly out ahead of content production. In other words, not all the things that the that at the shed can do are being taken advantage of yet by content authors. And this is also a, you know, organization that just bootstrapped itself and literally just got done, or is getting done with their first season. So now's a great time to continue to iterate and we have already iterated. But that sense of like this feature being sort of one full potential. But not all that potential is realized yet, it's probably somewhat tied to some staff report. And they would prefer that the tabs are like slightly less present or a little bit more subtle. Qualitatively. Our advice, my advice, to wrap up, if you're going to try to design a journey that is integrated like this, then it really does require close collaboration among multiple departments, at the institution among multiple vendors. And everybody doing their best to work on shared timelines, shared specifications, shared goals, that takes a lot of effort takes a lot of intention, it does not happen automatically. And it also requires continual iteration. But I think it's the right way to think and I think it's brought a lot of benefits in terms of outcome and also just in terms of process actually just in terms of building ways of thinking synthetically across the institution from its very first days. Thanks a lot.

Unknown Speaker 33:41
Hello, so I'm going to talk about the Whitney mobile guide. And this is going to jump around a little bit. But I'm going to start off just by being really clear about what I'm talking about before getting into the kind of hows and whys of it. So what is the mobile guide, it is a web based experience for serving interpretive content in gallery, think audio content, sound descriptions, ASL videos, that kind of thing. It is a holistic replacement for a previous iOS app, and audio guide rental devices. And it is an extension of whitney.org. And to really drive that home, it is here, this is where you go, this is where it lives. It may feel like a different experience, but it is fundamentally a part of our website at whitney.org/guide. So a little bit of context for the Whitney two, we post a lot of exhibition content online. We do this for a number of different reasons. One of which is probably because we work with a lot of living artists. And so there's a desire to provide a lot of contracts around what they're doing and their process. And related to that we put all of our audio guide content online and have for kind of the modern era of the web for the Whitney going back to 2009. connected to that our content needs to be as accessible as possible. And I mean that both in the sense of if you can make it to the Whitney or not make it to the Whitney you should still be able to To hear our interpretive content, and there's also accessibility kind of needs represented by the audio guide that are really important. And lastly, especially as like a midsize institution that used to be smaller and is getting bigger, like our processes really need to be sustainable, we still are fairly limited, like everyone is in terms of resources and staff. And so this is particularly at the forefront for us. So where were we pre mobile guide, in 2018, we had an aging fleet of audio guide devices and experience, the contract was coming up for renewal. And kind of across the institution, there's kind of the acknowledgement that these had really low take up rates, there was a lot of fatigue around marketing for it. So we kind of weren't doing it. None of these things combined in good ways. We also had recently launched the year prior a new web content management system that we built in house. So it's in this ecosystem that we had to kind of figure out what is the next step for our audio guide experience? And so what did we consider and I would say we considered basically everything. This list may look very familiar to some of you we trialed D two or a time before it ceased being a company. We thought about building a native iOS app, we thought about building an iOS and an Android app, we considered all kinds of hybrid and wrapped web sorts of experiences. We considered a web app or progressive web app. And we also considered doing nothing or running back what we had before. Neither of those options felt particularly good, but they are on the table. I would also say, and this is even slightly out of date. Now. There is the kind of reminder of what are all of our peers doing and a ton of these apps are no longer in existence. And it definitely was kind of feeling at a certain point, like if these other institutions, many of which are larger than us can't justify running a native app program like, why should we. So that was definitely floating around at the same time. So what did we end up choosing, we chose a progressive web app. And I don't want to kind of really deal too much in what makes up a PWA, a PWA versus just a web application, there are a couple of added benefits to them, which are represented here. But one of the biggest reason we ended up doing this was the web is widely supported. And since we already have a website, there is a shared code base for all of the platforms that may access when needed org slash guide, as we ended up kind of making it. And in terms of the PWA aspect of it, and the particular features that come with that there's growing industry support, kind of between iOS and Android. One of those features that was particularly appealing for us was kind of limited offline support. Moving to a streaming model, which is another kind of unspoken piece here. There are consequences for elevators and strange Wi Fi situations in the building. And so having a little bit more robust caching setup was really appealing. Another really big one is this would be accessible. And I don't mean that and that native apps aren't accessible. I mean it in that if you came to the Whitney and rented a device, you had no option to toggle on the screen reader functions. And we didn't release the app publicly in the app stores. It was an enterprise app. So functionally, there isn't screen, no screen reader support until we move it to BYOD.

Unknown Speaker 38:22
And that last point, BYOD, we still have rental devices. But the idea would be you could use your own device and get the exact same experience. And if you had any accessibility features turned on in your phone, etc. You could use those. And lastly, no download required for a website. It is still a tall order to ask people to download something they're going to use once. And we wanted to avoid that. So how did we launch. This is a very iterative process really even longer than this timeline kind of implies. We launched the bulk of the kind of bring your own device piece in the summer of 2018. Or we still had the old rental devices. But we started gradually pushing people towards using whitney.org/guide, the mobile guide. Technically it was out before that even quieter when we didn't sign at all. And this is prior to switching out hardware in the fall of 2018, where we had a kind of wrapped version of this on a piece of rental hardware. We gradually increased physical signage as I mentioned, down below, you can see one of the stanchions by our audio guide desk on the first floor stream for free when you go to org slash guide. And then next to it are kind of updated ticket stock that has a QR code on the back so you can scan and get into the experience. At the same time, we're also doing user testing with both specific and general audiences to adapt the experience in advance of kind of the full rollout in that fall. So what went well, the big thing is really the app design and architecture and there are a number of reasons for it. But the biggest one is just by bringing this in house. We had the opportunity to update an experience that hadn't been updated in years that was limited by third party platform on issues and everything else, so we really got to reformulate this without drastically changing the kinds of content that we offered. But to kind of provide access to them in kind of better ways in ways that were more reflective of where we are going with translations and other kinds of pieces, our of our interpretive content. You can also see on the far right, there's a I believe that's a sound description, or if not, that should be, but we did have some new kinds of content that we did get to launch that just wasn't the primary focus. But broadly, this is something that people were really happy with. By being internal, we could do what we wanted to do. And by also building off the existing CMS architecture, really, this was a really big gain too, because rather than have to work with a third party to upload content, and QA, that third party content, it's all done in the same content workflow that we were already doing to get our audio guides online kind of forever, as I've mentioned, sort of at the beginning. So it really shrinks the QA process and means that we don't have to work with somebody else's often. Another kind of big win. And this is going to be really confusing to look at, but is audience growth. And I will say the challenge with measuring any of the success is that we are so boom and bust based on exhibitions, that the Warhol show that we had really constitutes its own kind of bump. And so trying to look at periods before and after that are a little bit tricky. But broadly, we increased our kind of online guy, audience, mobile guide and audio guide, kind of old deal on the website by more than twice. And eight times since kind of the year prior in 2017, we drastically increased the BYOD rate, basically, since we didn't push it at all before and the experience didn't make any sense when you were on site. And we also improved the in building ticket rate pretty drastically, two and a half times over the course of 2018, and more than four since 2017. So the overall take up rate is now still consistently twice was what it was. A while back. Again, it's really complicated because of all the exhibition cycle stuff, but that it's kind of a win, no matter how you look at it. Staff support, internal audiences are really happy with the experience, possibly, again, because this thing hadn't been updated in so long. But a really nice side benefit of that as Visitor Services staff were way more interested in pushing people towards using their own devices, in appropriate moments when the lines might be really long for the rental counter, or the rental devices might be having issues or whatever else because they trusted that the experience was a more positive one. And similarly, and somewhat specific to the Whitney, our gallery assistants who are on the floor can direct visitors to the app without them having to return to the lobby where our rental counter is, which is a really big deal on an eighth floor museum if you might be on the eighth floor and not want to wait for elevators to go all the way down and back up. So that's also a really big one. And we also have signage, and Object labels that point people to this interpretive content and encourage them to get into the mobile guide.

Unknown Speaker 42:53
Another big benefit is by being based on kind of the web, we have to use our standard suite of kind of analytics tools and platforms. And for the Whitney a big one is Google Data Studio, which is a much kind of friendlier amalgamation of our Google Analytics and YouTube data. But it's been really, really great compared to the old kinds of reports that we get to be able to see what guides are performing well, what kinds of content is it the kids guide is it the kind of general stuff that we make what are people listening to, who's toggling the language settings and all these kinds of pieces also broken down by date by who's in building who's out of building session durations, how many stops all that stuff is really, really accessible by bringing it into our just our web analytics frameworks. And kind of where this links up to the before, during and after piece is, I haven't really been sure how to refer to this. But our kind of traditional audio guide section on the site where we just have everything since 2009. And we have done kind of a couple of different pieces of surveys and analytics, looking at other things to figure out that around 70% of people who are in this section have reported seeing the exhibition they're looking at content for and also around the same number happened to be outside the museum, they are not looking at it on their phones. And that kind of the obvious thing that it says there is then that people are looking at this after they have ended the Whitney. And so by building this whole mobile guy thing off of the same infrastructure, a ton of the UI improvements of the kind of language handling improvements. They all also filtered back to this piece, which is by users the largest part of our largest audience have our interpretive content in a kind of digital space. It's great to think of the mobile guide is the thing that you do at the Whitney but really it's the after you visit thing that is most people, even if they tend to be listening to only a couple stops. In terms of what we would do differently. anymore. More extensive Wi Fi testing is a big one. Being in New York, in particular with glass walls on your lobby means a lot of interference that we didn't necessarily plan for I would also say somewhat connected to that we would push to have less rental devices. The BYOD BYOD aspect of this was kind of far more successful and smoother than I think a lot of us at the institution were worried about. And we could have gotten away with having a way smaller fleet of these fiddly things that you have to manage. And I would also say clearly delineating responsibilities between departments. By bringing more of this in house, it doesn't necessarily solve everything, because there's still that kind of funny problem of visitors are using the devices, visitor services are getting the feedback when something goes wrong, and then that has to propagate back to us in digital media. And then the whole thing has to kind of go back the other way. And that could have been a little bit smoother, despite kind of the work that we did to make that work, it still was a little complicated. And with that I am done and open it up to questions for anyone. Forget how to,

Unknown Speaker 46:00
like on the ship app?

Unknown Speaker 46:03
What's the workflow for determining whether or not there are any prompting them? Are you? Like, where is that system? Like? How's that?

Unknown Speaker 46:13
It's connected to the ticketing system? Yeah, so it stores cookies basically. So if they if they're logged in, then we then we know they're a member, they bought the membership, the membership, pretty much can only be bought online, and that's stored in tessitore, which then means it's going to notice people like sanctuary, that would be like overhead that I wouldn't even want to participate in, if I have the ticket in my email to login

Unknown Speaker 46:39
to the site, like next group, all that stuff. Have you do you have any knowledge about how well that's working about people actually logging in? Right?

Unknown Speaker 46:49
I think that experiences they're engaging with? Yeah, no, I think that's, there's room for improvement there. Definitely. And exactly, I mean, it's not like there's a pop up that says you have to log in or something. So lots of people don't log in. So there's definitely lots of people who are there remember, or who have tickets, who are at the shed, who are who are not using this feature, basically, or might be using it, but haven't logged in. So, you know, we've ways we've been iterating on that, you know, we've made it more prominent on the app, the shed website that, you know, if you, if you are a member, or you have a ticket, here's a giant green button, you can press in the future, I have a dream and a written specification for like a magic Login link that could let you log in with, with one click from your email. But we're not there yet. But that's definitely you know, a gap. That means that some users aren't taking advantage of all the features that they could be taking advantage of. But like I said, on the other hand, they're getting it done anyway. So

Unknown Speaker 47:57
mine is everything a lot about this for getting rid of our mobile app at the end of the year. And think about ways that we can make the website more mobile friendly, and do a lot of things you guys are talking about. And it seems like in general, museums are kind of moving post after now. But I also wonder, like, at what point, you know, thinking about meeting people where they are, you're talking about Apple wallet, and something that people are just so used to using, you know, if, like, we should also be thinking about like Post website. And you know, like we get, you know, four times more people looking at our museum on Google My Business than we do actually going to our page, you know, and then with like schema and our events showing up on Google and all of that sort of stuff. Like so many of the interactions that are happening in things like with snippets, and Google Now and all these things, like tons of interactions with our museum or not even on our website. So I'm wondering, like, if, if there should be something at work, you guys are thinking about ways that you can start integrating more or just thinking actually, like, what happens?

Unknown Speaker 49:03
Are you asking why should we have one sec? No, no, no,

Unknown Speaker 49:06
because clearly, like we're thinking of doing what's the exact same thing you guys are doing? And it's clearly very important, and we still get tons of people visiting our website. But just thinking about sort of like, I don't know, is there? Is there like a post stage to all of this?

Unknown Speaker 49:24
I think it's both and, and, and, you know, some of that is like a search engine optimization question. And some of it is a social network question. And some of it is also is it like an email? You know, strategy question. I can say is like, I think you get the best outcome. If you think about that. Just try to think about that all as part of one story, you know, and have a shared vision across all of that. And sometimes those things, you know, one thing leads to another like, you get lots of traffic to your website that's driven from, from social or from email, obviously, and a lot of that is on mobile, for example. So thinking about all of that as B In part one journey is sometimes helpful. In the in part one voice definitely.

Unknown Speaker 50:03
I would also just definitely say like, as somebody who has feels like I'm always kind of looking Oh, is there another more standardized service for a lot of this stuff like one of the continual hang ups with our kind of interpretive content is that something might make a ton of sense for audio, but doesn't make sense for ASL videos or sound descriptions. And it's like, I have the appetite to separate those things out when it's like meant to be like a holistic kind of you take what you need to interpret the art in the gallery, and tell Spotify or the content management systems that other orgs are putting together and trying to share, like, handle all that stuff. Like, I still feel like there's a really good place for doing this in house. But yeah, if there was something great that like everyone used to get all this stuff in a standardized way, awesome. I'm first in line.

Unknown Speaker 50:53
Oh, pretty good. I would say some of them work better than others, the ones in the back of the ticket stock were far and away the most effective, we also put them on walls and in the museum guide and other kinds of things. For a while there was some insane thing where it was like a quarter of all people entering the mobile guide, we're doing it via QR code. One of the things that I will say was especially effective though, was it was really, really easy for visitors services to take that and say rather than for especially visitors who might not be super adept at typing in URLs, like hey, just point your camera at this and it'll work. So it's still like a kind of mediated experience, but like it's way faster for everyone involved. And that kind of turned out to be worth a lot. The camera yeah, or even if they don't like you can tell them. And that's easier than open this up, type it out. Oh, you're under pressure. You're Miss typing it like all that stuff. I think there was a question behind you that I wanted to

Unknown Speaker 51:49
follow up on the QR codes because I read the article on Medium about a month ago. And I was really interested in how on the one hand, you read last study, some people are like, what's the Milliken QR code? I don't want to do that. And do I have to keep my camera on the whole visit? But it seemed like it was working for you. And did you do it as an experiment? Or did you think, Okay, this will help us track where people enter? Because you placed the QR code in museums so they can start to guide at any point. Yeah, I was very curious with it wasn't just sort of a Gamble's and like, maybe this will work even though people are not.

Unknown Speaker 52:29
Yeah, I would definitely say that we set it up as an experiment, it was largely prompted by one there not being another great option, NFC, at least the last time they were looking at this, you can open a website on the iPhone, maybe now you can, but like, there was not another great easy entry point. And with iOS 12, I think where they made it native, it was like, oh, since most are audiences, iPhone users, like maybe we can try this again and see if there's like a new era where this works for people. And so all the different QR codes in the building had slightly different URLs that they redirected to. So we could see, did you come from a stanchion? Did you come from a wall label? Did you come from a ticket? And it turned out that like two or three of them were the only effective ones? I couldn't even find some of the other examples, I think, as I wrote in the post, but I would say it's been kind of borne out as really because there's almost no cost to doing it is I think the alternate way to look at this, like, did we get people? Was it a problem if they saw it, and they didn't understand it? Probably not. But we got enough people in and it's like, during Warhol, there was something like 15,000 scans of that. And it's like for an audio guy, like our audience isn't huge to start with. So it's like I can't discount what is like a normal yours amount of traffic coming in through that QR code. And so we've kind of institutionalized it in a couple different ways and have much kind of easier ways now to make them. But it really did start as an experiment.

Unknown Speaker 53:53
Are you at all concerned about how easy it is to spoof, perhaps a QR code? Basically, by creating a new QR code and putting a sticker over something to do you help people with Android users say, Oh, well, here's how you can put this into your phone. Yeah,

Unknown Speaker 54:10
those are, those are two really good questions. We are not particularly worried about spoofing, I don't know, maybe we should be. It hasn't, like in practice looked like a problem. So far. But again, it's still kind of early days for reintroducing this stuff. So if it does come up, like I think we'll change our tune. In terms of Android use, like it's really hard to find a universal thing here. And so it was sort of like if this makes sense to you great, but we also never print the QR code absent like the wit unit org slash guide URL, so it is never on its own. There is always another way. And so if you don't understand it, your phone doesn't do it, whatever. Like you have another option. It's just trying to be quicker about it.

Unknown Speaker 54:53
Using this new format, you doubled to take a break from your previous what you're doing, whether it's ranking devices Hmm. My question is what was the actual take of maybe pre Warhol during Marvel rapid? Overall?

Unknown Speaker 55:08
Yeah, it was around 2% for like the previous five years or so. Yeah. And so it's really, it's, again, really weird based on the shows and how things were pushed. But it is generally three and a half to 4%. Now, which again, this is all like different museums, different numbers, like, heard other museum numbers that are way different and better and worse ways. But like that was good for us. So and we didn't change the content is the other piece of this. We got this with the same strategy otherwise.

Unknown Speaker 55:40
I have a question. For I'm from one of the institutions that has a somewhat similar storytelling device. I'm curious how closely worked with the curator on the content creation? And was there any resistance to kind of approaching that more pared down version of territorial story for content?

Unknown Speaker 56:01
Great question. We actually the producers, then the designer on the primers worked super closely, and they embraced it, they really loved it. And I think they were looking to be relatable to general audiences. I mean, obviously, the creators want people to come to their exhibitions. So they really looked forward to telling a different story, especially for this last one, the curator was super engaged with this propaganda point of view, on Maximilian the first, and that was kind of his vision, and we were able to translate it. So luckily, so far, it's been smooth sailing, but they've been really and in, in creating these three primers, other curators want to be able to use these tools for their upcoming exhibition. So it's been a great proof of concept to get them involved.

Unknown Speaker 56:53
Were you able to assess that people were using it as a primer? They're using it

Unknown Speaker 56:58
before or during? Or

Unknown Speaker 56:59
after? We do have a survey on the page that says, Do you feel prepared for your visit? And most people have said yes, but it's quantitatively and then through user testing, and we ask a lot of qualitative questions, and asking people how they understand.

Unknown Speaker 57:16
And I was just curious, how, how would you market our people,

Unknown Speaker 57:25
we are trying a few different ways through social. So doing pushes on social media, pushing it on all of our visit planning experiences on the site, and then partnering with visiting partners that the museum has like TripAdvisor or NYC go, and promoting the primer in those places are really reaching people where they are like, if you're planning to visit the Met, if you're a one time visitor, you're most likely going to go and on Google on TripAdvisor on those websites before maybe coming to the med website. So if you have the open access to see it there, then we hope that people say they're so really going for that visit planning user that's going to those websites that get access to it, just go for it.

Unknown Speaker 58:10
So there was nothing on site in the exhibit.

Unknown Speaker 58:13
We did have QR codes in the beginning and the end. So we're trying, we don't have a lot of engagement with them. But because the Met is so huge, and the exhibitions itself, there's a multiple entry points, we're trying different placements. For play it loud, we did a QR code at the end. And that didn't get a lot of engagement, for embrace of things we put in the beginning. So that gets more so we're playing with the placement of it. And we can put in the tickets because our tickets have QR codes unfortunately. Organizations

Unknown Speaker 59:06
Yeah, great question. So it's still part of our website. And the primer itself is featured on the exhibition page is just a different kind of audience. So who's the audience that we're serving? And where are they accessing it from? The exhibition listing page is more of an institutional reflection of what the exhibition is. So it includes donors and the curators point of view and press and everything about an exhibition. So it's sort of the end all be all of exhibition, where this is for where you can learn about an exhibition, and it's targeted towards that casual browser, versus the exhibition page has another specific user. So creating two experiences based on audience engagement.

Unknown Speaker 59:55
Um, I don't know I'm, in what sense what do you mean Like Yeah. If it's a learning tool Yeah, I think it's too soon to tell I'm hoping I'm hoping to or creating consistent user language. So from the homepage and the exhibition page and the plan your visit all of those pages, where we say learn before you go, so that that is the place where you learn something before coming. I think we need a few more. And now that we have three, there's going to be like a destination with all of them so that people once you see more than one you can know that this is a feature that you used to learn before coming.

Unknown Speaker 1:00:40
I think that's about it for time if you want to ask us questions personally after I think we might be here for a few minutes, but thank

Unknown Speaker 1:00:45
you