Effects Of Shelter-In-Place On Museum Website Visitation: Findings From A Cross-institutional Study

For many museums, website visitation traditionally supported on-site visitation. What changed with shelter-in-place? We will present cross-industry trends revealed by an analysis of quantitative and qualitative data aggregated from eighteen institutions across the country, some small, some medium, some large. We will compare and contrast with deep dives into the data of the J. Paul Getty Museum and the Whitney Museum of American Art.

Transcript

Marty Spellerberg 00:00
Welcome, everyone. This is effects of shelter in place on museum website visitation. My name is Marty Spellerberg. I'll introduce our panelists in a second. But before we start, I'd like to thank Microsoft, the who is the registration Assistance Fund sponsor, XL, the Ignite sponsor, and all the other sponsors listed on the program schedule for helping us make this conference possible. For this session, we are using the q&a box for questions and the chat for technical issues. After we show our slides, we'll have some time for questions. And, okay, that's the housekeeping. Let's get started. So, welcome, everyone, thanks for coming. How we're gonna do this, there's gonna be three, three presentations, and then a q&a. The first presentation will be myself and Grace Poole. Then Tim will speak about the experience of the Getty, and Kahn will speak about the experience at the Whitney, and then we'll have the q&a. So...this is a cross institutional study. Early in the UN pandemic, as lockdown was sort of coming in. I thought, wow, this is really, this is really a notable time, we should we should do something to capture this and to learn from this. So some caveats about the study is that this is a cohort of 20 institutions. So not 200. It's a US heavy cohort. And it's a our heavy cohorts. So you know, keep that in mind, as you're looking at what what we're going to show. We did a both quantitative and qualitative, qualitative research here collected that data, the quantitative was in the form of Google Analytics data. And then qualitatively, we interviewed the participants. And then we use the what, what came out of the interviews as a lens to examine our our quantitative. So thank you to all the institutions who took part. I mean, we we wouldn't have been able to do this without them, obviously. So here's our first bit of data. Grace, do you want to? Do you want to talk about this one?

Grace Poole 02:56
Yeah, absolutely. So as Marty mentioned, we collected both quantitative and qualitative data. And one, one of those data points for the qualitative data was top pages for the participating institutions. And for because the institution's provided their own URLs, these top 10 pages of variety types and names. The one way that we could kind of make this into data that we could analyze, was I coded all of the participating institutions top pages into the categories that you see on the right, ranging from digital content and virtual tours, to education and the about page exhibitions in person events. And then that data was divided into the three periods, A, B, and C, that you see at the bottom. And we're basically for all of the visualizations kind of tracking in institutions year over year. So an institution was compared from their own 2019 to their own 2020. And we were basically just trying to see if there was a shift in what people were looking at in terms of their top pages. And you can see from the visualization that that there were a lot of shifts in terms of what the top pages included, even over the course of the pandemic.

Marty Spellerberg 04:47
Thanks. Great. So this one. This one we're looking at, we're looking at sessions and this is the entire cohort. The black line being the average Of these of these 20. The, of course, you what you can see is that the most institutions experienced a dip here, this is when lockdown went into effect. But a couple of institutions did not. The this orange line in this yellow line is the yellow is the AR 21. The PBS TV show, and this orange line is is the mat. The pink line here, the salmon line is the Getty. And we're gonna hear from Tim, a little bit. But we thought that it was really interesting that the institutions that the vast majority of institutions saw this dip in, in March. So we wanted to look at those institutions specifically wanted to take out the three that that I highlighted and look at this cohort. So in a in a for profit context, there's this idea of competitive advantage. And what competitive advantage means is that it doesn't mean that you're the at the top, it doesn't mean you're the best, performing the best, but it means you're performing above average. And for that reason, it's important to know what the average is. And so then that that, that average, we'd call that your benchmark. And then you would look at the practices of the of the organizations that are above average, and you would see if any of those practices you'd call those best practices, and you'd see what you could learn from them. So there's a little bit of a correlation, correlation and causation tension there. are, are the organizations successful because of the practices? Or are the practices just indicative of that organization? But so so what we're going to do over the next few slides is show certain practices. And the way we we, the way we came to these visualizations is we we interviewed the the participants. And then we went through the interviews, and we listened for certain practices. If in the interview, they said that they did a certain practice, then we associate that institution with that practice. And then the line here is the average each line is going to be the average of the institution's who we highlighted in that way and associated with that practice. And then the black line is our benchmark from the previous slide. Grace, do you want to say anything about about this slide?

Grace Poole 07:50
As Marty mentioned, when we interviewed all the participants, we were trying to kind of gather a holistic picture of what both the pre pandemic looked like for different institutions, both stylistically on their website, but also in terms of content. So we asked a variety of questions ranging from a pre COVID period to the first few weeks of lockdown, and then the future of digital activity, their website, and then I coded, though that information to make it quantifiable. So this visualization is based off of one of the first questions we asked in the interview, just asking them, what was the main purpose of your homepage prior to the pandemic? So what did you kind of prioritize for your landing page of the website. And as you can see, most in participating institutions in the study use their homepage as a space to give general information or to prioritize the visitor who is planning a visit to their institution. And then a smaller number of the participating institutions minus the big three or the Met are 21. And the Getty said that they use their homepage in order to highlight current and upcoming exhibitions.

Marty Spellerberg 09:31
So this one is this one are practices that institutions undertook in directly in response to COVID. I mean, I mean, I think we all saw these. We won't spend too much time on this one. But you know, the creative response page, create an ad at home section. I'll just go ahead and move to the next one. So this one is created new content. Grace, do you want to see say something about this one? Yeah.

Grace Poole 10:10
So kind of the bulk of the interviews that we did with the institutions were trying to gather what content based practices they're doing during lockdown. And so that ranged from Blue, which is new educational content, red, virtual events, yellow, new video content, and green, which was increased usage of their institution blog. And a variety of institutions did one or more of these new content practices. And that was something that we indicated in the coding.

Marty Spellerberg 11:04
So this is the creation of new content. And here in the next slide, we contrast this with reverse repurposing existing content. And as you can see, this is one of the few the few visualizations where the lines are three uniformly and dramatically above the benchmark line. Grace, do you asked anything about these practices?

Grace Poole 11:33
Yeah, so the existing content practices were all practices that in the interviews, the institutions indicated that this was content that they had used for previous exhibitions, or previous programming that won the pandemic hit, they pulled that out of the archives, or out of out of their, I guess, content management systems to try to repurpose into more evergreen content. So people said that they reuse publications or articles that people have published previously existing video content, whether that was educational or kind of event programming. And then the bulk of people said that they reused and repurposed their educational content.

Marty Spellerberg 12:31
Yeah, I thought, I mean, we all remember that the period early and earlier in the season, when, you know, we were just all scrambling to serve our audiences and, and, and bring things bring things that would be useful forward to them. And seeing this, it really struck me that the institutions who, for whatever reason, have we're in a position to or had chosen to, to create evergreen content, create content, in the before times, that they brought into the pandemic with them had a dramatically different experience, than than the others have a more positive experience. In this time, according to this way of looking at things. here's, here's a, just a, just a little bit of a little bit of a summary. I think, you know, we're all we're all very familiar, one of the one of the things we talked to the participants about was, what they use their website for, and, and, you know, every almost everyone says, it's about facilitating a visit, you know, people want to come visit, they come to the website, and we want to help them that, and a number of institutions specifically would put, you know, would highlight exhibition content on their on their homepage. And I think we all know from experience, the, the, the sort of the unending churn of the exhibition calendar, and always creating content for the website that that supports the exhibition calendar. And, and I thought it was notable that, you know, here in this endemic time, there's actually, there's actually another use, we have museum websites, and we got to look at it. It's people who are never gonna visit your institution. I wanted to point out that the Met for instance, has to two entirely separate streams for their, their digital creation. They have a team that works on facilitating visits and encouraging visits, and how People who are going to visit and then they have a team that works on digital content, that for people who are never going to visit in person, and I think what the pandemic gave us was an opportunity to see what happens to really be able to look at that audience, that audience who isn't going to visit and see how they use museum websites. So, you know, my big takeaway from this is that even if we're, this is editorializing a little bit. But even if we, we get back to normal, we get back to receiving visit visitors and purpose in person, we shouldn't lose sight of the the needs of the visitors who are not going to visit and making sure that we we create evergreen content that will be an asset, an asset for serving those visitors, and then also an asset for us in these situations where we can't. We can't welcome people to our doors. But okay, those are my slides. So I'm going to stop sharing. And over to you, Tim.

Tim Hart (Getty) 16:19
Thanks, Marty, I'm going to give a little bit more of a deep dive into our museums experience. I'll give you some examples of what worked for us and things that didn't work. And I'll also also show you where the audience grew. But in places where the audience didn't grow during this COVID pandemic work from home period. First, let's look at a slide that gives the overall digital reach for the game. So this is digital engagement over 12 months, and I'm showing overall engagement and high engagement. And the way I'm breaking that out, you can see on the left. So the the high user engagement is including things like followers, subscribers, sessions, engagement that's more persistent. And overall engagement is showing things like social media impressions, media coverage, social shares, ads, and I'll break those all out for you on a later slide. But as you can see, we experienced that dip in February that Marty talked about, we were starting to lose overseas visitors already in February, of course, in March, we closed but you saw a dip in our overall digital content reach with the decline in visitors and that held true, really throughout this period. But because we're collecting a lot of data, we're able to see where digital reach expanded and where it contracted. You'll you'll notice that in December, November, December, we were running at about an overall engagement of about 22,000,022 to 23 million. And then that dropped down to 13. But 13 wasn't an exceptionally low level. For us. It's not that different from what we were seeing in August of 2019 or September. But in this period, in November, December, there were there were a few incidents that drew audience to us. One was, well, it was the fires that were here, the Getty, we had a very successful exhibition. But we also had a newsworthy event, we were getting a lot of media coverage. And that drove our overall engagement up. But then you see with the beginning of COVID-19, and our our closure at the Getty, we jumped to 44 million, nearly 45 million overall engagement. And then following that in April up to 53 million, while the high engagement really didn't grow that much. And so I'm going to give you some examples of why that happened and, and tell you where things grew and where they didn't. So near the end of March, we launched something called the Getty Museum challenge. And the Getty Museum challenged asked our audience to recreate works of art from the Getty using only things found in their homes. And we ended up getting 1000s, maybe even 10s of 1000s of submissions on Twitter, and we got a lot of mainstream media coverage. And I'll break out for you how that impacted the overall reach to digital reach. So what did that look like? It was a challenge where people at home were supposed to take a famous work of art, and then recreate it with things they had in their own home. And the results really were pretty fantastic. Some of them really entertaining. They were very creative. We we've since launched a book, which give some of the highlights from the Getty Museum challenge. A lot of them were just really artistic taking a famous work of art that we might all recognize and personalizing it and recreating it from home. There were some really amazing results and even got their pets into the act. One of my favorites is this little chinchilla. And as you can see, it was not just really creative, but it was very personal. And it was something that got a lot of media coverage, plus a lot of social shares. And what you see here, in the top part is that, mostly it impacted our audience on Twitter, that on Facebook and Instagram and YouTube, we didn't see a big impact. But you can see here in March, as we were closing for COVID-19, right around the same time, we launched the Getty Museum challenge, we saw a huge spike in Twitter impressions from just above two and a half million jumping up to 17 and a half million, and then it declined pretty dramatically. And pretty quickly. Engagement had a similar rapid jump from February where it was, you know, hovering under 500,000, up to over three and a half million. And again, that steep decline. But what was interesting was that followers, which would be a high engagement, audience, followers never really changed. So this this sudden, dramatic jump in impressions and engagement didn't really result in a lot more followers for us something that we would have loved to have seen. Web traffic is a bit of a different story. The gettys web traffic, what I'm showing here is a Google Analytics report for pageviews. This is page views for entire website, and you're looking at 2019 in orange, and 2020, the same period in blue. And you can see here in April, this really big jump in our web graphic that corresponds, you know, with a with a lag time of a few weeks, to the release of the Getty, the Twitter challenge the Getty Museum challenge. And then it was more sustained on getty.edu that really then even it was in Twitter. And I'll give you some examples of why that was so but what was happening was that the the museum challenge on Twitter was generating a lot of buzz, it was getting a lot of coverage, a lot of earned media, and people were coming back to our site, to look at works of art, and use those works of art in the challenge. And I'm going to try to there, I can move the move the video, the the here, following the Twitter challenge. And you know, moving into October, November, we saw another spike. And so this, this represents the art challenge. Here, this decline in pageviews really represents the fact that no visitors were coming back for the summer. And the summer is usually our busiest period like it is for many museums. And the traffic on getty.edu in pageviews was lower than it was last year because we weren't getting that summer visitor numbers of summer visitors that we typically get. But then, towards the end of this period, we launched an adult a new exhibition experience called 12 sunsets. And that 12 sunsets exhibition experience had a similar impact, though, to a lesser degree as the Getty art challenge, where there was an initial large spike A lot of it due to earned media, and then a sustained number of high pageviews, and then a kind of a slower decline. And the 12 sunsets, it looks like this. It's a it's a website that we've got a sub site that we just created, based on photographs, Ed reshade, took his archives of Sunset Boulevard. And the way this works is that you can drive every shade, frock down Sunset Boulevard, and see not just all of the images of sunset Sunset Boulevard, the distance of Sunset Boulevard, but also over time, you can switch the years when these photos were taken, and see the change in Sunset Boulevard over time. And it's really been a fun experience for people. And they're coming back in large numbers. So web content, not just in those areas, but it performed better across the site. So again, taking this whole period into account March 1 through November 2. And looking at the Art directory, and this is where all of our object images live. You can see that it jumped from roughly you know 3.3 million pageviews in the previous year to nearly 7 million pageviews this year. And it also increased as a percentage of the website of the total pageviews on the website. This is our blog, Iris had a really dramatic lift, Iris is more timely, the information on Iris. It was getting a lot of pageviews and visits as a response to the museum challenge. But not only was our art and our blog performing better, but also research content. So what we were finding was that with this lift in the Getty Museum challenge that we also had a lift in research content and Marty was talking about how he Have evergreen content on your site. Then when you have a, an initiative or a campaign or a program like the Getty Museum challenge, that evergreen content, it all gets a boost. So so there's something there, that's taken advantage of a sudden rush of attention that comes from a really a very clever tweet, a very clever Twitter campaign, research vocabularies on the web, the gateway to our search, content all increased, so not just views of our, our art content. Now, subscribers, really didn't increase during this period. So if you look here, in March, roughly, when we had that giant jump in impressions, and website graphic, there really wasn't a change in new subscribers. these are these are all publications, their digital publications at the Getty, and I'm representing here, those publications, subscribers over time, and there really wasn't a lift. And that's something we would have liked to have seen. But the Getty Twitter challenge, and our 12, sunsets project didn't result in that. YouTube video views also, we didn't get a big increase there. So what we're seeing is that the channels, so whether it was you know, Twitter, or the website, they, the traffic, there was a lift for other content in the same channel, and that Twitter gave a visit lift to content on getty.edu. But that Twitter doesn't give a lift for us, YouTube, or to Instagram. We had a big jump in advertising and earned media. But the advertising jump, the ad reach was really before this period before the period of March and April. And that the jump here, the increase here that you see from roughly, you know, coverage views of around 3 million here, one and a half million here, increase in 22 million or so in April, this is coverage of the Getty Twitter challenge. This coverage really led people not just back to Twitter, but back to the website. So it was that virtuous cycle where something that becomes popular in Twitter then becomes popular on the website. And then we get coverage and earned media. And that reinforces it and the views and the pageviews increase yet again. So just to recap, some of the key points. We did have those big audience spikes during COVID-19, like we saw that Marty gave in his chart, those social media was really the catalyst and nearly every type of content performed better, but not things that we might consider a high engagement. Visit content for us used to be about 40% of our total web traffic. So when you're looking at our our page views for our website, and seeing that they were steady during that period, in the summer or down a little bit, it's really down a little bit after a loss of what used to be 40% of our visit traffic. Something that's happening at the Getty is that we have support across all levels of management for a robust digital presence. That's really important because that means that we're ready to receive new audiences. So if something happens, like the Getty Twitter challenge, we already have, we have laid the groundwork with evergreen content, whether it's exhibition content, or things like 12, sunsets, to receive that traffic and keep it within our content ecosystem. So the overall web traffic would have declined in the absence of physical visitors, but the new web presentations then boosted that traffic again. And I'll hand over to Colin.

Colin Brooks (the Whitney) 28:58
Hi, everyone. My name is Colin Brooks. I am the senior developer at the Whitney Museum of American Art. In New York, I use he him pronouns. And I'm going to be talking about winning out of word without the museum. So really over the last year, there have been to whitney.org, there has been the pre COVID whitney.org, which implicitly or explicitly was really built around the idea of building interest for our artworks, the shows that we hit on and getting people engaged to the point where they wanted to come and view the museum. That was kind of determined by both the things that we did by choice and the things that we sort of did implicitly based on just what were we spending time on. The second witness at org has been the stay at home winning org and this has been a really big question as far as like, what are people still using about the site when there isn't that I'm actually going to go to it option anymore. And apologies if you hear my radiator is going a little crazy they are here and making noise. But to answer this question I wanted to look into what is the data that we have while limited kind of say about winning in an org usage. So here is the homepage of the site on March 12, the last day that we were fully open. If you can see this, the the real headline option here is via Americana, a show that we had just opened a few weeks prior about Mexican muralists and their impact on artists in the United States. And everything that we were doing online, whether it was social, or ad based, or it just the stuff that we were promoting on our homepage and elsewhere was really focused around getting people interested in this exhibition. And so that obviously really changed once the show that had just opened immediately kind of didn't open anymore. And so what I am looking at throughout this presentation is 20 weeks of closure that starts shortly after we fully closed and ends about a month before we reopened that very limited capacity. And the idea there is to avoid some of the visit planning that would have happened. Once we announced that we were going to be open again. So this is a little arbitrary, but hopefully is a useful look at what a closed museum website is. So what happens immediately, we lost half our traffic 50% drop and users sessions page use. This is very fast. And taking a look at this chart where you can see those three metrics map from the start of the calendar year, there is a very sheer cliff post a bump for opening new shows to that mid March area where things really dropped. Similarly, looking at some specific parts of the site, we have our visit section which has a lot of logistical information about kind of what it means to come and visit our hours tickets, that kind of situations that really flatlines mid March somewhat similarly, our exhibition section where you might go to see what shows are on what is that kind of look like to read descriptions that also dropped pretty heavily. And that includes archival content, or exhibition archive that goes back sort of bodily to 2006 or earlier, that dropped off pretty heavily relatedly transactions, which for us are ticket sales and membership sales, just completely zeroed out. Post close. And maybe the most nuanced piece at the moment is our guides our audio guide section, which without really getting into it, there's kind of two components. There's the stuff that we push people to on site for interpretive content in gallery, and then the stuff at home for old shows or that you might want to look at later. And even the fact that there is that archival section after this spike for the new shows opening, it's still gradually dropped over the course of our closure. So maybe more interestingly, what didn't crash? The big thing is the collection as far as large aspects of the site that did all right. I don't know if this is something if you ask me I would have anticipated or not. But for the most part it held steady or increased in usage. Absent any other changes at the time, we had a slight increase in people using the online collection, we had a larger increase in Sasson sessions and pageviews and session duration. And one of the things that I was really interested in was trying to figure out what are people doing differently like is are there certain kinds of things in the online collection that are as much as nothing's a benefit, but like beneficiary to this? And well, a lot of that is reading tea leaves. Specific artists did seem to do better in terms of traffic. And as much as I like don't think you can really take anything away from this. And it's the top 15 artists during that 20 week period. The thing that stood out to me poking through this was that it seemed like a lot of the artists that we consider kind of Greatest Hits artists of our collection did well compared to artists that were tied to shows that you could no longer see like you can Roy Lichtenstein. Lichtenstein is on here and Eliza Lu who both had strong presences in shows dropped whereas artists that weren't particularly featured but are always kind of present like hopper and Warhol and Calder and O'Keefe did pretty well comparatively And so trying to wrap that all up into something useful as far as what we could do with the website in this period, I tried to do a little bit of, I guess I'm calling reactionary feature development, which is probably in general a bad thing. But at this moment felt like a real response to what is useful to people on the site, and what can I change to try to make better. And as much as there are so many bigger picture things that I would like us to do with the collection, that are really complicated institutionally, whether it's trying to do kind of more thematic entrances to the collection, like the gardens to to Chicago, or different handling of highlights, or different kind of sub collections like the mat, absent being able to really make those things move at this moment. And I did at least try to get us to make the initial experience sorry for my watch going crazy, a little bit better. So you can take a look at this later when I figure out how to share the slides. But there is the kind of old option on the left and the newer collection on the right. And besides just moving things around a bit, trying to bring the images up a little bit and clean some of the filtering or on the collection, I was really interested in trying to give another access point to this collection. And what I really wanted to figure out how to get in here was a way to randomize what you see, because so often, if you come to a box like this, you might not know what you want to search for, you just kind of want to see some stuff. And so we ended up adding sorting, which you can now do reverse chronological or not, or by title or by artist, you can also choose by random. And I think that's gives you a really different sense of what this 25,000 word blob is when you can kind of just page through and see a pile of stuff you didn't expect. And while that's really small, I did want to highlight it because this is a set of changes that we made entirely in response to seeing this is being used, what can we do in terms of UX and UI, when institutionally we are very limited in what we can do kind of curatorial. In terms of other things that did alright, videos did fine. Don't ask me to try and explain the YouTube algorithm. But just taking a brief look at watch time and views. We had a spike around when we posted some new content for vitta. Americana, in mid February. But even after that dropped off a little bit enclosure, this is better than the early part of the calendar year or last year looked that it still rose a little bit above the norm for us. And we don't get a ton of views. But this is enough of a spike where it did feel like there is slightly more engagement here happening. And it tracks a little bit with a wider idea that people are watching more videos. And it seems like we got some amount of that kind of bump. For a little bit of context. These are both videos that were in the top three to five on watch time and views over that period. And they I think they're pretty representative of what was being watched. On the left is really classic video that we posted in 2008 on Alexander Calder performing his circus moving a little fingers around. And people always watched that. And then a newer video about some of the muralist in the show. This is David securus, that also was doing really well during the period, which we launched on February 11. Another section of the site, and I put this in quotes, because it's a very relative boom is art port. And for those of you who don't know what art port is, it is the home of born digital art on Whitney not org. And what I think is notable about this is this is a project that we've been doing for 20 years. And this is a screenshot of kind of one of the archives there are now multiple because it's been rebuilt so many times. But it's a place where you can go and see our as intended if it still happens to run sans Java applets, and everything else. And in general usage of this is up by about 150%, across any metric that you can look at whether it's users or time or whatever else. And I think that's both because this is a way to see real art at home. But also it speaks to the relevance of some of the projects that we've had. And I wanted to spend just a moment to highlight one of them, which is this work called looted by American artist. This is part of our sunrise sunset series, which is a thing we've been doing for about a decade across multiple different versions of the site were at sunrise and sunset every day, New York time for 30 seconds. There's an artist takeover of Widener dog no matter the page that you're on. And for this one, we boarded up all the images on the site. And this is a direct like corollary to what was happening at the same moment. IRL, where museums were boarding up their windows and doors and so American artists like got us to dim the site to dim the text and to do this thing just across when you dark for those 30 seconds. And I don't want to try to say anything, you should go read what they have written about it, even though American artist is by design pretty unbelievable, which I find fun. But it is it is just so connected to what is happening physically to the digital version of Whitney Not a word. And I don't know, it speaks to a different level of relevance in this moment and a lot of the other things we posted. So check that out, I think it's a really interesting project. Lastly, just a few miscellaneous things that I thought were notable looking at the data that may or may not hold true across institutions. One is that we are no longer really mobile first in terms of usage, everyone is on desktop now 64% to 45%. Prior, we're more international than we've been in the past, whether that's because of the drop off of domestic tourists or otherwise, versus new audience traffic, I don't know, that's kind of complicated. And it's probably a bit of both. And then also, in terms of kind of some of the duration metrics were both an average session duration and time on page over the period by about 30 seconds, which is quite a bit for us. And I think notable, in part because a lot of the things that used to take a lot of time, we weren't doing like the thing that took the longest in the site generally was buying a ticket or a membership. And so to zero those out and still end up with longer duration on the site, I think really speaks to whoever is left using this thing is using it a little bit different way than they did before. And that feels pretty notable to me. And that's all I really had to say about when he entered Oregon, its behavior. And I am happy to turn it over to I think we have things in the QA and maybe Marty will do a little bit of running, Matt, thank you.

Marty Spellerberg 41:59
Thank you, Colin and Tim...this was fantastic. So one, one thing I want to say before QA is that this project is continuing. So you know, in the next step is we're going to write a paper about that about what we've learned in the data we've collected. And I also think it would be very interesting to continue this work into next year and do like a follow up like a road to recovery. So I'm going to paste the link in the chat. This is if you're interested in this work, and they're just want to follow along and hear about findings as we publish them. Or if you even think that your institution might be interested in joining the cohort for the next phases. There's a forum there that will allow you stay in touch with us. So I know people have been putting some questions in the chat and in the q&a box. Grace, do you want to? Do you want to moderate that?

Grace Poole 43:11
Yeah, sure. Um, so thank you, everyone, for coming to this presentation and listening to Marty and I and Tim and Colin about our experiences with this research project. So I guess the first question would be for Marty. And this is from Kim falls, she or they said can you tell us how you chose the selected organizations for the study?

Marty Spellerberg 43:42
Thank you. Great question. I put out an open call. So really, pretty early in the pandemic, or during lockdown. I started to go out with this with this call. And and so it was an open call any institution that wanted to could join the it was promoted on Twitter and is promoted through the MC n cigs. And I ended up speaking to something like 8075 or 80 institutions. And it was it was about 20 that, that that came through data in the end. So it was kind of a people kind of self selected in. We didn't I didn't mean to exclude anybody. But that's what we got. We got 20 that way. And and and the more you know, the more we can spread the word about this project, the larger the cohort can be in the next phases. And in the largest cohort, the more robust, the findings.

Grace Poole 44:53
Cool. And so the next question is for Colin. Natalie wondering the Whitney website is super organized with the information about virtual events and educational content. They're wondering if you could share the process of figuring out what might be the best UX experience for audiences in terms of organizing this information?

Colin Brooks (the Whitney) 45:19
A sure this is either Oh, man, I wish I had a better answer for this because the answer is sorted just like we did what we thought worked. Yeah, I would, I would note a couple of things about this, it took us a while to get the Whitney at home sort of section out with information about what we are running. So I would say maybe we did a good job. But we were not a first mover on this by any means. So we had the benefit of being able to see what a lot of our peers did. And that was sort of to our frustration internally, with the roadblocks that were kind of put in front of us, I would say in terms of like how we organize things. Some of it is just based on we run around CMS. So there's sort of like the natural ways of how we organize events that this kind of just applied to. And then there's also a small amount of development work that we did do to try to forefront online only things a little bit differently than we would handle things that had a location on site. But for the most part, we really had the kind of positive option to see what other people were doing.

Grace Poole 46:23
Well, and towards the beginning of the presentation, Bruce asked, Can you explain how you distinguish between visit planning, and exhibit information in terms of the homepage? And I'll take that one. Because I coded that data. Most of the the distinguishing between those two categories really came from the quotes from participants in the interviews. So if a good example is the Nasher sculpture center, who I think is watching this presentation right now, they said in their interview, that pre COVID, their main objective for their homepage was always to drive on site attendance. So they basically said the homepage facilitated someone planning a visit. So when that came across in the interviews, I would I would code it as an institution prioritizing information on their homepage that did visit planning versus someone who said it was a exhibition based kind of code. Let's see, I think this is this one's a good question for Tim. So Aaron Richardson in the chat, was looking was pretty interested in the collections page, kind of jump. And looking at that correlation between collections pages, and students virtual learning, likely being being used. So students with virtual learning are using collections pages. And I guess anybody could answer this, but I'm sure that Tim has an answer to this. If you saw any correlations between that increase collections, page usage and student remote learning, oh,

Tim Hart (Getty) 48:27
well, first of all, while grace really put me on the spot by saying I'm sure Tim has an answer for this. I don't know what I did to earn that. Um, so this is a very tough question to answer. And it's hard to answer for a couple of reasons. First of all, we don't have great profiling data through Google Analytics. We don't have logins or authentications, to know who is visiting the content on our site. And we're, we're very cautious about conducting survey research with K through 12 individuals because of privacy issues. I think a web survey accompanying that content would be very helpful for understanding if it's K through 12 students using collection content for education purposes. I don't have that data. We do have. We have a lot of education content on our site. And I, you know, at the risk of Murphy's Law with demos, or, you know, trying to answer things on the fly, I'm going to run a report in Google Analytics right now, while we're talking just to see if our education content had a spike in visitorship during this COVID period, so if someone else wants to take the next question, I'll run that report for us. And I'll come back with an answer to that.

Colin Brooks (the Whitney) 49:50
I have a a couple of quick thoughts on this. Just based on the fact that like, immediately reading this question, I was like, Oh, this is on our like, unfortunate. Super backlog of things that we've wanted to figure out with our education department. And it's such a backlog that I remember it being a thing when I was in our education department four years ago. But we have wanted to do more kinds of focus groups with teachers, specifically around online collection usage. Because to Tim's point, like there hasn't been a great way kind of in the natural Google Analytics data to figure this stuff out. But yeah, it's it's a really good question. And I often don't think online collections are really designed for educational use, as much as they kind of are branded that way. And I wish we spent more time on this.

Marty Spellerberg 50:37
I'll say I'll say one thing about if you recall, the, the top pages chart that we we've been in our section with that the the collections usage increased in a great in a greater amount than the educational content. And I was personally surprised by that, because I know how how, how much emphasis was placed on educational content at the beginning of stay at home?

Tim Hart (Getty) 51:09
Well, so I, I've run that report, Google Analytics for the gettys education content. So this is not directly answering the question about collection pages being more used by students for virtual learning, but looking at our educational content. I can see that in fact, if you like, Marty, can I still share my screen? I think I can. Yeah, let me let me try that. So what we're looking at is a Google Analytics report for education content, on the Getty website. And so looking back to Jan one, you can see that it was it was actually fairly steady up until this April period, when we had the the Twitter art challenge, or the Getty Museum challenge, and it rose a bit. Of course, things dropped over the summer. But I think it's really gratifying to see that once students were back in class traffic came back to the levels that we would have expected and actually exceeded where they were this time last year. So let me just run that monthly. Hopefully that helps a bit. Of course, November is just a partial month, but looking at October. And I and I hesitate to run this for a longer period because I think Google Analytics is going to stall. But let's just say this, we didn't see a big jump in the use of educational content. But I think it's really gratifying to see that the education content maintained its graphic, and even increased a bit during this period.

Grace Poole 52:42
All right, there are some questions that I'm not sure. Any of us can answer, but are, of course, extremely interesting. One of them comes from Sharon. They asked, How did online events fare and relate to traffic? And has that changed over time. So I'm guessing they're asking the change before the pandemic, as well as over the course of the year so far.

Colin Brooks (the Whitney) 53:15
I was just speaking of this a little bit. It's like sort of interesting in that, like, we're looking at traffic, which kind of correlates in some ways to interest everything. But like all of the How did it go and who attended stuff is all actually buried in on the zoom side. So it's like a very different data analytics problem. But just for kind of rough reference. For us, we had like a drop during initial closer closure, and then kind of back to normal when we're running a full slate of virtual things, which is probably way less events than we would run normally, but had more online interest because they were online only. So it kind of is like and then back to normal as much as the programming is very different.

Grace Poole 54:01
I guess this kind of goes hand in hand with Sharon's question about events, but an anonymous attendee want to know if and I think this is more of an audience research type question. But do any of you (Colin, Tim, Marty,) follow the same kind of visitor profiling, which I think they might mean like a visitor journey, kind of when you develop online versus on site programs. And I know this is a little different because Tim and Colin don't necessarily create online programming.

Marty Spellerberg 54:42
All I'll mention, in case it's relevant here, the previous cross institutional study that I was involved with, where we look at visitor motivation in web in web visit and we use used a model a framework of visitor motivation by a researcher named john falck. That was developed with for on site usage by setting on site visitation and is widely used when studying on site visitation. And that's something I'd be happy to share. More information about, I thought it was very interesting to use. I was personally interested to use something that had a crossover between on site online and and some of the participants in that study, which I can send more information about and reach out and Slack, did run studies concurrently, on site and online. I don't know if any of the others here want to speak to their experience.

Colin Brooks (the Whitney) 55:57
I've never seen these profiles, or whatever you want to call them line up between on site and digital at the Whitney, despite hearing people be kind of interested in that for years. So I'm also very far away from doing programming for the most part.

Marty Spellerberg 56:21
Mmm, cool. Well, we have Yeah. Do we have any final questions that we want to? We want to cover here we are. Well, I should make that a question. Are there anything else we want to put the moment then I'll say that we are going to be on slack. So I'll go over there. And I'll start to all start a thread. So look for that, and we can continue to have this conversation. And I want to thank you all again, I want to thank Tim and Colin and grace for for having this experience with me. And I want to thank all of you for coming. coming today. So cheers. Thank you very much.