Recentering the User: A Study of Digital Publications from Four Museums

In museums today, we employ user-centered design practices while developing projects to define our audiences and align with their needs. All too often, though, we lack the time and resources to do the equally valuable post-launch work of user-centered evaluation—work that could take us from the theoretical to the practical by shedding light on who actually uses these projects, and how. This knowledge is especially critical in cases where content design methodologies span multiple projects, or even multiple institutions, as in the growing field of online collection catalogue publishing. This session presents the findings of an ambitious evaluation of online scholarly catalogues produced by four museums: The Art Institute of Chicago, the J. Paul Getty Museum, Philadelphia Museum of Art, and the National Gallery of Art. The study, conducted by Rockman et al, draws upon evaluation of web analytics, user surveys, and in-depth user testing and focus groups. By addressing questions such as how the digital publications compare to scholarly works in print, what platforms and features create the best experience, and how the digital interface facilitates engagement with museum collections, it aims to close the gap between the audiences we plan for, and those we are truly serving.

Transcript

Unknown Speaker 00:00
Hello. Welcome, everybody. Thank you for coming to recentering, the user a study of digital publications from for museums. And we're very excited to present to you today, just a taste of the findings from an ambitious study that we have the Philadelphia Museum of Art, the Art Institute of Chicago, the Jay Paul Getty Museum, and the National Gallery of Art undertook to better understand our online scholarly catalogs. And specifically the users of those catalogs. It's been a great experience working with this group of people over the last few months and we're really thrilled to share our preliminary findings with you. So before I dive in, I'm just very brief introduction. So I'm Katie Riley, the William T. ranny, Director of publishing at Philadelphia Museum of Art. We have Lauren maqam, Assistant Director of Production in the publishing department at the Art Institute, Greg Albers, digital publications manager at the Jay Paul Getty trust and Emily's office managing editor for the permanent collection at the National Gallery of Art. And then our fearless leader, Claire Quimby research associate from Rockman at all, who really shepherded us through the process, the final report has been published online, you can see unlike I didn't get the QR code memo that says like, going to be the MCN if QR codes are also have one up there. But there is a link to the report, the full report, which has been published using the Gettys choir platform. Thank you, Greg. And they'll be the URL on the bottom of each slide. But I would encourage you to spend time with it after the presentation, as you will see it is a lot to digest. So what prompted this, it really kind of emerged organically. At the PMA, we had launched last year, our very first online scholarly catalog. And but a number of people here, as you know, have been doing this for some time. And I think we're at sort of a crossroads, I think all of us were really wondering about what the value is of continuing to do this. In some cases where you know, people it's like, we know how to do it, but to what end. And I think that we had and we all new, it just kind of emerged out of conversations with each other, we had, you know, really the same kind of defined audiences for these publications. So it seemed like reasonable to actually go into a comparative examination of them together. And I think all of us had used had employed user design practices in in developing the publication's there, really, nobody had done the work, kind of on the other end to make sure like, we're really hitting the mark. And, and it was kind of heartening to know and conversations how little we all knew about who was using I mean, it was like, not good, but it was like, you know, it wasn't it, I think it melt made us feel a little bit better. And, and also felt like, this is something that we can really make some progress on. And doing it together will be so much stronger than us embarking on any individual about evaluation of our publications. And with that we got really excited very early on about that very thing of, you know, we kind of got together, because it's like I happen to be talking to, you know, Greg, like, it's like we we all and then I think, yeah, the National Gallery came on later because it was at like, you know, Mo, I think you got wind of it at something. So it was like, we just we shouldn't count on serendipity. And if we can try to create really more of a kind of network and try to build off of this as a foundation so that people can really share learning and try to spark a field wide conversation. So very briefly before diving into the results of the study, um, just so you can understand the structure of the panel will each do very brief introductions to the catalogs under consideration, we hope that you are all very familiar with them that you may not be, as some of the results will show from viability of our publication.

Unknown Speaker 04:18
We then will ask Claire, from Rockman, to do a kind of top presentation of the top level findings of this study and some of the recommendations that emerged out of it. And then each of us will kind of go through and again, we've we've really just got the full report in a couple of weeks ago. So we've all been starting to process this with our internal with our teams at our respective museums, to really see what we want to do about it. So this is just a you know, our kind of first glimmers of what we feel like we're each going to investigate in the months to come. And then we'll end with you know, how you all can get involved and hopefully leave in hoping we'll have a fair chunk of time at the end for questions and conversation because you'll see it's kind of a lot to digest. So with that, I'll hand it over to Lauren, since the Art Institute really kind of kicked this all off to begin with.

Unknown Speaker 05:14
Thank you, Katie. Hello. So this summer, the Art Institute of Chicago released its 12th 13th and 14th catalogs on the OSCE toolkit platform. The OSCE toolkit was developed by ima labs in conjunction with the Art Institute of Chicago as part of the online scholarly Catalog Catalog initiative, which I think the MCN audience is more aware of than any other audience, anywhere. But yeah, that kicked off in 2009. So for this evaluation, we concentrated on two catalogs, Monet paintings and drawings at the Art Institute of Chicago and Matisse works on paper, sculpture and textile at the Art Institute of Chicago, we we evaluated two catalogs in order to be able to analyze an older catalog with a longer history of analytics and a little bit of a better, larger user base Monet was, was released in 2014, while also studying a newer catalog that features updated design, and all the bells and whistles that have come to define these platforms. Our Matisse catalog was launched just this past June and includes art, various media, so it really gave us an opportunity to show those things off. All of our catalogs feature zoomable images layered and annotated images, citation tools, and a book like navigation that displays pages of text in reflowable columns. So you can see that with the Matisse example there. The Art Institute's dreamed of a study like this for a long time, from the early days of the development of our of the OSCE toolkit, that catalogs have relied on certain expectations of the field that we wanted to test and verify. We were curious how the landscape of digital publishing has changed since then, and how scholars were interacting with the immense amount of information that we're publishing over this platform. Our Gauguin catalog, as a sort of extreme example is over 830,000 words. So yeah, so we hoped to better understand user expectations in order to be able to calibrate our internal capacity to continue this this work. And I want to say that one of the things that we loved about this particular study and the the fact that we were all able to do this together, is that we really didn't want to approach it as a comparative study in a traditional sense, because for us, one of the really important things was not that we would learn like whose tools are better, or like which direction we want to go on based on platform, but really use all of these different ways of publishing to understand the field as a whole. And so a lot of the of the direction that we went within the questions that we were asking, was to really try and get users to look at what we were doing and use their own imaginations to help us think forward. So yeah, very excited to go into this with all of you.

Unknown Speaker 08:29
So the National Gallery of Art focused our evaluation on Dutch paintings of the 17th century. This catalog launched in 2014 as the first of our online edition series of digital systematic catalogs of the permanent collection. The publishing platform that the gallery developed is integrated in our main collection pages on our website, rather than being a standalone entity, and this architecture remains a primary distinguishing feature of our catalogs. We've used this platform to publish six catalogs to date either in full or partial launches in the platform's first five years. We chose to focus on the Dutch catalog for this study, because it's received sustained attention over the years with several rounds of additions or updates after its launch. And because it's representative of the other four catalogs that deal with paintings that we've published in contrast, we recently use the platform to publish the galleries Alfred Stieglitz photography collection, uses the same platform. But that catalog has diverged a bit in functionality because of the different demands of that corpus. The online editions Strategic Plan presumed that at the five year mark, we would regroup following the initial wave of launches and establish a new plan for the following years. However, to this point, we hadn't engaged in any substantive user evaluation post launch so we had a lot of questions that we were hoping to examine in the study and Key among them were whether our target odd Chances are indeed finding and using this resource, particularly given the collection, catalog integration, what were the effects of collection, catalog integration? And what features should be preserved or change as we approach the new strategic plan?

Unknown Speaker 10:17
All right. So for our, for our part of the study, we chose to look at a Roman mosaics catalog. It's one of five that we've published since sort of starting this work. It was launched in the spring of 2016. And it's, we did use, we're using something that's now called choir, which is an open source. It's not open source, yet, it's a software platform that we are working toward open sourcing, so that other institutions can use it. And so part of our impetus of we wanted to know, obviously, for our own selves, how we, how can we improve the catalogs that we're publishing for our own users, but also because we're using choir and because we want it to be available more broadly, and we want people to use it, we want to be able to make improvements to choir that will then learn from the study that we can then pass that on to catalogs even beyond the Getty itself. Roman mosaics and the other catalogs we publish, are unique in that they feature. They have multiple formats. So while everyone's got different formats, we've got a few more, I guess, is what I'm saying. So we have the online format, but then we do PDF as well ebook, and we have ebook formats. And then we also do a print version that comes from the PDF. And it's all from the same source. But it means that the catalogs are available in as many different places as possible. Also, in terms of the design, Roman mosaics, all of the catalogs that we've been doing have been kind of prototypes, and that we've been building on each one the knowledge of each one to kind of do the next one. So Roman was x is actually fairly different from what we're doing right now. But I wanted to, we wanted to look at it as because it was a sort of smaller catalog than compared to some of the others we've done before. But it had some features that were not present in some of the other catalogs. So like Lauren was saying, we didn't need to do an A to B comparison across all of our catalogs, but rather, what kinds of features and what kinds of navigation items and what kinds of essays and writing happen in these various catalogs that are resonating with readers, whether they're happening in our catalogs, or in others. So we wanted to know, what was what we were doing resonating with readers, what improvements we can be built in into those those publications and choir. And then also we're wondering, we we have a sense of what we're doing and why we think it's cool, and why we think these are interesting and why we think what we're doing is a good approach. But we're wondering, we wanted to know do readers understand that as well. It turns out, they don't really sell more on that later.

Unknown Speaker 12:41
And then at the PMA. As I mentioned, we launched our very first online scholarly catalog in spring of 2018. And it comprises just a sliver of the John G. Johnson collection, which the museum holds, it's approximately 1300 works. And this was published on the occasion of an exhibition, but was intended to sort of be the basis of a more systematic launch of a more systematic publication on the collection. So this feature at about 60 object entries, six scholarly essays. And it was developed after extensive benchmarking and formative evaluation, luckily, being able to build off of the experiences of other museums and what they had done. It was developed in partnership with design for context. And in the you can see some of the guiding principles that really emerged out of that, that work which I can claim no part of I arrived after that, that was very unhappy to see the work that the people in my museum had already put into it so thoughtfully. But in the finished publication, you can kind of see the expression of some of these principles and some specific features, which to our point, you know, some things that are unique to our publication, we do employ DUIs as persistent, you know, persistent to ensure persistence, viability and stability. Probably the most, the unique feature of ours is the integration of our of digital digitized archival materials. This was done in collaboration with a huge digitization projects by our library and archives. So if you click on in a footnote to this Bernard barons, and to John G Johnson, it takes you to that actual object record in the Library and Archives you have primary source material right there, which is wonderful and also causes as you might imagine, some confusion and navigation issue. So I think that for us, we felt everybody was immensely proud of this. It was the team that had never worked together for the first time. It was the first collaboration among publishing library archives and it we had brand new two wonderful, front end developer and web designer who were brought in thrown into the churning waters of this, like in the middle of the project brand new to the museum brand new to it, brand new and new role. So, for us, I think that we knew that it was it was the first there was going to be stuff that we would need to that we would need to fix some stuff that we needed to work out on the process side, it was always our intention to do evaluation and know that we would roll it into the next round, because as I mentioned, there are nearly 1300 objects in the Johnson collection alone. And research is already underway on planned systematic volumes on Netherlandish and Italian so it's the train has left the station. So we really know that it's important to get it right for those future things. So that kind of so you can understand what it is that all of these people were we asked them to evaluate, I'll hand it over to Claire, who can actually get to the meat of it.

Unknown Speaker 16:02
All right. So um, yeah, on to our methods and findings. As the teens mentioned, the study was really intended to give voice to the user. And our research questions focused on users in different ways. So who they are, what they want, what works well for them, and what they think of digital publishing and the scholarly value of these catalogs in general. And then finally, at the end of the study, we wanted to take time to reflect on the best ways to measure the success of digital catalogs. I think that's something we'll probably continue to do, because it kind of got cramped at the end of the process. But we wanted to provide some guidance on what we learned that could be useful to researchers in the future who want to get feedback on their own catalogs. So our methodology, and forgive me if I'm speaking evaluators speak too much, feel free to ask questions. But we sought feedback from both existing users using a pop up survey and from target users using an email survey. And we use mixed methods approach meaning quantitative and qualitative data. We use quantitative methods like analytics and closed ended survey questions so that we could draw some real conclusions through strength in numbers and make comparisons across the four catalogs and get those nice, like quick look at performance through numbers. And then the qualitative methods helped us dig into the whys of the user experience. And some of the more complicated questions that don't really lend themselves well to a closed ended question where you have to select A, B, C, or D. The qualitative methods like our focus groups, and the homework exercise that those participants had to do, allowed people to say what they really think. Like this particular quote, we want to share it with you all. While immediately convenient. digital publications are bound to destroy western perception of culture, and downgrade us all to a literally pre historical and pre critical stage, highly technocratic, but devoid of human sense. And we really appreciate when people took the time to provide their feedback in that level of detail, and with that level of emotion. And participants shared lots of gems, and most of which were less foreboding than this and much more kind and, but we appreciate the critical feedback all the same. I'm a big fan of the quality of data for reasons like like this. So what did we find? It's really hard to pare down all our findings for you. So we're gonna I'm gonna do my best and about four slides. But first of all, that the catalogs are attracting a large and very diverse user base, our analytics data. I don't have a graphic for this, but I'll just read it out to you. The catalogs are receiving anywhere from 3000 to over 100,000 visitors in a year, depending on how the catalog is integrated into the parent Museum website. And Emily talked about that special unique structure that is pulling lots more visitors to that catalog because of the way it's in integrated with the collections pages of the NGA. We use a pop up survey to ask about visitors professions. And I for one expected to find that users were going to be in the catalogs target group of scholars, researchers, museum professionals, graduate and undergraduate students. So a lot of those top ones on that bar chart. Also librarians and archivists, journal editors, those are all kind of the target group that these catalogs were designed for. But we found that in fact, it was a pretty even split between that target group and non target audiences including artists aren't enthusiast teachers, museum visitors, museum volunteers casual browser, so this very diverse user group using them for different purposes. And we also wanted to know, well, and so important thing coming from that is just to think about who these palettes are being designed for. There's the target audience, but then there's all these other people too. And what are the implications of that and how future catalogs are designed. We also want to note that for various reasons, our data and user professions primarily comes from the NGA and the AIC catalogs. And only during a limited timeframe when we were able to run the pop up surveys, because museums don't like to assault their visitors with pop up surveys for long periods of time, for good reason. So it would be interesting to see if this kind of finding holds up over a longer timeframe with other catalogs.

Unknown Speaker 20:56
We also asked a lot of questions about the catalogs digital tools, because an obvious benefit of these catalogs is the quantity of information that they make accessible to users, and through technology that can exist in a print catalog. And we found that participants loved that the PMAs catalog included all that extensive archival information. The people in the focus groups were really enthusiastic about that, and talked about it breaking down silos, which was really neat to hear from them. Conservators and those with an interest in artistic process really liked seeing like the layered images and the AIC catalog and the infrared scans of artworks that are in that catalog. But others were overwhelmed by too much technical information. So we found that finding the right way to organize the catalog and layer the information is really important. Another thing that's really important to the participants in our study was the ability to zoom and download high quality images, the image really was key in the mind of most of the users. And focus group participants were really quick to notice if one catalog provide better quality images than another. So they were very attentive to that. The catalogs also provides citation tools and participants, they sometimes quibbled about the format of those tools. But they pointed out that not only is the citation function, a really great thing for students, it also draws attention to the fact that these are scholarly works and that they deserve to be cited properly. So just by having that feature there, you're drawing attention to the fact that it's not just any old website, it's a scarlet scholarly work. So by providing citation guidance, the catalog is essentially asserting its validity. So that's a nice benefit of it as well. In terms of design preferences, this is probably where participants like to talk the most. Whether or not we were asking them about it. But the four catalogs in this study had very different designs and layouts and participants preferences and feedback on design and was extremely varied. And the the gift you see right now is the page turning of the Matisse catalog produced by the AIC. So that book like format, where you click on the arrows, and you can navigate page by page, much like you would in a physical book. So yeah, participants feedback was all over the map a lot of the time, what appealed to one user might completely turn off another person. But the reassuring thing is that no matter if people really liked one design over another, they really seem to adapt well to the different designs. So even if they had critiques, they were generally able to find what they were looking for. So despite all the different opinions, we did pull out some themes that kind of tended to reoccur from their responses. For one, participants generally didn't like the book like navigation of the AIC catalog. And this was an interesting finding, because in past usability studies they have done, participants liked this feature, because because it reminded them of a print catalog. And, and having it like a book, I think probably gives, or in the past gave people this comforting familiarity with that resource that they knew just in a digital format. But we found this time around in 2019, that people described it as cumbersome and they said they'd rather just be able to scroll down a long page of text, rather than physically click from one page to the next to advance through the material. Another commonality in the feedback is that participants want cues all the time to tell them where they are in the catalog. So with was 830,000 words that they need They help to know where they are in the digital space of the catalog. So things like breadcrumb trails, which are, I forgot to circle it, but it's like up at the top where it says nga online editions, and then it points in such paintings. So it's like this little map that tells you exactly probably all of you know what it is, it was a new term for me. But having that was really handy to people and having persistent links to a contents page that really lays out everything that there is to see.

Unknown Speaker 25:32
People really liked that. And in a similar vein, they didn't want to spend a long time searching for the special digital tools. So icons that were labeled clearly tended to be favored over really discreet or clean kind of designs. Even you know, it put a little more on the page to look at. So you can see the mgas tools in the corner, they're circled, just providing a quick spot to access all some of the different digital tools offered by the catalog. And then here's a look at the PMA they have these white icons that appear on each artwork entry page. And if you hover your mouse, it gets a little tooltip that tells you what it does, and people like that. So they could always look for it for in the same spot to be able to cite download, share content, or perform a search in this case.

Unknown Speaker 26:28
Alright, so another big area of concern for the study was this overall, this big question of assessing the scholarly value? What's the good of these catalogs? What to what's the value for the users? Do they think that these are scholarly resources? And overall, we found that users do you feel comfortable assigning this material, so 80%, so they would feel comfortable assigning these catalogs for their work. And so the the catalogs really passed an important test, and we were really happy to see that number so high. At the same time, some participants showed sort of a selective trust of what the catalogues contained. And the names of these museums conveys authority. So there's, there's value in that people look at the names of these institutions, and they're going to trust the content to some extent for that reason. But for some people, there's still a stigma against interpretations provided by museums, they, they might expect some types of biases to be built in because the content comes from the museum. So that was really interesting and interesting to dig into. During the grad students, it was grad students. It was interesting to dig into that in the focus groups. It wasn't just grad students. But um, and but it was, it was really, really fascinating to hear people talk about that. And so while people really love the rich level of technical detail on artworks, they really trusted that information, the stuff that wouldn't seem to have any room for bias to be built in. When they were looking at something like an interpretive essay. Some people were grad students are more likely to say like, Well, yeah, but that you know, it's the Getty or it's the PMA saying this well, what would other scholars say? I'd be interested to hear. At the same time, some of those kinds of concerns were relieved if the user recognize the author of an essay. So really, calling attention to the author can be an important strategy, especially if it's a well known name. Participants were also interested to hear that most of these catalogs have been peer reviewed. And it's not wasn't immediately apparent to most users. So really highlighting that is one way to further elevate these catalogs. And I think, yeah, those are the two big ones the peer review, and yeah, the authorship. And the last major finding we want to share with you is concerning the permanence of digital resources, not too long ago, researchers were hesitant to cite online resources because they were viewed as less trustworthy than print and would be just because the citation process is difficult and annoying in the case of online resources. As the previous slide showed, attitudes towards online resources are shifting. So the vast majority of our participants said they'd feel comfortable citing these resources, and that's really important. We also saw that among our email survey participants, there's just really high numbers of people who are already browsing and using these catalogs in their work. Participants were also less wary about the fact that digital resources might change over time. So this call we've seen in previous days that it causes anxiety. You know, if I cite this material, but then future people can't go back and see what I'm referring to. But in fact, a lot of our focus group participants said that they hoped museums would be adding new content, as a new information as it becomes available. At the same time, they really want to have clear information on when things have been updated, and what has changed. So they're still thinking about permanence, but it's just some of the anxiety is gone. So doing things like providing permanent links to content are important. The nga has the previous versions for download, and maybe some of the others do as well. And when people notice those sorts of things, it really gives them more security in inciting these things for their work. All right.

Unknown Speaker 30:52
So there's a lot to digest here. And, you know, the, the report that Claire has so ably written is, is very long and contains a lot more questions than we've been able to present today. But a couple of the takeaways for the Art Institute, I mean, right off the back, the book, like navigation is a big one. And although this may seem like a, you know, dig of the OSCE toolkit, it's actually a very reassuring thing for us. And so all of these findings around the perception of scholarly value have really represented for us the direction that we were, that we were hoping the field would evolve in, obviously, by producing all of this content. And so there's a lot of reassuring information around these ideas of scholarly value. The reviewers pointed to the Art Institute, specifically when talking about, we couldn't find who the authors were, we couldn't tell that it was peer reviewed. And of course, there aren't really a lot of conventions for like really signposting that upfront. So that's something that we're going to think about, with author names, yes. But with the exact terminology around peer review, it's a little bit less. So it was a little bit heartbreaking to hear from conservators who were like, I looked into the to the entries, and I just wished you had actually given me the X rays, or you'd actually given me the IRS. And they're right there, they just couldn't figure out how to use the layered images or the various tools. And so we're definitely going to think about better ways of indicating how to use tools, not just on a how to page, which is very important. People really liked seeing how to page but also right there on the page where the tools are. And then linking was a big one. Different users were interested in seeing linking better linking between the catalogs and the collection sites. And of course, the resources at Philadelphia were were a great example of how that can work with archival materials. But we have a terrible, terrible linking between our collections pages and our our catalogs right now. And it's not generally generating a lot of traffic. But we have an all new website as of a year ago, that obviously wasn't around when we first launched these catalogs. So we're still working on how to get these two systems to talk to each other. But everybody's excited about the content. We specifically got a lot of great comments on the the writing and the the content of the essays and entries. So you know, the Art Institute has two more catalogs coming up within the next year. But as we near six years of publishing on the OSCE toolkit, we will be looking to update OSCE and or move our existing and future catalogs to a different platform. So we're going to take these findings in hand and figure out what's next for us.

Unknown Speaker 33:51
So at the National Gallery of Art, the biggest areas that we see for future consideration revolve around the collection, page integration and how it shapes users engagement with the catalogs. The quantitative comparison of the traffic across the catalogs from the four institutions was I mean, probably the most startling finding to me on a quantitative level. And I think it's something that's going to be in the back of our minds throughout the rest of our analysis of the findings. We began this study, like I said earlier, wanting to know whether target audiences were locating and finding this resource. The answer seems to be yes, which is great. The findings reinforce that our collection pages carry an awful lot of weight for both specialist and non specialist audiences in really large numbers. So we need to continue balancing the distinct content needs of both as we're evaluating the changes that we're going to be making to the catalogs. And we also have to be, I think more responsible about making sure we're signaling to users what they're getting and where they are when in there in the catalog, for example, to the users in this study, it matters when you know that you're looking at peer reviewed content. So we need to be really mindful about signaling where it's peer reviewed and where it's not. If the transitions between our catalog and the rest of our website website are essentially seamless. We want to get more data on our users and their journeys. We want to improve the catalogs viability for target audiences. And we want to take a closer look at current referral traffic and see whether we're capitalizing on that interest across all of our catalogs. Some of the problems that users pointed out, were caused by the integration in the collection pages that included users struggling to understand the full scope of the catalog, like, what are all the entries in it? What are the tools that are available, what are search and filter features that we have present, they want clear navigation channels. And also, we have duplicative redundant tools. In our catalog. A single object page might have, for instance, a search field that's specific to the UI catalog and also a search field that's for the entire nga.gov website. Users rightfully called us out on that for being super confusing. In terms of features, I was particularly interested in user feedback on reader mode, which is what we call a to frame view that lets you view text side by side with images and footnotes. Most sections in the catalog can be viewed with reader mode, toggled on or off at user choice. But the entry essays, the catalog entry essays alone force you into reader mode. And as it turns out, reader mode is really divisive. Some people love it, and other people called it this is a direct quote, pretty terrible to use. It seems pretty clear that if you're using a laptop, you probably hate reader mode, and are going to end up downloading a PDF out of disgust. But they love that we offered PDFs. So that's great. We also know from the survey now that 55% of respondents to the email survey report using a laptop so it's clear that we need to improve the experience for laptop users as we turn to our next catalogs including French paintings of the 19th century.

Unknown Speaker 37:18
All right. So I didn't put on the on the slide. But one of the funniest, clearest feedback we got on our focus group comments. So we got Claire, get, you know, we had we got all the comments and all this stuff back sort of pages and pages of it. And we use in the Roman mosaics catalog. The curator who wrote it uses the word comp or Rhonda as a headline, which I think just means comparisons, right? Like, or something like that. Three different people were like, What is copper Ronda, like, get rid of that? We're like, they hated that word. So that's one thing we're going to try to tackle. It might be actually the hardest thing we have to tackle is trying to get them to stop using copper Rhonda. But really, I mean, who knows? Yeah, yeah. The so one of the things is, I wrote your signposting navigation. So with ours, we have the table of contents in two variations in two different places in these catalogs, for a number of reasons. But it doesn't neither neither neither of which are good reasons. Because it doesn't work for for our users, it just was really hard for them to find the contents. And then to know where the contents were, once they like how to get back to those contents and why there were two. We also want to know I want to work on making links and traveling along the catalog more, sort of knowable ahead of time, users would either find that they'd click on a link, and it wouldn't do what they expected. Or they would click on a link and they would go somewhere, and they wouldn't know how to get back or where it had taken them. So trying to be more or like is this a link out of the catalog or just to another page? Is this a link that's just a pop up that it's gonna not go anywhere, it's gonna be fine. So trying to be more thoughtful about setting and giving our readers some expectations about what's going to happen when they do things. Because what we've I think some of the comments we saw were, people just stopped clicking on the links, they don't want to explore, they're not going to find those things we have there. If they're scared to click on something, because they think it's going to take them somewhere. They don't know how to get back and making it easier, easy for readers to return to where they were is the other thing. And then the second bullet there is show them the money. And that's what I was referring to earlier, when I did my spoiler, when I first started talking was that we have these features that we're working into our system, in the way we publish, that we're really proud of and think are important. And that sort of the multi format thing. And the fact that we have, the way we publish allows us to track changes to a catalogue down to the character level, and that we can share that back with people that we have a very specific policy about revisions and revision history, that these things are built for permanence, that they're going to be that we want reader that they're built in a way that we think will last much longer than a typical website would. But of course, we're not doing a good job of exposing those things and those features to Our readers that we think they'd appreciate. So we want to do a better job of that, all with a mind, though of I think, this notion that people, I think it came out in the catalog in this in the report and study, that while they want this information, the users want these features, they want people to find them, they also don't want it all in the way either. So I think we're gonna have to find a balance of how do we exposed these things at the right times and the right ways. And so that's going to take some further work and testing again. Yeah.

Unknown Speaker 40:26
And so at the at the PMA, I think that we won. I mean, we're, we were thrilled in general, that there was a positive response, I just, and I'll reiterate this, again, that, you know, we're all we're always focused on what's not working, we're focused on that, like, terrible quote, of the end of, you know, Western civilization. But, you know, in general, it was like, there was a lot of value, there was even greater value for people after they had, you know, they kind of went into it like, this is probably worth, you know, generally feeling good about it. And then after using them felt even better about them, you know, like, it's part of some of the questions we had. So it was really gratifying for us, you know, especially this being our first effort. But there were, there were certainly a lot of things. I mean, one of the most striking things for us was, you know, when we were able to look at the analytics and see like how people were getting to the different catalogs, they were getting there in very different ways. And we have almost no, we have almost no like results coming from search to our catalog, it's almost all direct traffic, which doesn't tell me that it's like that there's that many people. So it's like, you have to have the URL and find it to get to our catalog. And we've already been talking about why structurally, that is the case and working on improving that. And then also, because it is embedded sort of within our overarching website, I think that there was a lot of that what I spoke to, before that, that idea of that advantage of you can go from, you know, the object entry to the the object page in the collection search, you can go, you know, to archival materials, like a lot of confusion of like, Am I in the publication, my out of the publication? Where are the boundaries? How do I, you know, finding their way back. We also don't haven't learned talking about this for the Art Institute, like right now, if you just do a search in our website for, you know, one of the paintings that's in our catalog, you you do not get a result basically, like you get like the object entry for that. And there is nothing on the object entry that tells you hey, person who just searched for the student Leister, you're probably really interested in this, like more complete entry, just no idea. So and some of these were things that were on our tick list that we knew we wanted to get to that we just, you know, couldn't do for the first one. So, so it, it reinforced some things that we already knew, but that gave us new motivation to address them. And I think that this is something that I think a lot of people talked about, like, I think that people really want us to hit them over the head a lot more with this is what you do, this is where you are, people really did like the kind of more Brout ours because it is a scrolling navigation, it is sort of more web based people seem to appreciate that it's like this is I expect that I'm going to be enter a world of interconnected content, and that I like this aspect of it, and that I can like browse and search and find these connections and make my own path. But they also were just like, but how many Bosch paintings are in this, like, I can't really figure that out, you know, like they can't, there were just some things where it's like, trying, because there wasn't an index, because we kind of felt like, well, people will search us you don't need an index That's so old fashions, but like, they really want it, they want to kind of understand the boundaries. So I think that that's an you know, a lot of these things are, we can already think of easy ways to to make those solutions. And I think that also this idea of, of scholarship, people, a lot of people you know, DUIs are much more common in, in the sciences and other part in academia and other fields, less people were less familiar with them. But you know, the whole point of them is that it, it allows for, you know, persistence and viability, and it should, when it was explained to people what they were, they thought, Oh, that's great. That helps reassure me that I should cite this. So again, can we just explain this in a more overt way for people as they're using it? And also, for purposes, this was, as I mentioned, it coincided with the opening of an exhibition. So there was a kind of time constraint on this in a way that there wouldn't, presumably will not be for future, the systematic catalogs, which precluded peer review, but I think that we just have to this is something that actually is just we've been talking about in our print publications that we don't also don't have a standardized practice, and is really something that it's just like we have to make the space to make This happened, that regardless of it being digital or not, so there you know, there are kind of also a lot of really interesting things that we've talked about that we've learned in the process of this that are absolutely applicable to like all of our publications, regardless of I mean, just in terms of content and expectations. So

Unknown Speaker 45:22
So, as good digital publishers, we are digital publishing our evaluation of digital publications. Thank you again to Greg for so ably putting this together and choir. We say choir enough. So I mean, as Katie mentioned, at the beginning of our presentation, we invite you to read this, look at it, take it apart, ask us questions. We really hope that the data here can be a springboard for more conversations and more study. But to continue this conversation in a more organized or formal way, Katie, Greg, Emily and I have put together what we are now calling the museum publishing digital interest group muku dig. Anyways, we will be hosting regular conversations around topics relevant to digital publishing and museums, and we plan to hold our first call in January. If you're interested in this endeavor, please sign up through this GitHub page. Thank you again, Greg. And we look forward to talking to you more about this in the intervening months, but especially in January.

Unknown Speaker 46:32
All right, I think. So we have we have a robust 12 minutes for any questions. As I said, like this, there's a lot that we were not even able to scratch the surface of. So please. Yeah.

Unknown Speaker 46:56
So I'm curious like with the analytics that you have found that almost half were categorized as other? How you guys are planning to go through that with the next steps? Are you thinking about broadening your content so that you are reaching other audiences? Are you thinking about going even more specific to scholars and thinking about how to reach the scholars more?

Unknown Speaker 47:25
I mean, I think one thing is that that that finding, I think of very much in tandem with the finding about people who wanted different parts of the of the content, but wanted to be able to skip over others. So like, there were some people who really loved the in depth curatorial entries, but I kind of wanted to skip over the the technical information or people who were really into the conservation, but want to just get both out of the parts. So I think that it displays to what you were talking about, Greg, in terms of the in depth content is useful and is wonderful. But if there's a better way to parse out where you are in the catalog, what is featured, and to jump between different sections, it'll be more applicable for more people who can then more easily navigate what's there.

Unknown Speaker 48:12
Now we have one thing that people liked in our publication is that each one of the entries has at the very top has 150 word overview. So and then they have, you know, it has in depth exhibition histories and references and other things. But at the top of every single one, there's just it's like the elevator pitch for like the century. And just for people who are skimming through trying to decide, do I want to really dig into this, they found that really helpful. So I think something like that signals that's like, that's probably not going to be in, it's helpful to a researcher who's just trying to find a bunch of stuff. But also for somebody who's just happens to be the I've stumbled upon this, which again, doesn't really happen with ours, but does happen with others, you know, is is a less, it's not immediately, like, hitting them over the head with, you know, detailed biographical information about another landers painter, you know, it's so for, it's kind of helping to address that for that audience. And everybody likes images. Yeah, regardless, all audiences really like images. So investing in that is really worthwhile.

Unknown Speaker 49:16
And that data on target, it came from the pop up survey on whether it was in the target group. And I believe we ran it to see if both groups could find what they were looking for. And there was no difference. They they were finding what they wanted equally well. So that's encouraging.

Unknown Speaker 49:32
Given that a lot of the feedback that you shared here was revolving around like the user experience issues. I'm wondering if you can shed a bit more light on what were the resources that were available when these like test publications are being designed or devised, you know, how much kind of like prototyping or you know, focus grouping or were the conditions limited so that you know it's now getting surfaced through this study. And then based on the findings of this, how is that helping your respective organizations think about resourcing to implement these, you know, seemingly like no brainer or so obvious design improvements.

Unknown Speaker 50:13
I mean, I'll say that, and actually, Jasmine, who's here did a lot of the user testing. So there was, as I mentioned, we were able to benefit from a lot of, you know, we weren't part of the original OSCE cohort. So we already had kind of that final report, we had the frankly, green and web report that had been done in like 2015. So all of those things were kind of inputs that went into, into developing it. And they did a survey as with the target audience of scholars, to kind of talk about their preferences. And then throughout the design process, they did do user testing in like midstream. That was, you know, focused on it, you know, on usability, navigation and things like that. So I think that the, you know, we felt we felt like we had had done that, and felt kind of good, good about that in the work that had gone into it. But also, you know, until you have the finished product, and it's a naturally, you know, it's a limited group. And also, I think the thing that was exciting is this sort of comparative thing, because people can say, like, sure, that's fine, you know, that image tool is fine. But, for instance, the people really, really liked in the National Gallery, that you can compare whatever you choose, we have a comparison tool, but we predetermine which things we think are worthy of comparison with the image. And so people could look at ours and say, that's great. But then when they see the national galleries, it's like, oh, but that's better, you know, so and they wouldn't naturally, they might not necessarily offer that up, spontaneously, if kind of what's working, it's fine. But then when they see something that's working better, so I think that that's part of it, and others can speak to whatever.

Unknown Speaker 52:02
I mean, both the National Gallery and the Art Institute were part of this Oski, the online scholarly catalog initiative, which was a long term, project that into that had many phases, and there was a lot of development, there were 10, institutions, eight, nine institutions. That participated. So there was a lot of there was a lot of thinking cross institutional thinking that went into the development of some of these platforms earlier on. But it's, but it's been a while since then. And I think, you know, yeah, some of the things that people don't like about ASCII now, were really popular at that time. And we didn't even talk about this. But there's like a login feature where you can take notes, and you can like, make little sticky notes and like, annotate your whole own copy. And like, everybody was like, Oh, I can, we got a lot of comments that were, I can see why you did this. But, you know, scholars have their own way of taking notes, they don't need a special way of taking notes for a certain platform. They just have their way of note taking. And so it doesn't make sense for them to like create a login and go in and do things. And so you know, it's, we think we like there's some things about ASCII that I think make us laugh now because we're just like, oh, that's, that's ASCII, or that, like that's a particular type of publishing. But until we could actually ask people and go through six years of publishing on the platform, it was hard to determine what was going to stick out what and what was going to change. So

Unknown Speaker 53:36
yeah. I just want to add, actually, to the resourcing question you asked, which I thought was good, too, in terms of like, how do we take this and then advocate for resourcing to do the work? For us? I think it's actually more a question of prioritizing. So this gives us a way of saying the like, these are things that we want to tackle this is like, we know all the things we can do. And then some, and it's a question of like, which ones do we want to go go after, and we want to go after the ones that are gonna have the highest impact for the sort of the greatest breadth of things. So this is going to help us rather than add new projects, just prioritize the ones we do, I think, for us,

Unknown Speaker 54:12
and for the gallery, I'll just add that because of the collection, page integration, were anything that we do in response to the findings house, as long as that architecture remains the same. Anything we do has to work for the rest of the website, you know, we don't get to just change functionality change tools for the catalog without it, you know, potentially populating out to every single other collection object page. So it makes the whole process a little fraught a little bit.

Unknown Speaker 54:46
Yeah. So a lot of these catalogs a tremendous amount of information has been abrogated in them. And obviously we spoke to a lot of people and I'm wondering how you're addressing the permanence issue. Obviously, the technology It's used right now. So it's become obsolete. Like, what is your anticipation when that should be occurring? And what are the sorts of plans going forward? And

Unknown Speaker 55:08
how do you address that big issue? Thank you. I love that question. It's like I pay you money to ask question. Well, say one more time. Yeah. So that's a great question. And that's, so we mentioned the online scholarly catalog initiative. So that happened from 2009 to 2015, there was a final report that was written after that, if you haven't seen it, it's online. And we link to it actually, in this current study. But one of the main findings of that of the OSCE thing was, at the end of it, one of the they identified remaining challenges and that longevity question was, was a key one of those. So for us the way we're addressing that, and we, when we started doing our own publishing and our building choir, we, we wanted to tackle that head on. And the way we're doing it is through a couple of means. One is that choir uses a plain text format, so that the core content stays in plain text regardless, so if everything else breaks, if the online thing breaks, if the if the PDF breaks, whatever, we at least have the core content in a non proprietary format. But then that the website itself is built in a way that doesn't require a back end or a server running to maintain it or to serve it out to users. So really, it's just a set of plain files on a server somewhere that can be downloaded directly, they don't, you don't need to update a WordPress instance, are everything. So we're trying to do some web things as well, that will increase the lifespan of that website. And then of course, the multi format. So so the fact that we're printing out a putting out a PDF, and an ebook and a website, and then printing a copy as well, are all hedges against loss, essentially. So that's, that's how we're trying to tackle it. Because in the end of the day, it is a website and yeah, the technology is gonna change, browsers are gonna change some, you're gonna lose some functionality, but we want to make sure that the core content remains accessible.

Unknown Speaker 57:03
One way that the PMA publication addresses this is that it's actually built, there's a sort of a custom built pipeline, that pulls Word file. So it's works with like the native functionality of word and using the A lot of the format and native formatting of Word, along with comments that have shortcodes in it. So if people are familiar with choir, like a lot of the kind of markdown stuff that you might see embedded in something inquire is actually put into the like, comment fields within a word document that's paired with metadata, metadata, spreadsheets, which are run through a pipeline, which generates a JSON file, which is then used to build the website, and it's pulled together with, you know, image files and other things. But effectively, what that means is that the Word document is always the is always the master file. And you actually when you update the, the site, when you update the the the built publication, the design publication, you make the updates in the web, in the the Word document, and then rerun the pipeline. So it it effectively make sure that you always have, and this is like a kind of like holy grail thing and publishing because the word doc quickly becomes obsolete and you kind of, you know, you move, you leave it behind as you like, work through other parts of the design. So and that was very, you know, purposeful that was that, you know, part of the reason for that was to have this that, you know, regardless of, of what happens down in down the road that you have, you know, all of the composite parts that make up the publication remain intact. And we do have, as I mentioned, because we have DUIs and there is a system that sort of set up with this idea that we've pre registered kind of multiple, so that if you do an updated version of something, you would be directed to that new updated entry if there's a substantive change, but you would be able to, to go back to the previous version that would be like, it's something that just because we haven't updated it, it's not something that people are aware of. But again, we could communicate better.

Unknown Speaker 59:11
I think we're at five o'clock and we are our drinks across the I don't want to hold everybody thank you very much for coming.