Staff are Clients, Too

In 2018 the Los Angeles County Museum of Art launched a custom developed system, called the Move Object Request Application (MORA), designed to track the movement of art throughout the museum and replace a previous decade-old work order site. LACMA’s robust acquisition, loan, and exhibition calendar averages over 140,000 art location updates a year. Museum staff use the new web-based system to communicate with art preparators and collections managers about anything from gallery rotations and viewings to unpacking crates and sending objects for conservation or photography. To develop this application, the team performed extensive requirements gathering and employed an iterative, design thinking approach to the project. Rather than upgrading the previous system - with all the same assumptions - to a newer iteration, the project began with the question “What problem are we trying to solve?”. Adding to the complexity of deeply ingrained workflows and departmental dynamics was the fact that the primary user department operated in a mostly paper-based environment, which provided an opportunity for creative solutions in the usability testing stage of development.

Transcript

Unknown Speaker 00:00
Hello, everybody, welcome to our session.

Unknown Speaker 00:02
I'm gonna close the door. Okay, thank you.

Unknown Speaker 00:08
The title of our session is a somewhat provocatively tongue in cheek named staffer clients to applying design thinking to an internal facing app that's still cool, largely born out of discussion that we were having internally about the fact that there's a lot of talk about human centered design and design thinking when it comes to public facing apps. Rarely does that amount of resources and attention get Thank you. Is afforded to internal facing apps that actual museum staff use on a day to day basis. And so today, we're gonna go over a project that we launched in August of last year called the move object request application or the Mora app. So we'll be referring to that quite a bit. And my name is Yvonne Lee. I'm the head of collection information and digital assets at LACMA.

Unknown Speaker 00:54
My name is Meredith Steinfeld. I am currently the digital platforms manager and archive specialists at the Museum of Art formerly of lackluster I was here when the project kicked off.

Unknown Speaker 01:04
My name is Amanda dear off and I'm the TMS database administrator at LACMA.

Unknown Speaker 01:10
So with that, I think we're actually just gonna go ahead and kick it off with the very beginning the background and context to this,

Unknown Speaker 01:16
the great location update war, which is, it's not too far off from the truth to be honest. So back in or around 2011. And earlier, there was a kind of a bare bones web form that LACMA used to both request that objects be moved and to execute their movement. They went through a single person and such there were significant bottlenecks. And there are so there was really no control. And in a siloed notoriously territorial and hierarchical department, the bottlenecks were amplified request for transparency for speeding up the process or for democratizing the process were not considered. This form was reported to have changed the old up the old database when the requests were complete. None of us were here, slash none of us were using them at the time. Therefore, none of us can confirm. When LACMA switched to TMS in 2012, the form was rerouted to be able to pull data from TMS but it was a one way relationship. And there was still that kind of siloed territorial aspect to locations. And we tried to roll out a software form of object relocation but were met with pushback. And we weren't actually asking much we were asking that the people that who are responsible for physical location location changes, communicate with the people who managed the databases. And but at the time, the head prepared or didn't want staff using TMS at all, for no other reason that he didn't want staff using TMS at all. And there are many reasons not to want to use TMS. But not wanting to do it for not wanting to do it is like this tautological inefficiency, that just like doesn't make sense. So we tried to introduce a paper system. And he said that was too much work. We were not asking much. We just wanted to know what was moved, where it was moved, when it was moved, and by whom. And the webform didn't have the capacity for that output either. So really, pen and paper was kind of the easiest way to go with the time in 2012, and 2012. And so the idea of kind of fixing the process, we kind of kept kicking it down the lane and kicking it down the lane and just didn't really have time for money, or the bandwidth. Just because we had just launched TMS and then we launched the dams, and it was just this whole thing. And by the tummy kind of we were told by the powers that be that something needs to change. We were encouraged by our information systems department to consider an off the shelf product. And that makes sense. They're potentially cheaper. If you know the software exists, why would you reinvent it? You, you get standard development cycles, you get upgrades and patches, you're not wholly reliant on developers because, you know, we've I'm sure many of you have had the issue where you're handed this bespoke product, and then it just becomes a brick because nobody knows what to do with it. And we also tried to talk to other museums and said, like, what do you use? Like what are your digital tools for relocating objects outside of TMS and we either got no we use pen and paper or it's not great, but we do X a lot of shoehorning of software that's not necessary. really appropriate for the job, but existed kind of like a, you know, just there

Unknown Speaker 05:11
so then we had to, once we realized that we did really need a bespoke system, we had to then argue for the money for good development, and probably should have invoked Chatham House rules on this slide, but I don't work for LACMA anymore, so that's fine. So the UK has reputation as a very brand conscious institution. You know, the Art and Film gala where Keanu Reeves just announced his new girlfriend like that was at LACMA. That's like a thing that happens. They like to spend their money on high profile projects, you know, we trucked a rock through Los Angeles, and an egg a giant egg from Guatemala. And it's not to say that these projects are unimportant. But they're it's it's a very flashy institution. And staff whose jobs are infrastructure don't get that kind of consideration when your job is rote work, when your job is the steady state sort of work. Nobody actually sees what you do. And whereas staff who get to ship these really flashy projects, they get accolades. And they get the money. And it's understandable, and that these projects bring visibility to the museum, which brings money because funding is really volatile. But the unfortunate part is that when your job is only steady state work, and you see the other people getting the money, and the tools and the attention, you start to breed resentment. And infrastructure staff such as our preparers feel really overlooked and underappreciated. You know, C suite executives and C suite really just see art go up on the wall and down from the wall. And that's really all they see. And they don't know, the different things that has to happen. And I don't want it for any of you who are in administration or C suite or are executives. I'm not trying to attack you personally, I just want to be clear that it's when your focus is such a bigger picture, you don't get to see the nitty gritty, and that's really what was happening at LACMA. But it bears noting that, you know, that they're like that they needed help. And administration understands that there's a human element to it. They understand that, you know, people manage a database, and people run servers, but not what it takes or why money has to go to them. You know, for those of us who do this work, we're a hand wave or we're fairies, it just happens, we make things appear. We're incentivized to work out of the goodness of our hearts or desire not to get fired, equitable pay for labor or proper funding, or recognition for other or other fancier, more visible people. So in the face of all that, for administration, who likes glamour projects, who view labor with at worse apathy, or disinterest, we needed to argue that our art handlers didn't have the tools they needed. And moreover deserved funding for a project that could actually be released with fanfare because this wasn't an egg or a rock, or Keanu Reeves, like, there's going to be no press coverage. So why would you want to pay for it? And why would you want to pay in the hundreds of 1000s of dollars for it? You don't. So at the point that we had their year ish, they asked for a rough estimate, which is bad. development costs, what it costs, you can't say you can't put a price tag on it, because developers cost what they cause. Or they're going to put a senior developer, a junior developer, they're going to put two What's the timeline you don't know. And while you might be able to see like, roughly in the 10s, or hundreds, you don't know where in the 10s or hundreds, so when budgets tight, every cent counts. And the thing is, if we didn't give them an estimate, if we asked for too much, they're not going to give us any money at all, that's too much. And you have pen and paper, you know, and if we ask for too little, then we have $30,000, which is great, but $30,000 isn't gonna get you anything like literally nothing, maybe it'll get you an investigation. And this was only if they're willing to say yes, and more on that later.

Unknown Speaker 09:40
And, you know, the like there's not collection information and digital assets as the department and LACMA is under the registrar's not under it. And so as such, when we were kind of coming to them saying we're going to need your cooperation, we're gonna need your collaboration. We have this project need to help our art handlers We were really put through our paces, because our IT department had been burned by vendors who develop products and left them with no support. And we had to like we went through that pretty recently. And we understood that they were nervous that and they felt like we were being Cavalier. But we also felt like there was a territorial element to the reticence. And because he does under collections, and even though their departmental scope extends beyond TMS, beyond digital assets, there still was a sense that this department wasn't staying in their lane. And since leadership didn't necessarily see collection information, as tech workers, just people who use technology, we have the added issue of trying to prove that we do we are tech workers, and helping people using digital tools for collection purpose is is our lane. And then getting the kind of collection staff and the people who actually needed to use this to buy in, you know, everybody knew that there was an issue, everybody knew that there are holes in the workflow, everybody knew that the location process as a whole was just kind of this black box where people are just constantly chasing people around in the hallways. Where is this? Why is this not up, and then it also has like knock on effects to the website, where if we hang an exhibition, but the paperwork doesn't get back to us, then the exhibition doesn't look like it's online. So everybody was theoretically really excited to have this. They didn't want as much human intervention. They, they wanted something different. But you know, fundamentally, they're asking for an easier and more transparent process. And when everything is kind of so broken or so difficult, it's really easy to get people on board. In theory, the difficult part is getting the people who owned that kind of siloed territorial area, to be on board and to understand that they have to democratize the process. And more on that later, too. And kind of one hurdle right off the bat when we were doing these kinds of buy in interviews, is that despite the desire to fix this kind of issue, there's a wild variation in tech literacy among the stakeholders that we identified. You know, it's hard to explain a concept in the abstract, it's even harder when people to whom you're describing a concept. see technology as the hand wave, The Who, you end up needing to heavily manage expectations in the conversations, regardless of user's ability, I'm not going to build you a unicorn, and this isn't a silver bullet. And you need to suss out the broader issue, user might spend half the meeting complaining about a single exhibition, or even a single colleague, and you can redirect but ultimately, the more constructive approach is to understand why they're complaining, it's okay that these conversations turn into therapy sessions or even blame games, it's part of the process. And that is why we hired experts. Theta put up the money ourselves, in part because we had a consulting line item. And also because we really needed to be able to exert control over this project. We were the most impartial department or at least the one with the broadest agenda, we needed to be able to try and pay for this so that we can really control how the project went. So we did the whole three bid process. You know, and even though we had staff who could do this work and had done it on a similar scale, we felt that a third party would be more successful. People are willing to be candid with consultants, there's no need to preserve workplace harmony when you're speaking to someone who doesn't work with you, and has no agenda other than fact finding. And addition. Even a department with a bird's eye view of the museum's such as theta can be too close to an internal process, especially since it can be hard to extricate personalities from process, when a key player in a workflow is a missing stare. That is there's a desire to design around them. Even when you know that that's bad design or you fall into the opposite scenario and try to Field of Dreams a solution regardless of the players, you build what you believe is the ideal workflow and you hope they'll show up because it's the right thing to do. So we thought of vendor kalite matter. They never actually worked with museums, but they had done a lot of architectural evaluations. So they actually sent their CTO His name is Ben. And he spent a week on site interviewing as many people as possible, sometimes more than once, as well as daily check ins with Sita and for the nihilists who only showed up to the meetings because their boss told them or the staff who were so profoundly territorial that they barely contributed to the interviews because they didn't want to lose any of their power. Ben found them on their cigarette breaks and smoked with them. eking out unnecessary conversation in an organic fashion and getting a modicum of trust. Ben is a health conscious nonsmoker. Like I want to point this out like not a smoke does not smoke does not smoke. And I'm not advocating that you risk lung cancer in order to connect with your constituencies. And you shouldn't make that a criteria for your vendors. But this kind of illustrates that gaining buy in isn't limited to meetings and training sessions. And so at the end of all this, and hopefully, he's never going to sue us about that. He gave us a formal evaluation and the entirety of his personal notes. Okay, so now we have Ben's notes we have people talking to this sounds great. Except for like, remember how we talked about how we were asked to give a development estimate. And we had to give a number and that number was really, really wrong. Because who knows, because they want to get paid for their labor, and we want to pay them for their labor. So because of this bad idea of providing a rough estimate, and even when a key stakeholder stepped up, and gave us extra 10s of 1000s of dollars, we still couldn't afford the bid to actually build the software solution.

Unknown Speaker 16:15
You know, and unless you're, you know, like some fictional world, or I don't know, the Getty, you know, you're you're not immune to financial woes just lack Mazhar under the veneer of privilege, but they're still there. So when we realized that that was a thing, we also recognized that we had to take on some of the work ourselves to kind of D scope the bid, and be able to actually afford them to build. And LACMA does have some very talented developers, but it's just not feasible for what they had to do to attack on that additional project. So we viewed our their engineer or lack was engineer, I say there are I don't work there anymore. It's very hard for me to kind of act as a partner. So we discussed hoped, and we removed wireframing, data modelling, and user interviews from the scope of work. And my colleague, and they're in the IT department, James Attali and I kicked off the wireframing with just a few basic images just to get a sense of what we're getting ourselves into. And also, I've never actually used wireframing, I'm a database manager, I don't know anything about that. And so I just kind of dove right into the software. But what I was really unprepared for was one, just how many wireframes I had to make, and to what kind of questions they raised. So we had Ben's notes, and we had this deep understanding of what we were doing. And we have the proposed statement of work and our own institutional knowledge. And I still couldn't build the full deck. And I just kept circling back with several users. Though, I found ones that were able to kind of have that abstraction or ability to conceptualize the application. And I would sit with them, you know, showing them the step by step by step frames, you know, some users had a vested interest in certain functionalities. And overall feedback was more difficult to solicit. Because all they want to say is kind of do X kind of do X can do X, you know, and then on top of that, when I would go through these things, just different areas of the workflow that needed to be covered, kept coming up, and I kept having to remember and it was got anybody who does wireframing for a living, bless, because that was

Unknown Speaker 18:31
tough.

Unknown Speaker 18:32
You know. And, you know, there also situations like, well, we'd put a request in, but the location is actually not going to change, I just want you to put this in a box. And I ended up at some point like sitting on the floor with Amanda, like surrounded by my printed out wireframes just like rearranging them until the process made sense. And we ended up with 50 in our deck, along with the revised statement of work. And just see what they could build that I just wanted to. And we were just trying to do this, not just to get the cost down. But also because we didn't want any kind of scope creep. If we were able to really kind of cover every user scenario, then scope creep, or at least we hope scope creep would be kept to a minimum. And I just want to give a quick cautionary tale. Before I hand over the reins to you upon is that when we were soliciting this bid. When we made these 50 wireframes. We made this revised request for proposal. Somebody asked us Would you consider sending this out to a dev shop that will just build it like not any questions, not any interviews or conversations it's that you send them the specs and six weeks later you get a thing they will precisely what is outlined in the request without dialogue, user acceptance testing, or any kind of best practice handed off. And then you receive your software, no support agreement, no warranty, or no onboarding. Don't do that. Even if you think that you have 100% of everything specked out, don't do that. Nothing is 100% specked out. And if you think that is possible, then you are 100% likely to be in possession of an expensive brick.

Unknown Speaker 20:29
cool with that, let me go into basically what will be a pitch for design thinking. So the next section that I have is build test, except rinse, repeat. So at this time as January of 2018, I was brought on as the digital project manager at that point, Meredith had already left us for Dartmouth, we were very sad to see her go. And I had been given sort of the current version of the software in its very early discovery phases, still with some of the development scoped out in the form of technical requirements document and, and a series of data models and some matrices and just kind of odds and ends, and was given sort of the mandate of go go and build this thing, you know, and make sure that the users understand what this process is like. So the next slide is something that I actually found on Reddit. So if you're familiar with Kubler Ross is seven stages of grief, this is five out of seven of those. And no matter how empathetic and thoughtful, any one of us could have been about this process, there was certainly several of these in play throughout the process of the development of this product that would eventually become Morrow. Sometimes a couple thrown into there together. So I would say that it was a major learning curve, not only for the staff who were being possibly empowered for the first time to ask what is it that you need to get your job done at the end of the day, but also for our technical staff as well, because we are somewhat traditional in the sense that I get the sense that earlier projects very much followed that sort of Cascade waterfall, like by the time that it launches, you know that it's at least six to eight months. Not only overdue, but also obsolete, right? The way that museums work, I feel like every workflow, there's the idealized version of it. And then there's the reality of it, which branches out like a fractal, I've been told not like a tesseract. But like a fractal, because that's actually how the work needs to get done. And this system needed to somehow be able to be wide enough to close the arms around all of the different iterations that people's workflows would take on post launch. So the major steps in this development process involved, knowing who the users were and checking back in with them time and again, putting iterative prototypes in front of them, which was very new for them. A lot of them didn't even realize that our department works with two, if not three different systems. There's the production or the live system, there's tests, and sometimes there's even a beta. That was completely new to them, they had no idea that there was so much work that went into it. And of course, not only did they gain exposure to that little bit, but they were invited to participate in these different phases. So that was very new to them. And like Meredith had alluded to a lot of their work is very tangible. It's very physical, and sitting them down in front of something like software or something techie, and asking them to think abstractly and conceptually about their work physically, was definitely a challenge, I think for a lot of folks. The next thing was, of course, communicating new and changing business rules, because as we kept working on this project, a lot of things started moving around. So the former head of the department that was our primary stakeholder department left very suddenly after, I think, a 27 year tenure at our museum. And this person was reigned over his department. And when he left that left, a huge opening and a lot of room for movement. And we saw during this time, a new department came in with a lot of different with a lot of changes to the way that she wanted her department to work. So that was something that we had to sort of address mid midstream. On top of that, we knew that we were gearing up for a major capital campaign, if you haven't been following up, LACMA is closing down the 1965 William Pereira buildings, demolishing them early next year, and then we'll be building a new permanent collection gallery space designed by piers on Thor that is set to go up roughly around 2024. So this would of course, change the way that people would work. Because the new permanent gallery space wouldn't have been built. All of the storage would be taken off site to various locations, and we weren't sure how that would affect the main bread and butter of this application which is moving on work from shelf a to shelf B or gallery C.

Unknown Speaker 25:05
Finally, it was continuing with the Oh, I'm sorry, step four was communicating that this is an MVP. So there's a deep dive tomorrow, actually, that I'll be attending about getting beyond the MVP. But this idea that there is such a thing as an MVP, or that you would release a beta and not like a beautiful finished product was absolutely news to most folks who thought that we were going to rebuild Facebook in some way to support their, their workflow. We did not deliver I don't think that we're ever going to deliver. But now I would say we've been consistent enough with the communicating or around what an MVP is that folks note. First of all, this is the first but also the first in a series, I think that that's really important for getting user by him, especially for staff who often feel like they're neglected and told that they have to use these tools that don't fit museum practices at all. Finally, it's continuing with the training hacks and workarounds until we secure resources for v2. So we're actually gathering user feedback. We have been, in fact, sometimes we've been trying to dial down the volume of the feedback. But we've been getting a lot of really, really good feedback from people who clearly are using the system day in day out and have ideas. Part of the managing of this user feedback has to do with user groups who think that they're sort of the only people using the system, when of course they're not, databases tend to go ahead and delineate workflows and also make things a lot more democratic. Like you said, Meredith. And it's, you know, if you build a lot towards one particular user group to favor their workflow, oftentimes, the trade off is that you're building away from other people's user groups. And so being sort of the thoughtful stewards of this database in a way to develop so that it would meet a lot of different different focuses needs. So the very first one who are the users, right, based on the old system that we had, that was, I think, 12 years old, at that point, like it was a static, it was like Google Forms, but it was static, and you couldn't change it. And once it's submitted, it went off to a place that nobody knew where it went, actually. So there's some problems inherent with that system, but it had been developed 12 years ago, and there are people who really weren't comfortable with all of the jank Enos of that product, they really loved all of the idiosyncrasies, and they were terrified to let that go. But from that initial form that we had, we had an idea of who these initial stakeholder groups were, and they were our preparers, collections managers, some folks in conservation, curatorial, and registration. As we started sort of shopping around, news of Maura, we started getting a lot of requests from departments and divisions, we never thought would be using an application to move artwork, these included Education Development, which were that was, frankly pretty surprising to us. And so we had to think about what would be the right sort of level of access for them to be able to know where they are works at at any one moment, and to be able to request that it'd be moved from one side of the museum, to another side of the museum or to an off site storage location. Also, as a lot of educating our users on what software development looks like in a way that hopefully would make their heads explode. So it was this idea of getting different types of stakeholders in a room together just so that they could physically see it's not just you and your workflow, which is very important that I want to take in. But I also need to understand how that's going to play with your workflow, and how we can sort of build a system for that. And I think that that was really interesting, and definitely a step towards people understanding. It's not happening yet, but it will happen at some point, or it's not happening. Because of this, maybe there's a way to capture that outside of the system or offline. It was working with a lot of digital immigrants. So Amanda is going to talk about this a little bit more. But when we launched the project, or the product, we had thought that we had assumed a certain workflow, even with all of the testing and the user acceptance trainings that we had done, we assumed that people would be somewhat timely in their updating of objects getting to a computer, we were wrong. So one of the things is that for alerts and notifications for this system, we erred on the side of more rather than less so that people would be encouraged to go back into the system and check all the messages that they were getting. And I got a panicked email, not an email, I got a phone call, I got a panicked phone call from someone saying I'm supposed to be checking into my email once a day. I used to check it once a week, what am I going to do about this? Seriously, and so it was about thinking about Wow, this is really a huge change for you to be able to use the system. Let me show you how to build a filtered inbox for your for your email account, you know, and here's some things that you can go ahead and use in the language of that. So that that which is important to you will definitely come into your inbox but everything else you can disregard that safely and know that that's okay. And that you know, we're not punishing you in any way by making use of system.

Unknown Speaker 29:44
Acceptance definitely was not just technical, but it was also coaching users a lot of therapy sessions, I would say, and getting them to really ask for the things that they want. So sky was the limit blue sky, we knew that there was D scoping to be done later on, and managing of expectations but really, we wanted to know how people were going to use the system. And so there were requests for sounds, colors, integration with Google Calendar. You know, so many different sort of iterations of how this product can be endlessly hacked, in order to create what these users wanted, that it was really interesting to get them to kind of think about it, and really think about how digital tools sort of pervade their work their day to day work. Finally, it was about selecting the right types of individuals and getting them in the right combinations into a room together to talk about what their workflows were. Some people are very, very focused on what their particular needs are. For some curatorial staff, it absolutely depends on their curatorial department and what they see. So prints and drawings or photography is going to be really different from say, decorative arts, you know, and getting those two people or those two curatorial assistants in the room together to talk about how their workflows might be served by the same system, I think was interesting, not just for us, as the administrators and the builders of this tool, but also for them, hopefully. So the next step was iterative prototypes and users, ours is not a very, I don't know, is it safe to say that it's not a terribly tech literate? Yeah, for the most part Museum and so I like I weep when I look at pictures of people doing paper prototyping with their, with their users, cuz I'm like, That's not us. So I believe that our museum is about 500, folks, 500 staff of which probably three fifths or so have some sort of interface with the collection, and so potentially could be users of Mora. These people had never seen a wireframe before, had never seen a paper prototype. Like I said, they didn't realize that there were different environments in which software was being developed. And so the very first session was really, I hate you the word interesting, because it's such a non word, but like, it got a lot of varied responses from folks, when we put paper prototypes in front of them. And I asked them, do this thing, do this one task, and go ahead and click on the paper to show me how you would do it. And they're like, but it doesn't go anywhere. And I'm like, Yes, but it's important for me to see which button you're gonna click, and they're like, but it's a piece of paper. That was a lot of fun. And also I really useful, I think, just to see all of the different ways. And so one of the one of the things which Amanda is actually going to show you walk you through the product, I know, we're kind of burying the lead in that we're not showing you the product until the very end. But one of the things, one of the things that we built in was that there needs to be at least two different ways to get something done, because there are at least two different ways that people would approach specific discrete tasks in Morrow, through the paper prototyping sessions. The next thing that we had was a click through prototype. And then we gave them access to a test site. So again, even though it had tests, blazoned on the side of the of the site, people would forget that they were in the test site, and they would go ahead and make changes thinking that they were going to do actual work that would write back to TMS, which is our database of authority for our collection, it didn't. And so we eventually had to blacklist the site for all but a handful of users that we could entrust with us. No, but I mean, like, these are our users, right? And it's important that we build these because it never occurred to us that you know, having it's not flashing, but it's big, and it's red, and it's bold, the word test on it would, would not deter them from doing their regular work. And yet it did. That's what we found. And so we just had to adapt to that process. The constant user feedback in the managing expectations, which was if we don't ask for the feedback, how are we ever going to get buy in? I think a lot of folks were used to this sort of power dynamic where you would build a thing for them on their behalf, it would be delivered, and you would not touch it again for months, maybe years. And they were just okay with that. This was inherently meant to be something very different in which they were asked time and again, is this something that we can work on for you? Is there any way that we can hack this system, not necessarily with light matter, but through in house resources to be able to make this a better experience for you? What would that look like? So that was something that we actually continue doing. And oftentimes, we encourage our users to hack the system in ways and sometimes we don't encourage them to hack it, but they hack it anyway. Which kind of secretly makes me proud because I'm like, see, you look good to you people like you're getting all techie. Now, this is awesome.

Unknown Speaker 34:11
With all of the changing workflows with all of the changing conditions of the head of a department leaving who was our primary stakeholder group, and the upcoming capital campaign project, there were all of these different business rules that we kept throwing at our vendor to try to capture at any one time. So this is just a smattering of some of the different templates. I'm happy to share with you. I have like a folder that I collect of like different types of templates that are useful. But we went through 14 different versions of the data model, we went through eight or 10 different versions of the functional and technical requirements document. And we came up with a business rules document, which actually I think is probably one of the best documentation templates out there that we used, which provided a use case, something that refer back to the Functional technical requirements, a real world example. And then some idea of what the interface should look like. So it kind of ties in and is sort of the glue for a lot of the other types of documentation that we're using. And then, of course, swim lane or flowchart whatever it is that you want to call it, which outlines what is never going to be the actual way that things get done, which we would soon discover in testing, which is actually the next slide. So are actually it's not but it's scoping and securing user buy in. So the users were not tech savvy at that point. They are slowly becoming more so and feeling I think emboldened to come up with a different asks, some of which are some of which are really creative, and clearly not sort of people who feel like technology is not inherent, not intrinsic, what's the word that folks? Oh, intuitive, right, these are, these are non intuitive sort of users. And so the asks that they come up with, I would have never dreamed of it in a million years, but it's valuable nonetheless. And so having a space for that. That CETA team, which is the collection information and digital assets team, did a ton of dogfooding did a ton of QA testing. And it was you know very much the ad one artwork to be moved from point A to point B, add negative one artwork, add three artworks go into another task, come back to it. And I would say that we all probably earned our black belts and QA testing during this time. But that was really important to get a sense of what this would be because our users were inherently apprehensive about a new system that was technical instead of analog or paper and pen. So it was really necessary for us to build at least some idea of what that workflow could look like, at least at the very outset. And then from there for them to be able to customize it as they needed it. Finally, it was dumb enough so that it could bend with emerging business needs, because there were a lot of unknown unknowns during this process. And so we wanted to encourage people to be able to think about all of the different ways that they could break the system, and they broke the system quite a few times. Some of it was really great, because we understood that there would be things that would need it that needed to get changed. And other things we were horrified by we were like How could this have passed between the initial smoke tests, the harness tests, the dog fooding the QA, all of that there were still certain glitches and bugs in the system that would allow people to add the same item 500 times to to a record or record. Yeah, all kinds of where we were like, it would never occurred to us to do this particular series of tasks that this one user did means like, like, yeah, know how you did, yeah, I would never occur to me to use technology in that way. But this person did. And I was like, frankly, I'm amazed. I have learned something today. Thank you. You know, and when I actually made the discreet offer to two of our most creative testers, possibly to go into QA and QC for like video games, software development, just because they were they had such a knack for this, it was amazing the things that they discovered. And we're actually really grateful for them, because now we have a better sense of what product we actually launched when we launched in August. So launching.

Unknown Speaker 38:03
So now we got to actually do the thing. So with the application launch, the team, the CETA team started a series of training sessions organized by user group and department, each session was orient was oriented to the new system. And then we walked through the mock work order system. We also published the Murrah Guide, which is a Google site that is both user manual and procedure documentation. And then once all the training was done, we set them loose in the system. And we've found that all the training and documentation in the world cannot erase a decade of institutional muscle memory. So LACMA went from this, so appear on the screen on the left side, that's a work order. So that would be printed out. It's a Crystal report from the original system, which again, only pulls data from TMS it doesn't put any data back into it. This was printed out, put in a physical file marked for what month it was going to happen with the week. And then that file was pulled every week. And that piece of paper was handed to an art prep to go forth and conquer. That means that the information could be a month old, it could be three months old, it could be a week old, it could have been printed yesterday. And then on the other side of the screen, that is a change of location form. So this is the thing that we fought so hard to get people to do, which is they would fill this out. So they would take that first piece of paper, and they would take that second piece of paper that was blank. And they would mark it up with what the object was. And then some sort of other identifying information, the old location, maybe not always the new location. This is actually an altered form where we had when we started putting things into containers, which I'll talk about a little bit later. And then the last column was I'm making the final move of this or I'm done is putting this on a table and somebody else is going to move this and also put that information in the database. And so they would take their first piece of paper and mark that they had finished their work and give it to their supervisor. And they would take the second piece of paper, if they filled it out. And then it would somehow filter its way back to see it. And then we would enter it into the, into the database. So we went from this to this. Thank you, Yvonne. I appreciate that. So obviously, more is a digital system. And in this system, we have work orders that are searchable and organized in multiple ways. So again, you're not getting a piece of paper handed to you that says, Do this, this, this and this, it is you go into your dashboard, and look at what is scheduled for the day. Also look at what's scheduled for the next week. Also look at the stuff that has been done or needs to be done or maybe possibly hasn't been assigned yet. And it's going to have live data that is pulling from TMS. So whether it was the work order was submitted a week ago, a month ago, whatever it's going to have updated locations as of the last hour. There's also a clickable calendar that you can click that you can look at and look at how things were scheduled for the day, you can see your open requests, your in progress requests, everything like that. Also, just as a tidbit, this screen changes slightly depending on what user group you are. So somebody who's going to be entering work orders, they're going to see slightly different options than somebody who is going to go in and complete work orders. So obviously, we're going from two separate ways to conceptualize your work and understand your work. And the degree of change for users range from manageable to radical. So despite all the care, and thought that went into the design of the application, the act of moving from an analog centric to digital centric workflow resulted in some digital growing pains. Natives versus immigrants. So for some users, the act of simply checking their email every day was foreign, as Yvonne said, let alone checking into a dashboard to identify what was scheduled for the day. So basic digital literacy needed to be included in training in addition to onboarding to the system. So that's something as simple as how to make a bookmark how to save your password in your browser, so you don't have to click on it every time that you enter it every time how to make an email filter. How to, I don't know, well, some for some people, some people like how to put email on your phone and use the app instead of logging into a browser like all of that kind of information. So even for the people who were digital natives, this some of them still struggle to adapt to the new way information is organized and presented. Like I said, institutional muscle memory runs deep when you've used the same system for over a decade.

Unknown Speaker 43:01
Project centric to add action centric, so due to the more nuanced capabilities and new integrations, more organizes data and functions differently. Work Orders in the previous system would capture an entire project or round trip journey for multiple objects. More I can only capture one move or leg of the journey per work order. So I can say move this thing from point A to point B. This means that for a round trip, one, two work orders or possibly more would have to be entered depending on how many stops of the journey that thing is going on. So this necessitated a change in established workflows, and to some users more data entry, which everybody loves. New data standards. More does validation checks against TMS before completing updates to prevent dirty data from going into the database. If it fails, if any of the location updates fails that check it throws an error and it won't complete it. This is something that a lot of users never had to deal with before any sort of like check on their work. In fact, for some users who use the exclusively analog workflow, they had never even interacted with a database before. So handwritten in the old system, the handwritten update locations would be entered by people who would catch those mistakes or those errors, either figure them out or go back and ask more questions. For clarification. This was further complicated by a feature called containers and TMS that was slowly becoming more prevalent in the database do the due to the inventory project that we were doing in preparation for the new building. So containers in TMS prevents the same box from basically being in two locations at once. So when you're entering a location update for an object, if that new update means that there are still something something's in that box on a different shelf, it's not going to let you do that until you either move those other things out of that. Box or move everything in the same move to the new location with the box. So, both digital natives and immigrants struggled with this level of detail that was required. And collaborative data entry, the people responsible for the data entry for each work order, it changes depending on the situation. But regardless, it's never done by just one person. So, for example, if someone's having something deinstalled, they might know the shelf that that object belongs on, and we'll put that information in, or they might just know the room, or they might just know the floor, or they might not know it all. And that person has to find a place to put it in addition to putting it away. So it's up to the implementers to put in that final location. And if additional objects are shuffled around, because of that move, those objects need to be added to the work order, and also had their new locations and put it again, something that was not really considered before this new system. So as more rolled out, the feedback rolled in, and we developed a list of adjustments that were called wishlist features, because branding is everything. This was additional customization. You heard me say that already? Why are you laughing so hard. This was additional customization that ranged from cosmetic to structural as pain points developed with the changing workflows. So like we said before, the new system was low was up putting in updates in real time insofar as however long it took for somebody to get back to the computer and complete the work order. So that could be hours, or that could be weeks. And if other location updates were made. In the meantime, this would disrupt the location history. So this was the most difficult feature to develop, but also the most important. Additional dashboard views directors wanted the ability to see all of their active work orders in all states in order to track all of their projects quickly and easily. And this was more assists a symptom of past issues with transparency and communication that made some users feel like it was necessary to monitor all of their projects to make sure the work was getting done, but a relatively easy cosmetic fix. And for many users, it was more practical to refer to objects by the artist name or the medium, rather than the object number. So these fields were added to the more database and made accessible and all views because before we had the object number, we had the title, and we had the dimensions. And then we had an image if there's an image in the database, there would be a little thumbnail. But most people would say, you know, move the Monet or the, you know, the oil painting rather than something like a title.

Unknown Speaker 48:02
So lessons learned, here's the part where you guys learn from our experience in our mistakes. There is always more work than you think, and more emotional labor required than you anticipate. And you guys can jump in if you have thoughts and feelings about this, because I know you do. So it's also really important to manage the expectations of your stakeholders and your collaborators. This includes the division of labor and the expected end results, because there will be constant revisions to the plan. data literacy is the name of the game. It is something that you may assume is inherent, but probably isn't. There's a very fine line between being too prescriptive and too open ended. You need to think about when to provide guidance and when to step back and expect to provide participants with a crash course and software development and what technology can do feasibly stakeholders will ask for the moon when they can't tell the difference between a rocket and a golf cart. So we've got we've got a few minutes for questions, if anybody has something, but we were asked to wrap it up a little early because there's no buffer time between this session or the next one. You can also just find us after two Yeah, that's also very possible. Yeah. Sure. Yeah. Sandra,

Unknown Speaker 49:23
I had a question about the training question. Do you guys said that you mentioned that you had like a manual with guides. And one of the things that we've struggled with when we're building out our internal software, it's like pushing people to go to those. Yeah, an external site. We tried like user guiding where people would like literally installed an API. And nobody really like wants to do those. It doesn't replace at the end of the day and you sitting down

Unknown Speaker 49:52
so we already have a history of writing this kind of documentation for other systems. We have a similar thing for TMS and our dams which is resource base or what we call Odin. So the way that I use it is it's like, it's kind of like reading receipts. Like if somebody, if somebody calls me or emails me and says, Hey, I'm having a problem doing X, the first thing I'm going to do is is cool is chill. This is how you do it. And by the way, all of the information that I'm telling you is in this guide, the next time they email me, they're like, Hey, I'm having a problem with x. I will say, here, that's no problem. Remember, we talked about this, here's all the instructions that I wrote. And it's already here on this nifty 50 page. And then the next time it's go read the guide.

Unknown Speaker 50:36
I'll also add to that. So anytime that there was additional feature work that was being done, we would communicate that by referencing these updates have been made in the helpful guide that is at this URL, please reference this if you have any questions, if you want a refresher, we're more than happy to walk you through a training.

Unknown Speaker 50:51
Also put good did you use a Google site to make your thing put in Google and put in Google Analytics because you will be surprised how many people like access that like I consistently like like, a TMS is the main system that I oversee and and it's been in use since 2012. I still get a very solid 200 to 300 users accessing that guide every every month or so. Okay, I think we might possibly be getting kicked out or there if there are any other questions. You guys can find us. We'll congregate somewhere out here. Thank you,

Unknown Speaker 51:30
guys.