Can’t Touch This: the Promise and Perils of Gestural Interaction

Unknown Speaker 14:25
everybody, and welcome to filter in here. I think I'm gonna share my screen now. Let me try that again. Okay. Which is everyone's favorite part of the runs of the virtual conference. As we're all sitting here, getting set up, getting situated, all those good things.

Unknown Speaker 15:27
We're ready to roll. Yeah, I think it sounds good, we have one, one whole minute, technically, join us good go. Uh, yeah, let's do it. Um, so we'll get started. Um, we're gonna do we're doing a campfire session today so we're going to try to present for 15 ish minutes and then we're going to open up the floor for questions and discussion we'll actually have some prompts as well, but if you see something in the presentation, or if you have a hot burning question right now. Go ahead and type it into the chat. I'll be happy to read through those and get there or a call on people if people would like to turn their camera on and ask it we are open to any and all participation. So as a quick intro, I'm, I'm Brett Renfer Creative Director at looking at, we make interactive experiences, mostly for museums. I go by he him pronouns. I have a my white male with clear glasses, wearing a blue striped shirt with the funny crunchy part today. I'm calling from Brooklyn the ancestral lands that when I think people.

Unknown Speaker 16:40
Well everybody I'm Kim, I go by the pronouns, she, her, and I am of Asian descent my black here, and also a little bit of blonde wearing kind of a red jumper today. I'm also calling from Brooklyn, and I'm an associate creative director apple today. Okay, so our presentation today is called Can't touch this. The promise and perils of gestural interaction will be kind of diving into our learnings from our previous experiences of utilizing gestural interactions and observations that we've had and shortly followed up by a discussion so that we can really dive into and see what's in the future for gestures and actions.

Unknown Speaker 17:24
To be fair, we start with a little loose credentials. So of some past projects so in my previous life, I worked at a place called the rocker group lab. In 2014 we did this giant project for Google where you could filter, 10s of 1000s of images using your presence and gesture. I'm sharing this not because we think it's like an amazing project, but it was a little bit of a, here's how we learned what we know. So for this one I made a custom gestural engine wrote her own code installed a bunch of like custom cameras. It was a lot of craziness that I do not wish to do over, so happy to dig into some of those learnings today.

Unknown Speaker 18:05
And in my previous life I worked with local projects and also too, with the Cooper Hewitt museum to create this gesture based interactive called gesture match. The idea was that you will be utilizing your body to almost search their archives, and depending on the gesture that you've posed it will bring up a collection item that really was related to that gesture. This was in 2016 So this is when machine learning was definitely not a thing, all of these gestures were manually drawn to detect the data source so like bread I don't want to ever do this exercise again, but a lot of will be addressing this in our talk, but the technology has been advancing that something like this is actually quite readily readily available to us to execute. But a lot of learnings from here as well. And we'll be addressing that in our presentation.

Unknown Speaker 19:01
So recently, um, like a lot of people, you know, during the pandemic we started thinking about the kind of past, present and future of kind of touch free design and gestures are a big part of that so we did a lot of different kind of r&d and experiments and thinking and writing. And this is kind of like a piece of that puzzle as well. So these are some of the experiments we did, and I should say as a brief pseudo definition, when we were talking about gestures today we're talking about interactions that use your, your hands body, face, combination of nav above that has some kind of antecedent, in the real world so head do something that might be meaningful, even outside of a system, and trigger some type of virtual content. So we'll go through each these kind of pieces individually, but really our talk is about, here's the sort of things that we should all be considering when work concepting designing installing, and all the above a gestural basic installation so we'll go through each of these individually, and then more happy to revisit or talk to any of these in the discussion session.

Unknown Speaker 20:11
The first point that we'll be addressing is with any kind of interactive experiences, we need to think about where your audience is coming from and the demographic of your visitors, and the additional layer of complication with Gesture interactive is this idea of like how much exposure they might have of these gesture based experiences. The most likely scenario is that unlike touchscreens. Most people wouldn't have had any kind of level of exposure. Today, majority of the people know how to interact with touchscreens because it's so ubiquitous within our daily lives, but with Gesture it's such a new experience, a new interaction that is not fully integrated into our daily lives that we might need like I have your hand inviting visitors to help them understand how to engage. And then, even though visitors might not have had any first hand experience I think another thing to consider is that through sci fi movies they would have been exposed this idea of what these like magical gestural tech would look like Minority Report being one of the kind of the poster child for it where, when you think of gesture basic interaction, you think of Minority Report. So in that case like there is this kind of level of expectation where they want to have this kind of rewarding output or like this magical feedback and have ample amount of abilities, they can possibly have with gesture, so it needs to be quite rewarding and really propel people to keep doing it. A really good example that is actually in existence today is this connect game called just stands where it gives you quite a bit of like playful animations and feedback loop in order for, for them to teach you these like very complicated movements because it has, you know, it's almost like teaching you how to dance and when you execute the right move. It gives you that kind of spark and that reward to help the users, keep trying it out. And then, kind of, finally, another thing to consider is that there will be audiences that might actually do get it, um, we do live in this very heavily Tik Tok driven Instagram filter world where gesture is now kind of more integrated into these apps. So the younger audiences might actually have had the chance to try it out on their own kind of devices and technology. And so just considering that level of demographic as well. So the thing to keep in mind is that it is quite a bit of a spectrum of like there might be people who have absolutely no exposure, no idea what this gesture interactive is versus people who might have had a little bit of a touch, might have traded on these like event spaces and things like that so you need to consider like, what is the best onboarding mechanism for the type of visitors that you have and what mechanics might be the most effective. You really can't assume that, like someone knows anything and if they do, they might actually have very unrealistic expectations of what they experienced you're actually providing so at times you might actually need to over explain, be very clear as possible and the actions that you want visitors to take So, an example of that is like how to dance the robot dance and it's an almost like an instructional manual that we see right here and it might be. We might have to consider eating as overt as this stuff like almost diagramming out the gesture that we want the visitors to take

Unknown Speaker 23:38
when it comes to the technology or a little bit of like the how. There's a lot of different things we can say, but one of the big pieces is just the idea of thinking about change. So preparing for change over time, especially if you're considering doing something that might be permanent. So if you look at a timeline of even the last 10 plus years, things have changed a lot, you know, in the kind of the before connect era, BK there, this is possible, but you're doing things with the more heavily customized kind of computer science heavy background that connect in 2010 Suddenly you had pretty good data, you can see someone's skeleton you had some of these things off the shelf, it's kind of democratize these more, you know, university based tools, and the most recently are seeing people use AI machine learning algorithms to extract the data you need for gesture based interactions or even do gesture recognition automatically and sometimes with even cheaper, off the shelf kind of webcams and hardware, and so most recently, you know, Snapchat, and Instagram and others are baking kind of gesture recognition into their apps, some of which are kind of free asterisk for us. But I'm very you know dependent on the kind of output or things you're making. So regardless of the technology you kind of end up with some version of this recipe. You think about using hardware of some kind to Capture people usually some type of camera. And then you have this like multi part software part where you're kind of, you know, looking at the data and extracting something useful. So whether you're cutting out people looking at their skeleton looking for their hands, something like that, you have another part that saying let's match what I've extracted to a gesture, yes or no, this is happening or how much or how little is happening. And then you have your content you're reacting to that data and creating something beautiful, and hopefully educational and all those things with that. So the last, you know 10 or so years, this sort of recipe look like this now you'd use a computer, and probably a Kinect or real sense or ACEs action or whatever depth camera was your favorite. You are using some kind of off the shelf algorithm either a free one like the Kinect SDK or others like new track to extract a skeleton, typically, or some kind of data. And then you're using a part where you're matching it to a pose so you know the thing that Kim was talking about with Gesture match like matching that to content are saying, Are you or are you not doing something, and then of course the last part is typically custom, hence magic. Um, so, as another prototype from our, our explorations around gesture and touches design. Our colleague audio Fernandez made this one so this one is, you know we're looking, we're using you can see that connect in the foreground is connecting to, and computer, it's looking for this binoculars gesture, and the moment he does it, it kind of pushes the software into another mode and starts to react to his position in another way. And then looking towards the near future. You know I think there's this really interesting thing where these pieces are becoming somewhat cheaper although sometimes they're a little trickier to use but you're starting to be able to use, you know, a computer and a computer could be something more like a Raspberry Pi and it's cheaper and off the shelf, you know, the simple color web camera, because you can feed it into, you know, a super smart AI hopefully someone else's even trained it for you. That is just looking for gesture so not without problems now without this plenty of things to talk about with AI and machine learning, we're really just to say that not only is this system somewhat getting simpler, but also it's really different than it was before and so I think that idea of considering which pieces of this recipe are sort of swappable over time, you know, if you're installing something for five years, you really have to think about, can I change the hardware can that part change which part is fixed which part is variable. And are these things all in service at this being like a good experience. And as a quick aside, you know there are other ways that this is being sold. Hardware is another one right now we think this is like too cumbersome, like asking your visitor to put on for six or more little wearables to recognize their gesture seems like a lot, but the little gift in the foreground is HoloLens is like built in gesture. Now, those things are promising and there might be more cool quick pieces of hardware that are simpler to get started and play with. We want to say, Yeah, we're excited to see where this goes

Unknown Speaker 28:03
with the technology I'm like kind of the experience as a whole, we need to start to also think about the affordances that are needed for those type of technology. There's a great quote by the Kress ban from the MIT Media Lab that kind of addresses, three different kinds of affordances that we have to really consider, and this is from an article that he wrote and critique of the gesture gestures that were interactions that we're coming up with the like your personal phone, and this was his critique for like, you know, in order for gesture interactions to be really effective there needs to be a decent amount of real estate. You need to have the right levels of precision and position, and whether the application is appropriate is the right kind. So that's something to kind of pinpoint here that we're, we were as we were kind of like investigating these gesture interactives, we realized that when people are moving their bodies they need quite a bit of space like we don't want people to flare their arms and hit hit each other. There were some accidents that didn't quite happen and even our body Stairmaster size that we realized that oh, there needs to be at least kind of a six feet radius around each person, if it's, you know, a kind of a multi user collaborative kind of interaction there, there actually needs to be more because, again, like I said, we don't want people to hit each other in their faces. And another thing, another layer of consideration is that there might need to have kind of a designated space for privacy so I'm doing a gesture and things like that can be quite awkward, it could be quite vulnerable so that's also something to have in mind. In addition to just like the real estate that individuals need depending on technology there are very specific spatial requirements to make sure that the sufficient data is being captured. And we need to take into account of that of like how much of the limitations that technology might have and how we can accommodate that within the space that we are providing. And then another thing to keep in mind is like once people are including each other, um, that might quite feel nice for the visitor but the point of view of that tactic, it actually might not because it will hinder the capturing of the data. And then the final element is that everything needs to kind of work together and what surrounds the experience really do matter, thinking about the scale of the content on the screen. What kind of printer graphics will actually help guide the visitors, in addition to additional call to actions on to be interactive themselves so everything needs to be very cohesive and you need to work together in order for gestural interactions to be very effective. And then the next kind of like bucket of items for us to consider is, we can't leave this out content, content is super, super, super important in kind of like thinking about what gestures really feel appropriate. The thing to have in mind is that gestures have meaning, like you doing that action, it can provoke kind of a certain level of mood, a type of role or even a character that you're playing within the story, so you have to really be critical on what kind of gesture ties in with your content. The gesture that you are doing is actually part of the story. You have to consider like how are you positioning the visitor and what kind of role are you asking your audiences to play, But the strength in doing that kind of critical thinking around the types of gesture that you are tying to within the content that you're providing is the fact that this is encouraging kind of a multimodal learning it's a very different way of experiencing your content on beyond the touchscreen beyond the reading beyond just a printed graphic so it really emphasizes the experience of immersing the visitors in the content that you are providing. And then finally, this is kind of like a must in the process of investigating those gestures that you actually have to try out and you acting it out and on, I'm doing the gestures themselves to realize the limitations that I might have or the, the character that you're actually putting the visitors into might not quite feel right within the content that you have so try it out, body sermon. I did a lot of them, I felt really awkward but that is also part of the exercise of you trying it out, will showcase what kind of gesture is appropriate.

Unknown Speaker 32:29
That's the moment where we're jealous if our coworker from New Mexico's work from home life where Kim is in her apartment and she's like, I'm so you know the last piece you want to talk through is, is among the most important is just thinking about this idea of access and inclusion. There's a lot of different aspects we even talk about whether some of this meaning is even very western centric, but we'll focus specifically on accessibility to start. So, you know, starting with this PC and we use this and other kind of like toolkits to think about designing physical casework and physical interactive, to think about designing for reach, but I think it's really helpful to look at and think about when you're thinking about gesture as well. So, thinking about, you know what, gestures are available to this person in a wheelchair, thinking about what is that Canvas is actually available to them, but also noting that even in this diagram this is considering someone who has full range of motion of their arms so how do you think about a range of abilities of users and visitors, and can they participate in your experience and what ways might they participate. So one way to start to think about that is just name out alternatives you know so, as Kim said, once you start to try things and start to act out and put yourself in these positions you start to see these issues, and sometimes it might just be a simple solve so, you know, on this, this prototype was thinking about this kind of grabbing or selecting content so ideas original prototype and the left, you have to have full range of motion of your arm and fingers, realize we can just use time, you know, and Wally is updated on the right you can see it doesn't really matter this sort of what he's doing with his hand, or, you know how his fingers are position you can still do something that within almost the same amount of time to do a simple kind of gesture to be selecting something, you can even see the cameras look closer to him. He doesn't have to use his arm as much so I think it's, it's not perfect but it's an example of a way to start to think about ways you might refine your gesture to be more accessible for more users. So, these slides are imperfect or incomplete metric, but you know starting to think about as you're looking at all these different aspects. Once you think you have something that fits perfectly naturally with your content, string to think about can a huge amount of people perform this gesture. Why or why not, well people wants to the gesture even, you know Ken noted this idea of privacy, that's a piece of the puzzle but, you know, more people feel like the gesture is appropriate to the content. And are there ways in for everybody, you know, if some things require a different level of physical exertion, are there alternatives. And lastly, are those alternate modes just as special and considered. Do you know how what are people going to feel like if they can't do the gesture and have to do something different that feels more like it's been slapped on or added at the last moment, I'm trying to think about this suite of inputs and options for people that give a lot of range ranges in for people on and thinking about again access and inclusion from the start. So really our talk is not saying don't ever do gestural interaction or it's way too hard. No, it's saying that there are like really all interactive experiences they, they sit at this intersection this tension point that it probably isn't even more complicated and diagram than this but the things we talked about is, is it possible technology, you know, is it possible spatially, think about this considerations does it actually make sense, the story you're trying to tell, and does it include the maximum number of people and are there ways in for around a huge range of people. So your amazing idea hopefully sits somewhere at the central points, but things like body storming things like prototyping and testing are the ways to kind of get at each of these and push against these tension points. So hopefully, we're going to kind of open up the floor here. Hopefully this will give you a suite of tools and ideas and things to think through. So I think we'd love to start first if people have questions for us, and then we have some prompts, as there are there and so again you can use the chat and I can read them out, or if you'd like to unmute and camera on or off is totally up to you but please feel free to jump in as you'd like. Oh, Kim, you're muted.

Unknown Speaker 36:40
I think I'm gonna stop sharing for now just so that we can see if people decide to jump in. If not,

Unknown Speaker 36:48
Robin, go ahead, looks like you just unmuted.

Unknown Speaker 36:51
Yeah, yeah, I have a question, you know, could you talk a little bit about the skills needed to design a gesture based experience you know how MANY, how MANY people on a team do you need to have. Yeah,

Unknown Speaker 37:14
I think, um, it really depends on the type of product that you are pursuing, but usually it's probably, it's good to definitely have a tech representative. A person who is very acquainted with the technology that's available and then a person that's a little bit more design and user interaction focused, and then another person who might be kind of keeping an eye on content so those kind of three, I mean, we had a four, you know diagram but those three disciplines all kind of being represented is actually quite important. And I see right here, one of the comments and technology comments in question was kind of around accessibility and having that in the beginning and ideally the three people who are involved in the project are constantly having that discussion as well. Sometimes that loop that we do bring in a contractor who has a very specific skill set in keeping an eye on accessibility, and if that is kind of a primary goal which it most likely will be, we bring in a contractor if we have the budget to make sure that we are taking into consideration of all the ADA compliances as well as like the range of mobility that we will have to adapt our, our product for

Unknown Speaker 38:33
how complex How complicated can adjust your get.

Unknown Speaker 38:38
That's a really good and a really hard question. I would say that, similar to the access and inclusion, and I see Cassie are Henry's too so jump to you in a second. It is a little bit of like how much can people take on from a cognitive standpoint, you know how much are they learning something like even looking at those dance games you know they break out something that to a dancer might be, this is just one move, but this is now in like five steps or more, I think it depends on the duration, interaction, the type of emergent behavior so sometimes like that pterosaur example, you know once you kind of figured out to put your arms out, pretty much anything you do did something in the game, so it's, it was complicated in the output it could create but not complicated in the gesture. And when it comes to the technology, it's really dependent on that platform, some things it's like you might use be moving too fast for the camera to see you, which is a real consideration. Kelsey I think you had to first comment in the chat box you so and I see your hand raised. So go ahead and jump in.

Unknown Speaker 39:41
Thanks. Yeah, nice to see you, the presentation is a I've, I've had some struggles trying to work with Gesture stuff myself and it's really interesting to see the considerations you've all taken into account. I was curious, one of the biggest challenges I came across was how to explain the gestures to visitors. So I was just curious what kind of like signage or instructions, you provide to the people who maybe aren't familiar with gestures at all can understand without it being like a booklet.

Unknown Speaker 40:17
Oh, I can. No, go ahead. I think we might be about to say something, it's it's almost like as close to full scale, and like a rotoscope animation as possible. You know it's like you want to see somebody doing it, especially if it's full body. Um, and from there you just get more into this kind of like universal design of iconography or action I think it's depending on the steps, depending on the type of gesture, and as much as it can be like embedded in the experience, the minute you walk up, you're kind of assuming that people are starting from zero, and you're giving them as literal as a guide as possible. And sometimes you borrow things from video games, you know, video games are all about training and leveling up, so you might do the gesture as a part of like a, let's get started kind of mechanism and if it gets more complicated. You could stage that over time.

Unknown Speaker 41:08
And we went over it very briefly in one of our slides about affordances and how like the surrender graphics is actually really, really important. The very like the low hanging fruit here is indicating where people need to stand like people want to understand the space that they can occupy him so having that clearly marked is really important. And additionally to like, what the product or like what the app is actually delivering you on the onboarding side of whether that's like a rotoscoped actual person like doing the gesture that you want people to do, or if it's like you know iconography, in support of that if there's, you know, redundancy within the printer graphics that's also very effective. I have a little anecdote where like we did a, we did a very, it's more of a presence based game that we created for like a Maker Faire that one of our clients was hosting and they invited us to participate and it was very low five we just had the game and we're like, oh yeah, people understand it, nobody understood it. And so we used, post it notes to see what would be the most effective in explaining what type of you know presence or action we wanted people to take and the most critical part was, was exactly that, like making sure people know exactly where to stand, what the instructions are all the different details of those printed graphics actually made a difference.

Unknown Speaker 42:33
This question from, from Kristen around whether we found that people, visitors learn more by using these gesture based technologies, I would say, um, it's, it's nothing that we necessarily have objective data around. I would say that there's gestures and things that both in our own work and things that we've seen that are simple and straightforward but allow you to kind of play. People just send us them more time with them you know it is novel, as long as it's a low, it's like that idea of the low floor high ceiling and it's really easy to to walk in and just start doing something. If it's fun and interesting people might spend more time so I think you could extrapolate that people are hopefully learning something more over time through that. It depends on the gesture and the experience but that's I think the best kind of measurable I would sort of have the next question is around whether or not we have an example of an experience we've designed where we did include an alternate alternative to the gesture. Came out of strengthen it'd be kind of explicit. Yeah,

Unknown Speaker 43:42
I think I mean, a lot of the prototypes that we shared was actually a, an r&d project that we had internally because we were realizing oh there might be some, you know, because of the state of the world today, their touch might be quite if there was a world where we were afraid that touch might go away so we explored all these like various different ways of incorporating gesture into our apps, and this was basically by doing that exercise we found a lot of these kinds of observations. So unfortunately we don't have like a specific example that utilizes the alternative gesture but that is something that we are considering we actually the one other projects might actually become a real product and so that is in our forefront into considering what kind of additional pressures might be more appropriate for the visitors, if, if they're a little bit more if they have a limited mobility so we'll keep you posted.

Unknown Speaker 44:41
Yeah and I would say, candidly, I think there have been more times that project has started as a gesture and then the things that we talked about, especially the appropriateness the content we more pushed into something more either static or presence based, you know that it's kind of like you just need to walk up or do something very requires less mobility, or there's kind of a fallback that is like a touch or alternate interface.

Unknown Speaker 45:09
Um, the next question is from Jasmine. Um, can you share examples as to how you've measured user experience in the past was gestural interactions, whether that's pre launch user testing or during a run of an exhibition, our experience, um, I think it's a little bit of both. So one of the, we, we launched kind of a presence based interactive for the Henry Ford Museum and one of the as part of the process we did a big kind of user, user testing on site with the visitors to make sure that we are, we're gonna do a good job. We did a little bit of AV testing from there and so making sure that that's incorporated into when you're designing I think is pretty critical on this kind of like measurement of user experience is a little tricky because there will be post launch and pre launch and post launch metrics that we might want to establish. Yeah, that's kind of like the the tricky part it's, it seems like Jasmine you're jumping in. Nope, okay, I mean just hearing things now.

Unknown Speaker 46:18
But yeah, I think, echoing Kim said, testing things, and testing with real people really early on, and we've done it. I think one thing that is kind of playful I think about is we've done it fully analog as well. We did do a gestural experience that was about becoming a sea creature that unfortunately didn't make it to the grand phase but for testing that one, we realized that to create a prototype that actually worked where you could walk up and have skeleton tracking hand gestures that like did different things, we'd have to make so MANY decisions in the prototype that we were basically making the final experience. So instead we actually decided to recreate this thing, based on this old Marx Brothers movie that was like a pretend mirror, so people actually just dressed up as the creatures and mimics, gestures and held up like physical printed content or the content takeaways, And it sounds insane but it worked really well because we could react live, we could have this back and forth and we did a traditional kind of users get a little bit onboarding, they come and do the experience and they sit down with evaluator and have questions about content takeaways and things like that. And we found really key points that we're feeding into when we actually write code and make this thing. So, definitely think about ways that you could prototype without technology to.

Unknown Speaker 47:31
Thanks. Yeah, sorry I didn't hear the prompt earlier to jump back in but I, my, my team and I are like our department is trying to think more about how we can advocate for testing during the creation of interactives and so just for, you know we're coming up with like us trying to come up with a set list of metrics and like KPIs that we can use, where we create interactors but we've never done anything in the gestural space at least I haven't. So I'm interested to see like, are we interested to know if you're measuring like how long someone is engaging with an interactive if there's like, like how you measure between, like, on the data side how you measure between experiences when there might be multiple people standing in front of the screen so I don't know if that's like database or if you have a list of metrics that you commonly referred to, but that would be really interesting to see.

Unknown Speaker 48:21
Yeah, we didn't really have a standard we are. I mean, we actually have even done things like using our the same camera and giving anonymized but still kind of analytics style, data, and typically it's just depends on the experience so even in the binoculars when you saw, there is the first part of like is someone there. Did they see it, and then did they do the gesture, and then how much time was spent before they walked away or did something else so I think you can start to gate that experience to think about the duration and whether people got over this threshold so same with the Henry Ford, It was a little bit of did someone walked by this column so it was motion based, and then did someone actually there was like this threshold you could cross to uncover the content, so that when we had sort of states that we were measuring, but it you know it's all the things about, is it actually two people, they kind of crossing ways, those are without investing in things that people use in like retail stores or tracking customers like that is how some people do it, we are just looking for, we're less scientific, to be honest with that kind of data because mostly we, that's okay, but I think I would put it in that sort of way, it's a little bit of like, did people see it, do people do the gesture, how much time, like how far in the experience they go to and just thinking about what are the sort of states you're getting into.

Unknown Speaker 49:37
And if there's like a kind of a reveal or a final moment it's that's also a really big, great gauge into how long people have injured to get to that point because usually with gesture based interaction that reward is kind of a content moment that you are actually delivering so that's also a great kind of metrics for you to

Unknown Speaker 49:55
measure, and the endure is a great segue into Josh's question, which is this is have you found that use fatigue is an issue with gesture based interact is 100% I mean that's the thing that like Kim was talking about the body storming it's like once you try doing something. I mean I don't lift enough to do all these gestures like you're just like oh I'm doing this for like five minutes my shoulder gets tired, and it's it's all kidding aside, it's like, it's super important to do those before you go down in a world of making something you know and thinking about. Are you doing something that's momentary, and then you're kind of resting and sitting back with content, like what is that relationship between active and inactive. And there's, again there's that just physical fatigue, there's a cognitive for too I think.

Unknown Speaker 50:38
I mean, yeah, it's one of the biggest critiques of the Minority Report gestures is the fact that you're constantly having your arms out that you're having to do these finger movements which is, I mean, accessibility wise it's like completely not even feasible but even doing those body storming exercises you soon realize like, Oh, I just want to do it once and I just want to be done, like I don't want to keep doing the gesture over and over for that to be kind of the norm for you to get the reward, so having it's a little bit about like that cadence of the experience as well like what's, what's the rhythm that we're, we're allowing people to engage in and then rest. So, yes, user fatigue is definitely a thing.

Unknown Speaker 51:17
You know I don't do this. Oh, there's another question. Go ahead.

Unknown Speaker 51:21
It's okay. Keep going. And we'll cover here.

Unknown Speaker 51:25
I just, I was just remembering, one of the first gesture based experiences that I saw which was at the Cleveland Art Museum, a long time ago, where you could stand in front of a screen, and you were supposed to mimic that you could choose an artwork and you were supposed to mimic the pose of the artwork and the, the, the learning behind that was how you felt in your body when you did that gesture from an emotional standpoint, you know, so it made you, which, which then fed back to, oh if I feel like this emotionally then that's how that person in the painting is supposed to have felt or that that person who was carved in the sculpture was supposed to have felt, and I thought that was a really effective use of gesture, you know, for a for a kind of learning that's, you know, that you it's embodied learning there which is really not the same as learning it intellectually you're suddenly feeling it in your body. So I was just pointing that out as a good example of a gestural experience.

Unknown Speaker 52:29
Yeah I think that's kind of the biggest benefit of these gesture based interactive I think yes there are lot of considerations to be had and I think it's it's it's a difficult process for you to really strike it strike the right balance and have the right, interactive, but it could be really effective if all of those elements are, have the right synergy and we'll remain with the visitors a little longer than I think you've touching the screen, for sure. Kelsey had a question, have you established some guidelines for when you find gesture interaction to be effective versus other types of interaction. Um, not like a formal guideline, I think that was definitely the process that we wanted to get to, for our r&d project, but these are, these are kind of like the five pillars that we would say that is kind of a guidelines to you know make sure that the gesture is effective it's, There are a lot of components there, we don't have a strict rubric just yet. And I most likely have a hard metrics for us to like really pinpoint and like concretely put down but we're hoping that these are, this is definitely like kind of the considerations and the guidelines that we will be utilizing to create Jestro interactive, and then I think as Jasmine I think it was Jasmine that pointed out about the metrics of how do we build in some metrics to make sure that we can iterate on top of it.

Unknown Speaker 53:51
Yeah, I think that, well, the only thing we didn't talk about was, there's also just like the duration of how long it's something is going to be installed. Most of our work is permanent, so we have a pretty high bar about whether we think something like that is appropriate. Um, but I think that's a piece of it's temporary, there's a lot more you can do. So we have about five minutes, so I'll try to go through a couple more of these. I think I'm Christian you're you asked a question that we weren't thinking about as well it's like do we think that gesture based interactive will become more commonplace in museums. And it's really, it's a really hard answer I think for the next year or two, because of people starting capital projects, you know, or restarting them post pandemic I absolutely think is is going to be like a crazy explosion, at least, present space are people considering just your base, I think people are similarly thinking about, you know device as gesture or a stand in from like you know having a non touch or approximate based interaction. I think it's, it's hard to say because it's a little, even in the technology is not looking to establish here's the best and easiest and quick way quickest way to do it I think until there's another thing that goes beyond like I need to write C++ to make this thing work. I think that after that special I think there will be more because it is really interesting and appropriate and I think that it depends on the museum and the content too, so this sort of a non answer but I think we're thinking maybe

Unknown Speaker 55:21
we're hoping I'm only out of question around how inclusive gestures are for differently shaped and differently sized bodies. It is a concern that gestures might not read larger or smaller body bodies in the correct position from a technical view. Um, it's, it really depends on which part of the which part of the dark, date, which part of the body we're actually tracking the data on the part that we actually from previous experience on Brett, you probably have more context than I do but like height actually became kind of a tricky positioning, because of the angles of a camera and how much it can Capture. There are ways around it of like whether it's actually, you know, detecting more of the upper body or the lower body or things like that, but that also goes back to like accessibility again so it's a, it's a tricky line to balance, whether like people are a little bit more heavier set or lighter set that has absolutely no, there's no kind of limitations within that it actually it was more about the height of the person if there's like a four year old that comes on versus I guess six foot five man, that's a very different kind of data set that the technology can Capture though so that was kind of the tricky part.

Unknown Speaker 56:36
Yeah and I think Ellie to your follow up question about users and chairs, too. I mean, I think they most of the things. Most of the hardware and software does it okay job with people in chairs, I think you kind of tend to prioritize upper body, because of that then because you'll see us, has a really hard time, at least in my experience like differentiating like legs or sometimes they get kind of confused so I think it's, it's really important to test in this case, and you know it's the thing that's really hard, like, in a way we all become accountable for some AI machine learning that some other big corporate company has done, right, like it's like the reason those things are not good are based on whether they trained with a really open data set and a lot of other people, but doesn't mean that we as the, you know whether the museum or the designer or somewhere in between, like, you don't get a pass from your users because of that right So, typically I would say it works fairly well it's really important to, to work on all the spatial parts of that to you know to make sure that even the positioning of your camera can have a big difference within that, and then do the testing, you know, and I think it's really, there's a lot you can do with just even saying like I'm going to, you know, get an Xbox and try it or something like that to like before you even get started on your own, but it's a really important. Important note.

Unknown Speaker 57:55
Okay now, one minute session is done.

Unknown Speaker 58:00
I will just say thank you for all the questions I mean I think these are all really helpful and represent a lot of the stuff that we are up against as well so we will we'll post the slides, we put them on the skin, and then we will add them to the Slack as well on the experience design thread so please. Oh, I think we're both on this site too so if you have another burning question. You can find us there and then I'll be in the 315 wrap up as well.

Unknown Speaker 58:24
Oh good,

Unknown Speaker 58:25
good. Yeah, thanks so much this was really informative especially you know design, defining those four areas of consideration that people really need to think about technology access content, and what was worth one space as yeah space. Thanks a lot. Thanks everybody. Thank you.