Transcript
Unknown Speaker 00:00
length by time period, and movement, we won't be doing that these videos will instead look at our theme
Unknown Speaker 00:09
places and spaces.
Unknown Speaker 00:12
Art and identity,
Unknown Speaker 00:15
transforming everyday objects,
Unknown Speaker 00:19
Arden society
Unknown Speaker 00:24
these themes in an attempt to bridge the gap between modern art
Unknown Speaker 00:27
and you, I think that modern art can meet you where you are, and allows you to work with what you already know.
Unknown Speaker 00:31
Or says
Unknown Speaker 00:32
interestingly, as much about you or me, as it does about the artist himself. There's going to be pictures, there's going to be these people, there's love, there's death, there's this guy, and there's this thing, there's a chainsaw. This is why this museum, this is why the subject is so rich, because there's such diversity in it. If you find a foothold and get started, you're never gonna finish, it's going to be another way to look at modern art.
Unknown Speaker 00:54
Suddenly, it opens up a greater dialogue about connections not only between the works of art, but to things that we already know. It just gives you another entry point to really engage with the art that you're seeing.
Unknown Speaker 01:19
So God,
Unknown Speaker 01:22
you know, we're not the first ones to be doing 3d printing, we wanted to see 3d printing as a tool to evolve textile making.
Unknown Speaker 01:31
It needs to obviously look and fits beautifully, but at the same time, we want it to be practical.
Unknown Speaker 01:37
Why is this tracksuit differ from every
Unknown Speaker 01:39
other tracksuit no demand.
Unknown Speaker 01:43
Most often, you see garments and exhibitions on mannequins, but mannequins are difficult what your garment looks like, and what the mannequin looks like maybe seeing two different
Unknown Speaker 01:52
things, the challenge that we put to ourselves with sustainability initiatives, can you completely reimagine every feature of the garment? And can you do it in a way that then generates environmental value and positive impact on the community at every step of the way,
Unknown Speaker 02:04
the need for textiles is one of the most basic human needs and one of the most ancient or always.
Unknown Speaker 02:21
All photographers make choices, whether it's how you frame something on Instagram, what kind of camera used, what kind of prints you make, how you share that
Unknown Speaker 02:29
picture,
Unknown Speaker 02:30
the power of photography is way more complicated than people admit to, you have to learn to understand how images are constructed.
Unknown Speaker 02:38
I got into photography, because I wanted to know something that other people didn't know what makes a single image important anymore. How do you make that image that people are going to run to museum for what is it about
Unknown Speaker 02:47
me? What are we bringing to this photograph? I think you have to contend with a lot. And that is a very worthwhile experience.
Unknown Speaker 02:54
When you think of photographs, as processes, not as products, you start really understanding what they really need.
Unknown Speaker 03:01
How's the process different?
Unknown Speaker 03:02
I want it to be not about grabbing the moment but about dancing with the moment collaborating with the moment,
Unknown Speaker 03:07
can you see what I see? No, no one else can see it. You
Unknown Speaker 03:11
can stop something and look at it in a way that you normally wouldn't see it. And I think that's part of the real fascination with still photography, it lets you step outside yourself to kind of look at the world in a different way.
Unknown Speaker 03:25
Okay, so it's gonna be half a second.
Unknown Speaker 03:35
I think our work is often a response to something and that something is often a problem.
Unknown Speaker 03:45
I thought it would be an interesting site to do a work that just requires an action and that was simply to crawl. There's something more powerful about taking the thing that is the most abject thing and making it the most valuable thing.
Unknown Speaker 03:59
It all happens in one swoop and ignites in a way that it all happens at the same time.
Unknown Speaker 04:07
It was that 1% but I thought
Unknown Speaker 04:10
it will succeed and it did.
Unknown Speaker 04:32
We all think about these explosive moments of painting and you just witnessed some. But really, de Kooning alternated, those with some very long periods of careful looking at his paintings are really playing with this tension between all of that conceptual perfection and manual imprecision that we have because well, you're human. You make lots of mistakes. Kusama typically, not always but almost always at work. On prime canvases, there's not one way that Kusama made an infinity net painting. There's not one way that Martin made a gridded composition. So we're just exploring one approach here, feel free to experiment quite wildly from this. If you can hear that sound. That's what a decoding painting sounds like.
Unknown Speaker 05:30
Okay, I think we'll go ahead and get started. So thank you everyone for joining. I'll go ahead and share my screen for the presentation.
Unknown Speaker 05:45
Can you all see Well, okay. Okay. So thank you, again, for joining, what you just watched were five of our course trailers for online courses on Coursera. And we'll spend the next hour in an open q&a discussion about momos online course program. And the will specifically focus on MOOCs, massive open online courses, really looking forward to open discussion. And to hopefully, I think there'll be takeaways for other platforms and digital outreach more broadly. So I am Kelly Cannon, I used to pronounce she her. And I am an educator at MoMA. And I work with a really amazing, very large team of internal and external collaborators on our courses.
Unknown Speaker 06:44
And so I think during this session, this will be in my presentation. And I'll start with a broad sort of overview of some of the aspects of our online course program, ranging from production iterating, on courses to launching, activating and building communities, to enrollment revenue, and how you measure success in an online course. And this is an experimental session is really seen as an open q&a. So please, throughout the session, feel free to post your questions in the chat. And after the presentation, we'll get to those. And really, just, like I said, hoping that this will be open and collaborative. So welcome, any and all questions. So the first thing we'll kick off with is what, who and the what, why, and who, especially because audiences really are at the starting point for MOOCs. So a little bit of background. What are MOOCs MOOCs are massive open online courses. They are asynchronous, and momos are on the Coursera platform. But there are many platforms, such as cadenza, edX, future learn, etc. And most MOOCs are set to being free to audit but about two to 3% of learners pay roughly $50 per certificate. And then there are a few other options such as subscriptions. And if anyone is currently trying to enroll in the courses and are saying that there's a free trial, we can also get to that at the end to but they are completely free to audit, so you can always click that. So our courses are non credit bearing. And over the years, we've gradually made them less academic more of an informal learning environment based on feedback from our learners. And for several years. Before starting to make online courses on Coursera, we had been making online courses fee based online courses on the smaller learning management system. But we were looking for something that would be mission aligned that would help us reach more learners at a far lower cost point, oh, and also free. And so in 2012, we partnered with Coursera. And we launched our first course for teachers. And over the three years after that, we developed three courses for K to 12 educators. And then in 2015, we moved to creating courses for a larger general audience. We now have nine courses with 1.4 million cumulative enrollments, from 1.1 million unique learners. And as you can see here, 64% do not identify as students and 41%. Report, they're employed full time with 68% between the ages of 25 to 44. So what we're really seeing are that many of learners in our courses are working adults, their lifelong learners, they're often fitting this in in their free time and they're often doing it on their phones to an our top 10 countries of enrollment are listed here and they fluctuate a little bit over time, but these are relatively stable. It's over this year and last year, so that's us, Italy, India, Mexico, UK, Brazil, Spain, Russia, Canada, and Argentina. I also want to note here, too, that in 2020 at the start of 2020 Of course, enrollment was at 700,000. And that was cumulative from 2012 to 2019. So you can see that the course enrollment has doubled this year, added the same amount of learners this year that we added in the previous seven years. So it's been a really huge course growth. And we'll get into a little bit more of that as well. So MOOCs have also for us been an opportunity for global brand promotion. And you can see here that the courses are really attracting new audiences to the museum. This is from our pre core survey, and sorry, the screenshot here is a little bit cut off. But basically, what you're seeing is that in these three largest bars at the top learners, most learners are reporting that they recognize the name MoMA, they're somewhat familiar with MoMA, where they're familiar. But the very smallest group here is reporting that they're very familiar. So what we know is that learners are primarily coming to our courses by searching for painting photography, not by searching for MoMA. So we're bringing new audiences into the MoMA community through these courses. And happy to take questions Also, during the q&a about sort of how we understand that they are then deepening their engagement with MoMA
Unknown Speaker 11:11
through the courses, and after they take a course. And this also speaks to why one of the reasons why MOOCs are important for museums is that they help make your museum more accessible both geographically to those who can't come in person, but also by helping people find an entry point that may feel less intimidating to them. We know that a lot of learners are reporting and one of the outcomes that they gain from our courses are gaining confidence in visiting a museum or talking about art. And that has actually been a really nice thing to see come out of the post core service, which we will turn to the end. So the self reported motivations also reflect that learners are here for personal enrichment, ideas and inspiration, what you can see new techniques and processes and what you can see here in the orange bar, but midway through is also a chance to participate in something. So All right, great. We'll continue on. Sorry, again, for that interruption. I'll try to just speed up so we can get through more. So here we go. We've got I'm moving on to the next one, I think a question on on a lot of people have is can money Museum, my glam institution make a MOOC? And we'll talk now about producing and iterating on MOOCs. But I'd like to think that, yes, all institutions can make MOOCs at different scales. And I think the key question here is tapping into existing resources, and having dedicated staff on this as well. So, to start with just an overview of what kind of content is in a MOOC, we've got in moments MOOCs, we include films of artists, designers, curators, we have audio slideshows, or, or artwork slideshows with voiceovers, we do email interviews. And those are particularly easy lift. We have text and image slideshows. And then of course, we also share a lot of readings and resources. And we've been very fortunate over the years to partner with momos publications department, which has offered us what they consider to be exclusive PDFs excerpts of catalogs, so the intro to a catalog are particularly relevant essay, etc. And we'll get into that a little bit more. As we're planning course content. And this, I think, is really one of the key points here we're thinking about synergies across all of our museum platforms, and the priorities moving ahead. So for instance, we're thinking about upcoming exhibitions and labels that are being produced for that exhibition and how we can adapt those texts as the artwork texts in the course. We also make heavy use of Momo audio stops. So adapting audio stops into these audio slideshows, paired with images. And then as I mentioned, excerpts from mobile publications, and we strategize course, content and topics across upcoming programming, education, programming, high performing content, such as how to videos, and most importantly, the learner service. So some of our previous course topics have come from identifying the key topics that learners requested in surveys, and some of them like one that we are currently working on, will be coordinated with an upcoming exhibition. And, and a side note about that, as we're, you know, thinking about an exhibition that's opening in the spring at MoMA. And knowing that many people, far fewer people will be able to see that exhibition in person. It's been really important for the mission to be able to share that exhibition with a wider audience and using the online course platform to do that in a way that's really deep and full, as you can do with an online course.
Unknown Speaker 15:01
And it's also, of course, very, very important to make it accessible. And so we've been fortunate that on Coursera platform, they offer some automated subtitles for all the videos that we then go back in and edit to make sure that they really read well, but also Coursera has a global translator community that adds subtitles and translations. And on some of our videos for our more popular courses, there are even subtitles and 30 language 30 languages on some of the videos. And we are working on alt texts for all of our courses. And we have even experimented once with translating, of course into Mandarin. And I'm happy to talk more about that. And we are currently translating one of our courses into Spanish. So these are ongoing processes for us. And we are always learning more about them. And another aspect of online courses is making the content participatory. So we mentioned in a previous slide that there are films, there are audio slideshows, their email, interviews, and texts, but then sort of the second half of all of that is how do you bring in that interaction that engagement with learners. So we do that through quizzes, which we've structured as practice quizzes or memory checks, because everyone loves a quiz. But they don't like it if it stops them from moving through the course. So they love to know that they're getting it right, but they want to make sure that the it's not too tricky for them to be able to continue moving through the modules. And lesson reflections are a way for us to ask learners to take that moment and pause and think about what they're learning in the course, and what questions they have. And we in some of our courses, do that through the discussion forums. And so we will also share prompts periodically through it and the discussion forums are very, very active way for learners to speak and engage directly with each other too. We'll get into that a little bit more. Here's another if there's also, in addition to thinking about content synergies across platform, the second I think most important idea to take away from this is that the launch is just the beginning of a course program. And what that means is that we with each course after the the brainstorming and the production and the outlining, we then focus on creating a launch plan. And the launch plan sort of sets us up for how we're activating the course for the years after the launch. So the goals of both launch and activation are reaching new learners and also re engaging existing learners that are already in our Coursera community. So we are thinking about visibility during the launch on the museum homepage on social media, on all Eazy E newsletters that we have access to across the museum, really going wide with that, thinking about QR codes on labels. We've just started we were we just started to experiment with that the day before the museum closed in March. So TBD on data from that. We're also partnering with our press team to issue press releases. For these, we're partnering with membership to think about how we can offer sort of additional entry points for members, and then releasing these on YouTube as well. And a little bit more about YouTube, our YouTube team at MoMA has been a key partner for us in these and over the years, we've strategized the videos that we produce for the courses so that they will work really well both for the courses and for YouTube. And there's a secondary, very large community there on YouTube that these videos are released to and then once the course is live, then we continue to activate on both YouTube and Coursera through live q&a is with the instructors which here on the left this GIF you can see a conservator at MoMA Leon on the left, they're joined in with Sarah Meister, one of the instructors of seeing photographs for a live q&a. On the right here you see a video demo of Corey Augustine, the instructor of in the studio, and we filmed additional demo videos and release those to continue activating that community. We also another point here is just the YouTube as a place where audiences already are. And I think that's something that will be shared again and again, has been shared throughout the conferences, finding audiences were meeting them where they already are. And so in the past, some of our other activations have been blog posts on medium or Q and A's on Quora, or Reddit AMA is and those don't all have to be new production. For some of our medium blog posts, we were able to take interview content that hadn't ended up into the final course and readapt that into a blog post. So sort of looking at what you may have generated during the production of the course but that didn't get published in the course and finding new homes for that as a way to activate it. After it's live.
Unknown Speaker 20:03
And during the since March, we have also started doing a lot more office, our q&a is on zoom, and none of them have had the technical difficulties that I had this time. So they happen to everyone. And so we we did. I know that this is this is another topic that was discussed in yesterday's panels. But we we hosted 23 virtual events for learners between March and July, which was obviously not a sustainable pace. But it was a sort of rapid just response to trying to be there and be present as we were seeing that our enrollments were growing rapidly. And so it was, it was wonderful to get curators conservators from throughout the museum, to join in and be part of those virtual events. And to have that direct one on one or not one on one, but one to many, but having learners directly ask their questions to the instructors and getting a response to them. And they can be broad, really broad questions or very specific ones like here in the in the screenshot, you see this is our conservators answering questions about mixing acrylic and oil paints for the in the studio post war abstract painting course. And here, I just want to mention one more very special activation, which was in 2018. For the in the studio poster abstract painting course, we hosted our first ever exhibition of artworks that learners had made. And in that course, learners explore the techniques and processes of seven abstract painters, post war abstract painters, and they are invited to make their own works and share them back in the discussion forums. And what we had found was that learners really enjoyed sharing their images and created this sort of informal credit environment where someone might say, like I'm having, I'm struggling with this aspect of my painting, can you help me out, and other learners would chime in, or they might comment on the use of color or composition. And so we sort of building on that, and the really wonderful sense of community that come up in that course, we created an exhibition, we had a three week open call in the course only that yielded 500 submissions. And we put all of them up in slideshows. And if you know me, you'll recognize that this space here is the is the Coleman Education Center. And so we had an opening there, on a snowy January, weeknight and 200 people came, including seven learners who had actually traveled from abroad to visit their exhibition opening a moment, which was it was just incredibly special. And for the weeks after, people would come to the museum to find their work on the slideshows. And then they'd actually like call me down from my desk, and I'd come beat them, it was just a very wonderful time to sort of see in real life, this community that had been built through the course. And based on the success of seeing that people really enjoyed responding to image prompts and creative prompts, we began adding those to the rest of our courses. So instead of, instead of just having prompts that asked you to respond and type out your thoughts or your questions, we started putting in videos that our other colleagues and education were creating about creating artworks at home, we added those into the discussion forums and ask people that if they made if they made an artwork that they would share an image of it back where they would just share visual inspiration that they were looking at.
Unknown Speaker 23:30
So a key question, and this sort of will wrap up the presentation aspect of this, and we'll move on to questions. But a key question that I think everyone wants to know is how do you measure success in the MOOC. And this here, the ultimate measure of success is happy learners. like you see here, Shakira posing with her certificate from you pen. And there are various ways that we measure that so many of you probably know that MOOC completion rates are very low. They're typically around two to 4%, across many platforms, and it's higher for paying learners and ours are in this range, too. But we taking that as a sort of base point. We measure success through a number of different factors, both quantitative and qualitative. And some of those are so we started the completion rates, what we're looking at is changes to those completion rates. For instance, in the course that we launched in January, called what is contemporary art. That was our first course that did not have a peer reviewed final assignment. And that was one of the key changes from other courses that we had produced. And our completion rate on that course is much higher than in other courses. And so even being able to measure and sort of tweak and then assess how those changes can affect the completion rate is helpful for us as one of many measurements. But we also are looking at cross course enrollment, to know that once someone takes one MoMA course are they then going into another course. And then continuing on in the MoMA community we're looking at, we look closely at weight ratings and reviews. So we, I think an important thing for any new program, of course, any virtual program is to be constantly looking at the feedback that you're getting from the course. So having, whether it's by monthly quarterly, doing a deep dive into those reviews, and pulling out what learners are saying back to you about what they've taken from the course what they enjoy, but also what they what were pain points and what they would improve in it. And then and then and then sort of filtering those through a strategy to find what's what's actionable. And what's a priority to change. We also look at attendance at virtual events as a way of measuring the success. And we're looking for Momo, one of the most important aspects is really the activity in a discussion forums and, and seeing that learners are engaging with each other and that they're responding to the prompts. And having dialogue in there. And then and then as I mentioned, survey feedback to is another really important aspect for us. And we have taken survey feedback in the past and done two course, we call course experience improvement projects, which are really doing that deep dive three years after a course has been launched, doing a very, very deep dive into our survey feedback into our reviews or ratings discussion forum posts, and then sort of distilling out some of the key changes that we can make to try to improve the experience and then using these more quantitative measurements, like ratings changes and completion rates to assess the success of those changes. And this is here just a little bit more about that. So we have, we have both both midcourt pre mid and post core surveys to gather feedback at each point. And because completion rates tend to be so low and Luke's we don't depend on a post core survey, because that would get so few learners, but we depend on the mid core survey to try to gather as much feedback as we can to help bring learners past that midpoint of the course. Going into the future. As you can see here, this is just a screenshot of some survey feedback from seeing through photographs, before our most recent refresh of it, which was in September. And we could very clearly see from learner feedback that they wanted creative activities. They wanted opportunities to submit their artwork and seeing through photographs when it had originally been launched, only had discussion forums with text prompts. And so we made that a priority of our recent refresh was to add in image based prompts and every every one now my audio has come back on my computer. This is my computer is taken over. Okay, so I think that is it. And we will now turn hopefully successfully, I can unshare my screen after a second go challenges will now turn to the q&a.
Unknown Speaker 28:11
So
Unknown Speaker 28:14
all right, Can you all see? Come back on. Okay, so I think I'm back on the computer. There aren't two of me anymore. Um, All right, great. So let's see, um, maybe I have been planning on sort of start, Neil just start on the, at the top of the list of questions. And sorry for being a little flustered after the technical challenges. But, um, let's see if I can find the first one. So. Okay, so the first question was about what made you choose Coursera. And were there any other platforms you considered? What made us choose Coursera. So that was actually before my time. And it was back in 2012. Coursera, had actually recently launched at that time, my understanding was that Coursera MoMA started on Coursera. We started as part of course, there's teaching portal. And that's why we began with three courses for K to 12 teachers. And gradually over time, as Coursera grew, and as its audience grew, we could see that the learners in our courses were not just teachers, they were also more general audiences who were looking for who were getting confused by content about how to teach with art. And so that explains sort of that transition from gradually from entering as a teaching platform or entering as a teacher portal platform into a full general partner. other platforms we considered. And I can't unfortunately, speak to this because I wasn't there. But I think our sense is that Coursera has a very wide reach, and that it's available internationally. And even though it's a lot of it's mostly universities, it has, in the last few years also grown to include industry platforms, like Google and Facebook are creating professional certificates on it. And there are two other museums on Coursera and exploratory on the day and age. And I think we, over time you have assessed whether Coursera is the right platform for us. And I think based on the reach, based on seeing that people are still coming to these from by searching for painting, or photography, and not searching for MoMA. That is, it's still felt like a really great platform to be on. And definitely the partner support has been wonderful there. And the changes to the platform have been, are growing all the time. And they're definitely expanding and improving. So it's been easy for us to because there is a very, sort of straightforward content management system on the back end. So let's see, I'll go to the next question. But just want to make sure you all are able to even see and hear correctly. Just give me a thumbs up. Okay, great. Okay, so, um, do we create the same course content for different age ranges. So we're assuming that based on our based on the demographics data that we have from Coursera, and the dashboard are, it's about 68%, that are between the ages of 25 to 44. And then after that, I think the next largest age group is 16, to 24. And I know that the term the phrase general audience is not really a helpful one for anyone. But I think when we're using it in this context, we're we're responding directly to what we're seeing as the demographics so. So that is the that's, that's, I think we're thinking less about you danger, or less about age ranges, and more about the sense that these are informal. These are not college courses, they're not credit bearing, someone is probably taking these in addition to working part time or full time. And that these have to be that sort of provide that cross between enrichment and personal creativity and inspiration. And let's see dismal money to pay Coursera for hosting. So the arrangement, I think, is different with different institutions. But there's a revenue share, so that some of the revenue that comes from the certificates is split between the two institutions. And Coursera also supports us on marketing efforts as well. Okay, so I don't thinking that I don't want you all to actually have to look at me scrolling through these questions. And maybe Eleanor, if you want to chime in to if you're there.
Unknown Speaker 33:07
And just as a fun side note to while we're doing this, I want to say that Eleanor is my mentee, and my previous mentor, Sema is also on the call. So just want to give a shout out to the MSDN mentorship community as a really wonderful way for people to make lasting friendships throughout this community, and really make it feel like a welcoming community. So maybe Eleanor a few. We can just kind of take turns going back and forth between reading off and all I'll continue reading these off. And then if you want to also pull them out of the chat too, and share them. Sure, yeah, we had Jacques who, okay. We had jack, who was kind of echoing at Miranda's first question, and I think he kind of answered a little bit. Um, let's see. Thanks, everyone. As I'm scrolling through the chat, it's really nice to see all of your comments as I was troubleshooting to and encouragement.
Unknown Speaker 34:10
Okay, so a question here that I'm seeing is, too, so do we know how far along learners typically get. So a few years ago Coursera actually posted or create a new dashboard for us, the learning the learner funnel, and that was a real sort of revelation for us was to see how far learners were getting into our courses. And I want to definitely emphasize that the enrollment numbers that I mentioned at the beginning, are very, as you can tell by the two to 4% completion rate are very different than the numbers of learners who get through it. And, and that I think is is just part of sort of the standard funnel for MOOCs. We know that there is a drop off of probably about 50%, from enrollments to even getting through The first full module, and then beyond that it drops off to 17 for two, etc. With variations, but it is that was that seeing that audience funnel was actually the prompt for us to add in those pre mid and post course surveys and then to use that to iterate on the courses to try to see, you know, where's the drop off? Where are the pain points? What can we what can we add to the courses to help bring learners throughout. But I think an important thing to know with MOOCs is also that our goal is not is not to get learners to the completion, it's to support them in having a good learning experience, wherever that may be within, within the course and the platform. So sometimes that can be taking one week of the course. And then engaging in the discussion forums for that. And that may be enough for a learner, but it's sort of about how they would write their own experience in the course. And then ideally, you know, if they are then drawn into engaging further with MoMA community through visiting mama.org, or visiting her YouTube channel, or engaging on social, those are also ways that are a sign of success of their engagement in it, but, but really, it's primarily, you know, back to that Shakira image, it's, it's really about making sort of a happy, satisfied learner, wherever that may be in the course. Okay, so there's another question here about how are the MCs budgeted funded. So I said, within education, I'm on an interpretation, digital learning and audience research team. And so we have I, for a long time have been the only staff member who is full time for online courses. But we actually recently were just able, thanks to very generous funding from our core sponsor, Volkswagen, we were able to bring on our first ever fellow for digital learning. And that I think, is just another really wonderful sign of the success of the courses and museum and sponsor investment in them, and commitment to the courses over the long run. So staff time has been one of the most important aspects of how these are budgeted and created. And I also work with my boss as well, on these two. So we, the two of us manage the entire program with our third staff member now. And we work across the museum with many people, including our video team, including curators conservators, and other people who are speaking and serving as instructors in the course. And then we work occasionally with an external writer. But now most of that I think, has shifted to being as as all budgets for all museums are changing, we are doing most of that production in house. And so during in the beginning of March, when we shifted to remote working, we were able to create course teams for the first time, which was a really wonderful way of working with more staff throughout education, more staff throughout the museum, and creating teams of about four people for each course. And that includes the three of us who are staff members who work on these, most of the time and so. So for some of the courses, that means two curators plus us plus one staff educator, for others, that means bringing in conservators and between us, we plan and activate the courses and bring those folks on for virtual events as well.
Unknown Speaker 38:41
And we are, I'd say that a good way to think about our costs of production versus revenue, our that think our revenue that we make in a year from the courses is about a third of the cost of producing a course. So over time, you end up sort of bringing that the courses end up paying for themselves that revenue over time. But we are also very grateful to have Volkswagen sponsorship. And and then of course, it can't be underestimated any point, the staff resources and staff time that goes into creating them to. And let's see, there's a question here about do you offer the materials to learners after the course? If so, do you have information on how the materials are accessed and used. So after someone takes the course they can stay enrolled in the course for as long as they like. And we actually encourage learners when they're done taking a course or whatever point they sort of stop moving forward in it to just still stay subscribed to the Course Messages because that's how we activate them. So if you have dropped off after week two, but you stay subscribed to our messages, you will continue to get virtual event invitations. Before COVID you would also get invitations to in person, exhibition events or talks and actually there was a great moment February, our last in person event for the courses. When someone said like, well, I'm here because I got this email, from fashion design, I don't know what that is, but it was really excited about about this in person event. And so they came out, they said part of that. And then actually that that is something that we've seen repeatedly. In 2018, we held a few in person events for our course seeing through photographs. And we saw that when we asked learners only ask the attendees, when they had first taken the course, about half of them raised their hands and said that they had taken it in 2016. So two years later, they were still getting the Course Messages and staying, staying involved with the course and being part of that course community. And that may not have meant that they completed the course, but they were still kind of staying staying bolt of it. So let's see. Another question here. What are some surprises you've had with content that worked? The best surprise for us? And I think you may have dropped this into the chat before we got to it. But the best surprise for us was really seeing how successful image based prompts were, and the discussion forums that, in many ways, changed our course program. We previously had, as I mentioned, you know, a question might be sort of respond to this quote, in, in the readings, and then after we launched into studio only in 2018. And we saw for 2017. And we saw how active and supportive and communal, the discussion forums were around creating and sharing either your own artworks or images. We have read on as many of our discussion forums as possible so far to add in those active image based and creative prompts. And that changes the course experience dramatically, actually. And then from there, we've seen that that's okay. That's the kind of key that learners are looking for. Or that some, some portion of the learners are looking at a big portion of learners are looking for that. And so then, knowing that we can, that sort of shifts like the sense of this being about ideas, inspiration process, and that then becomes a higher priority for us when we're interviewing artists, for other course videos. So that sort of all gets folded into this cycle, a content production. And let's see, so another question here, is how much time is spent with each course over time? These don't appear to be a one and done kind of approach. There's ongoing maintenance or community engagement. Yep, definitely. And I think that that's, as we all here know, maintenance is key for for any digital product, I think one of the benefits of being on Coursera versus hosting our in in house LMS is that we don't do the technical maintenance on the platform that is all handled by course, there's very excellent and responsive technical support team. And so we are really able to focus our changes on
Unknown Speaker 43:04
iterating and improving on that course content. And so that happens, I would say, quarterly is a goal. But we're getting there, as we develop those processes for doing that. And at this point, we are Coursera platform offers item feedback on the back end where someone can just kind of flag if something is incorrect, and those are the first priority, put out those or fix those errors first. And we do that on a on a weekly ongoing basis, just to keep that up for maintenance. And then Apart from that, in the start of in March when we were seeing the enrollments grow so fast. When we created those course teams with other staff from across the museum, one of the things that we started with was asking the course teams to help us facilitate the discussion forums. But we've gradually found over time that facilitating the discussion forums was because of the volume of posts was not really the best way of approaching that. Because you're like if if one of our colleagues in education responded to someone's discussion forum posts, then it would quickly kind of get lost in 10s of pages of other posts. And so that's when we began putting most of our energies into these virtual events, and specifically office hours that are structured as q&a so that those are a good way of getting, we've seen anywhere between 100 to 600 people in each of those. They're a good way of getting the instructors to directly respond to audience questions, but then those are also recorded, added to our YouTube channel unlisted, and then those links are added back into the courses and they continue their reach continues to grow. And audiences who signed into the course after that virtual event happened can still access that those Q and A's and let's see so double enrollment this year. It's really impressive. Do you have a sense of where the majority of that audience has come from online search members? That's a really good question. I tried in advance of this presentation to find out what those demographics were and how those have shifted. And unfortunately, I'm not able to, with the filters that I have on my side, but we do know what we were seeing. I mean, at least in terms of countries, what we were seeing at the start of it was that it was it was very, it was, it was really important to us to see that. As people were beginning to post in the forums and say that they had just entered lockdown, they just started working from home that you could see them relating even there was one particular incident instance of, of someone saying, responding to a post from Italy, and saying that their country had just started lockdown as well. And so like this was a new experience for them. And they were finding connections with other people who had started that a few weeks before them in the forums. But we don't actually know about whether they are coming from. Actually, I'm sorry, let me rephrase that. So one thing that did happen during this was that press actually picked up on these. And what we could see a very clear link was between was on, I think it was maybe April 7, Id magazine, which is a large sort of fashion and style outlet in the UK, they posted about our courses, they had an article that said MoMA just launches five free online courses. And what it was inaccurate, because we didn't just launch five free online courses. But it was fun to think that they thought that we could act that fast. And then from that enrollment had been growing at a much higher rate than before. But then once that article was posted, we saw a huge uptick, particularly in fashion as design and what is contemporary art, which of course speak I think, to the audience in it. And then from ideas article, a lot of fashion and design press picked up on it. And that was an effect that to this day fashion, as design actually has a higher enrollment rate than I think it's now bumped down a little bit. But before March, it had an enrollment rate that was
Unknown Speaker 47:24
slower, but now it is among our highest, so that that effect is been ongoing. And so press is very, very key. And definitely I think I mentioned it touched on it. But that was right during the technical difficulties. So when we launch, we do try to issue a press release for that. And that has actually been a very key part of the outreach on this when we launch seeing through photographs in 2016. That was our first press release for an online course. And it went viral. Particularly with press in Italy, picking it up. We're not entirely sure why. But it did mean that Italy became our second highest country of enrollment for that course, which was an outlier. For the rest of our courses before that point. And let's see, I think there's two more questions here. Do you have any advice on marketing for through social media, social media tending to be built on short term engagement and loops being built on long term engagement? Yeah, I think so we've actually, we've been very fortunate to work very closely with our social media team at MoMA. And so as we're planning the course videos, we're planning them. Like, right now we're working on a new one. And we're making sure that we are producing social assets as we're producing the main course videos as well. And so thinking with our course, thinking with our social media managers about how to adapt that for social because that's a different approach than MOOCs. So for our MOOCs, for instance, we might have, let's say, 10 videos, for social, we'll pull up five fanatic, five themes from those videos, and then make excerpts that touch on five themes. And that can be a way of sort of speaking more to that in a thematic and ongoing learning sense on social through stories. And we've also been working really closely with our social media manager on on sharing with her what works in the courses. So knowing, for instance, that people really love to share images of their artworks. Trying to do that on the social as well. So sharing back when people are posting on social that they have made an artwork sharing that back on wellness, when people are posting in the courses that they've made artwork showing up back on the social as well. And let's see, so is there an institutional imperative for other departments to collaborate on these courses? And that is a question that actually speaks really well to a point that I think I forgot to make when I was speaking before, but having allies in your course production is also a very important part of our program. Sarah Meister, the instructor of seeing through photographs, was our first sort of curator ally on making a course back in 2016. And, and she sort of demonstrates, I think, the the amazing way that having someone who commits having instructor or course team who commits to seeing through the long, the long life of a course and staying involved as an instructor and as a community builder, really improves the course experience for learners. So she works with us, she's she is one but all of our many of our courses, instructors do this. They work with us on Course Messages, they before March semester would invite learners to public talks that she was giving around the world she traveled, and then often she would actually negotiate discounts for them to be able, if it wasn't a free talk, she would find a discount for learners, which was a really nice way of just kind of opening that up. And then as often as possible, she would engage in live q&a is on our YouTube channel. And she has said that being the instructor for the course, has changed the way she thought about her audience as a curator that she used to think about it as like this small group of people who might come to see her exhibition or read her catalogue. But now she thinks about it as a worldwide group of people who come with many different interests and motivations to learning about photography and about art. So and that is really we've been grateful to to find allies in the museum who can support this on an ongoing basis. And that's a very, very key aspect of this also,
Unknown Speaker 51:39
is if if launch is just the beginning, then finding course instructors who are in it to to also learn from their audiences through q&a to over the long haul. So I think there's just two minutes left. Are there any final questions that you all would like to pose or any final information? And I will just say that, I welcome further questions after the session. And adding in here, my contact information as well. And again, apologies for the technical difficulties. I know it was a little bit disruptive, but I hope that we were able to to get back on track well, so. All right. Thank you, everyone. It's great to read your questions and to see you all