Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Studying Kubernetes' Users with SIG... Gaby Moreno Cesar, Josie Pynadath, Tasha Drew, Carl J Pearson

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Studying Kubernetes' Users with SIG Usability - Gaby Moreno Cesar, IBM; Josie Pynadath & Tasha Drew, VMware; Carl J Pearson, UserZoom

Part of our charter is to make sure that the upstream community is building for users of Kubernetes, as well as for the builders of Kubernetes. To further that goal, SIG Usability has been engaging in user research and a jobs to be done study, to better understand who is using Kubernetes, what they are trying to accomplish, and how our project is meeting their needs, and where we need to improve the project and the user experience. Join our SIG's update to learn about our latest research results, what our plans are for the upcoming year, and how to get involved in the upstream usability team as a contributor!

A

Hi everybody and welcome to studying kubernetes users with sig usability. Today we have gabby tasha, josie and carl, um and we're going to quickly introduce ourselves and then dive into the content. Gabby. You want to go first.

B

Sure my name is gabby moreno, cesar I'm a co-lead for the usability and I'm also a product designer at ibm.

A

Hi everybody I'm tasha drew I'm director of product incubation at vmware and then josie hi.

C

My name is josie, I'm a product designer at vmware.

D

Hey everybody, I'm carl, I'm a senior ux researcher at userzoom.

A

Cool and together we've been uh leading a bunch of various efforts in sig usability, um so some quick housekeeping uh just to kick things off if you're wondering how to get involved in our upstream sig. Our chairs are myself gabby and himanchu.

A

I've included links here for ways that you can quickly find uh interesting content, so our homepage is on github, so just sick usability, and so this area has basically a little project plan and more information and more links to documentation. um We also have sort of our landing page under kubernetes.

A

If you want to see what our latest meetings have been about, uh links to youtube's all of our meeting, all of our public meetings are recorded. So that's this uh google docs link that you see right here. um We also have a slack channel that we're all in the kubernetes slack, and so you can easily join our slack channel, which is just sig dash usability and finally, the most important starting point to get involved with our sig is to join our mailing list.

A

This is a google group, so if you join the google group, you will see all of our calendar invitations, and so we have monthly meetings where we all get together. We kind of talk about our latest research. If anybody has a new topic, you're very welcome to add it to our meeting agenda and then we can discuss it as time permits.

A

All of these meetings are recorded, but the single easiest way to get started is just to join this google group you'll get the calendar invitations, you'll start getting links to youtube and comments about our agenda so yeah. These are the quick, easy ways to get started with sig usability and now over to gabby to kick off the meat of our talk.

B

Cool thanks, sasha, um I'm actually gonna pass uh pass off to carl to talk a little bit about our jobs to be done. Study um you know, carl will tell you what our jobs to be done. Study is all about, um but this is kind of the first project that say usability um kind of like took on, and so uh last year uh we when we were at kubecon europe.

B

We showed y'all kind of where we were at the very beginnings of it, and so now um we're gonna uh present just a little bit about what jobs to be done. Theory is about and then where we are in the project.

D

All right awesome thanks gabby yeah, so we are at the kind of middle of this process right now for jobs to be done, framework, which is a kind of research where we look at who our users are and what they're trying to get done. One quote that I think sums up. The high level theory very well is that people don't want to buy a quarter inch, drill or drill bit. They want a quarter inch hole they're.

D

Looking for the outcome of the situation, not necessarily the tool that gets the job done so to flesh that out a little bit more. The reason people are interacting with products and services is not necessarily because they want to interact with that product or service. It's because they're trying to get a specific job done with that product. So what jobs to be done is focused on is looking at what these jobs are and boiling them down to their most essential pieces.

D

So, even as the technology changes, people are still generally trying to get the same core job done, even though the names and kind of underlying technologies might might change for for how they go about that.

D

So by looking at the the job to be done rather than the technology, it lets us get a really good picture of what users want and it makes innovation a little bit more predictable and a little more scientific and trying to understand what exactly people are looking for, rather than just kind of hoping or guessing so jobs to be done can be used in a a lot of different ways um and it's kind of a longer process and, like I said, we're kind of in the middle, maybe the beginning of the middle is the most accurate way to put it.

D

So. First, we start with defining our users, which is part of a survey that we launched um some time ago, where we looked at people's general backgrounds um and how they use kubernetes and some kind of basic facts about their technical and organizational environments.

D

And we use that to feed right into recruitment, because we actually need to talk to real people, to the real users of kubernetes to understand what they're trying to get done with it. That's a big challenge with user research and gabby's going to get into the details of that challenge a little bit down the road here, but so we found all those users and then the next piece is to uncover the needs of those users. So we started with one of the most basic tools and a user researchers.

D

Toolkit is just interviewing users and then we have a couple ways: mental models and job mapping to analyze all that data, um and then, when we have a sense of what the actual jobs are, we go into the next phase of prioritizing to see which of these jobs are most critical for us to look at and then finally, we apply those learnings. So I'm going to walk through the details of these steps.

D

D

So, like I mentioned, the users were somewhat already defined, we're just looking for people that use kubernetes. So we started by collecting some background information and this contact information to get these users to interview to figure out what their, what their core jobs are.

D

So the the data collection process is, is very qualitative to start. So we use semi-structured interviews. We don't have essentially we're not reading off of a questionnaire. We have topics that we want to follow, but we really try to let the users lead us down, what's most important to them, so we're not imposing what we think is important, but rather finding organically, what's actually important to these users when they're using kubernetes in their organization and their their work on their own.

D

However, they might be using it and we look at not just the technical pieces or could you scoot back for one sec, not just the technical pieces, but also how kubernetes fits into an organization how it fits for their personal goals. So it's everything beyond just the the command line or visual interfaces they might use, but everything that surrounds kubernetes entirely.

D

So we get all these user interviews and then we transcribe these interviews for coding um in this qualitative coding process.

D

So we have kind of a two-part process. The mental model piece is not necessarily part of the traditional jobs to be done approach as other user researchers might know, but we wanted to include this, so we could really be sure that we're focusing on user behaviors and not just attitudes.

D

So when we go through these interviews in a first pass, what we're doing is we're extracting all the things that people actually do, not just what they say, but uh how they actually behave around kubernetes and whether that's directly interacting with the interface or you know, dealing with other teams. Onboarding kubernetes could be anything so try to stick to this core tentative of user research. That is, to focus on the behaviors over the attitudes.

D

When we get all of these behaviors extracted and we'll end up with hundreds and hundreds of behaviors, this is kind of an example version here. From the author of this method.

D

Is you get all the tasks down below so, as might be something like start a cluster or you know, create uh create a new user role, something like that and we sort of organize them together and stack them up to to get a sense of kind of bigger areas like security or onboarding or team communication, all the things that might go into using kubernetes and get a big map of everything that people do when they're using kubernetes.

D

When we have all of these behaviors, the next phase is to bring them into this kind of specific framework. That is the jobs to be done framework. So we can map out what exactly the job is, that people are trying to do when they're, using kubernetes, so they're employing kubernetes to get a certain job done. So what is that job? So to do that, we have to create job statements and then map those job statements to job steps. I know that's a lot of jargon, but I'll show you what that looks like right now.

D

So, let's say, for example, we have a behavior that we have in the mental model map like on board developers that are new to the team. So that's a good behavior, but um to bring it into the jobs to be done framework. We bring it into this sort of table view.

D

That is something that's a little bit more actionable and structured. So we always have a direction involved with the job, whether trying to minimize something or maximize something. I generally stick to minimizing something, because it's always easier to move something closer to zero. If you're trying to maximize it's harder to know exactly, you know how high is high enough, so you have a better end point with minimize so from onboard developers that are new to the team.

D

We might change that and to minimize the time to onboard developers that are new to kubernetes, so just creates a very nice clean way to take what is often a really messy set of behaviors. I chose a really easy one, but there are some that are a lot more complicated to be sure that they're fitting into this jobs to be done framework that we can then use to put into the job map, which is the next part.

D

So we'll have many dozens of these job statements and then we put them into this typical map. Here we might adjust this a little bit that are associated with everything that somebody does every job. Has these eight steps we're there defining the problem, locating what they need to do the job preparing for the job doing a final confirmation before they actually execute the job, then they have to execute that job monitor how it went if anything went wrong, modify it and then conclude that job.

D

So we have some adjectives here underneath each of these that kind of show common phrases that might start for each step. So, for example, prepare is often something like uh set up uh proof of concept or organize documentation, something like that and monitor as verify track and check. So it might be. um You know, check pod health or something like that, so you can kind of just imagine all the ways that the many tasks involved in kubernetes will kind of fall in under this map.

D

So we'll have a really good sense of kind of the cyclical nature of all the jobs involved with using.

D

Kubernetes, so the next piece is, we have all these jobs and we're sure that some people have these jobs and that they're real they're things that we actually learn from users, we're not just making these up. This is empirical data, but it's qualitative, so it's hard to say which ones. These are a problem for most people, which ones are really important to focus on.

D

So this next part is where we quantify this, and we use this by creating a survey that takes in each of those job statements all those little pieces that go together and we try to rank them with a survey.

D

So imagine each one of these dots is a job statement, so something like you know, create a new cluster or something like that or on board a new developer to kubernetes.

D

So each of these job statements will be ranked on satisfaction and importance.

D

So we'll put the job statement in front of somebody we'll say how satisfied are you with this at your job right now and then how important is this for your job right now and what we get is this map of all these job statements, and when we look at the lower right corner, we can see that's where they're really important tasks that are really unsatisfying to do right now, so that kind of creates a a high level and empirical theoretically back sense of the jobs that are what might need to be targeted.

D

What people are thinking is very important, but aren't able to do very easily right now, so it again it brings innovation into a more sort of scientific space than how it typically might be.

D

And that's all how the process works and, of course, with research kind of half of it is doing the research. The second half is evangelizing that research and putting it into action. So these findings will yield these really interesting and useful jobs that are prioritized, so we can see what's really important and might need to be focused on, but this is typically, as far as I am aware, pretty much used in a for-profit context and not really hasn't really been used in the open source space.

D

Yet so the way that we will implement these findings and the way that people will use these the way that we can make them more shareable, we'll all be brand new. So there's a big question mark here, because this whole process is pretty fresh for the open source software space. So we'll be kind of writing the rules for this, and obviously just when you're communicating the findings. Only half of it is the researcher. That's communicating the other half is the audience, so it'll be a very collaborative process.

D

I'm really excited to see how we share all these findings out and the way that we can make sure that they get the most use if they can. So I'm definitely excited to report back on this when we have more findings to share and then also how the communication process goes as.

D

B

Cool thanks carl, um so I'm gonna go ahead and talk a little bit about.

B

um You know the tools and just the research process that we've been following carl gave a great overview just about what jobs to be done is, um and so now I want to share with you just kind of like what it looks like um for for an open source uh special interest group.

B

um So at the beginning uh there were very few of us, maybe like one or two working on this study. uh This was about a year ago, and the cool thing is that um the team working on this has actually grown um as contributors kind of hear about it and um want to help out and so right now. This is.

D

B

Know where our contributors are everything you know, no one is really in the same time zone, and so um you know when we were thinking about some of the goals for for our research process. um Obviously, you know we're an open source community and so uh being open about the process and open about the opportunities. um No matter, uh no matter where you live was was really important.

B

um Also, you know for me, I'm a ux designer and so it's my first time contributing to open source. um I have a lot of research background, but um you know maybe kubernetes and I work on a kubernetes product, but um you know the people coming in. um If you don't have experience with open source um that that should be okay and we'll kind of like support you through um through the process, or maybe you don't know as much about kubernetes.

B

um That should be okay as well, because um if you have a really strong research background, we can use that for this project as well. So that was um some of the things that we also wanted to to consider and then also the fact that our contributors, you know some of us- are able to contribute part-time uh to this project. But some of us are only able to contribute on 10 or 20 time, um and so just taking that into consideration as well.

B

um So you know how research um studies start we had to do recruiting, because we wanted to talk to people that use kubernetes for their jobs, and so for this, the cncf uh twitter was able to help us out and.

A

B

Know we developed a screening survey um to kind of look for these people that are working companies that use containers that use kubernetes um on a day-to-day basis, and so through that we were able to kind of build up our our list of people that were willing to speak to us.

B

um We try and templatize everything, so you know everything from our outreach email. um You know like hey. uh We we'd like to chat with you. You filled out our survey. um We try to also automate as much as possible so we're using calendly uh to allow participants to book time with us.

B

um Initially, we were only reaching out to north american people and now we've added more slots so that we're able to reach out to european folks um one of the things that our this team also worked on was this um data data consent form that was generated um by this tool, that a really cool group called uh the research ops uh global group, um basically like they're kind of this, like slack community of really talented researchers um that are really passionate about uh just how to oper operationalize user research and design.

B

So they built this really great tool that you know. If you put in the goals of your research project, they will generate um a form that basically tells participants how their data will be used um and all that good stuff. You know that it will be recorded and what happens to the recording and who can access it uh and everything so that they kind of have informed consent.

B

So once they uh review the form- and this is all through calendly- they review the form and they sign up. um Then we'll coordinate with some of our some of our leads, uh someone that can interview them, um and then we also leave room for one to two shadowers. um So that's all that also kind of like allows people to sit in um take notes and also just like learn from the process.

B

So before we before we um sent out emails, we had developed a kind of like a standard jobs to be done. Discussion guide.

B

um This helps to just kind of like document the list of questions that we want to go about asking, but I think, practically speaking during the interviews, um we're really interested in just having like an organic conversation with people about how they use kubernetes and just kind of like discovering um with them what that looks like in their day-to-day jobs, um so that, but this is a great kind of like uh guideline yeah we thank them.

B

um You know free kubernetes t-shirt uh who doesn't like a kubernetes t-shirt, and then um this is kind of getting to the part of the process where we are at um as we interview people we're also at the same time, uh you know trans, um transcribing and analyzing, so we're using tetra insights for this um tetra insights.

B

If you upload a recording, it will transcribe it for you, which is really cool and for all of like the infrastructure jargon, or they can like kubernetes terms that come up those need just some lightweight cleanup, but for the most part it does a pretty good job, um and so we just go through.

B

um We clean it up and then we're in the process.

B

uh This is where we are right now uh we're in the process of um analyzing some of the interviews and just kind of like um distilling through what people say, uh and you know someone has an opinion kind of like trying to like dig deep into what behavior that opinion uh is reflecting, uh and so where tetra insights is awesome and that we're able to kind of like read through a transcript and just tag behaviors and um just like make notes and all that good stuff.

B

So that's that's where right now and and, like carl said we're we're about halfway halfway ish process, um it's ongoing and what I'm gonna do now is I'm gonna pass to josie and josie will walk us through uh opportunities to get involved with the sig.

C

Thank you so much gaby. The usability sig is composed of a lot of really wonderful people with a ton of industry. Experience from different companies. uh Gabby next slide.

C

We meet virtually once a month and you can get a calendar invite by joining the google group, so come, say, hi and introduce yourself if you're interested in getting involved. You can come share your ideas or submit a usability issue that you've seen or you've run into.

C

Yourself, if you're interested specifically in the jobs to be done study, we have a large range of opportunities to get involved depending on your comfort level and experience. Shadowing and taking notes is a really fast way to onboard. As gabby mentioned. We'd love help cleaning up the auto-generated transcripts.

C

It usually does a great job most of the time, but then sometimes the program will translate kubernetes into goober daddies, so any help we can get there would be really valuable and if you feel really comfortable and have a lot of experience, you're welcome to help analyze and tag the interviews or lead a user interview.

C

I was really nervous when I started. Kate's can be a pretty technical space, but it turned out to be a lot easier than I expected thanks to the support of the other designers as well as the pre-work that was done by gabby and carl.

C

uh Now we know that kubernetes is a pretty technical community, but the usability sig is is a bit different, we're mostly researchers, designers and pms, and it's not that we're not technical. It's just not a prerequisite to get involved if you're here and you want to pitch your own ideas, you're more than welcome.

C

In fact, the folks on this team are really really nice and they'll probably support you through the process. So a couple of examples of initiatives you can lead include design principles for kubernetes. You can come up with your own research frameworks.

C

You can invest time in coming up with user archetypes or personas or roles you can even think of design. First issues for non-technical members.

C

If you're from another sig or you'd like to drive an initiative for another sig, we're also here to provide support so, for example, the contributor experience sig has mentioned that they'd, like support figuring out how to gather user feedback for beta features.

C

If this is something you're interested in we're more than happy to help, you drive that um and that's it. Thank you so much for attending our talk, if you have any questions, feel free to drop them in the cncf chat or come find us in slack in the sig usability channel.