GitHub Satellite 2016, 13 Sep 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Staying sane with small incremental releases - GitHub Satellite 2016

Description

Frustrated by huge code reviews? Blocked by something your co-worker is refactoring? Nervous about shipping? These are all symptoms that your releases are too big. I'll talk about how to make sure you ship small incremental releases so you keep momentum, increase team happiness and become a mean shipping machine."

Ron Cohen leads engineering and product at Opbeat where he's adamant about building efficient engineering and product teams by breaking down silos, minimizing process and increasing trust and empathy.

About Satellite:
GitHub Satellite 2016 was a celebration of the latest and greatest in development. Watch the recordings to learn how engineers, founders, activists, and beyond tackle the latest and most difficult challenges in software.

For more, join us in San Francisco on September 13th-15th for GitHub Universe, three days filled with the creativity and curiosity of the largest software community in the world.

For more information, go to githubuniverse.com.

A

Hi welcome to my top. It's called staying sane with small incremental releases, and that is partly what I'm going to talk about today. It's mostly when I'm going to talk about my name is Ron Coan I helped start a company called upbeat where work today, you can recognize us by these giant coffee pots. That's a logo!

A

You can find me on Twitter and get up at Ron, Coan and an upbeat. We do next generation performance monitoring. So if you're building a web application and you're worried about performance, you should check us out. It looks a bit like this. We have pretty graphs deep insights and I mobile application, so you can stay up to date. While you want to go okay, so first I'm going to tell you a story, a story, a real story that happened at upbeat and and then I'm going to explain to you why small incremental releases are better.

A

Why you're going to have a better life if you start releasing small incremental releases and I'm going to tell you how to get there so I'll give you actionable tips on how to get really good at stripping small incremental releases? And why do we want to ship small incremental releases? What is it?

A

Where is it that we want to go nirvana where we ship more cool stuff to our users and our open source collaborators, and where we sleep more, so we don't get woken up in the middle of the night, because something isn't working so that that's why we want to go, and mostly it's about increasing velocity of our development and increasing the quality of the code.

A

We output now fair warning if you're building for the apple app store a lot of the tips that I have here are not going to be applicable because apple takes awhile to review your application. Unfortunately, okay story time, so it up eat we've been working on this amazing new feature. We'd spent three weeks on it and we were super excited to finally ship it and it had 39 commits and 20 changed files. It was pretty big, very significant feature and we were really really excited to get it out.

A

It also happened to be that there were six bug fixes in this thing, we're about to ship and also for improvements. It just happened that we fix some bugs and made some improvements, but of course, when we finally shipped it, this is what happened. Everything was on fire and it was very, very uncomfortable. We've been working on this awesome new thing for three weeks, but it didn't work so we had to roll back and fix it and try again and it wasn't it wasn't a great day. We had been very excited.

A

We sort of build up momentum, everybody was ready to push the button and then work, and so we rolled back, we fixed it. We try it again and this time it's sort of worked I think we tried two more times before actually got out, which was horrible. A horrible experience and in general, rolling back stuff is super stressful. When you ship something that doesn't work, you get really stressed out. It's a really bad feeling. It's also super anti-climatic.

A

You've been working on this awesome thing for three weeks and you're very, very excited to finally get it out the door and it doesn't work. So in total it's just really demotivating. It's a it's a bad situation, so we asked ourselves what happened? How how do we end up here? Why did it fail and what became obvious was that all this stuff should have been shipped individually and in hindsight it's obvious ship. It individually. It's not a big deal, just get things out 11 of the time.

A

So it's simply too much too big of a release and there's a lot of release. It has a lot of reasons why big releases fail. In this particular example, the code review took half a day and it wasn't very thorough.

A

How do I know it wasn't very thorough because I did it and you might have seen this tweet. In my experience, it's pretty accurate. So if you have 10 lines of code, the code review is going to uncover 10 deficiencies. But if you have a 500 line review, it's just going to say looks good to me ship. It no problem, and so this is an experience that absolutely felt this is something that really resonates with with what I've experienced, but there's actually also some actual data on it and so smart bear software.

A

Did this study on 2500 code reviews and 3.2 million lines of code and they found sort of a correlation between the size of the code review and the number of defects found. So the graph shows you the number of detected defects per thousand lines of code on the one axis and the number of lines of code on the review on the other axis and I'm, not sure that these dots really sort of merit.

A

This very smooth, fine line that they've drawn on here, but that does seem to be a correlation, so you should really be shipping small incremental releases. It is something that we've learned the hard way and I think you'll find that if you start shipping smaller mental releases, you are going have a better time. There are the reasons why you should be shipping small incremental releases. One of them is that shipping feels good.

A

It's an amazing feeling when you press the button and this new cool stuff goes out to your users. We want to do that more. You want to feel good, more want to be more happy.

A

Another reason is that when something does break, if you ship the small release, it's much easier to answer these questions what's changed and we can fix it.

A

If you get really good at shipping, small releases you'll end up in a place where each release only involves one developer, then it's very easy to answer. Who can fix it? It's a good situation. It is where you want to be, and really it's about accountability. If you have small releases, accountability becomes much more clear. You know should fix this, because that's a person who shipped it and.

A

An upbeat we've adopted this policy and it's something that we've had with us from the very beginning, and the idea is that if you build something, then you're the person that presses the button to make it go into production, so you're there in the office or working remote, pressing the button and stuff goes out to production. That means that when stuff breaks you around to fix it.

A

It makes it much easier to answer what changed and we can fix it. Two other reasons why you should be making small incremental releases it's easier to roll stuff back. So if you've only changed a tiny thing chances are your users might not even have started using the new thing yet because they haven't realized, it's gone out. It's a very bad situation. When you roll something out to users, they start using it and you have to take it away from them again because it doesn't work properly so to get a bad experience twice.

A

First, I start using the new thing, it doesn't really work and then you take it away again double bed and there's also more parallelism in the team less blocking, because your yours, you're deploying stuff all the time.

A

Okay, if you're convinced organist sort of cool good, how do we get there? What do we need to do to start shaping small incremental releases?

A

That's a few prerequisites that you need to sort of check that you need to do you need to do one of them is you need to be able to do reliable and fast deployments? So you need to be able to trust that when you press the button, the code actually goes out properly and there's no service disruption for your users. That's extremely important.

A

Another thing is right by the way, an upbeat it takes us one to two minutes to get code into production, and it's one comment at your type: that's all it takes.

A

Another thing that is really important is good automatic tests. Now you want to know that when you press the button, the stuff that goes out actually works or be confident to pretty high degree that it actually works, and so, if somebody has to click through your whole application every time you want to ship something to figure out. If it still works, it's not going to work like you can't do small releases, then so you need automated tests, they need to run quickly and they need to be reliable.

A

Finally, you need low overhead code reviews. If a committee of three people have to review your pull request and all say thumbs up every time, you need to chip something it's not gonna work. If you only get your stuff reviewed every other Monday, when there's a full moon, it's not gonna work. You need a lightweight process that enables quick and low overhead code reviews.

A

Okay, so I mean you have that never see a show of hands hmm like four people, okay, I was gonna, say. If you already have that, then you can pat yourself on the back so you're for people. You can pat yourself on the back good job um and we actually have this a top it. So we still went into trouble and we were asking ourselves what's going on. We have all this good stuff in place, but we still run into these huge release this once in a while.

A

What's going on, so it turns out that this is the train of thought that can lead to big releases, so you're working on this thing, let's call the feature a it's amazing. You can't wait to get it out. It's gonna change, how people think about sliced bread and it uses module be so your thing is feature a is brilliant. It's the code is clean.

A

Crisp features are made, which is the features amazing and the tests are great and it uses module B, which sort of does what it's supposed to do, and you took a little peek inside module B and you find that the code is pretty messy and the tests aren't that great and the API that you're coding up against is not exactly what you wanted it to be. So you think to yourself: okay, while I'm doing this really nice feature a I might just go ahead and refactor module be just well I'm at it.

A

In fact, I module be no big deal and there you have it. You have a big release and you've made a mistake. That's a problem! What you really should have done is finish feature a ship. It get it working get out.

A

There then go back refactor module, B or drop what you're doing with feature a refactor module, B ship that and then go back to finishing feature a and so what goes on inside the developers head at this particular scenario is: can I just roll this fix into that branch that I'm already working on and the developer is thinking what is the risk of doing a big release and hopefully I've convinced you that big releases are bad and that there's a significant risk?

A

You saw the graph and then on the other side, the developer thinks what is the overhead of doing another release? What is the overhead of putting this fiction a separate release, and so, of course, you can work on minimizing that overhead. Remember the three prerequisites that I talked about. So this is the sort of thought process. The rational developer goes through em, so I encourage you all the time to think about what is the smallest possible release. I can make that makes sense every day. What is the smallest possible release?

A

I can make that makes sense. That actually makes a difference and it turns out that, even if you're building a single feature, you can most of the time ship it in separate releases. So you can split up that one feature and ship it in separate releases. How?

A

How does that make sense? How my user is going to use an unfinished feature? And this is where I talk about feature flags or how I learned to stop worrying and love feature flags who are familiar with feature flags, good bunch, cool? How many of you folks use them slightly less amount of people? Okay well feature flags.

A

Are these if-else statements that you put into your code and so that enables you to run the code, this particular piece of code only for a group of users that you've selected- and that means you can selectively enable half a feature or an unfinished feature for a group of trusted users. So you reach out to them, and you tell them hey this new thing: we're really excited about it.

A

We've been able to feature flag for you, so you can have a first look. What do you think it's a fast enough? Is it doing what you thought it would do em.

A

Now, if you're using feature Flags, you can suddenly ship a single feature over multiple releases. That's pretty cool! Now, the first time you use feature flags you'll, probably feel a little a little a little dirty, because it's not typically the way developers like to use these developers you'd like to do these weird, if it's if-else statements that I have in my code, but it's actually a really good tool.

A

So the idea is, of course, to get code into production quickly, get in front of the users, see what they think see if your stuff actually works. It also enables you to make much smaller releases with all the advantages that we just heard about, so you can get code more quickly in production, easier with feature flags. Now, of course, making small increments releases takes practice and it's a balance. You need to find the right balance between the overhead of actually making a release and then the risk in making a huge release.

A

So it does take practice and a good way to get better at it is to make it a part of the review process. So ask the author of the code when you're reviewing something if this could possibly be split up. Could we ship this in separate releases? What, if you put it behind a feature flag?

A

So what does feature flags look like in open source, hmm it's slightly different in open source and what you can do is you can ask some of your users to install a module directly from get up, for example, so you push to a new branch and you ask them to install this new cool thing. Is we've been working on and so it open source a little bit flipped around the user basically decides if they want to try out your new thing instead of you just opting them in okay.

A

I want to talk about simple, a big part of shipping small releases. Is this concept of simple it's important, so you want to ship the simplest thing. That's going to work, nothing simpler than that, but still the simplest thing and mike krieger, the co-founder of instagram, said this and I think he's very right. Do the simple thing first, so always try to do the simple thing. First- and this is a concept that really helped inform a lot of this design engineering decisions at Instagram and apart doing things that are simple, is not reinventing the wheel.

A

Will so there's different communities that I see in the development the world and some of them have a tendency to reinvent the wheel. There's this thing called the not invented here policy and the concept is that if it was invented here, then we can't use it. Some people have this idea for some reason and it's a problem. It means that they have to write everything themselves now, that is, that sort of goes against all the stuff that is open, source right. There's this fantastic world of open source, there's all this code that we can use.

A

So if you see this, it should be a sort of red alert on your radar. Why can't we use something that already exists? I'm sure there's some kind of open source module that helps us. Do this at least look into it, and it comes from this policy that the problems were trying to solve a unique.

A

What we do is special. Nobody else is doing what we do and it's wrong most of the time, almost all the time. Somebody else try to solve this problem before you and they probably made an open source module out of it. Oh there's exceptions edge cases, but most of the time, somebody else did it before you and they might even be smarter than you.

A

So when you hear a co-worker that says, let's build a distributed system to solve this, that's what we need. Your first reaction should typically be no, probably not some variations of this. If you want to deliver value to your customers or users, is don't try to build automatic failover systems, don't try to implement leadership, election systems, don't invent your own encryption scheme and the open source world. It probably looks more like this. Don't try to build your own JSON parser. Don't try to build a new JSON database.

A

If you want to live our value to your users, if you're just goofing out having fun, then by all means to whatever you want, if you're trying to ship more cool stuff to users and if you're, trying to sleep more and not get interrupted in the middle of the night, you should probably look into something that already exists and keep things simple.

A

So, in summary, you should practice shipping the smallest possible release. That makes sense. You should strive to get code into production quickly. One of the tools you can use this feature flags and then try to keep things as simple as possible. Don't reinvent the wheel, that's it! My name is Ron Coan I'm from uh Pete. Thank you.