Continuous Delivery Foundation cdCon 2022, 24 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: The Cost of Agility at Scale: It's All Fun and Games Until the Bill Comes - Tim Banks

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Keynote: The Cost of Agility at Scale: It's All Fun and Games Until the Bill Comes - Tim Banks, Principal Cloud Economist, The Duckbill Group

Cloud-native technlogies have enabled engineering teams to develop and deploy at lightning speeds. But without careful consideration of the economic impact of their architectural decisions, at scale, these decisions can cause more financial problems than they need to. We will talk about some common practices and architectures that become economically inefficient at scale, some better alternatives, and how to determine when it's the right time to reexamine your architecture.

A

Hey everybody, I'm tim banks, I uh was my plan- was to come up here and start out with like a nice, sia song and then do some karaoke. But uh when they booked me they said they wouldn't. Let me do karaoke and actually do a talk. So I decided to do a talk on something I might know a little about.

A

So we're going to talk about jiu jitsu, because no I'm tim banks, I'm principal cloud economist at the duck bill group. I have a long background in tech, starting with like back in the days when your infrastructure's code was a zip file, that you would just ftp from server to server and deploy going from operations, system, administration systems, architecture- and you know a lot of the grunt work. What people used to call cool stuff that was useful and exciting right, great stuff.

A

A

Now I'm an engineering consultant doing like cost optimization, so you know you start one place, you end up somewhere else and it's kind of fun, but uh one of the things that a lot of things I've been working on have had a lot to do with cost optimizing companies that have really been pushing to move fast, move fast, move fast, move fast, and then we kind of figure out what happens when the bill comes.

A

So we'll start here. So as you see it's a nice car show, we got some great cars looking there. uh Everything is all fancy. There's classics, there's antiques, there's fast car, like everyone knows, every single little thing that's been done to all these cars.

A

They can tell you, like all the specs, how fast everything's going everything is neatly arranged like everything is just looking nice right, and this is what it like when you see like a proud engineer, talking about something that they've built or engineering managers are showing off their architecture or someone, demos, a product or maybe someone's begging you for funding right, but you're, going to see a great great kind of display of everything that looks nice with all the hoods up and everything under there. Looking very very nice.

A

So like what are we talking about here at cdcon right, writing fast, deploying fast failing fast? That's what the conference is all about: right, continuous, just just move and so moving forward and getting everything done right. You know, because, after all, anybody that tells you that aren't running mvps in production, we're going to need to give them a wide berth as they turn around, so they don't hit you with their nose.

A

And so you know some engineering organizations they really pride themselves on their fast sprints lofty road maps. You know these modern architectures with, like the latest serverless shiny event, driven web 3 server farm to database, table aiml, blockchain iot on the edge 5g ultrawide 4 barrel, exhaust all-wheel single origin, api driven small batch locally, open sourced a5 grade black box, managed services that would make any devrel person salivate at the mere possibility of a case study.

A

But what happens when that is our focus like yeah, it's cool, it's hip. It makes for great content and accolades on the bird site and hacker news, but.

A

How does it end up? You look at the bill and it looks like this.

A

Well, you know everyone's got their different dysfunctions in their organizations. We all know that. That's a known thing: everybody has it. No one can say that they don't right, but there's a few common problems that plague velocity focused organizations that I've either been a part of right or I've been brought in to consult for so let's talk about some of these.

A

Who's there, who, if anyone's ever, worked in the kitchen at the end of the night and sees that you know your heart breaks a little bit right, who's ever cooked a big meal right when everyone was raved over and it was so good and you're like now you're in your third hour, the meat sweats and you're, like you, walk into the kitchen and see that nobody likes that it's a horrible feeling- and this is what it's like when you push your code to production, you pat yourself on the back and you head to the next ticket, without cleaning up your dev artifacts right or any of your running resources.

A

Now I get it it's not always that easy, there's a lot of organizational overlap and occasionally accompanying dysfunction. That makes the mechanics of spinning things down harrowing. Sometimes.

A

Not checking the fine print. What's the pricing for your managed services? Are you charged by the cp hour? You charge my number of invocations by bandwidth, all of them, none of them some now it's like you know. What do you mean that this wasn't covered by my reservation right? We had a customer that we dealt with not too long ago at the duckbill group, where they had started off with lambda invocations, and it really worked for small scale.

A

Everything was running fine, it worked, and then people started for some reason using their product right and then, as they somehow started, to use their product and became god forbid popular. They didn't change their architecture. They just kept scaling lambda right because that's what it's there for it just scales, you don't have to worry about it. Well, you didn't have to worry about it until the bill came and they're like wow. This is really expensive and they got charged for somewhere in the vicinity of.

A

I think it was 107 cpu years in a month from lambda which wasn't bad enough, but they also got charged for number of invocations.

A

Now that may seem like just like the cost of scaling, but we come in and we say like oh well, maybe if you would just run. Oh, I don't know four instances. This would have covered all that and it would have saved you, like 90 of your bill.

A

It's really important to understand. You know what cost what and why.

A

The surest way to fund size, your margins, is to have a data science team. Your data, warehouse and analytics will molotov cocktail your margins and roast your unit economics over the embers, but we want to run big data analytics right. That's what we do now in the year of our lord 2022, certainly, but if you're not giving a lot of thought to how you do this and just throwing everything in the data pile, it's it's easy to do.

A

Right, object, storage is so cheap, it's so cheap and there's a great way to find out why the product team at snowflake is giving the porsche of the month club subscriptions out as part of the company holiday bonus.

A

Now, let's imagine it's the end of the month right.

A

You just now found out that one of your production notes fell over and you had to add a new one or at the end of the month, and you just now found out that something ran out of disk space. You need to expand the volume right.

A

That would be ludicrous. No one would accept that right. No one would simply say, like oh 30 days, that's enough time to know whether something fell over right but organization after organization make their only interaction with their economic observability at the end of the month, when it's time to contribute to your cloud, provider's puffer, vest jacket, swag fund.

A

Okay, all right, like I said, I've had a rather inglorious career of doing all kinds of different things, but why am I up here telling you about this right? I'm not really that smart. I just really like to get paid routinely, and I can't do that if I'm giving all my money to kelsey hightower, but there are a few things that we see in common in our in our in my line of work that may help you out when the bill gives you sticker shock.

A

Observability is the key right. You need to know what your infrastructure are. Infrastructure costs are as close to real time as possible. You need to tag your resources consistently check your bills check your billing dashboards, find your trends, find your seasonality, define a new to economics, get your cost revenue goals and be mindful of these metrics in your sprints and your deployments.

A

After all, how can you know what the projected costs are for your changes? If you don't have these targets already defined you're, not if you're not gathering data on it, you can't make projected costs and forecasts.

A

Projected costs right, you should be able, you should be making a cost prediction for what your changes are as part of your testing and approval process for any new features or new applications.

A

If there's a significant delta between your projection and the observed cost, you may want to look into that. You may need to go back to the drawing board and figure out why that is you probably want to correct that before you go into production now? This should help reduce the sticker shock for your new feature and make your finance team less likely to be drunk at noon.

A

Now the ease of managed services may not be worth the cost at scale right. You need to make sure you know what the point, what that point is where it becomes maybe not worth the squeeze in the juice, and you need to know how to switch to something different.

A

Now, this kind of broaches the larger point of not painting yourself into the corner when it comes to your architecture, one soap box I always stand on is that even if you're small, even if it's your mvp, you still need to think about what it's going to look like when this scales up to a number of customers and if you're, not thinking about that you're, going to make decisions that work now, but will hamstring you later so this can mean I mean you need to have some agnosticity in your architecture.

A

This can mean you have compute type agnosticity. You can have instance type agnosticity. You can have storage type agnosticity, you can have you know wherever your state lives can be agnostic, it can be local, it can be somewhere else. You can manage service here manage service there. If you are making decisions that prevent you from changing earlier or changing down the road, you are amassing technical debt almost as quickly as you're amassing actual debt.

A

Let's understand your data life cycles, and this is a big one right, throwing a bunch of data into object, storage and not really thinking about it can be expensive not as expensive as like putting in object, storage and then willy-nilly, willy-nilly and then trying to run analytics on it. Storage is cheap, it's really cheap. The compute is not remember that the less work that you do on your data when it's emitted means the more work you're going to have to do. When you do analytics, you are essentially aggregating compute and that can get pricey.

A

Clean up after yourself now I've got. I have five children. This is literally the thing I say the most anything of anything in my life is clean up after yourself, and it sounds so cliche and it gets missed a lot right.

A

Tear down your dev when you push to production, tear down staging if you're, not in testing spin down your build stack, is etl job done great expire, the artifact move it to a lower to a lower source here in an ideal world, when a branch is deleted, all the dev resources associated with that would also get deleted. Anything that's tagged like that. Will just go away right when something is pushed in production after something is pushed to.

A

Production staging should get automatically torn down and then rebuilt based on production when it's time to deploy please, for heaven's sakes, do not keep storage volumes around in perpetuity. Like those plastic bags, you were going to recycle back in 2013 that are still by your front door.

A

Have periodic architecture reviews see how your projections are comparing to what your actual bill is right, do cost optimization sprints make sure your tagging is consistent. I've said that a few times, but I cannot stress it enough, make sure your tagging is consistent.

A

If it's not go back and fix them right, you have time make time to do that, sweep through and turn off old things. Don't just leave them running in perpetuity.

A

Now, finally, none of this is rocket scientist because I'm not a rocket scientist, just a costly, old ops guy that got good at saving folks money, and you can typically realize some deep, decent savings when you, by taking some time to look at why things cost what they cost a time when the economy is shaky.

A

Taking these steps can make the difference between humming along smoothly and thriving or maybe having to send a low-key posturing, email threatening to fire, everybody that doesn't come into the office.

A

So in closing, I've been tim banks, principal cloud economist at the duck bill group, and I would just like to say to everyone here: it's in honor of happy pride month, red ted, cruz, black lives matter and trans lives. Do too.