Kong Destination: Automation, 19 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DevOps Is More Than Configuration Management

Description

In this session recording, Rick Sherman summarizes his DevOps evolution from a high level. For most of Rick's software engineering career, his experience with DevOps was all about configuration management. But after many years of experience with some key mentors, he began to see DevOps as so much more.

Read the blog version here: https://bit.ly/3xYMLUm

Check out all recordings from the Destination: Automation 2021 event: https://bit.ly/3AwzL9X

#DevOps #Observability #Config

A

A

Cool,, so I'm going to go ahead and kick things off because I have got at twenty five.

A

Minutes worth of content, and so my name is rick sherman.

A

I'm a technical evangelist at datadog.

A

and today I'm going to be talking about my journey through devops and how my perception.

A

Of devops has changed over the years.

A

Before I jump in there and to give you literally a 30 second intro to who I work for,, I.

A

Work for data dog or a monitoring and analytics platform.

A

We provide full visibility across your organization and that spans end to end, and that's.

A

From your infrastructure and network, to your applications and services, all the way to.

A

Your end, users., and so the idea for us is to enable everyone,, ops, devs, security,, even.

A

Business folks,, so they can have a shared understanding of your systems,. Your.

A

Applications and how your business looks from a technology perspective.

A

But strangely enough, I get to give a talk about myself and honestly, my journey,. What I.

A

Learned along the way,, what I wish I knew along the way.

A

There are going to be some details that kind of feel like a resume.

A

Please don't get hung up on them., I'm literally just including them there for context and to jog my own memory.

A

So I'm going to start this particular story about 15 years ago, when I started at juniper.

A

Networks in professional services.

A

and so juniper builds networking equipment for the largest internet and telephone.

A

Providers of the world.- and I was really focused on the software products at juniper.

A

And one of those things that was the thing that was my focus was policy engine and.

A

Identity engines and configuring all of the user related things on the networking.

A

Equipment.- and this is what turned me onto a ton of foundational technology.

A

I had done some windows and citrix and things like that before,, but this was the big.

A

Leagues for me. this was clustering..

A

This was high availability.. This was databases and ldap and distributed architectures.

A

Right., look at this little bit of screenshot here.

A

and we actually had individual components that did particular system services.

A

We spread them out and scaled them and did all this interesting stuff that you might be.

A

Doing today, in other platforms.

A

But this was my introduction to all of this.

A

Juniper also had at that time some proprietary ish,, you know, programming, language called.

A

Yunel script., it was really about manipulating xml.

A

So xslt, some weird programmatic things with the statements and all of this stuff.

A

But it was really obscure and there was only a handful of us in the world that were.

A

Really even utilizing it.

A

But because we had this language and because I had a background in this distributed.

A

Architecture, we had to manipulate some of our products to fit customer needs.

A

And so we had a large tier one internet provider,. You know who they are,, but I can't say.

A

Who they are. and once a year, we would upgrade all of their equipment. Globally.

A

And so they had manual procedures in the hundreds of what it took to upgrade each piece of.

A

Equipment.: here's how you check the before state.: here's, how you make a change.! Here's, how you validated., here's how you roll it back.! We automated that as a small team,, we went through hundreds of pages of documentation,.

A

Manual, steps and converted them into an automated workflow.

A

And we sustain that over a couple of years.

A

and so this was, again, my introduction into automation, the concept of automation.

A

And so my time, professional services was really core to who I became as an engineer.

A

Throughout the rest of my career. introduce me those foundational technologies., I was looking at unix.. I was looking at these databases,, but it also taught me about resilience.

A

And that's from both a technology, perspective, building, five nine carrier grade.

A

Solutions, but also personal resiliency.,.

A

Professional services is a really tough job to do because you're traveling in and out of.

A

Different customers with minimal context.

A

you're away from home for weeks on end, and you have to learn how to do that.

A

Also, being new to that space, being new,, that scale of of company, I'd really learn how.

A

To communicate effectively.

A

multinational corporations,, not everybody speaks english as the primary language, taking.

A

Obscure and complex technology and writing it into documentation that everybody could.

A

Understand. and so a lot of those soft skills were also developed at that time as part of.

A

This be kind of the theme. I'll talk about what I learned and what I wish I knew along the way. I took for granted just how important my mentors were to me at that time.

A

I had a couple of folks that really took me under their wing and made sure that I was.

A

Successful., I can ask him a technology question,, regardless of how basic or simple it.

A

Was, they answered it very graciously..

A

They made me feel very supported and encouraged.

A

They helped me with the living on the road, life. They helped me with the professional.

A

Education., all of those things really enabled me to be successful.

A

And I just going back and looking at this, really understand just how important they were.

A

To me.- and so I stayed at juniper,, but I did switch roles,. Eventually, I got off the road.

A

And went into cloud operations and so juniper had acquired a company that had a saas.

A

Platform, that was in a private cloud..

A

We were going to go ahead and move that to the public cloud., and so this case, it was a move to aws.

A

And we were an operational team,, so we architected this new deployment,, but we were also.

A

Responsible for maintaining it and operating it over time.

A

And so aws at that time, didn't look like any of aws today, right.

A

There was a console,, but it didn't have as much. Functionality is as the api.

A

That was apis,, but it didn't have as much functionality. You have today.

A

And this actually turned me on to a tool from netflix called asgard.

A

And so at that time, we were needing to do auto scaling groups.

A

we needed to do bluegreen deployments.. We needed to scale out our applications.

A

That wasn't available in the console at that time.

A

and we didn't want to write our own custom software to deal with those apis.

A

And so this introduced me to this concept of open source software that we could take and.

A

Use and manipulate and contribute back to to a community.

A

And we did all of those things.. This is what really kind of opened my eyes from a very hardware. Centric proprietary.

A

Vendor to open source software to open source community to solve these, these problems.

A

We also did some monitoring at the time.

A

again,. We were responsible for the operations of the software platform,, but this was our.

A

And cacti and monit. we weren't really making scaling decisions. Programmatically.

A

We would take a look at the stuff over time, gut check it and figure out how we were.

A

Doing down the road.

A

But also part of that role for me was looking at configuration, management.

A

So we were deploying these software stacks into the cloud and we wanted to make sure that.

A

The configuration was as expected and we could change it at scale over time.

A

So then, I got to spend all this time researching these different configuration management.

A

Tools that were available at that time. so kind of the big four of that that period,, they all have different logos back then.

A

But that was now my introduction into dev ops,. This configuration management tool.

A

Ultimately,, we wound up going with puppet., and so I got to write a bunch of puppet code to manage all of this infrastructure.

A

And so this was now my introduction into public cloud, and this was my introduction into.

A

Dev ops., I didn't read the phoenix project.

A

I wasn't going to ghent..

A

We were a group of people that had a problem that we needed to solve, found tools that.

A

Supported and help solve that problem.

A

and that was it. so for me, devops was configuration management.

A

That's what I needed to do., but there was lots of other different things. I'd learned in there..

A

You see logging and time series metrics that might come in handy later.,.

A

But again, to kind of retrospectively look at what was happening in my mind at that time,.

A

Co-Worker of mine over coffee in the morning was trying to convince me that docker was.

A

This really awesome thing., and I thought it was terrible.. It was real early on alpha.. It only ran on like one version of ubuntu..

A

It didn't do anything that I needed to do..

A

I loved vms., I loved hardware..

A

This docker container thing is a fad.

A

It's going to die on the vine.. I think I think I was wrong about that.

A

But this is a good opportunity for me to introduce the devops lifecycle.

A

You might have seen this graphic.

A

if you haven't, you're, going to be really familiar with it by the end of the talk, because.

A

It's going to pop up a lot., but this was my view on devops..

A

If you were to ask me what devops was at this time,, it was my operational centric view.

A

I understood that there was devs and there was some of this stuff happening.

A

But for me, this was it.. I was in charge of deploying infrastructure, deploying applications and making sure they.

A

Worked. end of end of the road., that was it for me., but I did change roles one more time at juniper to a newly created.

A

Team called automation, engineering.

A

And our charter there was to make programmability available to the masses, juniper had a.

A

Machine interface at apis,, but we didn't really have a language that folks were.

A

Comfortable using., and so we started writing abstraction frameworks in python and ruby to.

A

Make that accessible to a broader community.

A

And so when I joined that team, there was a gap.

A

So some of that work had already been underway,, but none of it was being tested.

A

And so we were going to be delivering the software again to a large service providers, to.

A

Large enterprises., we had to have a high degree of confidence that this stuff was going to work and.

A

Ultimately want it becoming part of the core operating system., and so I took on testing.

A

I hadn't really done a lot of testing,, so I learned about unit testing and functional.

A

Testing and mocking, and what is code coverage look like and and how do we put this on the.

A

Github repos that everybody knows that we're doing the right, things.

A

And then we also we're starting to now use this language as these abstraction frameworks.

A

Matured. I got to go back to my old friends configuration management tools and figure out.

A

How to write modules in their language that would interact with our devices.

A

So I wrote some puppet modules, spent a ton of time with ansible as the company, as the.

A

Product is,, as the modules sometime with saltstack to work on new remote minions.

A

So this was the very infancy of net devops crossing over with more of a traditional.

A

Configuration management tool.

A

And so automation engine was great..

A

It introduced me a lot to testing..

A

I really got to dig into it.. I got to spend time with ci/cd..

A

We did it as an open source project, and so we didn't actually have a product manager.

A

So we got to play product manager,, take different issues and from github or different.

A

Customers or different sales teams, build roadmaps,, build the features, put them out.

A

There. this was, this was really a new set of skills that I hadn't really been exposed to.

A

Prior. but again,, we were really focused on a few use. Cases.

A

We were focused on our strengths at the time.

A

So it was really about configuration management..

A

We had some operational capabilities,, but we were pretty focused on what we were doing.

A

And so, again,, I wish I had the ability to spend more time with folks with different use.

A

Cases or more of the sysadmins of the world to try to figure out what they were doing.

A

But that's where we were at that, given time.

A

And so now this dev ops lifecycle for me, flipped.

A

I was no longer on an operations team..

A

I was no longer on call.. I wasn't going to get paged., but we were building the software that other folks are going to use.

A

We were making sure that it worked. we were making sure it was released out to them, how we were going to package it,. All of.

A

Those things. so now devops for me.

A

Kind of flipped,, but I wasn't really thinking about it as devops, right,. I was just.

A

Thinking about it more of this traditional engineering team that I was more familiar.

A

With. but I like the puppet so much that I decided to go work there for a handful of.

A

Years. and so I went there to focus on network automation.

A

And so I joined a puppet in the engineering team as part of release. Engineering.

A

And I got exposed to a whole depth of of software building and delivery that I had not.

A

Been previously exposed to., so software build, tool chains, working with compilers and ce and all of this low level.

A

Stuff, building virtual machine pools.

A

I was looking at how to build the puppet agent,, because the software we would put on these.

A

Networking devices with their weird closed operating, systems.

A

How could we actually run that in our virtualization environment?.

A

How could we run that through ci/cd pipelines?, and so this got me into yet another depth of software that I just previously hadn't been.

A

In and also started to show for me, the dev ups was mainstream.

A

Again, I came into it at twenty thirteen,, twenty fourteen and I've been around,, but I was.

A

Only looking at it from a very specific viewpoint.

A

While I was a puppet,, I got to go out to user conferences and speak with folks.

A

We did an annual state of devops report with all sorts of information about all of the.

A

Different tenets of devops and how different companies were progressing and where they were struggling. and most importantly,. I got to work with a lot of recovering sysadmins,.

A

As they would say. so I would get to hear their trials and tribulations, what problems.

A

They were encountering before, how they solved them.

A

I came from a service provider network and this was a totally different group of people.

A

Than I really got to work with. and so that cross pollination of knowledge and experience was was huge.

A

And I really learned a lot about the community and about what our mission was.

A

But there was also a little bit of confirmation, bias.

A

There was a net dev ops survey that floated around in twenty sixteen sponsored by some of.

A

The networking practitioners, not really even the vendors.

A

And at that time they were really focused on configuration related activities.

A

And so I really thought about dev optimal configuration management, standpoint.

A

The networking folks were still thinking about that way..

A

I was obviously on the right track for what we were building that did evolve over over.

A

Time., so you look a few years later, we're now starting to see that more folks were.

A

Looking at operational, related, activities.

A

obviously,, the lion's share was still configuration management,, but the winds of change.

A

Were starting to take hold., and so while I was in engineering a puppet,, I obviously learned a lot about software.

A

Development, building software that other folks are going to contribute to its scale and.

A

That working with that community, figure out the right integrations, how to how to move a.

A

Product forward., I do at that time wish. I had spent a little bit more time with the traditional sysadmin.

A

Group. I was really laser focused on on networking.

A

And so I missed some of maybe the broader use cases that were out there.

A

And so, if I go back to the life cycle,, I'm still mostly just on the side, here.

A

All right, we're building integration's, we're building software, we're getting it.

A

Through the pipeline. we're getting it released and packaged.

A

But I wasn't responsible for it.

A

and operational standpoint,. We weren't deploying this in anger,, as I would say.

A

I wasn't in professional services., we would get feedback from customers,, but I was part of the engineering team.

A

I didn't have a strong view into operations.

A

And so I did pivot at the end of my time, at puppet to alliances,, which is more of like a.

A

Corporate or strategic type of position.- and I was really focused on technology partners., and so what were the integrations that we were going to build?.

A

What was the time frame in which we were going to do so??

A

What was the business outcomes that we wanted from those relationships?.

A

This was a whole new set of information around justification of activities from a.

A

Corporate or a business standpoint and, you know, at the time kind of felt like I was. I.

A

Was leaving the engineering and putting on the suit, so to speak,, which was obviously far.

A

From the truth. I still still got to be myself,, but I got to really learn about planning.

A

And strategy and longer term horizons and the justification of how money was going to be.

A

Spent. and then I also got to spend some time evangelizing.

A

And so obviously that paid off, because here I am today.

A

But even at that point, I saw a lot of our partners were doing things with containers.

A

That were really starting to do a lot of stuff in kubernetes., and I kind of felt like it was: maybe them just doing it, because it was cool.

A

They're also doing a lot of stuff, a block chain, and I felt like it was kind of cool,. So.

A

I might have missed the boat on that one.

A

But so this added another piece to my my devops life cycle.

A

And this was planning, right., looking at the needs of our customers,, the needs of our company and the needs of our.

A

Partners and figuring out what it is that we were going to build moving forward after.

A

Puppet., I did a brief stint at a networking, automation company, and this was where my.

A

Realization about configuration management being the the only thing that mattered really.

A

Started to diminish., and so we were looking at things like source of truth..

A

We were looking at things such as change, advisory, boards,, looking at etl, tools,.

A

Interactive chat ops,, but most importantly,, this business justification problem.

A

And so we had different customers or potential clients come up to us and say, hey,. I am.

A

Really invested in automation., I'm seeing a lot of really great success with the tools that I'm building.

A

But I can't get buy in. either. My coworkers don't feel like they have. Time, management does want to invest.

A

Financially into it., how can I justify my activities?.

A

And it seems interesting to even be posed that question. If you're from a more of a.

A

Traditional sre or or even development, background.

A

But those folks are still pretty siloed.

A

there's a lot of excel spreadsheets floating around with all of their ip addresses and.

A

Cabling and everything., and so they were really looking for help.

A

That's not to say that they were completely stuck in their old ways.

A

You're, looking at the vendors putting out things like streaming, telemetry, grpc base.

A

Configuration. um,, I didn't spend a lot of time with it at that point in my career,, but I.

A

Do wish I had because obviously would come in handy for me. Later.

A

But still, my devops life cycle here was really around planning.

A

What were the activities that we were going to do??

A

How do we justify them?? What was our method of moving forward?.

A

And so in twenty twenty I moved over to datadog,, which is where I'm obviously at today.

A

But I didn't really know anything about observability and I certainly didn't know.

A

Anything about datadog.

A

I had seen datadog running around at conferences and the shirts, scanning badges and.

A

Giving out these really pretty demos., I knew some folks that went over to work at datadog,, but that was it..

A

So I decided that I was going to learn about observability and went about this company.

A

And so in doing so, I started with what I was most familiar with and that was network.

A

Performance monitoring., so I got free trial log in and set it up.

A

And I was looking at that network performance monitoring., so that's cool.

A

I get that.. Then I started looking at some other things. I was familiar with, like system.

A

Level metrics. so monitoring, cpu or disk or memory, or some of those things that honestly.

A

Had to do with with cacti years ago.

A

ok, great, easy., then I turned on logging., so I been familiar with with some competitive tools,, groc, parsing and and shipping logs.

A

But throughout that kind of like interview, process and educational process,, I got to kind.

A

Of see a glimmer of what observability was when I saw them all together.

A

So I had been familiar with most of these things already in the past,, but they were point.

A

Or individual solutions.

A

and I was mentally putting these different things. Together.

A

And so it it obviously worked out for me..

A

I got the job and I got to spend the last year or so really just digging in to.

A

Observability as a concept, datadog as a product.

A

And the thing that kind of surprised me the most is when I got into application.

A

Performance monitoring,, I was like really hooked.

A

I never really considered myself a software developer.

A

I was obviously responsible for some software,, but I never considered it to be my job.

A

It was always like I was an imposter pretending to write software,, but seeing upon.

A

Tracing and distributor tracing and how it all tied together with these different logs.

A

And these different metrics and tying it all together..

A

This was the type of observability that that I wish I would have had all along.

A

And I'm not saying that as a as a sales pitch thing.

A

It's just literally like this concept of observability, of tying all these different.

A

Facets together was really powerful.

A

And I could actually tie it back to previous use cases or previous times or something like.

A

That would have really come in handy for me.

A

and so datadog does a ton of stuff.

A

like I said,. I've really been digging into all the different pieces.

A

The output of that is really me just learning more about the different interactive parts.

A

Of the lifecycle and how we monitor them or how we observe them and how we exchange.

A

Information between those different silos,, because I had spent a lot of different time in.

A

Different silos., and so that was the last piece.

A

That was the final little piece of the lifecycle that I was missing. Was this monitoring.

A

Viewpoint. and so 15 some odd years later,. I finally filled out this entire devops.

A

Lifecycle., my mindset has really drastically changed throughout this evolution.

A

And I'm still learning., I'm still evolving.

A

I do get to use kubernetes now., I don't. I don't think it was a bad idea,, but I haven't really used, like, operators.

A

I'm not running it at scale,, but I'm liking it.

A

I'm liking the networking pieces.

A

I'm liking, some of the serverless stuff I get to play with.

A

So just continuously evolving, continuously learning.

A

But that's my story., that's my evolution.! Everybody else's is different.. You could be days, weeks,, months, years into your career..

A

You can have different different background, different focus.

A

Your story is your own., but I do want to challenge you to kind of think about what devops or what automation.

A

Looks like to you today., and also what is your evolution been as well??

A

Like writing. This talk was really interesting to me because I got to go back and dig up.

A

Old memories and look at old documentation and try to put myself back to that mindset and.

A

Draw parallels and comparison to where I'm at,, both from a personal perspective today and.

A

A technology perspective. seeing distributed architectures are 20 years old and how they look today,, maybe not.

A

Everything is new., maybe some ideas existed in the past.

A

but more importantly,. How can you take your journey and help others along their way?.

A

Right. I had these really amazing mentors early on and they were pivotal to my success.

A

So I would also challenge you to to help those others around. You share the war stories.

A

And kind of move this community forward.

A

And so I have literally gone 15 seconds over., so I'm going to wrap this up.. If you do want to learn a little bit more about observability,, you think any of this.

A

Stuff is really cool.. You can check out our free learning labs at learn. Dot data dog hq.

A

Dot com., you can also sign up for a fourteen day free trial at our website.

A

And if you're in your journey- and you want to learn more beyond just a guy in a purple.

A

Shirt talking,. You can also check out our careers page.