Cloud Native Computing Foundation PrometheusDay NA 2022, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Film Premiere - Inside Prometheus: An Open Source System That Changed Technology

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Film Premiere - Inside Prometheus: An Open Source System That Changed Technology

Witness the journey from inception to adoption of the technical triumphs that changed an entire industry’s approach to monitoring systems as we know it today. This dynamic documentary film takes viewers into the world of Prometheus and provides unique access to how it became a game-changing open source technology celebrated by hundreds of thousands of developers and companies across the globe. In the film are some of the most prominent pioneers, executives and engineers from the Prometheus community to provide a comprehensive and thought-provoking view into the origins and meteoric rise of one of the highest-velocity open source projects in the world.

A

Julius interview take one.

B

More and more of like normal companies uh went into a state where they had Fairly complex distributed systems. They were severely lacking. Troubleshooting skills.

C

Teams that used to have to deal.

C

You have these groups, people are running everything's manual and, of course, that means it breaks a lot. We.

A

Have more things to observe and more complex things to observe you.

D

Have so many more machines, you have so many moving bits and pieces. You.

E

Ended up getting woken up to be told that one of your 200 servers is feeling slightly unwell. That was very, very annoying.

D

Or there was no system at this time in the open source space which which could handle this.

D

F

Either too simple and not powerful enough too powerful, but not scalable or or too complicated for cloud native, the.

G

World was at the brink of becoming very, very fragmented.

C

So the market was really ready for something that would be a focused project on a monitoring solution.

H

At the time in 2012, I was still working at Google in Zurich in Berlin, SoundCloud was just emerging as this really cool new startup in the music scene.

H

Having the uh the aspiration of becoming the YouTube of sound had really great Engineers as well. um You know good Dynamic culture and just seemed like a fun place to work.

H

I eventually made that jump to Berlin and joined at SoundCloud. Another ex-googler Matt, proud, had joined at the same time, and we noticed that SoundCloud had a bunch of issues when it came to reliability, latency of the website and all that, and we were having trouble really figuring out, even where these problems were coming from foreign.

E

At the time could not keep up with instances coming and going all the time so.

H

We had hundreds of microservices and thousands of processes running on that cluster. Already changing hosts changing ports every day when developers rolled out a new revision.

B

Even finding out, if you're up or down, if you're under 100 servers, then there's always a server down. It's usually not bad, because your distributed system is designed for that. But you have to find out that it is broken, and once you know that now you have to find out what is broken and yeah. There was just no chance.

H

Soundcloud at the time already had built their own in-house cluster scheduler system, so this was before Docker even existed and then far be before kubernetes existed, so it was a very modern, dynamic and always changing Computing environment already, but it was very hard with the traditional tools to get insight into what is really happening in that cluster. We.

E

Were using statisty and graphite for metrics, but the integration between the two wasn't very good, so we could not really use metrics for alerting.

H

Graphite database was not really Built, For This highly Dynamic use case. So when there was an issue like a latency, Spike or Spike of Errors, we were unable to properly pinpoint it down to a specific process, or you know not knowing whether it is a specific process or the entire. You know group of microservices.

E

People had started catching on to metrics as a base of monitoring, but there wasn't a good integrated system that could work with this. The.

B

The monitoring wasn't ready for that right, so the technology was there. It was pretty immature in.

H

A way, but it worked in this more Dynamic world, you needed a monitoring system that first of all integrates the world of alerting and time series and then also have a more flexible data model that allows more dynamicism, and that has you know, on top of it, a query language that allows, for you know flexible dashboarding, but also powerful and flexible, alerting at the same time, so tying together. The time series and the alerting use cases into one system.

C

Google was one of the Front Runners of the cloud native movement right because they saw web traffic scale at a level. None of us saw at first, and so they had to scramble and really figure it out. Bergman was one example of them trying to wrestle with metrics data with time series, databases.

H

Matt, proud and myself had just both come from Google and we were used to this one monitoring system called borgmon at Google, which was used to monitor all kinds of different services and infrastructure at Google in a very flexible way, and we noticed that when we came to SoundCloud that SoundCloud already had a very Dynamic cluster scheduler internally. So at the time we looked at the different available open source monitoring systems and we were a little bit disappointed. They were not really great at tracking metrics in detail, especially in a constantly changing world.

B

There was a really really strong need to do something about it.

H

So, as kind of a hobby project, initially we just sat down in our free time and started building what eventually became Prometheus.

H

It was a very big, ambitious project. We were trying to build a complete new monitoring system and Paradigm kind of from scratch. So not only did we have to explain it properly to people who had never used such a system, but also build it. So I did initially have a lot of doubts. Whether this project would be worth it and whether it would succeed. We were not hired to build a monitoring system. We were hired to make SoundCloud more reliable, SoundCloud.

E

Is not a monitoring company SoundCloud has a relatively small engineering team. We could only devote so many resources to developing something like Prometheus.

H

But we did draw out the little dependency map of what we actually needed to do our job and we always ended up getting back to that. First, we need good visibility into what is happening in the systems. Otherwise, we cannot improve anything.

H

A year in that decided to leave SoundCloud and go back to Google and then I was alone on Prometheus for a while I'm just pushing it Forward. Gradually new people joined the project as well, so, for example, uh Bjorn rubberstein also left Google and I had known him already and he got interested in what I was working on. So when he joined SoundCloud, he also started helping out with Prometheus pretty early on.

B

If I had been like the manager of Julius and Matt back, then I would probably not have approved the project. I guess because it's just it sounds like crazy and it probably was crazy and we might just have been lucky that it all worked out.

H

The building part had so many different components, so we had to build a proper time series database. We had to build a querying system. On top of that, we had to build client libraries for people to actually get metrics out of their different processes and services and hosts. So it took a lot of faith initially both from us and for some people who supported us to believe that this you know at some point could actually start coming together and start really making a difference where this project would be worth. It.

E

The team that was building Prometheus started using it right away and very intensely.

H

We got the Prometheus server to a point where it started working pretty properly.

H

We got the internal cluster scheduler instrumented with metrics, so that people could see the CPU usage memory usage and so on of all the different processes in microservices, so giving them really the first time really good insight into what is actually happening, and we built a dashboard Builder back then called prom Dash, because grafana didn't exist yet, but you know I would say maybe one or one and a half years in we had enough of the things that really made Prometheus worth it coming together.

E

Adopting Prometheus internally was a gradual process. We.

H

Needed some internal Alpha testers who were ready for a very raw experience, so we found a couple of teams that were willing to experiment with it and use it to monitor their own services, and they also gave us great feedback on what they actually needed and the pain points they still saw with the system.

H

Then, once we had that prototype together, we were optimistic enough that we could actually, you know, introduce this at SoundCloud and make it work properly eventually, but.

E

We really only started pushing everyone to migrate when they started kicking over the graphite server again and again, while trying to record all the metrics and all the details that our old monitoring stack could not handle and we knew Prometheus could handle.

H

It I think everyone realized that this monitoring system was sufficiently different in every aspect to what came before that it made sense to actually you know, have this completely new system and that it also works much better with a new kind of dynamic, clustered Computing world that we were going.

B

Into we thought this is very helpful for what we are doing here, and it would probably be good if others use that, but we had no expectation that anyone except props, if you were ex-googlers and then very few, very nerdy people- might actually realize what you're doing here. So we had no expectation that this would change how the world is doing. Monitoring.

E

Before we went out there and publicly announced Prometheus, it was already open source.

H

We already threw it on GitHub on Day Zero, put it under an open source license and just made it available to the world and.

E

There were a few people at a few companies who had already started contributing and had started using Prometheus in their own company in production.

A

I, as personally wanted to go more, you know into into into this open source Community than just you know filing that issue and complaining right, I, really we were really heavy users of Prometheus. So why not helping the project and jump in and really steer that into the more reliable future right, I.

C

Have personally seen that projects that emerge from end users tend to do really really well.

H

So when we finally published Prometheus to the world, we wrote a blog post from the SoundCloud side, but also from box ever, which was another company that was already using Prometheus outside of SoundCloud. Two companies at the same time were releasing their blog posts, saying like hey.

H

We are using this monitoring system and it's not only being used at SoundCloud at the moment, one and a half weeks after our official announcement of Prometheus someone who we still don't know to this day, who actually is managed to get Prometheus onto Hacker News on rank one for a whole day and that really turned on the lights for the project. We saw that really in our GitHub stars and our chat channels and our mailing lists in the GitHub contributions. uh Just everything started: shooting through the roof from that day, on.

D

Yeah I looked at Prometheus I did a test run and within literally hours, I fell in love with with this system in.

C

Came this really focused really strong project and it just swept the crowd away.

I

It solves a real pain point for the industry. There are a lot of monitoring products out there, but only Prometheus was available as a free tool that really grappled with the challenges of dynamic, containerized, Cloud native applications. In the early days, Prometheus.

G

Made it much easier to add metrics to our project. It.

F

Has a simple user interface, a single binary and then ease of deployments, and this this is, uh you know, changed the entire way in which we approach consume and develop with our Monitoring Solutions.

D

For the first time in my life, I was able to to have monitoring data. I could actually do math on this data.

H

People said yeah, this really changed the way we monitor things and even the way we deploy our services.

G

Yeah the parameters, Exposition format is unique in a way because it is first of all human readable. So without relying on any monitoring backhand, you can take a look at the output and read it. It's an introspection and troubleshooting tool of itself also.

A

The pool model it makes things much more reliable because suddenly those collections are telling you much more and it provides additional context. Additional information. Try it on opposite. Try ask those applications to push data to your monitoring solution. Now things are very, very tricky. You have to think about buffering rate, limiting service, Discovery and lots of complex stuff that, ideally you don't need to worry about it.

G

Gave people a different level of flexibility and I think monitoring and troubleshooting is in a better place because I'll, probably just Exposition format,.

H

There was a lot of still ongoing growth, but also a lot of work, so we just put more and more work into making Prometheus more stable, more integrated with different systems, eventually having a 1.0, stable version and then a 2.0 as.

E

Prometheus gained adoption more services were instrumented with Prometheus metrics.

A

You know you can take any really open source solution, maybe kubernetes maybe MySQL, maybe you know mongodb and you will have Prometheus instrumentation for free all things together. The adoption grown because of Simplicity here they.

C

Did it just at the time when Cloud native was proliferating was becoming like? Was the new cool kid on the Block, and suddenly your old monitoring, Technologies or techniques such as APM were falling short, a.

H

Lot of companies started seeing the value in Prometheus and started using it. You know both in startups and in larger companies, so.

F

I became a end user and familiar with Prometheus at Uber, where we built a monitoring platform with Prometheus as a core that allowed us to build control mechanisms and end user configuration stories on top of that platform and focus on what really mattered to Uber.

B

So one weird thing was when the kubernetes folks noticed us the.

H

Kubernetes people noticed that it was similar to the monitoring systems they were used to at Google, so they chose to support Prometheus style metrics in all the different kubernetes cluster components that led to a world where you know kubernetes supported Prometheus very well, but at the same time Prometheus also supported native service Discovery and data collection on kubernetes. In the end that led to Great Mutual support between the two systems and when you were using kubernetes, you were probably also using Prometheus in some fashion within.

D

Google they had Borg in the late 90s early North and they needed something to to be able to monitor all this data and they created borkman for it. And this pairing is replicated in the open source scene between kubernetes and Prometheus.

B

If you want to claim there is a conspiracy, it really looks like it like Prometheus kubernetes are both like 10 letter words they're both of Greek origin like kubernetes, is aggregate abbreviated, k8s Kate's, you could abbreviate Prometheus, p8s or Pace or something I, don't know their logos that were they're all done right, like this blue and this orange is like perfect, complementary colors. It really looked like these systems were designed for each other and Matthias sometimes calls it like twin separated at Birth, but yeah. It felt really like okay.

B

Finally, they get together, they always Were Meant to come together. It.

E

Filled a whole in kubernetes, and it filled it in a way that was very familiar to a lot of people both in the kubernetes project, but also to a lot of people in the industry who had used this model before I.

D

Strongly believe that without kubernetes Prometheus wouldn't be as large and vice versa, because without Prometheus kubernetes you wouldn't be able to to observe all the stuff which is going on within kubernetes or through kubernetes. So.

C

Prometheus came in because observability monitoring becomes that much more important when your systems are Loosely. Coupled teams are working on small pieces without talking to other teams, so you really need observability at that time. So, hence Prometheus was a great fit, and today it is one of the pillars of cloud native.

H

We really were looking for kind of a neutral home for the Prometheus project. The cloud native Computing Foundation was just being formed at that time, and the only project that was in there so far was kubernetes.

I

So when I was talking to the Prometheus team about working with weaveworks and weaveworks adopting their product as part of our solution, I was also asking them. What were their thoughts about a foundation for the future?

I

I met someone called Craig mcclucky from Google who was working on the kubernetes project and Craig told me that he would like kubernetes to be in a foundation so that it could be an industry asset and not just a company asset. We talked about how the cncf could come into existence. That's why I said to Craig I think Prometheus would be a wonderful project for the cncf's early days and we agreed that would be one of the projects that we would try and recruit when the cncf launched.

E

By handing off Prometheus to cncf, we allowed the project to grow beyond the needs and the capabilities of SoundCloud Prometheus.

C

Is truly a special project, it has had a very strong velocity over the years and they joined cncf in 2016 and actually I, remember being in the observability space. I was working on distributed tracing. We saw Prometheus joined, CNC, wow, okay, we should try to join in too so it was really a like a stage Setter. The.

I

Idea was to continue with Prometheus autonomy and its own roadmap, but support it with a more broad umbrella provided by the cncf community. Marketing and other support.

C

We are grateful to have this project with such talented technologists in it with so much usage across the world to be part of our ecosystem. People.

H

Now saw that the project was really independent had a proper governance. It was associated with the other projects in that Foundation right kubernetes, especially the cncf, also just helped us in a lot of different ways, supporting our events like prom, corn or Prometheus days, but then also helping us with PR, helping us with Cloud infrastructure funding and just giving us advice along the way. As.

C

An open source project Prometheus has welcomed all kinds of folks to come and enhance it, and that I think brings more people more ideas and therefore a higher quality product. Prometheus.

F

Really defined a standard set of ways of collecting and consuming telemetry later with metrics and monitoring, and that meant that people could build on top of it and focus on their core monitoring problems.

I

All the time people talk about standards, standards are used in two different ways. Sometimes people mean standards like TCP, where everybody uses the same way to connect up a network. Another kind of standard is what we call a standard tool or a de facto standard. It means commonly used popular tools that everyone should understand. Once they've reached a certain level.

D

If you look at it today, if you look at it already a few years ago, Prometheus is the de facto standard for metric based monitoring.

G

Think about it. Hundreds of projects today support this Exposition format and, like each time, I see a new project in the cloud native space, especially they have some parameters. Exposition format, support, Prometheus.

C

Has retained a narrow focus, it does monitoring. It is a Time series database with a querying language, and it does those things really well doesn't spread itself too thin. I personally also think that a lot of the original, the ogs are still deeply involved. That is just awesome. It's it inspires many to join in.

C

It brings this continuity to the project and I see their passion when I see them at kubecon, Cloud native cons or at the Prometheus Advance prom cons and those folks care just as much today as they did five years ago seven years ago, and that's just one.

B

I was surprised that it became so popular so quickly. I enjoyed the work so much here with my colleagues at SoundCloud. It didn't even feel like a challenge. I.

D

Had made a decision to to roll this out for all the places where, where I would ever go, because it's just so much better than anything which was there before for.

C

All the people out there who are considering what should I build. What would what would resonate? There are other parts of the cloud native application development process that are not commodified that are not de facto standardized. Look at all those things and decide where you have the strongest inclination to build a successful.

G

Project there are projects where it's trying to achieve everything, but it ends up achieving nothing because there's so much fragmentation, I think Prometheus community and the core kept it very straightforward, kept it very simple, but yet achieved the build that, like really big, you know, ecosystem around the project. We.

H

Are still actively developing Prometheus just making it better all the time you know making the database more efficient, improving certain parts of the UI making it integrate better with other infrastructure components and service, Discovery mechanisms, and so on.

D

A personal lesson I took from Prometheus is that even if you have the best tech in the market in the ecosystem and I'm, obviously biased here, but I strongly believe that this is true. Even if you have the best attack, it's not enough. There are so many other aspects to success and it needs a complete mix of not just the technology. It also needs adoption drivers. It needs public speaking, it needs a healthy Community.

D

It meets other projects which you can align with and work together with it's never about one single person or one single project. It's it's an ecosystem of communities.

J

J