Red Hat OpenShift OpenShift Commons Operator Briefings, 16 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: High Availability PostgreSQL and more on OpenShift Jonathan Katz (Crunchy Data) OpenShift Commons

Description

High Availability PostgreSQL and more on OpenShift
Jonathan Katz (Crunchy Data)
September 16th 2020
OpenShift Commons Operator Hour
OpenShift Commons Briefing

A

Hello and welcome again to another openshift commons briefing today is part of a series of operator hours and um the good folks from crunchy data have long for a long time done great work around the operator pattern, and today we have jonathan katz here with us um to talk about the high availability, postgres um and a whole lot more on openshift and we're going to have him do a an overview of of the topic and hopefully a little bit of a live demo, because we'll put them on the spot and live q a so if you're, wherever you are, whether you're in twitch or youtube or facebook, or here in blue jeans, ask your questions in the chat and we'll relay them to him and have a conversation at the end about you know the future of postgres, sql and all kinds of good things.

A

So jonathan, take it away, um introduce yourself and um let's learn a lot more. Thank you.

B

Thank you thanks diane yeah, so yeah very happy to be here. It's a you know: it's been a while, since uh I've been participating in an openshift commons, you know I'm looking forward to the point where we can all get together again in person. So uh just real quick, a little bit about crunchy data. uh All we do is postgres and all we do is open source. You know very similar to red hat's model. uh You know we started historically focusing on how to securely deploy, deploy.

B

Postgres, which is you know, very important in you, know many enterprise environments and then you know, as you know, containers became a big deal and as kubernetes openshift became a big deal, we focused on how do you deploy postgres in those types of environments, which is a lot? What we're going to talk about today and, like I said you know everything we do is open source. You know everything I'm going to demo today talk about today's open source. You know we believe you know the best postgres solution is you know the upstream solution?

B

So, oh you know everything we try to do you know we. You know we give back to the community in that regard. So yeah, you know feel free to. You know, ask questions throughout you know I I'm very active in the postgres community. Actually I have a slide on that. um I've actually been I've been using postgres for about 15 years. um I've been active in the community for about 10, uh actually a little bit over 10 now mainly focused on advocacy.

B

So, for instance, we have the postgres 13 major release coming out next week and a lot of what I do is you know geared towards you know getting that release out the door. um You know before I joined crunchy data. I was involved in a you know, a few different startups in new york city and, of course, we use postgres as the solution there, but my background's actually an application development, and I think you know some of the things we'll talk about today will reflect that and you know for me.

B

You know it was sort of my journey from application developer to understanding more of the systems administration side and you know bringing that background of you know being always being the accidental systems. Administrator in you know the various organizations I worked for so, uh and you know you know beyond beyond that. um You know. I know I just love the open source ecosystem in general. You know I've been very fortunate in my career I've only you know I've been able to use exclusively open source, so um it's great to you know give back.

B

You know in a variety of ways, so you know you know maybe the back of a second. You know if we were well first off. If we were in a you know a regular uh conference setting. I would normally ask how many people are running postgres and then how many people are running postgres in production, but I also like to take a step back and just talk about. You know what it takes to run a database.

B

You know in an organization now it's one thing to play around with you know a data system on your laptop. Do some cool things on it, which you know, arguably, is what I'm going to be doing in the demo today, but there's you know more considerations when you're trying to run a database system in production. I mean it's very similar to what you might do with openshift. Where you, you know, you have multiple.

B

You know, high availability notes set up to make sure that if you know one goes down you're able to get your workloads over to another until you know the the original node heals um and you know- and you know- and you need to consider this- you know when you're running postgres I mean, or you know, even any database.

B

I mean databases, are you know, foundational applications, they're storing your data, so you need to make sure your data is. There is safe, it's stored securely and you're able to retrieve it and retrieve it efficiently as well.

B

But you know a lot of things can go wrong in production. um You know, for instance, your data might become unavailable. There could be a network outage. There could be. You know a road process, that's you know just tapping all the resources from the system. um You know it. Could you know it could be any combination of things. You know you know there are many cases where you know I would. You know, commit a denial of service on my own production database.

B

So there's the human error as well, so you want to make sure that your database remains as available to your applications as possible, but you know certain, you know, other things can go wrong as well. You know, for instance, let's say I drop the users table. You know that's going to cause a very bad day, so we need to be able to restore the database. You know back to the point where the user's table existed um beyond that. You know you might be.

B

You know using databases in your development environments that you do have one production environment, but you know to develop. You want to be able to bring up databases for your team, so you won't be able to rapidly provision and you know, maybe rapidly destroy.

B

You may also want to be able to create production or production. Like data or scenarios for your developers to work through, for instance, to troubleshoot an issue that is in your production, environment, so being able to clone and copy that data, is you know very important as well? And of course, if you have you know various you know data regulatory concerns you might be able to do that in a way that is safe and secure, you might be able to. You know, anticipate problems before they happen, so you might be able to monitor.

B

You might want to know that, oh I'm running out of disk, you know I need to provision. You know I need to provision more space. I need to resize my you know, pvc or you know or pv um you know. Maybe you know. Maybe things are not performing as quickly as they could be. You want to be able to you, know, troubleshoot and diagnose, which are your slow queries. You know and try to optimize them um and, of course you need to be able to manage.

B

You know you know who's able to access your data production. Typically, it's your applications or you know a database administrator who might come in and troubleshoot things, um and you know you also need to make sure you're securing the connections or are they over tls? And you know you know other all the considerations around that. So there's a lot to consider- um and you know I you know- I still try to layer it that you know there's.

B

You know multiple ways to get there, that you do have your stock postgres system, but you need to bring in an ecosystem around it. Now.

B

Postpress has a lot of these things available in terms of you know, what's built into into the core system- and this is you know- comes through really over 30 years of evolution of postgres post. You know I'd like to joke. Well, it's not really a joke. You know I'm about the same age as postgres. uh It's been around, you know it's been around for a hot minute now um and a lot has you know it's. It has now adapted and changed with the times. um You know.

B

One of the reasons why postgres has become very popular uh through the years is because of its license. You know much like linux, you know, there's no single vendor, um you know it's a very flexible license. No one can actually own postgres and the community has gone to great lengths to make sure you know this is the case because of this you know it. Actually, you know it. It's attracted a very healthy ecosystem of people contributing to postgres and through this and through real world use cases, you know it's. You know.

B

Recent postgres releases have brought it to you know future parity with other proprietary databases.

B

So an example I'll take an example from about 10 years ago, but uh streaming replication now and streaming replication is one of the foundational aspects, for you know: creating a high availability system that you're able to take your data from your primary server, and you know- and you know copy it over. You know in an ordered fashion, to your replica server and in a way, that's you know fast and efficient, so that came about 10 years ago and it you know it was.

B

You know the basic form is, basically, you know as soon as I get the data changes, send it over I've. I've no guarantees if it gets there, but I'm going to try to send it over as fast as I can then came. Synchronous replication, which is geared towards workloads that are you know, require you know that are are right, sensitive, for instance. I really don't want to lose a transaction.

B

I want to make sure it's copied to at least one other place uh that you know that came let's say you know roughly nine years ago, and you know I could actually probably go into a whole sphere about the difference between these two replication modes, their trade-offs, etc.

B

I'll save that for a later slide, or perhaps a question, you know you know, and you know, as opposed because develop you know further things were added. um Suddenly you had cascading replication where you could have a replica of a replica which allows you to you know, push things out. You know. Even you know, you know to you know, you know more distributed environments, and I can actually tell you that the postcast operator which we'll talk about you know it supports that and supports that feature in one of its architectures.

B

uh We, you know eventually we got something called quorum, commit um you know which is popular in the distributed consensus world where, basically, you can say as long as my you know, my transaction gets to at least you know. Let's say two replicas, then I can consider it written and if not you know, I'm gonna hold the transaction. Until I know it's you know safely written to you know a certain number of replicas. We also got logical replication, which you know created. You know you can create some.

B

You know really awesome things around that, but one of the important things is being able to online upgrade. So you know going from postgres. Let's say you know 10 to 12. I can keep my system online and basically you know flip it over like that reducing the amount of downtime that I need- um and you know, and so and and that's just one feature now. You know this replication has been very important to you know. You know ensure postgres can stay available now to properly use it. There's a few. You know, you know it does.

B

Take some work to you know, set things up. You know similar to writing uh an openshift manifest. uh You know there. You know the feature is there, but you know you need to be able to fill it all in, and this is where using automation patterns like the operator can help. uh You know with that task.

B

The the other thing is, you know, postgres, you know extensibility, you know beyond just having extensions themselves, and you know we're going to talk about those today where you know you can basically, you know, add on to postgres functionality, while still maintaining you know the open source core, um there's a whole library ecosystem. That makes it you know super easy to manage. Postgres you know be able to, for instance, monitoring it or you know you know handling. You know the aha in a really cool way beyond that.

B

uh You know, having you know, transaction safety, data integrity, ensuring that your data gets written to disk being able to detect if your data is corrupted. uh This is the page checksum feature and postgres. um You know it's, you know it's really made it. You know very trusted and the question late. You know when I first started the postgres community. The question I got most often was what's postgres and this was in the tech community that you know, people didn't really know what it was, and here it had already been around for 20 years.

B

The question I get now is not what's postgres, but how do I do xyz with postgres? How do I deploy it in this manner? How can I use this feature? And you know it's really showing the evolution that you know?

B

Not only does you know, postgres have you know a name recognition, um it's being deployed, it's being deployed in mission, critical ways and again, with my uh open source community hat on, like it's just so cool to you know, have really seen their evolution and it really goes the credit of the community, and you know the feedback we've been getting into the community to make sure we can continue improving the products, so um that is my spiel on postgres they're.

B

You know I was joking with diane that you know I could probably talk about postgres itself for at least an hour, and uh there still would be plenty more to talk about, but you know today you know the goal is to talk about. How do we run postgres on you know, openshift or kubernetes, and in a way that can satisfy you know all the all the needs you need to be able to run it into production, and this is where operators come in uh operators.

B

You know have really changed the way in terms of how you can deploy stateful applications on kubernetes. And what do you mean by that paper? Workloads really only have one job you need to maintain state. Basically, if you know I modify you know, let's say I insert a record: let's say it's a financial transaction.

B

I need to make sure it gets written to disk. I need to make sure that you know nothing's going to mutate that record in a way where it's going to invalidate the transaction or lose the transaction. I mean that would be pretty bad, so we want to make sure that we're able to create a system- that's resilient, that let's say you know my disc. Does you know crash out? Or it knows you know the block.

B

There's you know some corrupt blocks on the pvc, I'm using that I'm able to restore my data in a way and know that no, I didn't lose any of it, but in order to do that in kubernetes, you know we need to apply more knowledge in general, um and this is where operators can help, because operators can help encode specific information into you know into your environment, so that you're deploying your you know: you're deploying your stateful service in a way that you know matches what it does so postgres does things one way my school does things another way.

B

There's you know from a high level they're similar, but the specifics in terms of how they work are different and that's where the operator can capture that information. Additionally, you know to modify the state. You know it could be arranged for something you know very simple or very tedious to create a postgres replica is actually a little bit of a tedious process. It helps to have all that automated and it helps to you know, have that automated in a way that is able to do it efficiently.

B

um Adding a user, on the other hand, is very simple. You know I can add a user database no with one line of sql, but let's say I bring a new dba into the team and I want to give give them an account to all the databases. Well, that's a little bit more tedious and granted. You know if you've been doing devops practices for years. You can, you know, write.

B

You know your appropriate ansible playbook to get that all set up, but then you have to think about that from a maintainability perspective, because you know you're, you know you're creating something that may or may not be standard standard practice, and then you have to teach everyone in terms of how to do that. What operators do is that they create a consistent view of the world so that you know adding a user is the same.

B

You know no matter where you install the operator and that's where this pattern is great, because you know with having a healthy operator ecosystem.

B

Suddenly, you know we're able, to you know, create standard ways of taking actions on the various uh stateful services that are out there, um and you know this leads to the final point of you know allowing automated managed workloads. um You know some of this starts with just being able to do. You know the proverbial day, two operations handling high availability and having monitoring in place having you know, taking systematic backups and so on and so forth. But those things continue to advance, and you know some of this is forward-looking.

B

um Having you know having smart systems- and you know some some of these are already and when what I mean by spark systems, I mean you know being able to auto scale or on you know, auto auto tune themselves and there's some services that can do that and with postgres. You know to be able to deal with a database system like postgres to do things like auto scaling. You need to have a lot more knowledge because you know what does that actually mean? Do I need to scale vertically? Do I need to add more ram?

B

How does that affect my postgres configuration or you know, do I need to load balance things more? Is my workload really read heavy? How do I determine if it's read heavy? How what are my, what are my scaling thresholds and we have the ability to start amassing that information and start building out systems like that, but you know that that'll take some time. That said, you know with the way the operator pattern works. You know you know we're well on our way to getting to that point.

B

So you know I would say that concludes my spiel on kubernetes operators and you know, tie it into a topic, that's near and dear to my heart, which is the country postgres operator.

B

So, as I said at the beginning, at crunchy data everything we do is open source um and the operator is no exception. uh It's actually been open source since march of 2017, it's one of it's one of the early examples of a stateful operator. uh Currently, one our version four four. Actually we have a four five in beta right now, I'm gonna demo a little bit from that today.

B

This is uh the surprise demo I was uh informed about, but uh you know that surprise demos are all the more fun uh it's a it's level, five on uh operator hub, uh it's a certified operator. um You know you can go ahead, try it out today and you know going back to that original slide.

B

I mean I do like the hippo wearing the the coupe hat, but the idea with the operator is we want to support all the features you need to run postgres in production and it could be a single postgres, postgres cluster running production or it could be. You know, a thousand postgres clusters being managed by the operator. You know we've seen both use cases. Of course everything in between and to us. You know it was important that we're able to do that and you know I could go.

B

I can go and read through the list, but I you know and I'm rather going to cover. You know a couple things I hadn't touched on yet um elasticity being able to add and remove replicas. You know they could be for high availability purposes. They could be that you need a read-only replica. You know your your business intelligence person comes in and you know wants to perform. You know some read-only queries um and you know it could be. You know it could be for load balancing as well. You know it.

B

You know it just depends on what your use case is, but you know the idea was to make it easy that I can, you know, add an additional replica. Just by typing, you know pico scale hippo or whatever my cluster is uh leveraging. You know. One thing that's important too is we can leverage? You know. Kubernetes and openshift native objects to you know, create a you know, very sound system. So uh kubernetes has this concept of pod anti-affinity. Where you can assign rules to say hey.

B

You know I want pods of these types to ensure that they don't deploy to the same cluster. uh You can require it and say like don't schedule the pod, unless they're specifically deployed to uh different nodes, or you can prefer it and say you know, try to schedule the positive different nodes, but if you can't do it, no, it's okay.

B

You know scheduled to the same node, um but the reason we we went with that is to make sure that there's you know a higher probability or guaranteed probability that you can schedule your postgres instances to different notes.

B

um A disaster recovery. uh We use an open source tool uh called the pg backrest.

B

You know it's actually something that you know we support very heavily and the reason is that it was designed for terabyte skill databases that you know they saw. The office saw an issue where it was difficult to create an efficient backup solution and postgres, for you know very large databases and they sure enough they've solved that problem. um You know- and you know one thing- that we've done with the operators that we've built our disaster recovery solution around it. um It's great.

B

um I you know, I you know it's one thing that you know I've always looked to employ. You know wherever I've been employed, you know, and since the existence of pg backrest- and you know we're very happy to you- know- help try further use cases to it.

B

In fact, you know one of the benefits of doing everything open source is that you know the operator does help you know and based upon the feedback we get from the operator, it does help drive, features being added to pg backrest, so one of the big ones was being able to expire backups. You know or create a time-based retention policy.

B

You know saying like hey. I want to keep this full backup, for you know, 21 days, um you know that you know came from direct feedback of people using the postgres operator and ultimately, it was upstreamed into peachy backrest administration. uh One thing that I may or may not attempt to demo is being able to use the popular pg admin tool, which is a graphical user interface to administrate um the uh you know your postgres instance and uh pg monitor, which is a connection pooler.

B

uh One of the known limitations in postgres is that um the number of connections you have could have an impact on database performance and it has to be a fairly large number. uh That said, um first off, there's actually a big improvement for that coming in postgres 14, but you know that's at least a year away, and um you know you know until then you know there is peachy bouncer, which uh is you know, a connection bullies.

B

So basically you could have uh you know you know several hundred connections coming into peachy bouncer and then you know skill scale down when actually goes into postgres I've deployed it. You know multiple times in my career, big. You know. I was very happy that you know we were able to integrate it into the postgres operator and you know there's some other. This matter.

B

Benefits of you know future bouncer as well, where you can do connection state management or basically, during a failover scenario, you can have pg bouncer, you know hold connections until the failover is complete and then resume them.

B

Last but not least- and you know a big feature, four five is a full support of the open source pg monitor, which includes you know: lots of wonderful, the charts and uh graphs that are essential for monitoring postgres clusters, which we'll touch on in a little bit. So why should we use an operator?

B

uh Well, there's a lot there's. A lot of you know really good points on that um I'll, I'm just going to brush on the slide, because I'd rather get into you know talking about some of the uh the architecture and the the nitty gritty details of how the postcast operator works. But you know automation, standardization, ease of use, um and I think you know you know. I tried to discuss this as well as I could before um I do want to emphasize the ease of use.

B

You know the idea that you know we have. You know a fairly simple api or you can manipulate things from the crds or you know, even like a cli ui, you know our pico command line. Interface, you know is, is very popular because I can just do pico create cluster hippo boom. I have a postgres cluster pico, create cluster hippo replica count. Three. I have you know a high availability uh uh cluster. There. um I've also been involved in developing a ui around the operator, and you know it's.

B

You know that that's the longest story for another day, but you know by creating like a standard interface, it's very easy to create. You know even more. You know robust applications around postgres um scale and when I talk about scale you know it's not just saying like. Oh, you know the postcast operator can manage. You know a thousand postgres clusters. It's also scaling your workload, and you know this is something I always looked at. You know when you know, as an engineering manager is, how do I scale our processes across the team?

B

Can I create something? That's repeatable and standard that you know everyone can, you know, can be able to do and interface with so that um you know that's important to consider as well. um We certainly don't you know engineering type things to ensure the operator itself can scale, um for instance our multi-name space support. um You know we at one point: we were hitting limitations on it, but you know thanks to one of our very smart engineers. You know we can handle.

B

I know easily northward of 100 namespaces in the multi-name space mode. You know, I don't know you know we haven't really pushed the limits of it yet or we haven't found the limits. I should say um security, and this is the cool thing about running postgres and containers- is that you naturally get a certain level of security there and such that you're in a sandbox environment, um and that actually affords you to do you know if you want to do some.

B

You know cooler things like give people super user within your container, and by that I mean postgresql user, not uh not a root within the container yeah. You can do that and there's lower risk to that. um You know I might have my own opinions on what you should be doing, but the idea is that you know you know you're operating in more of a sandbox environment, and you know that affords its advantages as well, and you know just having the process isolation built into it.

B

Certainly helps when you're running in a in a multi-tenant environment- um and you know the flexibility- and this is one of the first things that really attracted me to what we were doing with the operator. Is that as long as you have a node, you can run it anywhere. You know it doesn't matter where your openshift nodes are deployed. Yeah, you can run a postgres cluster there and that's really cool, and you know I think you know in in this regard.

B

Like you know, I can you know I'll speak for the quenching operator, but the you know, operators in general create that unified layer where no so long as I have you know something that speaks you know kubernetes or openshift.

B

I can deploy my workload thing with if I'm running openshift, you know as long as you know, I have an openshift node somewhere it doesn't matter what cloud provider I'm on and what hardware I'm running on. You know I can deploy there. So that's really cool. You know one thing that we do when testing the postgres operator. Is we test it? You know we test in all sorts of places. We run openshift in one environment that openshift in another environment, and you know for the most part it just works.

B

So that's uh you know that that's my pitch for operators in general. You know, let alone the the postgres operator.

B

So how does it all work? um Well, you know, I would say, just look at the diagram, but going into a little bit. You know. Part of the operator pattern is that you have these things called custom resource definitions, and you know from my application developer perspective. I consider that your database schema your your customer resource definitions or crds. Basically, you know store the infrastructure of what you're going to deploy. So we have one called uh pg cluster.

B

The peachy cluster stores is essentially your definition of your postgres cluster. You know it says how many you know how many replicas do you want? What ports you want to run it on? Do you want a pg balancer with it? Do you want to collect metrics with it, you know and so on and so forth? And you know from that, you know pg cluster. You should be able to deploy a postgres cluster anywhere.

B

If somehow your whole kubernetes environment gets wiped out well well, if your data gets wiped out, then that's a different story, but let's say all your deployments get wiped out. Well, I should be able to redeploy, you know a postgres cluster. That looks exactly like what you had before from that definition.

B

So from there what the operator does is the operator reacts to what's been added to a custom resource definition or any changes in the custom resource definition and then applies those changes throughout your openshift environment.

B

We've layered a few things. On top of that we have an api server. That makes it a little bit easier to interface with the custom resource definitions as it aggregates that information together and that's actually what the pico command line client uh uses as well. Speaking of we have a command line. Client called excuse me pico, which is, um I got the eco client. It works across different operating systems um and it makes it you know super simple: to create.

B

Postgres clusters manage post clusters, modify them take backups, you know whatever it may be, and we'll demo some of that. We also have a scheduler which is used to uh schedule backups amongst other things, um and this is great because you know you should always be taking backups. That's another thing. I always emphasize that there's one takeaway from this talk please take backups of your postgres cluster.

B

You know you never know when you'll need them and maybe to further emphasize that um there was a time in my career like one time where I really had to make a point in time. Recovery like you know, there was something where we had to. You know, get the data back to certain point of time and replay it. You know and find certain things, and thankfully you know now that we have backups. You know we we had our transaction history. You know back to that point and we were able to.

B

You know, solve the issue, but please take backups, it's so important. So just uh finishing up, you know this diagram. um The most important thing when you're running a database is your storage? uh No matter what it is, your your storage is going to be. Your bottleneck. George has gone much faster, like ssds were one of the greatest things that happened for uh you know, database workloads, you know the one trick with ssds, particularly back in the day, was reliability, but you know you know, the recommendation would be, you know always have.

B

You know, run in like a raid 10 mode. um So that all said um you know the interesting thing in you know the kubernetes and openshift ecosystem. Are you know the variety of storage classes available um or even you know, people who you know? We have cases where people just use a host path, storage and you know, pin their. You know postpress clusters to a single node and you know ensure you know everything gets right into that node.

B

um But you know, storage selection. Is you know a very you know detailed involved topic. um The various storage classes have certainly improved. You know. Even you know, even since I started our crunchy data and you know it's interesting to see. What's you know happening there, um our team stays on top of it too. You know for for our various testing and uh compatibility purposes, but um you should always consider you know what your you know. What level of storage you need based on your workload?

B

Not everyone needs the latest and greatest and most expensive storage for their databases. I mean you know, am I biased, like? Yes, of course, I you know I love playing with like the latest and greatest and fastest storage, but you might not necessarily need that. But if you do, you should know how you're able to you know optimize uh using uh postgres. You know with your various storage layers.

B

So let's talk about high availability, which you know was you know the you know the original topic of this. uh This chat. This is a really important slide, because this explains how high availability works uh with the postcards operator, but it also shows like how cool it is. You know, whenever we get tickets about, uh you know: support tickets around high availability. You know one thing we always try to ask.

B

Is you know, how's your high availability set up in your kubernetes or your open shift environment, because your post with high availability with the postgres operator is tied to your kubernetes high availability, and this is the feature, and this is why it's really cool so kubernetes and openshift are backed by their own distributed consensus storage system.

B

You know, you know it could be fcd and you know ncd has its own high variably high availability system built in, but the reason why we leverage this is because it minimizes the amount of postgres nodes you need to deploy for high availability. So so, let's take you know the wrath algorithm. Now the wrath algorithm algorithm says you know you always need an even number. Sorry, you always need an odd number of nodes to be able to get high availability, and I believe you know, as you recall, the recommended number is five.

B

So running five postgres databases, particularly let's say they're multi-terabyte. Now, that's a you know, that's a very large footprint even running three. I mean I like to recommend three as the number that you should run, but you know that's still. If you have, you know, 10 terabytes of data, that's at least 30 terabytes that you need plus backups plus. You know your. You know your various logs and wall and you know other things that you need to store.

B

So this could be a lot of data floating around by leveraging the you know the openshift back storage system. You actually only need to run two postgres instances to get high availability and that's what's really cool because it does help you lower your footprint, but you can still maintain a safe distributed consensus, high availability, you know what I showed here was you know. I just showed three nodes here, and these are the actual openshift nodes, because you you know, the other thing you need is a uh you know, pg backers repository.

B

um You know, that's the third component for the high availability and the reason you there's two reasons why we leverage that one. Of course you need to take backups. Remember the one take away from this talk is please back up your database.

B

Two uh pg bacchus is actually used in the self healing process. So let's say my primary goes down and you know it sits down for several minutes. You know the replica is promoted.

B

We want to bring the primary back into the fold or the old primary it's going to be a replica, but we need to be able to catch it up to where the primary is so it can. You know rejoin, you know as a healthy streaming replica.

B

We can leverage the pg back rest delta, restore feature to efficiently copy the information into uh you know the the failed replica effectively reprovisioning it and bring it up to speed and then tie it to you know, tie back to the the primary so so now what does pg backus serve? As you know, part of you know the disaster recovery system. You know it does play a role in high availability as well.

B

Well, which is you know, which is really cool, because you know we can leverage all those efficiencies that are put into it to you know quickly heal uh components within the system, so the takeaway from this is that you know the post. Consulaberator does provide high availability, um but you know it's leveraging. It's leveraging it from you know, kubernetes and openshift, which you know is like I said you know, is really an advantage.

B

I guess you know the other. The other thing I should point out is that the operator itself is not providing the high availability, because that would make it a single point of failure. The postgres instances themselves are providing the high availability. You know they're communicating with kubernetes to make sure that you know to basically determine you know if there needs to be a a leader election, because you know primary is down or unreachable or you know whatever it may be.

B

So disaster recovery: how does it work? I would say that it works in. You know pretty you know in a pretty cool way and in in what we can support. So we can. Actually you know we we support, uh you know a multi-repository setup and what I mean by that is that you can push your archives. Your backups to you know a you know: a pvc, that's somewhere within your your local openshift environment, or you know you can push into s3 or an s3 compatible storage system.

B

You know, yes, will just mean I o, um and you can actually support both at once um and yeah and that's cool, uh because you know you can guarantee that your backups are being pushed to multiple places.

B

um Additionally, you know you're able to schedule backups as well, and you know the easiest way to make sure you keep taking backups is to keep scheduling them or to have a schedule them, so they can keep uh being taken.

B

However, you should also make sure your backups are being taken and you should monitor everything else, and you know this. This is a feature that um you know that I'm I'm previewing, for you know the upcoming operator release. That's you know coming out towards the end of the month. Is uh our our integration with pg monitor, so what pg monitor provides is a series of grafana dashboards um that can be. You know, written from you know a prometheus database.

A

B

It basically shows you your overall health or your database, and you know it can you know it can track things um or checks, a variety of things uh and probably to go into. That is a full talk in itself, but you know you know it's essentially the curated list of the key metrics.

B

You need to keep an eye on to detect, to try to anticipate issues with your system or you know that will give you the overall healthier system so, for instance, if you're supposed to take you know you know daily backups and the backups not taken. You know that you know the the top bar that you see on the screen is going to go uh red.

B

uh You can drill it down to your specific databases or specific pods within your cluster, um but you know the idea is that you know you have all these key metrics there take another one replication status. uh This helps you detect. You know what your replication lag is um if you're using asynchronous replication. You know this will be. This could become an issue because your lag is too much. You know you have you're at risk for data loss. If there's a failover event, um so you know you do want to keep an eye on that.

B

uh You know the other thing. The other thing with this is that you know this works with. You know all all the upstream components. So you know if you already have a prometheus instance. That's uh you know set up that you're using to aggregate your metrics. You know you can plug in the grafana dashboards and you'll be able to pull from that prometheus instance.

B

One of the dashboards that I want to show is actually related to getting pod specific metrics. So this was an issue that we had run into for a long time was that you know some of the key metrics when using a postgres database are related to your actual body utilization, and you know we had a hard time trying to pull these metrics out in a consistent way. um You know one of the people at crunchy data, joe conway, you know he's a post-request committer major contributor.

B

um You know you know, and you know now, a uh container enthusiast wrote an extension to postgres called pg node mx that can actually reach inside the pod and uh pull the information out of the various uh c groups, and it works for c groups, version one and version two and it can pull out. You know the specific you know, let's call them the os style metrics um from that particular pod. So I can see. Oh you know this. You know primary is used, has as much disk activity or you know it's currently.

B

You know cpu is currently pegged at 100 of it. Do I need to raise the limit um and help answer questions like that, and this is this is probably you know, you're not supposed to pick favorite features, but this is the feature I'm most excited for for the for the upcoming release, because uh particularly you know, it's not excuse me as an application developer.

B

um You know this. You know this is. This is where I always started to when troubleshooting my systems, because to me they're the steps that make the most sense. If memory utilization is getting out of control, you know. I knew that you know. There's probably you know there might be a runaway query and actually look into that same thing with, like you know, cpu was that 100. Let me try to find the process. That's doing that. Oh, maybe it's a really poor recursive query. I ran, which was often the case.

B

You know I could then go in and fix it. I mean, kill the process and then fix it, but um yeah just usage. You know, oh I'm at like 80 disk usage. You know, is there an unacknowledged replication spot somewhere? That's you know causing you know too many wall logs to be retained and, of course you know it's nice to be able to look at the charts and get all this kind of all of this information. But it's also good to be alerted to problems and uh we've pulled together our favorite alerts.

B

That will tell you, you know, uh you know what kind of errors are going on. You know, for instance, if high availability is you know, if you can't heal something you know something's wrong, the cluster isn't accessible and it's inaccessible for a certain period of time. um Now, let's you know, let's send a critical alert for that. The way it triggered this alert originally was I uh I did a a very terrible thing.

B

I removed the data directory from postgres and just like totally made it, I made the instance unusable, and that was good enough to trigger the alert. I was actually not running it as a high availability since, which is why uh it uh was impossible to heal.

B

um So that being said, uh you know we, you know, we've added. A variety of alerts you know is, is your disc filling up? um Is it replica? Is a replica you know? Is it lagging too far behind and so on and so forth? And you know we hopefully you know you find these useful and I hope you never have to respond to one of those alerts.

B

So um last but not least in terms of you know the walkthrough demo, then we'll try, you know a live demo. um You know adding the ability to administrate your database from a uh a user interface, in this case a pg admin. uh We actually did. uh We actually created an integration where you know you know we're able to tie the postgres user accounts to bg admin and keep them all in sync. So that way you can log your pga. You know admin data, pg admin instance and you know administrator postgres database.

B

So you know we. We try to check all the boxes in terms of not only having the the day to administration options available, but make it very easy for the people who do the the daily postgres work to be able to interface with their database.

B

So let's do this. Let's create you know this. You know high availability, cluster with monitoring and connection pooling, and you know all these wonderful things with one command, because that's really the beauty of all of it. So, first of all I went ahead. um I created a postgres cluster because I think we're going to do have some fun.

B

So, let's see so we have a so. We have a postgres cluster. um Let's inspect it a bit, so first off uh pico pico's, you know we call it the the command line, client for um yeah for interfacing. With the postgres operator, I mean you can integrate with the custom resource definitions directly. We find this very useful um to use the command line. Client and you know, there's a bunch of different things. uh For instance, I can test. If my cluster is up, I can see the the current disk utilization.

B

um I can scale it well. First, I can introspect it.

B

I can I can check on the health of my backups.

B

I could take another backup, because why not.

B

And I can scale it up.

B

So yeah, so so the pico, so the pico client is you can see. Is a swiss army knife for uh being able to uh handle? You know all these different operations within you know your daily postgres environment.

B

So, while that's going on to show there's nothing up, my sleeve.

B

I had created I'd also created pg bouncer, which I've been connecting to um it is, and I have a port for it set up to it. You don't need to worry about reading the screen since just a port forward.

B

I've also been, I also added some data to my postgres database to um you know to basically bootstrap it using you know a tool called a pg bench. So what I'm going to do is. I am going to start writing data to my database using pg bench.

B

And I will also say this is truly a live demo, because I'm going to do something I did not test beforehand, but I want to see if it works.

C

So how often, how often does a live demo actually break on you.

B

So typically, I rehearse them so the probability is very small. um This is uh this is uncharted territory, but in this case I'm going to try to purposely break it. um What I'm going to do is I'm going to kill the primary node while we're running a peachy bench and we're going to see what happens.

B

So let me find first off, let me find an available terminal window where I can do it.

B

I realize the font's probably really small here, but what I'm going to do is I'm going to run a delete on the primary.

B

B

Look at that so clearly we have the server connection crash, but it reconnected just like that. So I'm gonna, I'm gonna, put this into the background, so it keeps running so what's going on, so let's do a test on hippo.

B

So we can see that you know our. You know our prior main pg bouncer are up um and we you know the original primary. Was this node, the hippo blah blah blah blah. The new primary is hippo ac gj. So the highly ability works that you know we quickly. um You know, even even though we deleted the pod, which you know things can happen in coop world during openshift world, so maybe the pod does get you know rescheduled or deleted, but it came back up. The interruption was very minimal. um That's pretty cool! That's yeah!

B

You can't get much better with high availability on that regard. um Again, you know what I mentioned earlier is that if you want to avoid transaction loss, um you need to have synchronous replication set up, which is something that you know we do support from the uh the postgres operator here. Let me stop peachy punch because those messages are getting tiring.

B

If I want to create a cluster with synchronous replication, I would use the sync replication flag. Now I mentioned that I talked about the trade-off with that. uh The problem was taking this replication as it stands today. Is that um if your replica goes down, it's not your primary. If your replica goes down technically your primary goes down, because you need to guarantee that your rights are getting to the replica now with postgres 10 and beyond this itself.

B

The quorum commit- and I believe you know I haven't tried it myself, but if you do some of the you know advanced configuration with you know a postgres operator instance, you can probably get the quorum commit. um I I would also caveat your mileage may vary, but the quorum commit. You know you can make it a little a little safer to deal with uh synchronous replication in the sense of that you know, there's still like the performance penalty, uh I mean sorry there isn't like it.

B

There is a performance penalty still using it, because you need to guarantee that your rights get to whatever you know. Number of postgres instances that there are, but um you know it's not it's not necessarily going to cause. You know you know a downtime much like if just a single replica goes down your primary goes down, um but that said, if you know, there's certainly workloads that need it and you know you're willing to pay the performance hit because you're guaranteeing that your rights are going to be in multiple places.

B

I'd also say you know, leveraging pg backers the way we do. Those you know help ensure that your rights get. You know pushed out to you know the the back repository as well. um You know, there's always a little bit of a delay because you do it. You know you need your full right ahead. Log record uh written before I know it goes out there.

B

Some of the other things that uh that are that are cool about this um other than uh showing that high availability actually works. Is um I'm just going to create a p, a pg admin.

B

So take a few minutes to come up, but you know we'll try directly logging into the database via pg admin. If I'm able to get all my commands correct because again uncharted territory, you know you know, maybe to cover a couple of the other cool features. um You know you're able to tune your memory settings or to you know how much you know cpu or ram you want for particular node. You know if you decide that you know what you originally you know deployed your cluster with is not um sufficient enough.

B

You know you can update it later.

B

um What else can you do? uh You can add table spaces uh table spaces are good. If you want, you know, there's a variety of reasons where you'd want to use a table space so table space is an additional external volume that you're attaching to your postgres instance.

B

um You know for multiple purposes, one it might just be a very large table that, or a very large group of tables or very large database that you don't want to attach to your main uh postgres data directory. It could be that you know you want to take advantage of. You know super fast storage, for you know a particular data set. It could be that you just have a lot of data and you need to you know, spread it out amongst multiple pvc. So I don't you know the use. Cases do vary.

B

um So you know you're able to you, know, add tablespaces to uh you know, operator postgres clusters operator managed postgres clusters. You can set the size of the pvc that you want. um You could do that on a one-off basis.

B

um Do what other cool things as you can see, there's a lot of flags. I do encourage you to read the documentation.

B

I've tried to write the documentation in a way that tells a story that gets you going to where you need to go as always, because it's open source patches are welcome. You know people certainly do like to leave their opinions on the documentation.

B

We also support an easy way of setting up tls connections. You know this is you know. Probably I only lightly touched upon this earlier. um Keyless, of course, is a very important part of uh you know. Being able to you know encrypt. uh You know encrypt communications, make sure people aren't eavesdropping.

B

uh We try to make it as easy as possible. uh You just have to provide you know, uh you know a key cert pair and you're good to go. You can force you can and you can also force all connections to build your tls by using uh the tls only option, so I've probably stalled. I hope long enough. uh There should be a pga been deployed. There is, let me see if I can port forward to it. So.

B

B

One moment I'm going to say make sure I actually put the port forward correctly to my computer.

B

Yep alrighty, so now I'm going to shift my screen one more time or maybe two more times all right. So this is pg admin. um It's uh you know. As I mentioned it's a popular user interface for interfacing with postgres. Let me create a new.

B

B

Call the user hippo.

B

And voila, so this is cool because you know this syncs with the administrative dashboard. um I can, you know very easily log into my databases and see uh forgot, which one I ran, those uh you know the the pg bench tool on.

B

Mostly because I want to be able to run some queries against it, but I forget where I ran it.

B

Here we go alright, so um yeah, so here's uh here's, uh the uh there's all of our data tried it. We could try to query against it or I'll. Probably you know because I remember the syntax. You know I can create a table right here.

B

Insert some data.

B

B

Just like that- and you know again, you know what was really nice is that you know I was able to. You know, run a command on the operator and then you know have everything you know synchronized into the pg admin interface. So you know it's it's. These things are convenient and it's you know creating ways of you know systematically being able to um you know to be able to. You know, you know, manage a whole wide variety of different uh postgres workloads and needs.

B

So I think, with that um I will return to the slides. So first uh we saw we actually saw this whole thing created um and yeah I mean that's really. You know it's really all I have I can. You know, try to go ahead and you know do more live demos. See if uh see if I can actually break myself. um If I try to deploy the monitoring suite, I will break myself because I found something you know incorrect in my configuration. I stop and figure it quite exactly figure out what it is.

B

But if I can get that working, it's actually pretty cool to see all the charts and dials uh going but yeah. I think with that. I'm happy happy to take questions.

C

Hey jonathan, it's mike here from red hat. How are you.

B

How are you doing.

C

I'm doing really well, I I uh interesting demo. It's always it's always good to see, see the inner workings of something so cool, um but you know I manage relationships with lots and lots of software partners here at red hat and it seems like there's just database vendors coming out of the woodwork everywhere.

C

Are they all the same, meaning like how is how is crunchy, uniquely positioned out there to be better than the other 10 database vendors that are all popping up everywhere?.

B

Yeah, so you know, look, there's, there's all sorts of different solutions out there to you know. How do I store my data and you know it's certainly something I've seen through the years um and again you know you know you know, I'm biased I've been using postgres for 15 years. I've deployed been deploying it successfully in production for 15 years deployed, and I was running postgres now, even before there's replication or like any high availability guarantees, and you know I think you know one thing that you know I love about.

B

It is that you know it's it's a very strong and healthy, open source communities. You know similar to the way you know. Linux is similar to the way kubernetes is, and you know one thing that I've liked at crunchy. Is you know we? You know, we have adopted the red hat model and you know we found you know and we find that you know you know we like the fact that everything we do is open source and we're able to support open source that you know we can make the upstream.

B

You know the best stream so to speak. So I think you know you know what you know what I like about crunchy. Is you know beyond that? It's you know how we do focus on. You know what I said in this original slide. You know beyond open source. It's you know adapting to the modern technologies that are out there, ensuring that you know we continue to make postgres work. You know efficiently on openshift and you know the focus on security. You know a lot a lot of what I found.

B

Data security through the years is that you know everyone wants it, but they really only employ it. You know to you know to ensure that they're um compliant with whatever regulations that they are, and if you find that you're not doing things to keep your data safe and secure, you can actually run to you know a pr nightmare when you know data is breached, and you know, data security is a whole topic in itself. You know requires, spend you know several hours on it and I think you know.

B

One thing that you know we try to focus on is not only how to mitigate the risk of threat, but you know how to deal with you know what happens you know. Should there should there be a breach and ensuring that you can minimize? You know the overall damage of that. So um you know that you know that's what I love. That's what I like to say about postgres and crunchy. It's you know the fact that you know you know we love the upstream and you know that's. You know, that's what we want to support.

C

Thanks, uh I got one other question: the completely other side of the of the of the table, though crunchy.

B

C

B

Did you guys come.

C

Up where did you guys come up with it? We always hear lots of interesting stories about how different companies select their name, and you know the the the hidden meanings of why of the name and so forth. But why why crunchy white crunchy data.

B

So interesting, I thought you would have asked about the hippo because I think that's you know. That's the the bigger urban legend.

C

Well, that's right, I think that's, that's part of it. It's like you know there is, is a hippo crunchy. How does that work.

B

Yeah, so so, there's many so there's many urban legends around this um I'll. uh You know, I think you know. Crunchy data actually came out of a meeting where uh someone uh described you know what uh our founders are working on as very crunchy.

B

So uh that's uh you know that you know, maybe maybe the urban legend for that's a little less, but the hippo has many stories, um and you know the one that I choose to believe is that uh you know hippo. You know, hippos are you know fiercely protective of their bordering hole and you know given crunchy's data roots are in you know the security space you know having you know having an animal that is, you know. Very protective of you know your core acid. You know the watering hole your data lake.

B

If you will, um you know it gives a strong. You know sense of assurance that you know we're looking after the integrity and safety of your data. You know another one. That I heard is that you know hippo. You know when you see a hippo in the water.

B

You know you only see its eyes, but you know there's a whole lot going on underneath the water- and you know, that's sort of you know like a principle of security- is that you might only see one layer, but you know there's a lot of other layers going on um there's at least five other urban legends. I've heard around the name, uh but you know those are the two that I particularly like.

C

C

So what what's? Next? What do you? uh I know we're just about out of time? um How do people get in touch with you? Do they just send an email to jonathan or you know, call your house phone yeah.

B

Yeah uh well, fortunately, as you can, you know see by oh, I don't know if you're still sharing my kitchen or not uh we're fortunate that that house phone is does not is not actually connected to an outside line, but.

D

B

Email me, um you know jonathan.cats at crunchydata.com, you twitter me at jcat05, um go to the website. Crunchydata.Com see what we're doing um plenty of people come to uh the actual github repo uh uh crunchy data. Slash postgres operator uh certainly get a slew of questions there, but yeah you know, that's the you know, that's the easiest to what's next. If you want to get in touch, okay,.

C

And there was there was another question. Jonathan just I know sorry we're over time, but.

B

Because our crd.

C

Yeah, yes, shannon was asking about you know. Can you share your cr by any chance and I was like I'm not sure what cr is but shannon just it's the it's the uh that is what the uh it's, what you use to create the actual pods.

B

Yeah so the see so all this is in the documentation. Now we have a section just on the custom resource.

B

So yeah the series, the series, the definition itself, we actually we uh in our documentation. We have examples of how you can create uh custom resources.

B

If you I mean, if you I could show what one looks like real, quick uh one. That's already created.

D

Yeah, sorry, if you have just the mo it's fine, just kind of for the learning purpose to spin it up, it's kind of easy to have a sample um when you mention that documentation is at the access.crunchydata.com document. Right.

B

D

Oh there's the github one.

B

It's the github one.

D

B

Well, the one github we'll get well github, it's it's in the access.countrydata.com documentation.

D

Okay yeah. I found that github one and pointed to the quenchy data, but I would just kind of want to get a quick start and grab your cr and then I can just pop them on and then try yeah.

B

Yep, so this is so. This is an example cr. This is the cluster that I was demoing from.

D

And it is in the uh github, which is what you're saying.

B

Correct well the correct way it's in our documentation, so we so we have an example for how you can create a postgres cluster uh using a custom resource.

D

B

And one one of the reasons why, with the command line is you know it makes it a little bit easier? um You know it's much easier to type pico. Create cluster hippo then fill out the cr, but that said you know we do. We do provide examples for how to do it uh from you know, using just the the custom resource.

D

Okay, cool. Thank you. Welcome.

A

B

A

I think we came to the end of our hour here and uh it was a bit of a tour de force and- and I really appreciate that- uh thank you so much jonathan we'll have crunchy back again in a little bit time. uh I think next month I will have some of the folks doing some spatial gis, but this has been an awesome session. So thanks michael again for arranging this and jonathan for sharing your kitchen with us.

B

Yeah happy happy to invite people into our kitchen that we can only fit so.

C

C