Red Hat OpenShift OpenShift Commons Briefings, 20 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Database Disaster Recovery Made Easy: Building Metro HA Postgres Cluster - Annette Clewett(Red Hat)

Description

Database Disaster Recovery Made Easy: Building a Metro HA Postgres Cluster with OpenShift Data Foundation
Guest Speakers: Annette Clewett (Red Hat) and Andrew L'Ecuyer (Crunchy Data)

Abstract:
Getting your database-backed application up and running on OpenShift is straightforward, but what about deploying a mission-critical application that needs to have geo-redundant high availability and fault tolerance?
Knowing the right open source tooling to use can simplify the "disaster recovery" problem when running complex applications on OpenShift. By combining PGO, the open source Postgres Operator from Crunchy Data used for managing production grade Postgres databases, and OpenShift Data Foundation (ODF), providing resilient storage for OpenShift clusters, you can survive the loss of a datacenter with RPO=0 and RTO in minutes instead of hours.
In this OpenShift Commons Briefing, you will see how to configure operator-based ODF installed on an OpenShift cluster stretched across multiple locations to create a suitable environment for a geo-redundant Postgres cluster. From there, we will explore the fundamental concepts for building a high availability, multi-datacenter Postgres cluster. We will demonstrate location failure tests to show OpenShift, ODF, and Postgres cluster availability and disaster recovery capabilities.

Additional Resources:

https://github.com/CrunchyData/postgres-operator

https://github.com/CrunchyData/postgres-operator-examples

https://red-hat-storage.github.io/ocs-training/training/ocs4/ocs4-metro-stretched.html

A

Well, hello, everybody and welcome to yet another openshift commons briefing today, I'm really pleased to have annette kluett and andrew likure um andrews with uh crunchy data and annette is the principal architect here at red, hat and they're gonna talk about database disaster recovery and how to make it easy, and this comes out of the smart cities initiatives and I'm going to let them introduce themselves and their topic and they've got a great demo.

A

So I'm really interested to see that and how it all plays out. And if you have questions, ask them in the chat wherever you are on youtube and bluejeans, or on twitch and we'll relay them to the speakers and have some live q a at the end. So with that annette uh take it away. I know you guys have put a lot of effort into getting this one going. So thank you very much for being here today.

B

Yeah, thank you diane. So, first off, if you hear a grandfather clock in the background, that's me: um I'm annette clewitt, I'm with red hat uh in the data foundation, business team and uh be going over a lot of things that support this environment, um including the smart city demo.

B

C

Good morning uh yeah, my name is andrew lecure. I am the director of operator engineering at crunchy data in this role. I am responsible for the development and implementation of our crunchy postgres for kubernetes product, also known as pgo, which I'm going to be going over a little bit here uh this morning in the next few slides thanks, adam okay, to start out um just to give a bit of an overview about crunchy. So who is crunchy data yeah?

C

Well, really, at the end of the day, the way I like to describe it, you know our business here at crunchy data is enterprise postgresql.

C

You know if you're looking to deploy, you know, maybe if you currently deploy looking to expand your usage of postgrads or just venturing into postgres for the first time and looking to uh make postgres a big part of your enterprise. You know we, as we are the partner that can make that happen. You know we do that in a variety of different ways. You know, first and foremost, includes the expertise we bring to the table for everything and anything postgres related, um and you can see that on the slide here.

C

You know we have contributors internally here within uh crunchy that do contribute directly to postgresql and the postgres source.

C

You know, in addition to just a wealth of other vast knowledge and expertise we have within the organization, um but in addition to that knowledge, you know we also bring to the table the tools and technologies needed to ensure you can get postgres deployed in your environments in your enterprise environments, according to your specific requirement and needs, you know, so this not only includes bringing things like. You know: certified versions of postgresql itself, which can be deployed within your environments, but also providing the tools and technologies needed to facilitate deploying postgresql within your environment.

C

You know in a variety of different environments where you might be deploying postgresql, you know, and that includes cloud environments which we're going to be touching a bit on today. You know in the crunchy here we have a few few products along those lines. You know the slide. There did mention our crunchy bridge solution, which is a fully managed. um You know postgresql cloud solution, but today we're going to be talking about crunchy postgres for kubernetes, uh which is our kubernetes based solution um for managing postgres.

C

So what is crunchy postgres for kubernetes? Well, we like to describe it as declarative, uh declarative, uh postgres and really what this is. It's an open source solution. We also call it pgo um that allows you to facilitate and streamline on the deployment of production-ready postgresql clusters by simply declaring what you want your clusters to look like. So within the solution we have what's called a declarative api which allows you to define you know simply via a spec, exactly what you want.

C

Your production grade, um postgresql clusters to look like next slide, so to dig into this a bit. You know what do we mean with a fully declarative solution? You know um you know really what that means. Is you know anything you might need to deploy to ensure your database architectures, you know, are fully production ready to go um ready to meet your needs, your requirements.

C

um What we're talking about here is allowing you to have a convenient easy to easy to use way to define via what we call a spec. Exactly what you want your architecture to look like. So, for instance, you know to touch on a couple topics that are going to be relevant to our conversation today.

C

If you want high availability or if you need disaster recovery, for instance, the solution we provide makes it as simple as defining exactly what you need for those elements within a specification and from there you know this solution, crunchy postgres for kubernetes or pgo automatically um takes those specs. Configures deploys those databases, you know exactly how you have them defined. You know.

C

So, for instance, say you want a connection pooler, um so your clients can connect your database through a connection pool or like pg balancer, it's as simple as defining that within the spec and from there our operator solution. I will process that spec and deploy the components within your environment to make that possible. You know, and again I also touched on other functional areas. You know this applies to as well so disaster recovery.

C

You know we're talking databases here, right, stateful, applications, it's critical, that your database is safe and protected, so you want to ensure you can easily create backups and restore from those backups in the event of a failure and again that's what the solution is designed to do. It's designed to make those things seamless and streamlined where you know you can easily define backup schedules and get that data redundancy you need. um High availability is another big part of that architecture as well. You know facilitating the ability to easily spin up replicas that way.

C

If your primary database goes down, we can fail over keep your database available and ready to go and that's really what the solution is providing, at the end of the day, providing users with a convenient easy to use way of defining.

C

You know their database architectures across all pertinent functional areas, whether that's high availability, disaster, recovery, monitoring or whatever it might be. You know, and the operator takes control for that point and make sure that those uh you know database architectures, are deployed and configured as needed according to your specific requirements, um next slide in it.

C

So a big part of this solution. You know so it's one thing for us to be able to to take a spec that you've declared. You know you've defined what you want your database to look like and get it out there and running, but that's only one part of the process right once it's out there and running. We need to make sure it continues to be available, you know and that your data is accessible as needed.

C

You know, and that means ensuring that your database clusters can heal um as needed, um so to just throw an example out there you know um so so what this means is. This solution is constantly monitoring.

C

um You know your database environments, the databases it provisions to make sure they're healthy and conv configured according to your needs, um so, for instance, to provide an example say your cluster needs a connection pooler and that's what's shown on the slide here um and for some reason, um someone out there. You know chaos can exist in any cluster. Someone deletes that connection pool or that connection pool or deployment.

C

Well, what that means is the operator is going to immediately detect that recreate that deployment and make sure your connection pooler is available and ready to use. You know immediately after that occurred. You know and that's just a theme within our solution um in general, you know once you declare your cluster and the operator builds, you know and deploys a database architecture.

C

According to that specification, it's going to constantly monitor it, make sure it stays healthy and make sure all the the kubernetes resources are in place as needed to ensure that database remains healthy and continues to serve your your database needs next slide.

C

um In addition, you know the this declarative approach to defining your postgresql clusters. You know also enables git ops workflows, because what that means, at the end of the day, you're defining your databases using um simple yaml specifications that could easily be stored in any version control system. um That way, you know, because you know, as we know, database requirements differ across different environments, different needs, so your databases for development versus qa versus production, you know, might all be a bit different.

C

But by giving this declarative method of defining your databases, you can easily store your database configurations in a version control system where they can then be easily integrated into your continuous integration or continuous deployment pipelines. You know to easily provision the databases you need at any stage of your software development life cycle. You know to give you the databases you require and ensure the data is accessible, that you need for your users and applications next slide.

C

So another big piece of of this solution you know, is ensuring um you know that your clusters continue can can be updated um without interruption. You know so again, we said you know it's one thing: to get it to get a database deployed that is out there and running, but that's not going to be the end-all be-all for your database right you're going to need to reconfigure it you're going to need to tune it. Maybe you deployed it without a connection pooler, and now you decided you want one.

C

This solution is designed to make that all seamless, so you know, as you need to evolve your database. um Your database architectures your deployments over the course of of a cluster's lifetime. You can continuously do that without interruption. You know, and through a rolling update strategy that we've implemented within the operator solution. This means you can um safely make changes to your clusters and we will safely roll out those changes to all your instances, whether it's postgresql configuration changes, changes to the architecture as whole.

C

You know our solution is designed to safely and cleanly roll those changes out in a way that avoids disruptions to your data and ensures your users are still able to access the data that access the data they need next slide.

C

So, to sum some of these uh uh bits up, so you know what what are we bringing together here at the end of the day, um you know when we talk of you know, talk about the crunchy posts postgres for kubernetes. You know really. This solution tackles. You know the critical functional areas you need to ensure you have a production ready database system in kubernetes.

C

You know, first and foremost, is high availability right and we've touched on this a bit, but at the end of the day um you know you want to make sure your data is available when you need it and where you need, you know, and the postgres operator solution makes that seamless. So you know the ability to add additional replicas to your clusters. You know um the ability to fail over the replicas in the event of a failure and make sure your data is always available as needed.

C

You know that is implemented in a way that is seamless and basically transparent to the end user. You know so again, basically you're defining within our solution. You know the elements you need. I want high availability, you know I want multiple replicas. I want redundancy, you know, but we take that uh we take the the process from there and wire that all up to make it make it happen. You know we make sure those replicas are properly replicating from.

C

um You know the primary database we make sure they're configured to be able to fail over to in the event that there is a failure. You know and your data needs to remain accessible. You know, so that's a big part of our solution. You know because we're talking data here right and, first and foremost, you know we need to make sure your data is available um when you need it and where you need it and the high availability, high availability parts of our architecture are what make sure that happens.

C

But disaster recovery is another important piece of that too. You know, and this is another area the the operator greatly facilitates your ability to ensure you have a strong disaster recovery solution for your database, because at the end of the day we need to protect against disasters right. You know, chaos can occur, things can happen and at the end of the day you know data is paramount. You want to make sure your data is safe. It's available. You know that way.

C

If anything goes wrong, you can properly recover and get back to the place you need to be, and crunchy postgres for cooper kubernetes facilitates that you know whether it's making it easy to schedule. You know backup, so you know your database is, you know, effect effectively backed up using proper backup schedules, whether it's making sure your backups not only exist in you know or making sure there's redundancy in your backups themselves. You know that way. If you lose an entire environment, you don't lose your backups along with it and really just all.

C

At the end of the day, making sure those backups are um are properly taken and could have effectively be used to recover your cluster in the event of a disaster, which is what it's all about.

C

um So another important part of the architecture here too, is monitoring right, because we want to be able to detect problems before they manifest themselves in a big way right and that takes keeping an eye on the cluster.

C

One of the things crunchy postgres for kubernetes provides is an effective monitoring solution. You know it's a monitoring stack that once your database is deployed out there, you can continue to keep an eye on the health of your cluster and continue to tweak and evolve it according to your needs- and this also means identifying problems like I said before they manifest.

C

So you can get in front of issues with your data database and and uh ensure your data again is accessible as needed. Security is another big part of our architecture too. You know. Not only is it. You know important that your data is accessible, but it's important that access to that data is secure right and it's proper, properly locked down um and our operator solution um really builds in security from the ground up. You know, with all elements of our architecture, we're making sure at all.

C

Access to your data is done in a secure and controlled manner, whether that means enabling tls by default, using certificate based authentication, scram, passwords or whatever it might be.

C

You know our our solution is designed to be secure by default um and again when we're looking at deploying production grade clusters. You know that is important. That's a critical part of our architecture here you know, and the last piece here I want to mention- is convenience right, because you know at the end of the day, we want to make it easy for you to be able to manage your databases in these environments.

C

Right so say you have a production database out there that you want to clone into a cl a dev environment, so you can do a bit of testing or troubleshooting again, that is seamlessly done with crunchy postgres for kubernetes or if you want to customize postgresql or your architecture in any way possible. You know again, we make it easy to do through this. This solution next slide.

C

uh So, to sum up here you know again: we've touched on a few of these things already, but I'm crunchy postgres for kubernetes. At the end of the day, it's a fully declarative and get ops ready solution. You know we make it so you can easily define the architectures the other database architectures. You need.

C

You know in a fully declarative way and we take on the management of your database there to to create a fully seamless experience for deploying you know: production grade, postgresql clusters, you know, and that makes it you know that includes making it easy to get started too.

C

You know we want to make it so as soon as you spin up a new database. Not only is that database ready to use, but it's ready to be used by applications and consumers and end users.

C

So you know we do things like automatically provisioning secrets with credentials that you need, so you can just wire those things right into your applications without ever having to look up a username or a password or create postgres accounts again the idea just making it as seamless as possible to get up and running so your end users applications whatever they might be, can easily get up and running using this solution. You know an easy to upgrade. That's another big part of this too. You know, as kubernetes and openshift continue to evolve.

C

You know we're evolving along with it. You know, and we've baked in the the functionality to ensure that process is, is as seamless as possible for end users. You know, so you can easily upgrade your crunchy postgres for kubernetes solution. You know, as kubernetes openshift continue to evolve as well, get all the great new benefits that come with it. um You know with as minimal of effort as required or as possible, and at the end of the day you know when we sum all these things together. What does it really give you? You know it's.

C

It's production grade enterprise, ready, post rescue out right, you know, and it's ensuring we're. um You know allowing you to deploy the postgres clusters um that meet the specific needs of your enterprise, your specific requirements, their secure, lockdown. You know and make sure you have your data data available when you, when you need it, um that's really what it's all about at the end of the day, so um um so yeah. I think that pretty much sums it up in it.

B

All right thanks andrew.

C

B

Yeah so um I'll speak a little bit more about what andrew was talking about in terms of deploying the latest version of crunchy data postgres. uh Before we do that, though, let's speak about the storage, so the group, I'm in in red hat uh openshift our data foundation. uh This is um one of our main offerings- is openshift data foundation used to be called openshift container storage, and I want to point out here that this is very much built from upstream open source projects.

B

One of the main projects that does the orchestration that you may have heard about just graduated from cncf, as is rook and rook, is orchestrating the entire deployment of the other very mature upstream project ceph and the two of them together can be deployed in a kubernetes or or openshift environment, um and it is totally as as andrew is saying uh managed with within that environment and we'll take a look a little bit more later in the demo at some of the components.

B

But um you know two projects and being very close to openshift data foundation as a product. I can tell you that everything that we do downstream um is is basically created and tested upstream before we pull it downstream. So you know good good use of open source and then the the operator framework, which um you know, has become really prevalent and makes it really really easy to deploy.

B

um In the case of of what I going to show you in the demo deploying kafka um deploying postgres the crunchy deploying openshift data foundation, all this is is operator framework and um has has also matured. I'd say you know, I mean operators have been discussed for the last three or four years, but now I would say most most. Applications that are serious about being in kubernetes uh do do have an operator-based deployment and not only is an operator just to speak about it for a minute.

B

Not only is the operator deploying, but an operator is reconciling back to the state that you want, which is a very powerful concept in terms of of managing your applications.

B

So again, just to to look at the the the components we already spoke about, rook and stuff, um the the other one that we've included with uh openshift data foundation or used to be open shift. Container storage is an object gateway that has some nice features of being able to sort of bridge two different environments, say azure and aws, and have uh be able to mirror objects between two different clouds.

B

So all those three are going together uh to to form the storage solution that uh the postgres is is going to use as well as I you'll see in the demo that we're also have kafka so the so. The the way that we use the operator framework both for administrators and and users, is there's more than one operator.

B

So we have a what we call an ocs meta operator and that operator is going to sort of bootstrap the other two. So the the rook stuff is the one that I spoke about is an upstream effort and then nuba nuba core is also upstream, but the ocs operator is sort of you know again doing the reconciling doing the management, and it's it's always watching.

B

If, if the ocs operator is not running and ready, then something has gone astray and you need to to look at the other operators to figure out what has not deployed correctly. So it's a it's a multi-level management, but um works very nicely for getting all of the orchestration done and then maintaining and um also as as andrew spoke about being able to upgrade, which is one of the features of using an operator.

B

So getting to the the topic of this particular session.

B

If you look at the disaster recovery continuum, the the way we're seeing it um at red hat is that you you're, starting on the top right um back up and restore whether it be outside of open source and kubernetes, has been a solution for quite a long time. And the question is always: you know how much do you back up? How often do you back up and then from a restore pro point of view?

B

How long does it take to restore, and can you restore, you know back to a known good state, so it's it in terms of kubernetes and and what openshift data foundation has done. We've integrated highly with traditional backup, vendors, say: spectrum protect, plus or netbackup, or trilio castin, and we've done that via the the csi interface, which allows you to do snapshot and clones.

B

So that is quite mature. We released the capability to do that about a year ago, supporting the csi standard, and you know we. We can provide more information about that. But but as I go down, we have regional, dr we're calling that. Essentially, this is a multi-cluster solution and then metro, dr, so to have a little bit more context here.

B

As I said, for backup and restore this is done using openshift data foundation, the storage solution and we're we're doing it with snapshots and clones. So if I create a volume mounted to an application and then I want to have a crash consistent snapshot, I can use uh the the snapshot capability either either directly through the ui or a yaml to to back it up or I can hook it in to like.

B

I said, the traditional backup vendors are all pretty much now having a csi capability to both initiate the snapshots, as well as to use those snapshots to create clones for restore, and we have a whole list in red hat of isvs, that we've done particular testing and um created videos, solution guides and others that to to see how you do that. So, regional, dr, is a solution sort of in in development. um We have some pieces of it now at red hat, but it's really the idea that we have multi-cluster.

B

So the requirement here is, we have you know two clusters: uh kubernetes are openshift and we're going to asynchronously replicate the data for the persistent volumes between those two sites. Now there's a lot of stitching and a lot of orchestration that goes into that, because it's not just the data replication at the persistent level. It's also at the kubernetes object level or resource level that you need to um do replication some ideas.

B

You know how you could do that with with um upstream, is using like the valero backup and and restore apis, um but this is a solution that we we see um being available, um probably by the end of the year as a you know, sort of through the advanced cluster management, which will do the orchestration for creating multi-site and then be able to um also orchestrate the the data replication and the kubernetes um resources to be to be replicated from one cluster to another.

B

So getting to the left here down, there is sort of a from a storage perspective is a synchronous solution. So that's why we call it metro. It has to be um within a certain latency or distance. I usually say a couple hundred miles, no more than a couple hundred miles between the sites, and the other main thing here is that we have this arbiter or consensus location.

B

So the storage uh openshift data foundation, backed by staff, needs to have uh monitors which keep track of the cluster, and they need to have consensus.

B

We also know, or that ncd which is used for the control plane of open shift, needs consensus. Something like zookeeper, which is currently used with kafka, needs consensus. So you need a site that you can put a consensus node at and be able to keep forum, even in the case of a what I'm calling a data site failure.

B

So this is um a solution that is available now with the latest versions of openshift data foundation, and you know it it. It is a pattern which you do find where sites are a couple hundred miles apart and the consensus location can be probably twice as far away, maybe 500 miles, so it it. You know it's.

B

It is something you can do today to have resiliency and if we look at sort of what is the recovery time objective and the and the rpo uh recovery point objective, which is more about uh you know, did I lose any data?

B

It is possible to get a recovery point objective of near zero, depending on how how the applications you have really it's about the applications, because you know persistent storage is great and it can recover. But if you, if the application doesn't recover, doesn't matter that I have the storage.

B

So these are just some of the um sort of the ways that you could recover. There is one uh issue which it's not really a red hat or it's it's a community issue which is, if you are having a an application, pod mounting an rwo volume, and you fail the node that it's on the cube. It loses um status of that volume. It thinks it's still connected and it will not uh release that volume. So you get a multi-attacher and that is listed in a case there.

B

Workaround right now requires a force to delete that pod to to allow it to recreate on an active, um an active site, so just a little more sort of information about how this would lay out. So we see that, on the left hand, side we have the odf data replicas and we have the the monitors so at a minimum, each site would have um in this case two monitors and then you would have at the third site your your fifth model.

B

Your fifth monitor this is for the storage same thing for scd you could have a master at each site and then you have their third site and the the monitor pod actually can use a toleration to schedule on the master, even if it's unschedulable, so the other thing we have going here is we have the you know the the usual openshift infraservices those would be placed in in each data center.

B

So if we, if we lose a data center, um we see here that, because we have two data, replicas left uh the way that the the storage is configured for this arbiter mode is you. Every volume will have four replicas, as we showed in the prior slide, but every volume only needs two replicas and monitor quorum. To continue to be able to serve reads and writes so assuming the application survives, um then you're going to be able to continue to to take inbound connections at a very rapid recovery.

B

Again. This is based on how your application recovers, but the storage will recover basically almost immediately one way to make your application able to be what I call zone aware is a it's a relatively new capability. I think it came out in kubernetes 1.19, but it's been available since openshift 4.6, it's called topology spread constraints, and it does require that you have replicas.

B

um If you only had you know one instance of your application, it wouldn't really matter, but as long as you have two replicas, um you can use uh the topology kubernetes zone label to to spread your your replicas for your application among the zone. That can be a hard affinity so that you know you in some cases you don't want your your replicas to to fail over to the active zone, because the next time, if you fail you, you know you could lose a zone and all your replicas would be on the same zone.

B

So this is a really powerful concept um in the demo, you'll see where I've applied it to to kafka, and we also post crunchy postgres uses a very similar capability.

B

The application that we're going to take a look at to sort of challenge ourselves here is called smart city, green city and some of the colleagues in my group in openshift foundation.

B

Created this uh demo- and it um has a lot of pieces to it, so it makes it sort of challenging to to recover. But what we have here is um if it was deployed, the demo doesn't have two different open shifts or multiple openshifts, um but you have an edge environment which is basically collecting whatever data you're trying to collect, whether that be image, state or other data, then you run it through a model, in this case a license plate retrieval model and then you're going to basically put it onto the edge kafka.

B

Bus go through coffin, mirror maker, come down to the core kafka bus and then various applications are going to pull from that and then kafka consumers are going to basically take the messages and then write them into the crunchy postgres database. And since I don't really mention in the demo, I just want to say that um I deployed this using the latest crunchy data, postgres version version 5.0.

B

It has a new. um I think it's a new custom resource, postgres cluster and I found it um extremely. I mean I'm not making a pitch more than what editor did, but I did find it extremely easy to do to create a replica as well as make the the replica placement have a zone anti-affinity, and it was a really pretty easy experience. So this is using the latest postgres or crunchy postgres version five, the other thing.

B

If you've used crunchy postcrests before there's no longer, um you don't have to install a specific pgo client for for doing the kinds of things that andrew went over. You can use a cube, cuddle or openshift cli commands to do to do everything.

B

So following this second stage, so we start with uh you know getting the data at the edge um move it through the mirror maker.

B

Maybe do some some special things with the data before you pass it on and then there's there's some object, buckets involved where data is stored and pulled from and then eventually we're going to do some calculations- and um you know similar to I live in california.

B

I don't know if, but anyway, all of the california toll locations as you go in and out of san francisco area are now doing this and um they they. They basically recognize your get your license plate and then I I have an account and they charge me for going past. You no longer stop at any uh toll toll boost so for this situation, we're going to um challenge a few of our applications.

B

Crunchy data is um like I said it has a primary and a replica, and each one of them and we'll see in the demo how they're made to stay in their particular zone.

B

And we're going to fail one and see see that it switches to primary. The other thing going to do is uh kafka, so kafka is part of the solution. If kafka um is not able to recover, then it wouldn't really matter, because it sort of you know, has the messages and it's providing all of the other apps with what they need. So the way that did uh you can make kafka, currently is you you need to have zookeeper has to have quorum. So again, using a toleration.

B

I placed one of the zookeeper pods onto the arbiter consensus location on the master, so that zookeeper quorum will be kept. So if we lose two of the kafka replicas two of the zookeeper replicas we've still got, um and and within the coffered cluster config, you can set a custom kind of attribute that allows you to say that kafka can continue to operate with two replicas all right.

B

So I think, where it's demo time, the demo today is going to be for an application that is called smart city and, as you can see, it has a lot of different um apps that make it up the main components that we're going to look at being resilient is the kafka core cluster, which is in the bottom square, the postgres database using crunchy and the storage, which is not shown here, the storage from openshift data foundation.

B

So on the top we have what would be called edge locations. Those would actually be separate openshift clusters, but in the case of this demo, we're going to emulate that with uh what we call a safe node, so everything on the top will be on a single node that doesn't fail and on the bottom. This is where we're going to be failing, openshift nodes to see what happens.

B

So if we go to the openshift cluster and the the console, we can see that we have quite a few operators, starting with the red hat amq streams. That's going to give us kafka and we again we have two kafka clusters edge and core. We also are going to use grafana to look at the data and be able to see that the application is running.

B

Local storage is used by openshift container storage to create the storage cluster using ceph, open data hub operator is where we get the we launched the grafana as well as superset, which is going to be used for a dashboard that data scientists would use.

B

If we want to look at further the components we can go ahead and start with the nodes.

B

We have three masters and we have five worker nodes. As I said, the one on the bottom is the safe node. So it's not going to be failed. If we wanted to look at how they're divided they're divided with a topology label- and this topology label is also with all of the components used to be able to be zone aware and if we look at the the arbiter, this would be.

B

um This node would be at a location, a third site so that it could actually act as an arbiter and reach consensus for both xcd, we'll see zookeeper, as well as the um the storage. The the monitors need consensus. So this is our consensus or arbiter node.

B

If we then look at how the data nodes are divided, we have a label for them called data center. One data center, one has two worker nodes in it. The kafka core pods are here: the storage pods are here and then the last one would be the data center 2 and again it's going to be the.

B

Using the topology label, we define a second data center, so we basically have three zones an arbiter for consensus and then two uh data centers. That would reflect different sites that were within a hundred to two hundred miles apart.

B

We also can take a look at the pods and for the pods. Let's start with the edge kafka.

B

So here we have three that are going to be um our edge cluster and in this case, as I said, they're all on the same node. This is a safe node. That is not going to be powered off, and this is, you know, emulating a another openshift cluster that would be um collecting, essentially the the data putting it onto the kafka bus and then getting it via mirror maker over to the core.

B

If we now look at the core, so these are the core each one of these is using a openshift data foundation, volume for its storage, and if we were to look over here to the right again, the nodes are all different. So these are the four nodes that represent data center, one and data center. Two and there's a kafka pod, spread on each one. The way that that is done to make sure that um it's just not, you know arbitrary, how they're done it is done with topology spread.

B

So this was added to the kafka cluster. This is something you can add, and the operator will use that to to schedule. So we can see that we're using the topology kubernetes. I o zone in the top we're using it to make sure that that there are pods on both zones and then the second, where it uses the kubernetes io hostname that is to spread. If there's more than one pod per zone, then it will spread the pods among the hosts. So that's exactly what we saw.

B

We saw four kafka pods on four different hosts, so we know now that um the kafka is spread there. Let's go ahead and look at the storage. So for that we're going to use label.

B

And we'll start with the actual storage devices.

B

So there's going to again similar to kafka there'll be one per node, and this is again using that topology label and the way that the storage works is it is configured to for every volume to every volume is created with four replicas, but the minimum size that a volume can continue to support reads and writes is two, so we can lose two of the four here and the storage will still operate totally fine.

B

The last thing I want to look at here is postgres, so we have two replicas here. This third one is for backup, but these two replicas are again using topology spread and we can see here they're on two different nodes. If we inspect the how the the placement is being done, we can look into the animal here.

B

Here's uh how the topology spread is done. First, we're we're, placing it on a node, that's in either data center, one or data center, two and then second we're using the the pod anti-affinity based on the topology label. So now, let's see what happens when we have a failure.

B

So we've got a couple of terminal windows here and I want to explain what what you're looking at um top terminal window is just showing a view of the nodes right now um we have three masters. The next two workers are in data center, one by label, and the next two are in data center, two again um and master zero. The very first one is the arbiter consensus, node and it would be placed at a third site.

B

It can be at a higher um what we'd say, latency or distance, so it could be. Maybe you know 500 miles away, maybe more and then the two sites that have the the data center one data center- two they need to be. You know not more than maybe a couple hundred miles apart and then the last node is my safe node uh right under. That is a view of the two crunchy postgres pods and they are because of topology spread or pot anti-affinity.

B

They are in each a different data center or zone, and one currently is master, the one that is in data center two and the one in data center. One is a replica down at the left is a view of the storage. This is actually what we call stuff status.

B

It's looking very good right now, all of our monitors are in quorum and all of our storage is up on. The right is a kafka cat which is um showing us. The message on the topic called lpr license plate retrieval, and we can see that right now, um we're definitely continuing to get that. So what? What does that look like from a dashboard point of view?

B

This is what the grafana dashboard is looking like, and this is obviously a demo, and it's not really london right now, but um if you notice uh the the car images will be changing. That would be. You know: cars going by these locations, an image is collected, and then that goes to the model um through the edge hand it over via the kafka mirror and to the core and then store it in the database, and then this this particular grafana dashboard is pulling this data from the database.

B

So what we want to do now is we want to create a failure and see how long it takes to recover.

B

So to do that, I have over here v sphere and we're in a cluster called perf1, and what we need to do is take down data center 2.. Our primary postgres replica is on data center two, so we want to go ahead and take that down to show you know, sort of the worst case of what could happen.

B

It'll also impact kafka, and it will definitely impact the storage as well. So I'm powering down two nodes not going to power down the master at that location just so that we continue to have access to cli.

B

So now on the bottom right, we can see that kafka cat has stopped um we're also starting to see some things happen in the storage here. The storage has gone into worn and we are seeing on the top um that we have at least one node. That's not ready soon, we'll have two: it takes about 60 seconds. There we go and we can see that down and again the opposite right. The kafka is starting to recover um in the middle there, with the the the crunchy uh postgres replicas.

B

Now we have two masters, so it has already switched over and made the the opposite zone. Now that is active the master and we can see that kafka has slowly recovered. Now, if we look over to the left, um we can see that we do have. We still have quorum on our mons because of our consensus site, so we've lost half our storage.

B

We basically have switched replicas on postgres.

B

Kafka has totally recovered now with two of four replicas. Now, if we go back here, um it doesn't look like we're quite rick up there. We are so we we've now recovered and we are now continuing to receive images from the edge and this data is being pulled from the database. So by the time we got back here it had already recovered.

B

The recovery is somewhere in the range of a couple minutes. It's um it's quite quick.

B

So just to prove, though, that everything um really is down, we can look at the the pods and see what their status is now so that'll be rook dash, cef dash osc, and we see that we have two of them and pending, which means they're they're not able to schedule because they need to stay on the the zone. That's currently down there. The the anti-affinity is a strict anti-affinity for zone.

B

The monitors again are used to keep track of the cluster and again we have two that are pending three that are running important. One of them is on our consensus, node or arbiter zone at a third location. So it is, it is keeping the quorum for the uh the storage cluster.

B

Let's just go ahead now and look at postgres. So we'll look at the the label here and we have one in a terminating one running. The one in terminating is on the data center two and the one running is on data center. One we saw in the terminal that switchover was very, very quick within seconds and then, lastly, if we just look at kafka, we see that we have two running and those two are on data center. One and the two on data center two are in a terminating state.

B

um The two that are they're working are using openshift data foundation, um uh what we call rbd volumes and um that is keeping the the core kapha cluster, going the two replicas. So, in summary, to run our smart city application, we have, um the the images are changing, the counts are changing and we were able to recover. Thank you.

B

So I was just going to say these are some resources if you diane's going to make the side set available and there is a readme for setting up the smart city demo, it's not currently configured to be highly available, as I did, but uh certainly, if you're interested reach out to me- and I can give you the the deltas for how to make it highly available.

B

We also have uh a guide a couple guides on how to configure openshift container storage um for uh using an arbiter and um how to recover so and then uh andrew. Did you want to talk about your resources.

C

uh Yeah sure thing so the the links there should basically bring bring you to the uh the postgres operator um documentation. So you know the the bullet point. Second from the bottom there um that should take it. Take you to our various documentation. We have out there and then that final link is a repository where we have some examples. So we have basically a suite of examples out there that demonstrate different use cases. um You know of how to get the postgresql operator up and running um across a wide variety of different use cases.

C

So it's a great place to start for anyone, that's looking to experiment with the operator, so yeah.

B

That's the thing: that's actually the section I used as well. Again, if you don't see something um I know right away, I didn't see how to make it. uh The replica is unaware, but if you know feel free to reach out to one of us, if um you don't see something in those examples.

B

Okay, well, I think that's it diane.

A

There's one question that just came in um and and I and it was probably the question- I asked you guys beforehand, because I love the demo, the smart city demo, because you know everybody's been through a television with everybody's head, there's their license plates scanned somewhere. It's always interesting to see how it it all works in the background too, but um dan's sort of asking um at whether or not red hat consulting or maybe the data foundations group has done any of this as a service set it up as a service. Yet.

B

Not yet but um good question we, we are looking to take uh this sort of this. This is one of the demos that we're working on, but we're looking at taking these forward working with the openshift data hub group and and such so yeah again out, but yeah we're we're we're going to make these more available right now, it's just to read me, but um but looking to make them available in a in a more persistent way.

A

All right, well, I know a lot of work went into getting this um up and running and um making this demo work and making it aha. So I really appreciate the time you guys took to do this and to walk us through this and as always, crunchy um thanks for all the work that you've done, um one of the earliest users of the operator framework and getting postgres operators and helping us debug the early days of the operator framework.

A

So you have a place that's near and dear to my heart always um so I will definitely have you back and we'll be doing more of this um data open source, openshift data foundations, um talks um in the up and coming future. So look for that and we'll post this video up on youtube with all of these wonderful links and resources for you to get started. So please do reach out to annette and to the good folks over at crunchy, and we will look forward to hearing um about your use cases for this. These applications.

A

So thanks again everybody and have a great week.

B

All right, thank you. Thank.

A

You thanks tonight.

A