Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deconstructing Apache Hadoop: A Dialogue About a Cloud-Native Refactor - Anu Engineer & Marton Elek

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deconstructing Apache Hadoop: A Dialogue About a Cloud-Native Refactor - Anu Engineer & Marton Elek, Cloudera

Many existing applications like the Big Data stack is in a community debate over moving to K8s. They see it as the future, but there are lots of unknowns and complex problems to solve. This is a dialogue between two developers who are exploring and trying to finding the true path to becoming Cloud Native. In this talk, we will discuss in detail our experiences and issues that we have faced in moving the Hadoop Object store Ozone as a Cloud native storage solution that runs on top of K8s. We will take a deep dive into the advantages and challenges that we faced during this journey — having a cluster manager like K8s allows us to move away from managing physical details and instead focus on the storage part. Some of the challenges were things like security and how we tackle that.

https://sched.co/MPYI

A

Okay, hi everybody. Thank you for joining. This is a talk about cloud native refactor of one of the high.

B

Mass run, how are you oh I'm,.

A

No see sorry one moment: oh hi Anu, how are you I'm, fantastic I, haven't seen you in the last five years. What are you doing nowadays? Oh.

B

It's been a while since I met, you two are these days: I work on something called a patch, a ozone which is an object store based on the Hadoop storage layer and it this supports s3 protocol as a primary communication mechanism. It sounds.

A

Very good, is it as something like the Hadoop HDFS you are? They have some storage right, yeah.

B

We do it is the next generation of HDFS, which is primarily being rebuilt, for you know, scale, consistency and things like that. Yeah.

A

That sound sounds very good yeah because and I'm just I'm just trying to find a good object store. You know that it's maybe you have one or really.

B

Yeah, of course, you know anything that I write is really good. So if you are looking for a good object store, this is the place to come, but you know I'm just curious. What are you looking in an object store? What do you? What is your use cases well.

A

Okay, first of all, I think you got it so we are just playing roles, so this is the disclaimer so but the main reason what we are doing this, because I think this is the most effective way to tell you our story. Yeah.

B

So in the last couple of years, we've been working on taking the most successful distributed file system and make it work on kubernetes and in container and in cloud nary way. So we are going to take you through the journey of how you take a real application like HDFS and pour it over and make it work on something like kubernetes and one.

A

One reason why I think it's very important to talk about this one because I don't know if you, if you know the feeling, when you find a new technology and according to the according to the getting started guide, everything works well, because the hell over can be deployed with the new tool, but a real application.

A

It's a totally different story, so I think the head to pizza for the new version, the head to paws on it's a very good real, distributed application, because if we can do it, we can use it in a cloud knitting in the environment, then I'm pretty sure that any other state fool distributed. Application also can be used. So in fact, but we are both Apache Hadoop commuters and we are working on the hydro-zone. But no, oh sorry. What was your question? Oh.

B

I was just asking you: why are you looking for a new object store and what is a use case? Yeah.

A

You know that I'm working currently with machine learning models, and we just decided that we should move everything to the kubernetes I'm, pretty sure that I don't need to explain it. Why? But kubernetes says that bring your own storage and I have no storage, but I can bring so I'm just looking for a good storage- and maybe you can you can help me absolutely.

B

uh We do have an excellent storage. We have the reputation for it now in the most widely deployed, distributed, storage and I hope that you know we can carry it over to Oh. What is this? It's.

A

A very world yeah, that's just one question and and I can use your storage. Is it cloud native or or not? I have.

B

I have no idea; I've, never I mean what does cloud native even mean. Okay,.

A

So I have a plan: I need a storage and I'm pretty sure that your storage should be run in a cloud native environment. So why don't we do something together? So you know all of the details about the hood, Hadoop and distributed system. I can help with the cloud native environments and let's check that. What if.

B

You say so because I have absolutely no idea what cloud native is so, let's explore. Okay,.

A

So yeah, let's maybe we can start with the cloud native. It's very easy to explain right, because the only thing what I need to explain is a Google, and this is from the CLC Earth Charter. So the cloud native technologies empower organizations to build and run scalable applications in modern dynamic environments such as public, private and hybrid cloud solutions.

B

Seriously did some marketing I write, this I mean it doesn't make any sense to me. I hope, I think you cannot even repeat it without looking at it. Oh so.

A

B

Really sorry yeah.

A

I can get some help. I.

B

Know a lot of you guys who actually understand this very well, and you know: okay,.

A

I have a little definition before trying to review slide, so this is from the CNC F dot IO, and this GOG native computing uses an open-source software stack to deploy applications as micro-services packaging each part into its own container and dynamically orchestrating those containers to optimize the sauce utilisation.

B

Yes, yes, so I do understand, open source michaelis, I was in container and you know no pun intended, but it seems like a definition written for an engineer. My name is engineer, but you know still, oh, but still there is this part where you know this is too generic? I mean I, know these words, but what does it really mean to do a cloud native application? Maybe you should explain it like I'm five, maybe actually.

A

There are more than one approach is to explain, so the only thing what I can explain you is that what is cloud native for me? What is my my personal vision about about cloud native and.

B

A

Let's start it, you are a head of developer right. Yes, so you have releases, you upload the releases. The sources are there. There are binary tour files which contains all of the required drawer files and components and everything. So how can I? How can I run it in in kubernetes or Amazon Cloud under.

B

It and Sharon: that's how we do it. That's how we do it. Oh I.

A

Know that you do it in this way, but I would like to start it so any help or is it my problem or your problem.

B

A

I think that is the big difference. So as a developer, this is my responsibility to to support it to run in any kind of cloud, environment or kubernetes or just locally. The docker compose so I think this is a big difference in the in the cloud native area that this is not just a tour anymore right. We need to support it to run it and one way to support. It is what which one is very important for me to create the connection. So this is a team game right.

A

It's not just one application in a tar file and do it your own self, but one way to support it and run in multiple environments to make the connection and make it easier to run together with the existing monitoring logging application or in existing closed you're.

B

Basically saying we have to be part of the environment and be a good citizen to play in this cloud native or a good team player. Actually so.

A

This is a team approach as well, so I think this is very well alone and I can just you show any other close, the UI of any other cloud providers. So this is the usual user experience for a cloud provider right. You can just click click and you can start a new machine and everything is clear. What's going on and I think this is the user experience what we are looking for and this the before this, so this picture to connect or or project to all of the others?

A

It's um it's not just because we like to create connections, but we would like to provide similar user experience. Mota UI, it's not about the UI, but if we can provide the same usability or same observability even with just command-line tools, then I think that is could be very useful. So.

B

You're saying there's a large amount of consistency in the user experience where every software has some shared tenants, which we can all share. Is that what you see yeah.

A

So for for me so again, this is just for me, but for me, the cloud native, the biggest step in cloud native, that this is a improvement on the user experience and the users are also the administrators and the opera operators. So I think that's what we what we need yeah.

B

I heard somebody say yesterday that you know in the new Cooper Narus world your tools will change, but it's for the better, because you know you'll have more consistent, tooling and usage across all your systems, whether you are using Apache, Cassandra or Apache HDFS, or some other tool like cockroach DB you'll have a much more uniform experience, so it'll be good to play there and right now. This is how ozone looks like you can. Kubernetes, with cuboidal apply f docker.

B

If you want to bring up a pseudo cluster to play around with on your laptop, you run compose up and, of course, we still believe in running it on bare metal with the normal start, functions and I.

A

Think this is a very good example for this connection, so that the traditional Hadoop can be used from a spark high, but only with the Hadoop interface, but the real storage should be used with multiple type of interface right. The connections and one connection is the AWS interface. So this is the common language. Like the English write everything everybody speaks AWS CLI interface and.

B

This might come as a very great surprise to you guys that you know. Martin always tells me that Twitter is a foundational influence to our distributed systems, because we want to make sure that any of these startup lines fit in your tweet. You know it's a lot of times. We like acknowledge tutors influence on distributed file.

A

I think a good getting started guide should be fit in one tweet and not because we are hiding the complexity, but because it's we can manage the complexity. Ok, so I think we managed the deployment start. At least we can. We can start, but I have some concern that what will be happened after the starting, so that network could be one of the one of the problem really network.

B

I think it's a solved problem. What could go wrong.

A

Ok, let's try to understand first that what is what is inside your project, so how can it work? Oh, it's.

B

Pretty straightforward: there is a object, store metadata manager which contains a it's called ozone manager. It manages all the namespace, so objects like buckets and keys are managed in the namespace layer. Then there is a Storage Manager, which is a block level manager. So there is a separation between the block manager and the namespace manager and the actual blocks live on data nodes. It's a pretty standard architecture for any distributed storage system. Ok,.

A

I'm trying to repeat it, so you have some kind of slave nodes, which has the data and all of the others are managing the data and saving the metadata okay. So it sounds good, but there I have just one question that: how do you use DNS or is it a requirement for you? Yes,.

B

It is um what about DNS Oh.

A

Nothin problem with the DNS: it's the question that how do you use it right? We just.

B

Look up our name of an object. You.

A

Think that the DNS can be used just in one way right. Just click to the button.

B

Assume that the machines are already pre-configured, ok,.

A

Let me explain the problem, so let's say I have a kubernetes cluster, so you have multiple slaves right and you would like to scale it up and down. So this is in the right side. I have three instances and by default the instances have IP addresses no DNS all at all, because we separated the DNS from the instances. So you have one DNS. This is something like the load balancer and behind a load balancer. We could have multiple instances. That's very nice, yeah.

B

It sounds good, but it's a stateless applications. Only stateless application can do that. Yeah distributed file system cannot actually.

A

I, don't think that is it's about stateless or stateful. What I think is that you have a very specific use case because in a normal use case, all of the services are equal. But in your case you would like to get an information from a the metadata server that where the data is stored and you would like to directly use the data from the right data node, because you have a very sophisticated application, algorithm.

B

That is true, yeah.

A

So that's not about the problem of DNS, but that how can you access? Because here we have a DNS, but the instances itself can be can be used only by the IP address. But this is not about the DNS. So it's not about I know that it's a very old protocol, so it reduce it. Yeah yeah.

B

It shouldn't be able to be NS because it has been working from 1980s, ok,.

A

Still, ok, it can be used into kubernetes, but let me give you an other example, so you you mentioned that you have two metadata server right: that.

B

A

And is there any dependency between them? Yes,.

B

We have a very strict startup sequence: we bring up the SEM which acts as the cluster manager and then all the other services join onto it, like the ozone manager or the data nodes join, and then they become a membership protocol where we can.

A

Learn: I don't understand that it's about what I understood that you have one service which should be started at the beginning, right. That is.

B

A

You have another service which would like to connect to the other service right. That.

B

Is correct? Okay,.

A

The only problem is that I can't guarantee that they will be started in in this order. I just.

B

Thought of the deal was that kubernetes was an Orchestrator and my mind when you say Orchestrator I should be able to say just start this and then start that, just like an orchestra, yeah.

A

And how can you orchestrate me in this way, I, don't know.

B

A

Of containers just thought this: yes, so we are starting everything together in the same time, and this is your responsibility to be prepared if the DNS is not yet available, okay,.

B

So this is right, retry loops everywhere, because we have no consistent path to define.

A

That's a very good practice actually- and this is just a small modification. So it's not about not a very big problem with the DNS, but you should be prepared that in case of 30 try, you should invalidate your DNS cache as well. That'll.

B

Make sense, oh, it would have been nice if kubernetes actually had a way to say: hey here is a o CR, D and I. Have these dependencies and I would like to start it in a certain way. Okay,.

A

Let's focus on what we have now alright and another problem could be reverse DNS because, as I remember, had two pews these Kerberos, which depends on the DNS. So maybe we should check the security first and and let's talk about the security well, what kind of security do you have yeah.

B

So you know we rely on Kerberos, primarily because we need before we let somebody read a file. We need to know who you are and Kerberos is the de-facto protocol, where we can recognize who you are so you know, there's a directory backing it directory and stuff like that. You can configure ozone to work against it and when you read or write your access permissions are checked. So that's how it works. Yeah.

A

Actually I would be very happy to say that all Kerberos, it's so old school and it's so I, don't know it was used only that it at the previous century, but I couldn't say that we have anything better in the kubernetes I.

B

Have to disagree so strongly because it is one of the few really working protocols out there. I don't think there is a replacement for Kerberos out there. I don't know what goo Bernard is, but maybe kubernetes has invented something itself, but I don't think there is any user identity system which can replace actually.

A

Actually, what we have is the role based access control, but this is for the kubernetes resources, so it not for you it just for in the container Orchestrator and we have some kind of secret management, but yeah I need the real security or your project. So, but let me have a question so, as I remember the biggest problem with a very good Kerberos that for installation it could be very painful right if I, remember well, I should generate this key type.

A

Files and I should copy there, and this kubernetes is not this scoping game so completely.

B

Agree it used to be an extremely big pain point housing data nodes. We expected you to put a key tab file which acts as your identity inside the system because you're a service, nobody can type in your password and user name right. So you put that in the file and make sure nobody else can read that other than the service. So you had to do it 3,000 times before. You could go ahead, but that has been addressed in the ozone world, where we only use that only on two machines.

B

Only the metadata servers need that when ozone boots up it creates its own certificate. Infrastructure goes MPLS and uses a token based management based on Kerberos. So all the identity is managed that way. So, yes, we still have that pain point, but it's only two files: okay,.

A

So I'm not sure if I understood, but it means that I don't need to use the Kerberos key tops at all for the servers and.

B

None of the slave nodes need Kerberos, so.

A

Just for the clients and I can use the certificates everywhere. That.

B

Is true, just all the clients and the matter to measures.

A

That's good, so still we can. We can make the deal so okay, so the security is sold in kubernetes. An other way to provide transparent network encryption. Is the service mesh? Do you need one I have.

B

No idea what service meshes Oh Google.

A

So I can just explain it very easy, so this is from rest, IO IO. The term service mesh is used to describe the network of micro services that make up such applications and the interactions between them. Okay,.

B

You lost me again: I'm sorry, maybe I'm dumb.

B

Might be a very good idea: okay,.

A

Let's say you have containers right, you have different services in ports so and let's say you put a proxy and the reverse proxy to each of the pots, and the only thing what you know what you should do is just use this proxies and reverse proxies for all of the network communication. That's also just proxies reverse proxies on the service level and I'm sure that your question is about.

A

Why is it good yeah the main advantage of this one, because we can manage all of the reverse proxies as proxies from one centralized location and you can provide additional function? Functionalities, let's say blue blue green deployment or you can transparent a network encryption or you can just define any kind of security rules you can you can you may get more detailed, metrics, so fine right? It.

B

Sounds interesting but I wrong. I am I'm. Also can I also concerned about. You know the number of network hops in an architecture like that, because a distributed file system pushes too much data. It is not, if you're a database, you still can do that, but because it's data packets are small, but when you're streaming large amounts of data. I wonder if it'll hit me, but you know that's small.

A

Usually I have no problem with the Hawks if, if the hopes are olena right, so my hope that if nothing is free, so the complexity itself can be a problem. So service mesh is very, very powerful. We can do almost anything but to islas service mesh. For example, restyle I think you should install about 50 custom resource definition and more than 100 kubernetes resources so yeah. It provides some kind of visibility, but on the other side, some kind of consult still just installing I, didn't configure anything here.

B

Yes, so one of the other things that we found when we looked at this was the fact that you know traditional systems are used to the fact that there is something called gssapi and configuring Kerberos at a network layer is been removed, like it's very easy, I'm used to C or Java or stuff like that. I can just go, write this code and works where the G RPC up till now, I'm hoping that kubernetes moves so fast that one day kubernetes says.

B

Ok, we are going to support Kerberos authentication or a GSS like API on the G RPC layer, but when you bring in an established application with lot of security considerations missing out on Kerberos hits you really hard. In fact, you know there are some of the reasons why our protocol uses G RPC, based on whenever we use tokens. But when you do that one part of authentication between users to the metadata server we have to flip back into raw TCP. So it's kind of a bummer, but that's what we do so.

A

You said traditional applications, but I have an other kind of classification, so I think there are true kind of applications, the infrastructure applications and business application and I'm pretty sure that this ozone storage, it's an infrastructure application and it's slightly different from the business application, so, for example, in in ozone, I'm pretty sure that all of the all of the services are same. We have the same library for network protocol, so, for example, we don't need to we.

A

We don't need to use an external proxy just to catch all of the all of the networks, because one very big advantage of servus mash that it can work with heterogeneous environment right. Even if one application is written in gold, the other one is Java. You can get just catch all of the.

B

I I buy into all the advantages, but it's simply not easy to lift an existing application with a serious security consideration and just say: okay, let's go serve a smash. Let's go somewhere, you will probably have to make hard decisions. I.

A

Think we agree so I think you need just a small subset of of the service mesh functionality and maybe it's not the best way to introduce some kind of complexity just because the few required functionalities yeah. But if we don't use the service mesh, you should answer. What is the the still? We should answer the observer observability question, because one of the main advantage of the service mesh was that we can. We can check what's going on. So what do you have here? Oh so,.

B

What we have is a bunch of you know homegrown solutions over time, because we started looking for babe. You know much before CN, CF and other stuff came into being, so maybe that's a place where we should look at what to do. Yeah.

A

That's exactly so. This is what we already discussed right. So the good thing to connect to connect all of the components and be a team member with our project. It's not just to make it more popular, but it could be easier to achieve a few goals. For example, we have very powerful tools for log e log collections or matrix collections. So let's quickly go through what you have, and maybe we can just improve it a little bit. So what do you use for metrics? So.

B

Hadoop supports custom matrices and we have you know just a sink into which we write with no consideration where it goes to so it is, you know, very agnostic to water technology. We use okay.

A

I have no problem with the custom, metrics I think this is promote else. It's pretty clear that it should be used, because this is one of the most popular member of the CNCs ecosystem. So the only thing what I would like to ask from you is provide all of the numbers on an HTTP server. It's very easy right to have CCTV page I will define the format in a pro Mattel's friendly format, but it's a very easy format that.

B

Sounds like an excellent plan. I think.

A

We should use produce.

B

A

If you can do it, I can give you unlimited number of powerful visualization tool and you can show everything in there. Oh, okay, all of the metrics in your word, Oh, perfect. Okay! So let's go forward for tracing I think this is the pretty common question that what's going on. Why is it.

B

So uh Apache Hadoop has a really sad story here we used to have our own tracing sub project called Hadoop tracing, which was abandoned at some point and now deprecated, so I, don't think our tracing really works. I worked for that matter. More.

A

And more sad stories, it.

B

I'm not them, but you know this is one of the very sad stories.

A

How can we get the happy end with this approach?

A

Okay, so for tracing I. Think in the kubernetes order, CN CF area- we we are on the other side. We have more than one solution actually and the revealed emerged sooner or later, and but until that I think any of them are very powerful and very easy to do. Put two existing distributed application, so I just start suggest to start great any of them. So that's.

B

An excellent idea, in fact, you know we should just go into one of them and I'm, not exactly sure which to pick or maybe it's open, telemetry now or.

A

You can just use a coin right, okay, so last question: where are your log files? We.

B

Just ready to log4j and pretty much use, no collectors, you can configure the end users get to configure your collectors, and you know what is.

A

B

A

The best way, what I can do it's my.

B

A

Finally, you can see I.

B

Will say the same thing about the tarball, and you said it's not very good. Okay,.

A

In this case, there are many ports.

B

A

There are many powerful tools, so the only thing so this is the V means situation right. We have very powerful tools, just Co electorial, all of the Ryoka, all of your locks, so just look to the console, for example, or one file, and we can just forward it to at Central, Pacific central place with flu and E loci or anything. So, okay, I think we solved the problem. This.

B

Happens to be one of my most favorite parts of the work that we did primarily because having worked through a distributed file system and how we integrated the cloud native technologies and how how different the user experiences it's one of the few things which I think has positively influenced our file system. The fact that we are able to go look at other solutions and say: ok, everybody has their own speciality, we store files.

B

We give it back to you when you need it, but these other systems specialize in other things- and you know this whole, like Oh system I mean the ability to leverage. That is the most important thing. I mean you know. The strength in the numbers is what makes cloud natives so wonderful to work in so.

A

Small modification and very more happier develop.

B

The cost of doing in most of this on the screen is so low that even you have a really old legacy application. We would still recommend you to go and integrate all of them. Ok,.

A

So I have we have one more remaining task that the developers are already happy. I would like to be happy as well. Yes,.

B

I, what is that crime scene investigation.

A

Ok, yes also, can have to be happy but CSI, so you have something which can be started now in the cloud native environments.

C

And works very well.

A

Yeah but I need your storage. Do you remember you.

B

Need my storage and we have s3 interface, ok,.

A

So this interface is just for the storage. This is the container storage interface, vendor-neutral interface, or what.

B

You should stop using google and explain like 5 yeah.

A

I was pretty sure that that way we are not the next question. So let's say you have a storage system. I have I'm the container, Orchestrator and I will ask you to I will actually okay first start with the volume creation, so I will ask you to create a storage. I need five gigs. Can you please create one sure? Okay, that's a controller interface of CSI. It should be run in one place. So it's something like the master yeah.

B

We have a master okay.

A

So how can I use it from from the node or containers, so I will have a second question. A second request that can you please mount it somehow I can I can define the destination bus that could you please mount the previews previously created storage to this directory and all of the other tasks it's on on me, so that.

B

Is quite possible then.

A

We are fine again a small modification right and we are fine well.

B

Generally I would say this fine, but you know, CSI does have some issues and I know if there is any CSI committers here we struggled with flowing the identity through and I know that CSI generally is designed for a volume and volumes do not have identities, but you often talk about you know: file system interfaces like NFS or you know, object store and when a file system interface, the user identity, is extremely critical.

B

So the very fact that we have to go in and say: oh, when you're doing CSI, we will run in a administrator contacts. So what allowing you to do mount CSI? This means that it is very easy to. We are basically punch a hole in an extremely complicated security system that we have already built. So it's like you know it looks like CSI, is very new, so I know I'm, not saying that it's a it is a deal breaker for us.

B

We will support it, but it is a little more to go before it can be used in an priced place, because if you build something with that and if your user identity doesn't flow correctly, mostly somebody who's in your security group will come and say what are you doing? You cannot Bhagat.

A

I agree that there could be problems and security is a very important area, but I I. Don't think that that CSI is the panic problem. So DSCs I have a specified fide interface, so I.

B

Know I'm saying that there's a missing hall-worthy, it's very difficult to flow the identity through the CSI. Yes, I.

A

Think, yes, but I think it's most important to implement the data path because it's not defined right. I asked you to mount your story.

B

And I must say that she attended an amazing job where he said data is your problem? Go read and write gives us perfect freedom to do whatever we want. It's in the control plane that you know how.

A

Can you serve the day? The problem for me is the.

B

Assessed works, you just bound the volume and read and write, and it is how.

A

Do you mount my volume, oh.

B

We use a CSI fused driver which mounts this and allows you to write, read and write, and we are also going to support NFS and native fusil Aptus for more performance. Oh.

A

I like performance, oh, it sounds good.

B

A

It seems that we moved to a common direction right so now, I almost have the storage. Your storage can work in the in the kubernetes, and the original question was that what can we?

A

What can we get from this work where we are working where we can work together, and this had two peppers Gio's, and so my original expectation was just oh, this is a tiny salt technology and it should be refactored from the beginning to make it easier to work, but it turned out that with just small modification, very smart modification, small modification, it's a very first class citizen of the kubernetes. Yes, yes,.

B

So that is one small thing that I want to add. It might look like it was very easy to take an existing application lifted, because the stuff that we talked about is very simple, but we started our journey really early with kubernetes, probably at 8'o and we've been continuously testing. It takes a really long time to build a distributed file system and make it stable. Okay. So so at every point when, because we use so extensively, the testing and everything was done in kubernetes at every point, our thought process influenced by cloud native your journey.

B

If you take an existing complicated application and just try to lift it, it may not look exactly same because it took us two years to and continuously think and I trade through cloud native technologies and an another caveat is that Google reduces great and wonderful system and we want to actually be there forever, but it moves too fast.

B

Ok, so you know, features are coming in and if you take a dependency on a feature too early, like you know, I'm a file system, I would have been sitting in employee and then you would get deprecated and you Google in a CSI. So be aware of that when you take strong dependency, so.

A

Kubernetes is fast for you. How fast do you go fast, you're, the ozone manager they also on development.

B

So we I mean so there is a difference in writing. An enterprise-class software which gets deployed in a very massive scale and writing something which is which is a business line of application, and you know, file systems are specifically slower, much slower, so you know whether we would started working before stateful sets or daemon sets became popular and being used. So now by the time it comes in you'd, say: ok, let's start looking at this.

B

This is one of the reasons why we would not go too deep into service mesh, because at that time it was not a ature enough. Let.

A

Let me let me be more specific: when can I use it? Oh, the.

B

Whole zone yeah, so we are we're at version zero point four Oh still tagged as alpha, but we think that we will have our GA releases by third quarter or end of this year for availability. It's an open source things. No, you guys are most welcome to try it out. You can install it in a single thing and it basically is I would call it a spiritual successor to Hadoop HDFS, and you know we store large amounts of data. We typically store para bytes of data and various customers.

B

All the fortune, 500 is or at one point is used our software. So we are just bringing known pride tested into the kubernetes world. I hope that you guys will actually end up writing stateful applications and kubernetes and rely on us to address your statefulness. Okay.

A

Thank you very much to be here, and that was or hopefully, oh yes, hopefully it. It helps to to explain that how it worked and feel free to contact with us or just try out ozone as a bonus track. We also have something like between the hem and customize to deploy epic, a big data projects or any kind of projects, and if we have time we are happy to answer any question or any.

B

Questions at all.

C

My question is: do you think it's worth the hassle to implement all the obstructions instead of rewriting all the Hadoop jobs to be a cloud native.

B

The question is that is it worth implementing the Hadoop abstractions are just rewrite their jobs. The answer is that typically kubernetes or a runtime like an infrastructure is not enough because you might have very complicated jobs like Apache spark or you might have a complete layer of business writing Apache hive.

B

So unless you have an strong underlying layer which offers you these abstractions your rewrite spans very large amount of data, and then you would have to reinvent all those abstractions again, so it doesn't matter where you go, because the foundational notions of computing remains same so either you can go rewrite them or you can lift and move them in the kubernetes, so I personally think kubernetes as a new operating system. So hey. If it were used to work on Linux, it should just work on kubernetes. So that is why that is my personal take.

B

But you know, depending upon a field, if you're writing something very simple: it's you know case-by-case. So, yes, there is a possibility that it free writing, might just work for you yep.

B

A

Have two more minutes.

D

So you have HDFS and you are beyond right. What happened? What happens in this scenario? Yeah? That's.

B

A great question: he says the research TFS anderzian, what happens in this scenario, so the yarn guys are actually and I'm, not speaking for them, but young guys have actually returned a very smart scheduler on top of kubernetes I think it's called unicorn and they yarn is a very rich scheduling methodology, especially for jobs, and they are I, think will one of these talks in cube con where they will explain how yarn leverages Cooper Alice, so both HDFS and Beyond is actually moving to embrace kubernetes full-time.

B

Basically, our thought process, as I said, is that kubernetes is our new OS. So, yes, pretty much everything that you're familiar with well show up and I. Think all of the Big Data ecosystem is slowly moving towards this reality that you know what are you're used to right. You have a distributed database. Okay, so you know you, you have to bring it a weather, cockroach DB! Is there whether it is high? If you have a large hive installation, maybe you have twelve para bytes of data and high, so moving it over to cockroach.

B

Tv may not be the right thing. So no depends upon how use cases yeah, so yarn will come in pretty soon. Okay,.

A

I think this is end of our time, but we will be here and you cough a break is coming so.

B

Any more questions we'll be around.