Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Practicing Linux Crash/Panic Issue on Production and Cloud Server: Using K... Ben Shushu & Gavin Guo

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Practicing Linux Crash/Panic Issue on Production and Cloud Server: Using Kdump + Crash - Ben Shushu, Running Linux Kernel Group & Gavin Guo, Canonical

With the rapidly development of Internet in China, more and more servers and cloud servers deployed Linux systems, like Alibaba, Tencent. In addition, with the development of the Internet of Things and industry 4.0, more and more product development chooses Linux system as the basic platform. Although the Linux kernel is robust enough, the system crash will happen frequently. The topic of this speech is to introduce some experiences and of kdump + crash in Linux crash issues on our production development and deployment. We will introduce 6 experiments: Lab1: Panic caused by a simple null pointer Lab2: Access list head linked list that has been deleted Lab3: a crash issue on device driver Lab4: How to find the value of local variable and parameter of function through call trace and stack Lab5: step by step analyze a complex deadlock crash issue Lab6: Recovery function call-stack manually.

https://sched.co/NruN

A

Actually, it's okay, we're gonna start earlier and well. This is a pretty informative session, so lighter so we have dozens of slides. Actually, it's pretty bouncy and um well. I will do the first part in Kevin well responsible for the second half so.

B

A

From my parts, all I just start it. Okay start from now and I won't go into great details. Do to tell them it's even interested. Actually we have video recorded and, if you're interested, then you can follow us on bunch issue. Linux can tell our communities twists and actually the case M.

A

C

A

C

A

Yet just chatting well personally, I think this is pretty useful, especially for the production developments. Well, AI and clouds are our passwords now, but for the foundation of this platform, it's still elect Linux this building for exists or Linux, and some of the organizations are still need supports from Linux and if you're using Linux, then that means you're likely to run into panic and crash. So we know that there are panics in different ways, and so today, I'm gonna talk about it. I think this is the right topic for today.

A

Well, at the beginning, I would like to spend several minutes.

A

X 64 and 86, and the main streams are these two structs. So I pick one of them to.

B

A

On today, so we talked about a mechanism or methodology. Well, so if you develop develop a hang of it, then you will know there are structured data. I have a lot of slides, so I just skip the.

B

A

Intro and then chew, so we talked about the practicing. The next crashing panicked issue on production and cloud server. So for my parts of light introduced, they keep them in cash. I.

C

A

To have a brief deduction and we have nine practicing and examples that will briefly go to eight with examples and the final one. The ninth one will be very complicated and will be responsible by Kevin.

A

So why do we need okay? Stop Linux has been developed from for 28 years. It's pretty strong, pretty robust and have you ever seen a black screen in Linux. Well, I! Think it's not frequently seen and if we develop, if we deploy it a large skeleton, it's possibility, there's the possibility of black scale. So we're still considering the reliance of this system, and if we run the test, then then you will find the possibility of crushing the whole system. So.

C

A

Why the panic in the system? Well so I think this is how we developed the same.

C

A

Kisum, what's.

B

A

C

A

What we used in when we have a panic, then we will develop those screenshots and we save it later, who restore the situation and find clues. It's like what the police do at the murder scene, we'll save the crash scene and then for later research and for maintenance and developers, especially for embedded systems and for small phones, these developers in the personnel working on the optimization of products. They also need KTM, especially for our physical devices, and when your system is not responding. We use case M.

C

A

It's pretty handy and.

A

Your CPU and fixed thus might cast a panic in the system: they're not responding to reboots so case L is coming into play. So what do we need to provide you with a very good screenshots for research later well, if you're working on Linux, if you're developing Atlantic's, including operation and management and cloud provider, and totally all this rather than personals,.

C

A

Will have to know case and better.

C

A

Is a structure for concern when we have a crash where two cores one is called production core and the others capturing core, and we use K s EC to capture the cause of the crash. There was switch.

A

Developer screenshots of the memory and load it into local or network discs, and how can we provoke tit? We have multiple ways you can do it manually and.

C

A

Can also do it in a system you can set a kernel, panic, wash dark or Laura golf and or wolf memory, etc.

C

A

All works: what if our system is blocked? It's a court race, that's commonly seen for 120 seconds.

C

A

It's probably a deadlock or other things. So if you're using case em, then you can better solve this problem, so you would know who has the lock who's waiting on the lock? We will use case em to learn these informations.

C

A

Also, if we want to develop a case em, then we will develop it on deploy it on server, and here it's, my recommendation is to use the X 68 server, so it's pretty new and it's pretty solid nollie into Brik and and also we can develop it on arm 64 platform and I'm 64 platform is coming to use. Now.

A

We have a experiment platform with 5.0 and I'm 64, and the sympathic platform enable you to run all the tests that you.

C

A

How can we trigger in case I'm manually? This is how to do it after okay, sir, this graphic shows you how to use the max 5.

C

A

Write the memory documents- and this is using the crash tool to do the VM call and if we're developing the dam data, then this is pretty handy as a tool. We have the dozen of orders that you can follow.

A

Starting from here, this is an example. Roughly we have nine listed here due to time sensitivity, I'll go through several on first, it's pretty simple our Opie error, and this is the whole analysis process.

A

This is how we're analyzing code trace, for example. This is the parameter the first one, and this is the course this is the course for the crashing kernel and you.

B

A

It if you're interested after the session- and this is how we run the analysis so due to time, sensitivity, I'll just skip- and this is how we check on the second parameter test number two- is how we have access to a deleted list and restore the process of panic and experimented and experimented experiment number one and is how we visit a no none, none existing page- and this is a bug and also we're developing a driver if we use a ragam app.

A

This is what happened. Okay,.

B

A

Creating a issue- and we add one parameter in experimental, five composed experimental four and let's say the hardware is not on the want to be status or ideal status. Then there is a that log. Then a km can capture the medieval information. Experimental six is.

C

A

A

I'm 64 computing value and we have to analyze on the functional functional stacks and it's better for you to understand the functional stacks relations, so you can read through the handbooks of arm 64.

A

So this is the functional scheduling of arm 64, and this is how we organize the stacks, and this is the graphics. This is each stags and they have FP n lr and how we store it, and this is how finished computing and we have several equations.

A

This is three conditions based on which we can calculate and compute the relations in km due to term limits. I will just skip this is how can we come up with the relations of functional scheduling.

A

Number seven is like this, so how can we know its location in each tanks? This is very important for our on-site analysis. So first we have to understand our 64 and there are stacks and.

C

We have to also.

A

Deduct the parameters based on our recursive analysis and we have to compute our calculate on the location of each stacks, and this is before and after the experiment, this experiment is about. How can we come up with the second and the third parameters and their exact values? This is my previous slides, and how can we get the court raised.

A

Find the code of compilation and how can we locate the second parameters? This is arm 64.

A

Stack layout, so only five minutes so times limited, skip and skip.

C

A

Wanted ADA is pretty complicated, remotely related with what Kevin is about to talk and how can we use km and use the stacks to get the value of parameters and then we'll know who's. The holder of the deadlock so km is the key to solve the deadlock. We can basically get every information we need in the analysis process, and this is a self evolved experiment by my self and.

A

B

A

Get the rewrite log from M&S, yes, tread we'll.

C

Also need to get the semaphore.

C

A

A lock and if we have the two threads going on simultaneously, then there is a crash. Then how can we.

B

A

On this example,.

A

So briefly, this is pretty lengthy and we can see that in PS me we have four threads and non-traceable. We can analyze that one by one.

B

A

We will use this compilation codes to deduct on the clues that you want to find and you will find the location of this clue in different stacks and you analyze the logs and you had to know the locks very well and if you don't have a good understanding about the river logging, you can read on the blue bubbles, so I will skip this parts due to time limitation.

A

This is analyzing on the test process or treads. This is the call trace.

C

A

Have a thread P I, T member, and we have sir coded- and this is how the test treads get the lock and we are in analyzing the compelling code and we're also using the features of arm 64. For example, arm 64 have them X, 1, 9, 2, X, 28,.

A

Temporary change. We will have to save it for further analysis. We'll have two minutes so skip it.

C

A

We have cooldown.

A

Who analyzed each tread one-by-one see that exact location at that log and lyric happiness is gonna talk about the same thing.

A

So sorry to the time limit, so if skip these parts, and we can reaffirm the graphic on site, this is my self developed treads and how it became a deadlock briefly, this is summarization km can help us to solve the black out or the black screen issue.

B

C

A

Know the crash tool and how to use it.

A

You have to learn the basic process of an analysis using crash tools and you have the locates the parameters by compiling code and solve problems by doing this, and then there will be no black screen for you and now, let's invite Kevin an expert on kernel, and you will talk about a copper level that log issue and how to solve it.

D

It'll be a complicated.

D

Deep into this, like.

C

Assembly I will.

D

C

D

Concept in debugging or yes, I hope you can now not from it. So let's continue.

D

C

To see the detailed description of bases, but.

D

I think that, yes,.

C

This provides details.

B

The whole process I.

D

Have recorded in it, you can quickly it open and view it yourself.

C

D

Biggest client is a.

C

D

Only have walls over a hundred of Scipio use.

B

C

D

C

D

No Matty is enabled.

C

D

D

So I like to go inside and check it. So if you.

C

D

D

The leg disease, the SM vehicle function, yeah. If you see this time, it means that is stuck on CPR.

C

Because TOB is, unlike the.

D

Ordinary man see it use this.

C

D

Would be freshed yeah, it is stucked. You.

B

Can see in this line.

D

That is but ypi.

B

D

It get stuck because.

B

A

D

For example, I miss apos, zero and there's another CPU. You want.

C

And they want to frustrate the sub. You I.

D

Want to win it in the finish. If so, ask for me to continue on imp is stuck here and there's no response.

C

From that CPU this.

D

Is what you can see from a few.

C

D

We must get in it. You can see that.

C

D

The 5 years ago, they've been developing this for a while 50.

C

Lost including.

D

People from Bangalore- and they give us some consideration and we ask them to test it and we search it for a long time, but always having to come to a.

B

D

C

To and why the IP.

D

Is missing when you to wait for for a long time, I.

C

D

A right to page Twitter, page.

C

D

If I throw it and.

C

I will wait for you. It.

D

Doesn't come back our intro.

C

D

I interrupted our.

B

D

B

D

Were sending again.

C

D

Problem is not what I have thought about before.

C

Let's go y'all but.

D

You just chose your eyes.

B

C

D

C

D

Need to reflect.

C

The probability.

D

I'm using is that no, but why have do I? Have this kind of problem.

C

D

Endure refer to the VM I should.

C

Refer to the most, because.

D

The kinda wanna provide.

B

Us with detailed informations.

D

The total I think it would has not true, but.

B

D

C

Whether it's true or not, but.

D

The client they depend on you for the help.

C

D

I look to refer to the whole society.

C

As Chevy's.

D

C

D

Can be run on a testing machine, they can be verified.

D

C

Is not the production only assistance in.

D

This testing environment or made it when.

C

B

C

D

C

Three days it emerged. Yes,.

D

It's running very fast when we were producing it, you would team code.

C

To hit it, you.

D

C

D

C

D

The King s things and if you enable.

C

D

And if you in a boy.

C

D

You are panicky the south and you can find a VM.

B

Code yourself, yes,.

D

D

C

Is the instruction code for each under.

D

B

D

For each PT you feel puppy into a lot under file, and you are.

D

B

D

B

See what choice I know, which is still working so.

A

C

D

That who is working.

C

Idle because you.

D

Won't kiss you are just down: okay,.

C

D

Your stay idle, but the whole system is awaiting for this app to appear.

C

D

I this case M deep.

D

You can see that.

C

Procedure management.

D

C

D

We want to expand it a few monitoring. You are just a.

C

D

The similar pages together, yes to.

C

Merge them together and.

D

You can also stormy Amazon from the host that's being done.

D

B

Is the first victim is locked to an erroneous.

C

15, what's.

D

D

B

D

To be reduced or yeah, you can check it yourself later.

C

D

Boy, it turned it into always.

B

D

Continuous, if.

C

D

Both sides, it were merged the full page into two layers and to.

B

Make it to PD because when they are screening.

A

D

You are gues tacos.

A

D

And you cannot.

B

Tell which the.

D

B

C

D

Something happen, VMs see me, it appeared and the two-dimensional yes.

B

D

C

Page 4 of the.

D

What we go on to connect me, I, get it get it.

C

D

Yes, I am huge page and Korean. What is a big.

C

D

We want to find a wizard, it was a killer who gets the key.

C

D

I in a Bhanumati there was something happen: B.

A

Something's happening.

D

And in a beginning, I won't see me too much. You may think that my day is nothing to do with it it, but it has something to do with it. So it's find a bad tricks. It's no.

C

D

It's the auto banners. It's.

C

D

Very remote access, if it is now we were, will.

C

Put a pitch in to the local to.

D

Make the memory latency become produced.

D

It find in it's not visit. It happens to be a key in.

C

D

It wants to migrate this.

C

D

Wants to make migrating pages and you happened to find a cave what, but it yes, it should happening during.

C

D

C

D

But the page is no.

C

D

C

If you want to change video, it will be ok.

D

Should be updating, it need to be refreshed.

C

D

Be refreshed from pitch table and Oh from step by step.

C

Until we have a just the mention with the conclusion.

D

Microdisk a 10 pitch it has to over.

C

D

And more pages, so they can. Let me copy the by one single machine.

B

D

Alma pages they are not.

C

D

To one page, I.

C

D

Skip the details.

D

Yeah, we have already loaded our PTO, but you can see.

B

This example analysis because.

D

I were talking.

C

About two to three hours: if I go into detail, yes,.

D

How to get the documents interview real complicated.

C

Yeah how to solve this problem.

D

Threat enough, it.

C

D

To trace twice table one so.

C

Stable treatable, so when we.

D

Merging it isn't to the page, if it's not stable, you cannot merge.

A

B

D

B

Go to it! Yes, so it's forward. How is the.

D

B

D

The first day announce table, treats and the website has been screwing the toys. If it's not.

D

If they are merged into one and they are put in this stable treat.

D

They are merged into one page.

C

D

Linking is has over at oh.

C

Two point: three.

D

So if we need to send a team and.

C

Yes, there are too many complicated.

D

It will take a long time, I.

C

D

C

D

A list for the crush.

C

It's a command.

C

D

These procedures.

B

C

I want to process a it.

D

Was putting in a sippy.

C

D

Process a it needs to be put.

C

C

D

Torture is best to migrate to back to migrate.

C

C

D

You are processing your.

C

D

Thing in CPU for oh, but that is its you.

C

D

Can see on these two stages, two pages.

B

C

20S and one or two I'm.

D

Not the chest, a migrating.

C

Page to this not I, think.

D

Is allow, if you want to schedule.

A

B

D

B

Microwave no wise.

C

D

Migrate process.

C

D

It's not that easy, because they want to my order. The.

C

Processors and.

D

The city Oh cute was.

C

B

About it well,.

C

D

C

D

Not the keyboard I want to get.

B

D

C

A boat they're taking my balance and Jason pitch no.

D

It is the merge that together damage over 200 MB.

C

D

To do that, he open you fresh.

C

D

The Brandon quick.

C

D

C

D

When I view is it fresh, you.

C

D

See that migrate, ages, sniper.

C

C

D

Can see the bar is too long when the work work, what it's work.

C

D

Reverse MVP: we.

C

Want to put some pretty nice.

D

C

I play deeper and that this was.

D

C

You'll want to shorten your time.

C

D

C

Not have a code in.

D

C

D

Was rejected by the Philippines, so he wants to shorten the time.

C

D

You can see our discussions.

C

D

Send this to him and.

D

Him with a modulus was shortened.

D

The whole process, the whole story, is about it's like this.

C

D

Spencer, who asked.

C

D

Knows you can refer to this.

C

D

No, nothing can be obtained all I'm telling you how to do it in detail. Yeah. That's kind of my presentation.

A

B

A

So tempestuous our time is limited. So if you have any questions you're more than welcome to communicate properly.