SunPy Developer Discussion, 30 Mar 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SunPy Developer Discussion

Description

We have some bugs to talk about.

A

And I was running.

B

Do much to discuss this weekly? Is there much to discuss this one I'm kind of it a little bit arm's length, the last couple weeks not busy another way that safely we.

A

Were trying to get no point six out last week, but I didn't an error. There are a few blocking bugs one of which um Oh.

A

Somewhat, what's his name, the Mundek, so whoever he is- and I also use just fixed for me- it's very confident the other one. Is this lira tests bug it's probably a good place to stop 1359 Normandy's, oh yeah,.

B

I haven't ever I was away for the last four or five days, so we were discussing that mid last week of remember and haven't looked this instead. What's the latest on that I, don't.

A

Know it hasn't been any movement on youth my day, so.

C

C

D

A

Still failing for.

E

But for different reasons, as lasting, gradual said, um see the stack trace, let's see.

A

And that length is greater than file size. That implies the downloads.

C

Being corrupted, yeah I think that was just a temporary problem with him put it with that particular one.

E

Yeah that doesn't look like that's a standard error looks like yes, that was interrupted or something sorry.

B

Tissue with which it will prb for sure.

A

E

We run these tests together right now on my machine, so.

F

Let's just go to.

E

See what happens so one thing that we did talk about was very and dad noticed it originally at the time stamps. The millisecond came from tulare for not write due to a bog, basically- and I put in a PR that fixed that, but I may have made that start feeling.

E

C

E

It was something was being passed in him as an integer. That's all it was instead of a float, so the millisecond info was doing passed in an integer, so they only updated itself once a second, oh really yeah. So I.

C

E

Taped it to float, but the test presumably have the wrong realtek information in them. These are failing for me, okay. What kind of error is that assertion error? We.

A

Don't know I need to remove the to remove the x-files get the traceback.

B

What it exhale exhale.

A

Means that if it fails, it doesn't report the error, it skips it if it fails, it still runs the test, but it's noisy if it fails so.

E

It's kind of like we.

D

Know this death is broken. Yes,.

E

D

Exactly what it is.

A

D

B

A

Thank now class is now maybe that wasn't what I am confused.

A

Okay need these seem to be passing for me now.

A

Are these downloading any data to a temporary directory, or are they downloading them to.

C

A

It's download they're, not unloading into a temporary directory. Ali I think we should probably set up a way to make the tests run using a temporary data directory rather than the users directory when they're running from, because it's very difficult you're just going to purge my son by data directly.

E

So it's just over, does it do.

D

huh The Heather done no, my road test.

E

What is it test liar? A date.

A

A

E

I see yeah, it's just gonna download. It Bob put it in some payday 2, whatever like a room like those.

A

That's a pitt need half a code base by mistake.

A

Well now, that's interesting.

C

Is this still 1559 yeah.

A

So when I ran these the first time they fail. Oh wait, hang and now.

A

Sees anyone I, don't really understand what the problem is here. Yeah.

E

I'm running the test: oh my machine- this is taking a while and I seem to recall, while in a thread says that I ran them and they failed. Don't actually remember doing that, but I said I did muster the.

B

Last I remember was that gradual had come up in a different era to the ones you guys had been talking about and was going to work on it. That was a lot.

C

Meaner and okay, that was a downloadable file.

B

C

That's what it look like so today, I tied in the same day I had no problem obtaining the file. The error that I had was on start and end time stuff you fixed.

G

That, and did it well the stuff you get.

E

Out of the file should be correct. The test may not clean updated, so it's really.

G

Amazing it simply a case of the test is looking for the wrong numbers. You know. So if you just change the number so now you should be fine. Well, that would be the simplest case. I just want to check whether that is what's going on when.

A

You're running the tests, are you running the tests for just the like a package where you just run all the tests? I just.

E

Done on the test, which was a bit of mistake, yeah they can take a while. How are you running the tests? Tests mean execute and executing the mean there.

A

A

Specify package when you do that I just can't reward here's off something I, just.

C

Had to do sub theta, 3, dude, pi, dot, test and test under school, what I'm, gonna taste yep that works? Yes,.

A

C

A

You can also do this.

E

All the solutions are much smarter than what I just did see.

A

I've run this a good few times once, where it downloaded the data every time and once where it didn't and seems to be, passing with the one.

E

hmm Call I'm just going to execute these commands.

B

How do you become the road you'll about this since she's, probably worth at the most yeah? Probably a sleeper well right, yeah, but I didn't mean, as part of this mess.

A

It's like half eleven way m.

A

Yeah, okay, well I, was to the moment he's checking for he's. Checking that the difference in time between the two is less than one. Second Oh.

E

A

Seems to be working.

E

Yeah, it should probably better than that, though,.

E

B

You mean it depends what he s is trying to. You know what the funk is trying to test it just trying to test it's like petal, lower level it just trying to check that level 3 in that the decadence is less than a second. Let me yeah.

E

No I think he was just you know, essentially budging the test, so that, oh you know. One second is close enough, but I don't think that's a good solution, because you know to be exact. We should know exactly what the times that should be, but you can only know that for a specific file.

B

E

B

You can't just sort of download latest you have to have a very specific file because you know they're the nominal Caden to 20 Hertz. Well, if it's during a campaign, it could be a hundred Hertz right.

E

But hear this hi.

B

E

You know he specially picked a day. He knows what we know, what the first time time should be unless the day to get reprocessed. Of course, the data is being reprocessed. I was just finishing off being reprocessed as we speak. haha Well, that might be interesting.

B

And so the argument for having you know some very some has some small test data just so long as the function or whatever you're testing doing the right thing. Whether the data is the latest or not, is not really that important.

A

No, what do you need.

B

A

We need more offline tests, yeah.

E

Well, there's another weird thing that has just happened here where I don't weather supposed to be like this, but if I do lie right time range start, it returns the date-time object, but if I do, why are the time range endura terms of time stamp is.

E

C

Supposed to do that, no you want and we want some time now. You don't want a thing. You mean like a string. Well, my.

E

Own, if you type on it well, the end is the pandas PS lib climbs down the start. Time is a datetime datetime. Well,.

A

That's all manner of strange yeah.

E

A

Okay, so I've just checked back out to master, and it seems that the millisecond values in the parameters to the test- and they were the ones that he's reading from the file of different.

B

Yeah, so that's just a case that the values in the tests have not been updated.

A

My thing sounds the easiest solution. Yes,.

A

Let's actually check the file showing.

A

Maria pay us.

A

What files its downloading those nearer level many fits files. Oh.

F

Yeah this what's an attachment has to be wrong right.

A

The phone is blowing from right, location data.

B

Clear, how can you be right.

E

No, no sorry miss reading that file, mmm don't mind me.

A

It's interesting, you know it seems to be failing for the level 3 data. Oh okay, it looks like when someone set up the level three tests. They've just copied the mic word milli. Second value from the level two days are expecting it to be the same.

E

But it looks correct according to what does that lead.

A

To that appears to be the problem, the first test, his second two tests are roaming down on downloading different level data. So I don't know. What's going on there.

F

12-6 the devil three, the bumpy I'm.

B

Values below one minute should be here. One minute, one minute, Caden run an avid.

A

How the hell is time stored in these files yeah as integers.

B

In the Fed files, yeah.

F

B

A

Can top my head? Well, I just opened a level throwing fits file in astro bite and look at the raw data and column. One is time big, endian, integer, a level three yeah.

B

Well be minutes of day every minute of day those level three, it would be minute yeah, it would be if it is just an integer, then it's minutes from midnight UT. I think.

A

Why the hell is getting all up on milliseconds if the resolution of the thing is in minutes.

B

Well, hang on what number? What's the number of the first and second time, zero and one it was level 3? That means that's it minute, cadences! It's one minute, average data from this from the level 2 data, which is you know, the nominal cadence, whatever that happens to be so.

F

B

Will be no milliseconds in the time stamps that are created from the fits well yeah? It should be minute resolution unless there's a data gap, but it is, it is putting.

A

Minutes here this protein second, don't know why and what at what point we're not around somewhere in the reed step?

A

How is this doing the convert into? Maybe that's, maybe this is what the integer habit was doing.

A

It's still engine izing, the minutes, I.

B

Have to go in a few minutes, but if necessary, I can just look into the exact format of the fridge files and confirm tomorrow, no vivere, you do for.

A

The level 3 data, it's still passing in an interesting number of minutes, so I, don't understand why this is a little well.

E

Rightly think that level three is passing it's a little too. It's not passing. Oh.

E

Yeah, what so I don't think this problem level 3, but why a little too failing this question, but.

A

In the level 3 there's still micro milliseconds in there yeah.

E

They probably shouldn't be there if there's no actual millisecond information right. It's.

C

So many seconds in the time stamp, which is well mentally life, actually just calculate offsets from some face time. Then my best thinking of milliseconds microseconds, true illogical.

G

Yeah but I maduro.

C

What's going on well.

E

Looks to me right, yeah, the train staff. Can the test is not my own date for level 2 data? It's incorrect.

E

So you should be able to just change the value in the test if the value is checking for for level two, it should pass yeah well separate question whether it was a useful test or not, but I think that would fix it. I'm still really unclear as to what the test is really trying to achieve. I guess it's just trying to make sure. Let's say if you make some change to lie, rely curve right and somehow you've broken something this dude might find that out. Because can we weren't meeting the dates properly anymore.

E

That would be the purpose of a test like this say. Are you still getting the data out properly.

A

Right, it's testing the time passing yeah I'm anything else.

A

The test can probably do with being improved and also being made of line for a start, but moving pal yeah.

F

A

So it seems that when you pass an integer number, two Pandit or through pandas.

A

It's coming out with point 0 for seven milliseconds or whatever is on the end of the timestamp I mean it's the same for all of them. It doesn't seem to be a problem that, because the base time has point.

B

For maybe a millisecond has just adding a minute to the base time yeah, it's possible aight. That's probably fine.

B

Or if the facetime is in the kids files, that's an issue with the kids problem: yeah I, don't.

A

Know where it's getting that waste time phone start start start start start stop in the file everything it is getting that from date, oops on.

B

Way or the other from Dane arms that sound right, they do jobs in their Fitbit bar yeah,.

A

A

Yeah, it's in the it's in that start on that north point, four seven one four seconds is in the start time in the header.

B

Okay, good, that's fine!.

B

But and that's an issue with how we are having produced the levels.

A

In bottom level, threes, fine, so that implies that the is just a bug that is changed in the level 2 data. So let's quickly reload this level two days.

A

So that's yes, integer numbers! So what's time in level two later then loaded.

F

In ages now.

A

It's flown singly.

B

Isis inform second I would guess, but let me have to pull up a bug and confound that an app to run into minute.

C

So I think it is settings if you have a look at the IDL version of this. Could then they divide the number of sessions and that taken column by 86,400, which is 2014 60 x, 60.

A

So it is decimal time all right, tyson time through the documentation.

A

B

The fishbowl does it not have time unit and tell you whether it's minute or a second level.

C

Ready yet you know, I think it just read: underscored lighter yeah. I think it looks for some level market and then debates. The debates were ever than the time column by the appropriate number fourteen, forty or 86,400. So.

A

When you pass a float two seconds in date, interview no time delta object, which is what we're doing in the reed step. If any argument is a float and there are a fractional microseconds, the fractional micro seconds left over from all arguments are combined and that some is rounded to the nearest microsecond. If no argument is a float, the conversion and normalization process or exact.

F

Sounds reasonable when.

A

So we are buying virtue of the fact we're using date time rounding to the nearest micro. Second.

E

D

What is the resolution of that is that a problem do we need millisecond resolution and the maximum cadence is a hundred hertz.

B

So just do the sums and figure that out.

C

F

Early sessions.

C

We need together melly. Second, we need to get the up to a hundredth of a second clip 28. Yes,.

E

Well aware, the test is written at the moment. It doesn't matter right. The only thing it's testing is whether the first value that comes out of the file is equal to some make.

A

E

That we've manually written so what we might want to do, I guess for interest in the milliseconds, is when you'll check the first 10 values right and see, if they're all at the correct cadence and are exactly what there should be no.

D

Hang on a second is.

C

D

Least, second, is a thousandth.

C

Of a second and later goza, 100x100 klippel sake, since it's now rounding yeah, it's.

A

Rounding to like a second- so that's okay right should be this, isn't.

B

A program to a single micro: second: they should be fine cool.

A

Right, yes, in that case, it's just a case of fiddling if I using the tests. Yes,.

A

Having successfully just convince myself at this fact,.

C

That's good and if they did I ever get three process, then we may have to change those numbers again. Data.

B

Have been reprocessed- it's just not available yet, but it will.

C

B

To the dander the sample tangs change, I'm not sure I don't mean I don't know, because I didn't do the the reprocessing I I would imagine naively that they would be one of the things that would stay the same and the radiance values would change. But it's like I'm guarantee that, because I'm, just assuming, which is not necessarily best waiting on I, can ask or again like I, said the other. The other way is to take it offline and have have a sample fix Parker's.

B

A

Shouldn't have a sample file if we can make one that it's nice and small, and it also, we should still have an online test to make sure that if the data gets changed, we know about it. Fine.

A

But we should also be testing this code without an internet connection which we are currently not doing, and yeah I.

B

Have to run guys if you have any other sort of questions or conclusions about this. Just you can add it to their conversation thread on github and tag me so I get it tomorrow.

B

Sure another way then switch you and better luck have a good evening. Thanks hon, take it easy.

E

Yeah I think you were just filling the numbers on the test at the moment, but they're the most useful tasks ever.

A

I'm just gonna fit all the numbers. I'm yep. Oh hello, boy, wait! huh What's up, okay, nothing! It's gone mad! Ignore the crazy person! Hey.

A

A

I'm, a crazy person.

C

So that is, evil like this will need be 0.6 releasable.

A

Hopefully, that's another bug which I found in the process of trying to package a nasty one which I didn't have time to actually test enough to guarantee that it wasn't just me being silly. So I need to check that appears to be a config issue with which is not so much a code book as it is a n.

A

This is a packaging bug which, frankly, I was.

F

Stopped what start.

D

And now there is another JSOC bug as well. That's question I just like that.

C

Said the skips first fail: yeah, okay,.

A

In fact, that PR is just open for it.

A

Handy, that's a nice easy fix, I like it.

A

A

Think there's much else: I was um I.

A

Keep finding things that need doing to the 34 downloader.

A

I'm wondering whether saying a summer student loose on it is a good idea.

C

On unified download on with it mean why else is what else has thought I thought I was pretty close and what so much didn't we can't use a g-shock one. Can we well? We.

A

Had a bit and I quote that defected data downloading and file naming and stop side of it needs a fair bit of work, because at the moment with vso you can use the query response object from the VA's over to name the files and.

C

A

Can't do that with any other client, so having a unified, downloader doesn't actually teaches client dependent is somewhat stupid.

A

What does the juice I'll give you that gives you much less information? I mean, obviously, to some extent that is the problem you're going to have is that you get less information from.

C

You get different amounts of information from different amounts of information from different places. Yeah.

A

But I mean that so much can't really practically be avoided, but you can at least support the same syntax and we need a way to be able to inspect the response for the information that is available.

A

Then, at the moment, vso will doesn't, if there's a file, the same name there it doesn't handle it neatly at all. Just add downloads fight again put some point: zero on the end.

C

What's wrong with that.

A

It's not consistent with behavior we've implemented elsewhere in some papers to our, and, secondly, it would probably be more useful for it to either skip the file if it is already been downloaded or provide a way to overwrite the same file. You.

C

Need like a club or option yeah? Yes, some kind.

A

Of claw option.

C

Is that enough to block it, though I mean because, like the clobbering I can see the file naming being more of a problem than like clobbering or either zero like if the JSOC doesn't give you a file name and then that's that's a big problem.

C

What do people do anyway? What.

A

Are people doing now and Jacob gives you a file name? It just doesn't give you as much information about like the wavelength for the instrument, I stuff Nate.

A

And I mean some of the, so the like out download is just don't give you any information about the.

A

If you like, any information about the wavelength and stuff.

C

At all, so yeah well, like Gophers I, mean we're going to have to put more stuff in the resources for the like, and then in that case it does not any information, isn't automatic.

A

Yeah I mean well that is as days to some extent a separate problem bar.

A

The framework for the framework for the downloader section should really be in place before.

A

I think I've nearly got this working I fixed two or three failing tests.

C

In Captain, 59 yeah.

A

Okay, I, don't understand why this is different.

A

This is the same number.

C

So that ended you, you said there was each part of the time range was a different format of the data use, different data yeah different data tape. That's right, but I try.

E

C

Through the the constructor and I couldn't really see where that could happen, it must be something in well I'm, guessing it something in later, as a blister.

E

C

Curve yeah didn't these.

A

C

A

To be doing pretty much exactly the same thing.

E

We need to look at time range and see what it does, because it's the time range dot star at the time range not end.

E

So maybe there's some weird behavior in time range.

A

Did you say was I eat just giving a path this time spam yeah, but whenever the reason he's probably doing that is because, like at lifetime range is probably checking for? Are we checking for a sub? Club is instance of daytime and a plan does time range? Can timestamp is an instance of daytime it's a sub class at bedtime. I see you should probably just we get through past time, which is its supposed to catch. That.

A

These two tests seem to be doing the same thing. Just the.

F

Other day I, don't they this.

A

Is I need both the test liar a level on the test live a day of just doing the same thing, just ones downloading two different levels of Bayer, so I just delete the second one, I think.

E

So it doesn't seem to do anything different with it.

D

You are the weakest link, goodbye.

E

Yeah they're identical go from the level thing.

E

A

Someone added the level and just caught and pasted the testicle modified it to their own ends. Yeah, it's the kind of lazy thing, I do well.

E

What's the difference, the air with the bottom one, it was test on line.

A

One seems to be testing from a URL by explicitly testing a URL. Okay I, don't quite a wide. They again they seem to be basic and doing the same thing.

E

Well, I guess they want to test whether the creation of the works that way.

E

The excess passing I with tweet numbers yeah great.

A

Oh, that's got my modified change, love it. Let's come along for the ride.

C

Any other the easiest, POS and then nothing's, passing Lily.

A

E

A

Quite a while there I just opened 1367.

A

E

uh I think I can give that a thumbs up pretty quickly.

E

Yes, if you're Pisces uh-huh all right assume it does there we go as the fastest plus one in history probably is, but yes assuming it passes,.

A

So, okay with that testing, don't you get? I.

A

Recently modified the set up top, I think so that you can just make it run the online tests only just kind of handy, because they're, the ones that break all the time.

G

A

I'm going to see whether I can run all the online tests and they actually work.

C

So I has dumb and ducks right out of 22 pull requests now.

A

Once the Lord says all.

C

A

In fact, actually up milestones along one.

E

Just have a look at some of the.

C

Other stuff- that's in here so there's 13 20 years our backcourt mate.

A

A

Help there isn'ta 1320, open so issue, sorry 1561.

A

A

A

Except I think that now also needs to be employed to master, because he never changed that in master like we were planning on it, because we've branched off no point 6.

E

This is a poor, a question to master no.

A

This is a pull request into master, isn't it? Yes, it is what we shouldn't be, but ok, ok, this bunch at that attack, em.

A

Yeah I don't think he meant this to be a PR intern master right.

E

It says it's for version 9.5. So what yeah.

E

So this just oh yeah, just changes the up getting the most recent data feature that we're going to remove any way soon, but it's good to fix it.

A

Well, exactly that's things, the feature that was going to be removed, which is why we work at a PR at to master, but we branched off no point six before we removed it. Yeah.

E

I see, I think.

E

But every it's an easy, it's an easy fix, so.

E

E

Let's see who just keeps going back one day at time until he finds the fowler exist,.

A

E

Yeah that works.

E

I'm just gonna +1, it.

E

Assuming all the PR to the right place, soy de.

E

Back portrait corrections to go operational time ranges.

E

E

A

E

Please yeah change the lid operational dates, so the goes satellite.

E

This is the back work is.

A

E

So I guess the problem here is that he's adjusting the dates, but now and then just you get a fail because the sum of that goes files or just kind of screwed up right, and so he made a list of the dates where that happens: I'm not sure what we're gonna do with that list, but.

E

We could I, don't know if that's a problem, though, because if it fails it should they useful error like no father's not exist or I, think we.

A

Were going to make a like manual test that you could run to check which days were not available or something like yeah.

G

A

Run as part of this testing, but is in the test, folder can be manually executed.

E

Yeah I, don't think it's a big problem on, say: I mean user tries to create, it goes like her and then it says no, you know file, not five, then kind of you know, and it's not there.

E

That seems like a fairly easy to understand error, so I guess you could get weird things like said: they get key error on some files. That's probably pretty weird. I.

G

Don't know why you get that.

B

Anyway, it looks like.

F

Mm-Hmm, which one is that 1324.

E

They're, actually quite a lot of dates where file does not exist. Oh well,.

A

Get bad data yeah.

E

I mean yeah, something we can do about that.

E

It's no one's fault and the witch.

A

In speaking to joke about the SD hope data a chase off recently, we are still on the air. Wait yeah, it's terrifying.

A

Terrifying is there anything else major to go over I, don't think Simon moon, I think so.

E

But I should go. I have another meeting like 20 minutes.