South Big Data Hub Data Sharing & Infrastructure Group, 3 Feb 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CI WG demo: Syndicate with iRODS

Description

Date: 02/03/17
Presenter: Chun-kun "Amos" Wang
Institution: UNC-Chapel Hill
South Big Data Hub

A

On to Chun Kuen mo swang will be talking to us today about syndicate dr. Wang received a BS and master's degree in national Chun, Qiang University in Taiwan in 2009 and 2011. He is currently working towards the PhD degree in computer science, University of North Carolina at Chapel Hill this research interests through the parallel programming and system verification that will hand it over.

B

Hi this is Jin Qin. Our just call me, amis and today I'm, going to present the current status of the integration of syndicates and Iowa.

B

So, first of all, the syndicate is the story surface. That's build a coherent storage abstraction from the already deployed component, so it can be include all storage, l, edge, caches and deposit providers, and one of the nice features that it's fully decoupled education from providers and the transparently leverage them without modification. So it's not only offer consistent semantics across the multiple providers, but also over a flexible, a programming model to application, so the user can just define their own provider as a non stick functionality.

B

So we don't have to take care about what is the underlying are base? You are going to use so the next slide. It shows the Syndicates architecture so in according to I think it's indicated architecture. We have three major components and first one is the user syndicates gateway, so X Q stands both in this indicates gateway. The users Sinek has gateway, is interface with H edge cases for an advertising endpoint, so its response to the application, read/write request and six.

B

Secondly, there is a another component called refocus in this head gateway, so it's an interface with the cloud storage provider, so right now, syndicate, suppose Amazon, s3 or glacier Dropbox and Google Drive, Google, Drive and Iraq as well, but it's still under development, and so the river card syndicate gateway our response to rewrite a request from the peer user of blow us to declare gateways.

B

So if it does not generate any requests of if own, so it just provide a service and the last one is acquisition case, syndicate gateway and which provide interface with the data provider and maps maps and existing data set into a volume read-only directory hierarchy.

B

So whatever your database upload, the file to syndicate and acquisition gateway will grab the data from the database and then provide a read-only data to the user. Ok, so the following I'm going to show you the sentence as a simple demo of this architecture.

B

So right now, I have already setup. My virtual machines am I running all the experiments on the docker container and the god. Docker container is the virtual machine, so we can use the docker container to simulate the behavior of a distributed file system. So here right now, I'm already boost my docker container. So you can see here there is a 5 container right, one. It's a replica gateway and users gateway, an acquisition gateway and Iraq itself and MX represent Makati pass surface, so it's manager, metadata server and then control control that actually data location.

B

Okay, so right now I'm going to.

B

Log into the user day way to take a look.

B

Okay, right now, I'm in the user gateway and.

A

Then we somebody who's typing. Please mute their phone, we're getting some some.

C

A

Here with so speaking,.

B

Okay, oh I'm typing, so maybe yeah, okay,.

A

B

Sorry, okay, so we increment on several.

B

Several user tools and we put in the files so look at here- that we provide us indicate cache, get LS just like the normal file system, you use in the Linux environment.

B

So, first of all, we use la to see what we have in the city read file system, so I've provided my user register username and then my date huh body.

B

B

So right now there's nothing. It is in noise.

C

B

Now there is nothing exists in The Syndicate file system, because I didn't upload anything, so this window just show you I can upload something wrong: I rot so right now, I love into I, rot, container and then I think Savannah I already upload. A one I rot file is called I, rot files that 50 and I already uploaded, but I have to activate my acquisition gateway to death. The Syndicate is nowhere what kind of file you already upload so.

B

So so right now, I are going to activate acquisition gateway. So it's already accept and then provided on the data set to The Syndicate file system, and then we we execute the LS again. So we can see the file so right now we got okay. I see the Iraq file that we see. I already provide upload, one arm, I rot, they have database, and then we can look at take a look within use page on to take a root. Allah I.

B

Rot file data.txt.

B

Ok, so this is the automatic upload file for Iraq from the script. So this my are just just just a way. I I can get a date happen to make that file system and then the next one I'm going to show a flat bar. If we wanted to upload some files on the user gateway, which you can do this the first of all week, we asked a local file, subject e and then we just time this is a sample.

B

B

And then we use we can use the put.

B

The local file that you see and then we can try to write to the the side to write to our clinic has file season, but it turns out it will be arrow because we have to. We have to activate the reverse River cascade way. First, so these windows show you the result, our gateway.

B

So here we have to change it into we're, going to activate riffle away man.

B

So I do it again and then to see- and you will see the reaction from the acquisition of the Tripitaka way in this window.

B

Okay right right now we we upload, we upload our file local file that succeed. So we play this again. We take a look.

B

Okay, I see, there's, there's a low profile which is I uploaded from the user gateway, and what, if I, wanted to update my data database, like maybe we are going I'm going to just create another sample from Iran.

B

Just tap and then I use the I command, which is provided for my rod itself and I put into Irish server and then attempt I have to update my acquisition gateway.

B

And that that acquisition gateway to read the current status from iris again and then.

B

From my user gateway, I can see the that is the upload file for irods. So if we take a look.

B

Simple I rot. Oh.

B

Oh there's a pipe man upload, okay, but yeah. It turns out like it shows the we can get the latest a file formula, local local data database. So, let's all of my demo is there any question.

A

They open up Florida question.

C

How would you automate updates to your acquisition gateway whenever you put a file into irods, so.

B

I just write a show screen. So when I activate my irods container and it's already partying into the container execution come in 9, so it's all automatically upload every file.

C

So, basically, I could write a policy to automatically register everything into the acquisition gateway. Yes,.

C

Hey this is Jason. Is there a performance improvement using syndicate for data distribution over just say, asynchronous replication within our Uggs.

B

Currently, we don't have the we don't. We don't provide the performance comparison because we develop some instruction and put some shocking. There is still I mean there's a potential box, so we verify the correctness is our first priority and after we finish that we will to go in to look at the performance, II issue.

C

Cool mxx thing.

A

Anybody else have other questions.

A

Okay, well hearing no other questions. I want to thank amis for a great presentation, and thank you all. Does anybody have before we close for the weekend? Does anybody have any other announcements or things they want to share with the group? I see Jennifer hammocks on the phone and her Ser spoke project I think is pretty interesting that they were one of the spoke T eyes that were awarded. Does anyone want to jump in and add anything today? Oh.

C

This is Jen can I just a quick word about next meeting yeah. Please do I'm really excited, although barely following this discussion, I will come to representing a community that is mostly biologists with very few data scientists, so I'm hoping to have a bunch of data scientists next week. What I plan to do is describe to you our data and hope that people can advise me about solutions for our data. Just so, you all know where I'm going.

A

Karla's, that is that next week, what's the actual date of that Karl, could you jump in and give us specifics time and date right.

C

February 17th: this is a bi-weekly meeting, so in two weeks Jennifer will be presenting and also match Spitzer from the Open Science framework group.

A

Excellent and and we'll send out reminders of that as well as we get close to the date, but you all should have gotten that information in the newsletter that went out I think this week, Stephanie sent it out.

A

So look for that. We will be beginning to wrap up our demo and begin to move towards more real working meetings. But if folks have any recommendations for additional demos that you think we really should have it's not too late. You certainly could make some recommendation as we'd like to hear it so, and you have Carl's email, I believe as well as mine, and so please send us your ideas any last thoughts before we close for the weekend.

A

Okay, I know it's Friday, people are probably there so well. Thank you all speakers I really appreciate and thank the rest of you for for joining us, and and thank you all so Regan and Jason for asking great questions. We really appreciate it and with that we will close have a good weekend. All thank.

C

A

C