GitHub GitHub Universe 2022 - Cloud, 21 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Serverless Spark CI:CD on AWS with GitHub Actions - Universe 2022

Description

Presented by: Damon Cortesi

Interested in combining Amazon EMR Serverless and GitHub Actions to continuously test and deploy your PySpark code to AWS? In this session, Damon will talk about how to configure unit tests when a push is made, run quick data quality checks when a pull request is opened, and automatically package and deploy tagged releases to Amazon S3 for use in your Spark jobs.

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

A

Hi folks, my name is Damon Cortese and today I want to talk to you about continuous integration and continuous deployment for serverless spark jobs on AWS using GitHub actions. Let's get started so today we'll see how to use GitHub actions to both schedule and automate data quality jobs and unit tests.

A

We'll also see how we can use GitHub actions to package and deploy versioned assets of our spark jobs to Amazon S3 and then we'll also see how we can use vs code Dev containers to increase the pace of our local development when we're working on spark jobs for Amazon, EMR I know that's a lot, but it'll make a ton of sense when we get into the weeds.

A

So let's take a look at what this is going to be like if you're not familiar with Amazon EMR, it's a way to run open source, Big Data Frameworks in the cloud spark Hive, Presto, hbase Flink. All of those are different open source Frameworks. You can run on EMR. We strive to keep up to date with the latest open source releases within the last 30 days, or so we have different runtimes that we've added performance improvements to so just by running, for example, spark on EMR.

A

You can run your spark jobs almost two and a half times faster, most folks that use EMR also use S3 storage. So this is a really big Point you're able to separate your compute and your storage way back in the day. I was an hbase admin and I still remember calling the data center and saying hey.

A

We need to add, you know, 30 more storage to our hbase cluster and they said well good luck, we'll see you in six months or when the hard drives finally ship across the seas so being able to scale your compute and your storage independently and be able to rely on S3 to be able to scale up your storage as much as you need is a really big benefit here and then. Finally, it's easy and scalable with EMR. You can scale vertically, you can scale horizontally.

A

We have manage scaling where you can take this information and automatically scale a cluster in and out for you great system, but there's a lot going on there right. You have to figure out what ec2 instances to use you have to figure out how to scale the cluster and it can be kind of hard to use. So this year we announced the general availability of Amazon emrs serverless, and this allows you to run serverless spark And Hive jobs in the cloud.

A

All you do is pretty much say, run your job hit, go and we'll take care of running that infrastructure, scaling it up dynamically and shutting everything down when your job is done. It's pretty awesome. So what we'll see is how we can use an entirely serverless infrastructure to run unit tests, integration tests and deployments with GitHub actions and EMR serverless. Let's talk about the architecture here. First, we'll have vs code and we'll use a development container for our local iteration. Next, we'll configure a few different GitHub actions on the first one.

A

We'll take a look at push events, so anytime. Somebody pushes to the repository will run a simple set of unit tests that make sure the code is correct when somebody submits a pull request, we'll run a longer set of integration tests on the actual data, so we can ensure that the data processing is as we expect it. If somebody creates a new release, a new tag on the GitHub repository will take all the artifacts package them up ship them to S3. So then you can run that job in product option and use semantic versioning.

A

Finally, we're also going to schedule our ETL jobs using GitHub actions, because why not it's a pretty simple job, so we'll just schedule it on a daily basis to process our data and spit it back out to S3 the other pieces of this architecture that we need one. We have an Amazon S3 bucket for our artifacts and an EMR serverless spark application we'll have one of each of these for both a test environment and a production environment.

A

We'll also have IIM roles for GitHub openid connect, as well as the spark job will have its own role to access data in S3. The thing to note here there's no other AWS resources that we're creating not even a VPC, not even a Nat Gateway, nothing we're creating the buckets we're creating these serverless spark applications and we're creating these GitHub actions. So that's why I put architecture and quotes up at the top, because there's not really much to talk about here.

A

Let's see what this actually looks like here's, a diagram of what the end state of this is going to look like. So, on the left hand, side, we've got our local development, we're using vs code, of course, and we have an EMR Dev container, so we'll spin that up and we'll be able to do some local iteration using jupyter notebooks. We can explore data on S3.

A

We can write some unit tests and some basic jobs using Jupiter and using pi spark and then we'll push that up to the GitHub repository on those pushes on those pull requests on those tags. We have a set of GitHub actions in the middle. That's our continuous integration and continuous deployment section so we'll run those unit tests automatically those integration tests automatically and when you tag things we'll deploy those automatically both to the test environment on the right hand, side in AWS, as well as the production environment. So this is fairly straightforward.

A

What I'll do next I'll run through the actual code for some of the GitHub actions and then we'll dive into a live demo to see how we can take this from a completely new repository to deploying an ETL job in the cloud I'm excited about it. I hope you are too first I want to mention GitHub openid connect. This is a fairly recent addition, and this allows us to get AWS credentials without having to hard code any secrets or anything like that.

A

So when an action runs, it communicates with AWS and says I would like some temporary credentials for this specific role. Those credentials come over. They are time bound. So usually you know they only last for an hour or so and they're very tightly scoped as well. So those credentials can only access and can only write to the S3 bucket that they need to write to, and they can only start jobs on that specific EMR, serverless application on the AWS side as well, you can tightly scope which repositories are able to assume these IAM roles.

A

This is great because you know you can't just Connect using some random repository and happen to get some IM credentials right on the AWS side. You have to say this. Im role is associated directly with this repository and you can even scope it down to different branches in the repository. If you wanted to do that, so that's GitHub openid connect number two on a push event we're going to set up a basic unit test. This is fairly straightforward.

A

We'll just use an Ubuntu container, we'll set up the code, we'll check out the code and we'll install Pi test and then we're just going to run a pi test on our PI spark code. So this one's fairly straightforward on a pull request, we'll set up an integration test. This one is a little bit more complicated. What we have to do here is you'll see that AWS action to set up the AWS authentication. So this is the GitHub openid connect integration. I talked about.

A

This goes to AWS pulls in the temporary credentials, and now we can run both AWS CLI commands in here, as well as other things that access AWS resources. So this is actually going to kick off a spark job and EMR serverless wait for it to run if it fails, it'll fail the test and we can go back and fix our code number four. uh If we tag something, we want to build this and deploy it to S3. This is pretty similar to the previous one.

A

The only difference is we're not running any sort of integration tests, we're just zipping up some python files and shipping them over to S3 with a the tag, and that's it finally, a schedule so we're going to run a daily Cron job. This is something that will happen at 2 30 in the morning UTC time we can also trigger it manually if we want to- and all this will do- is kick off- that daily ETL job again. If it fails, it'll fail the job, we'll get email notified and we can fix whatever broke overnight right.

A

So now we're going to do our live demo. A few highlights to call out we're going to see what a bad pull request looks like, so we can make sure that those integration tests fail and we can fix our code. I'll show you how to take a GitHub repository from nothing to a daily scheduled, ETL job in a matter of minutes. Let me pull this out and I'll pull up uh GitHub, so I'm Decor on GitHub.

A

If you want to check this out, I will have this entire repository available, so you can go ahead and try this out on your own first thing: I'm going to do is I'll just go to GitHub and create a new repository. So let's go here new repo and I'll call it serverless Spark demo, awesome! That's available! I'll make this private. For now and then I'll just go ahead and create this empty repository, so I'll take this and clone it down.

A

So I'll just do a git clone of my serverless Spark demo and change into that directory.

A

So the first thing that I want to do is I just want to create a super basic unit test. That makes sure my GitHub actions are working in my python. Environment is working, so I've got a little bit of this pre-baked here, but let's open this up in vs code, so I've already created a couple files here. One in this Pi spark directory. I've got this test basic uh test. I just am asserting that one equals one just to make sure that this even works right and then on my GitHub workflows.

A

I've got this unit test yaml. So this is what we saw earlier in the slides. This happens on any push to this repository. It's going to change into the pi spark directory check out a specific version of python and then go ahead and install my requirements. Dev. The only thing that's in my requirements right now is this Pi test. So what I'll do is I'll go back and check all this in so I've got all these files.

A

Do a quick, git, add git commit with our good old initial revision, commit and then I'll push this up to GitHub. So let's switch back over here we'll do a quick refresh and you can see my code there and if we do a refresh again, we should even see that nice little yellow dot there. That's telling me a test is running. So let's go into the actions inside the actions. There's my initial revision and if we go into here, you can see my Pi test job right there.

A

So this is already running it's setting up python3710 it's going to download that it's going to install all the PIP dependencies. It's going to Cache those! So when we run this later, it'll run fast and then it's going to go ahead and run that Pi test job. So you can see that actually just finished running that was super. Quick I've got my one test there that passed in 0.01 seconds- and this will finish and when this does finish, we'll see that the job is green. That means we are good to go.

A

We can add some more code to this repository and we can make sure that as we're adding new code that it's working exactly the way that we want it to work. So the next thing that I want to do is I want to go ahead and just explore some of my data using pi spark with vs code. We have an Amazon EMR extension. So if I go down here, you can see there's this Amazon EMR extension and you can use this to both browse and monitor your different EMR environments.

A

So if you're using EMR and ec2, you can look at them there. There's my EMR and eks environment and my EMR server less environment. So I can even dig in here and look at some jobs that I've run previously and if they failed, I can see why they failed. But the other thing I can do is I can actually take this and create a new local spark environment, and this will be almost exactly like connecting to an EMR node in an EMR cluster.

A

So we can create a pi spark environment, we'll choose EMR, 6.7.0 and we'll use the US East 1 region. There's a few different options for authentication, we'll just use this EnV file, so we'll put some credentials in an environment file. What this does is, it goes ahead, creates a few files for us. So it gives me this EMR local file. That tells me how to use this, and then it creates a couple different Dev containers. So there is my Dev container Json and a Docker file that I can use to build my local Dev container.

A

Let me go ahead and set up step two of this, so that just sets up some credentials and puts those in there and now what I need to do is look at this EMR local I need to log into ECR, so I'll copy that pop into a terminal, real, quick.

A

This logs me into ECR uh Amazon elastic container Repository when I'm in there what's going to happen, is the specific spark image that I'm pulling is the EMR on eks 6.7.0 image that pulls directly from ECR? So when I go ahead and tell GitHub to rebuild and reopen a container what's going to happen, is it's going to pull that container image directly from ECR?

A

Make a few modifications to make this run well locally and then drop me into that container image and vs code, and then I can pretty much do whatever I would normally do in in EMR, cluster I can connect to S3 I can connect to the glue data. Catalog I can you know, start up a Jupiter notebook and run some Pi smart code. We also start the spark history server.

A

So if you want to take a look at your spark jobs, while it's running you can look at that locally too I can go ahead and close that terminal. What I'm going to do is I'll create a new Jupiter notebook now, because I want to browse some data.

A

So here's my new jupyter notebook I just need to copy some code, real quick to fire up my spark session, so that connects to my local Dev container and, like I said this is almost like being on an EMR node in an EMR cluster, so I if I wrap, spark and then, if I want to once that fires up I can do a spark.

A

Sql show databases.

A

This might take a little bit to spin up, but once that does that'll connect to the glue data catalog and show me all my databases that I have now I'm a developer advocate. So one of the things that I do is I maintain a lot of different statistics about the content that I put out there. I've essentially created my own data Lake I keep track of views to different GitHub repositories.

A

I keep track of views on my YouTube channel that I make I also sometimes also keep track of how many times I copy and paste throughout the day, and how many Chrome tabs I have open. Throughout the day, like I mentioned, I've been doing this data thing for a while I kind of have a little bit of a problem, but let's take a look at what's in that database, so we'll just do a show tables from Damon's data Lake and inside there we'll see all the different tables that we're Gathering data.

A

For so you can see, I've got this clones and popular in releases, I'm tracking, all my different GitHub releases and this views table as well. Let's take a quick peek at the data in that views table so we'll do a DF is equal to spark.table and we'll do Damon's data, Lake, dot views and then a quick df.show. So again, this is actually connecting to the glue data catalog. It's going out to S3 it's reading these raw Json files from S3, but this is all running just locally on my laptop right.

A

So this is one of the things that I love about vs code. Dev containers I've got this great contained environment. If I need to share this with somebody else, it's really easy for them to have the same exact environment and do exactly what I'm doing so we can see here. This is the views data there's my repository that I might want to take a look at there's the statistics column and let's talk about what that statistics column is it looks. We can't see everything.

A

So let's do one more thing here: we'll do a df.print schema and we've got tab completion here too, which is really nice, so we can see in the stats. There's two integer columns, there's a count and uniques. So this is the number of views and the number of unique views over a two-week time period and then, in this views array for every day over that past two week time period, we also have a number of views and the number of unique views for that specific day. So this can get pretty hairy.

A

To be honest, if I do a DF, select, stats, dot show I, think that's what I want to do. We'll pull out that column. Take a quick look at it and I'd want to do one more thing here: I want to do a truncate equals false, so that'll run spit out. Some data I'll run that again, so we can have that truncate equals false there. But this is pretty hairy data to look at you've got, you know account and um your uniques and then you've got this array of 14 other different variables.

A

It's kind of hard to look at to be honest, and you can see the size of that array. Changes. We've got multiple recordings from multiple different days. I do have a few views in Athena and Amazon Athena that I use to query this, but I'd rather have some nice clean data. So what I'm going to do so I can develop some some of this code locally as I'll. Take a quick sample of this data and write it out to my test directory.

A

All this is going to do it's going to read that um that spark data frame filter it down to my AWS samples, EMR serverless samples repository take about 100 of those records or actually exactly 100 of those records and dump it out as Json to my Pi spark test directory once we have that we'll be able to start coding a real live spark job against that sample data and then add some unit tests to make sure that uh you know we're coding in the right way.

A

So let me go ahead and create some of those files right now.

A

All right, so, if we look here, we've got a few new files, one in this jobs directory we've got a new GitHub views file I like to group my data engineering stuff together in different classes, when I'm working with specific types of data, so I've got a GitHub use class and in there I'm going to extract my latest daily metrics, so the sample data that we saw in the notebook. That's just extracting that data pulling out some columns. It's exploding that views column.

A

So it takes that array of views on a per day basis and makes a number of rows based on that and then we're going to pull out and rename some of those columns and then find the maximum number of counts and uniques on a per day basis per repo. So instead of you know doing a whole bunch of crazy views in Athena, I can just uh kind of have this code that spits out exactly what I want to be able to analyze. Let me save my other file here, so this will be my Explorer notebook.

A

So what I'm going to do with this GitHub views file is I. Want to create another unit test for it, so let's hop into the tests folder and in there I've updated this test basic. So we have this test that works, and now we have a test extract latest daily value. I've got my mock data frame. That's going to read from that sample. Json data I'm, going to execute my extract latest daily metrics and I'll.

A

Just do a few simple tests right make sure the count is equal to 115, make sure the minimum date is May 31st and then the maximum date is 9. 22. just make sure that the data is as I expect if I go into GitHub again in my unit test, I haven't changed anything there, so all I did was I added. My new class I added a new test.

A

Let's see what happens when I commit all this and put it up onto my Repository so again, I'll do a git, add of everything, add a sample data and update tests and we'll do a git push.

A

Awesome Let me refresh this page now, so we can see again, we've got our wonderful, yellow, dot there and our new code. So if I go under GitHub actions here you can see my unit test is already running. This happens. You know pretty quickly. Obviously so feedback is pretty important, but we can see here it's going ahead, setting up that python. Again it already cached that version, so it doesn't really have to install too much else and then it's going to install a new dependency.

A

So because we added some Pi spark code, we had to add Pi spark as a dependency. So it's going ahead installing that and then it'll run our unit tests. The first test we know, will actually work the second test. Well, let's see if it works, this one takes about 10 seconds to run there. We go actually 10.48 seconds that second test passed. So we know that we added our GitHub views class. We know that we're extracting the data and the way that we want.

A

Let's move on and see how we can add a new uh integration test when somebody opens a pull request.

A

All right so I added some new code here, magically um I added a new validation job, so essentially I'm going to take my spark session create this up. Read from my data lake table again extract those latest daily metrics and I'll do some assertions here, and this is going to be an integration test that runs over the actual data that is in S3.

A

So I want to make sure that you know I have the right count of Records I want to make sure these uh metrics fit within what they should be, and this is again not just the sample data, but the data that's actually over there on S3. So this is useful when somebody opens a pull request. You want to make sure that you know there's no regressions in your data pipeline or anything like that, so this is going to run over the actual data, I added a new GitHub workflow integration test and on the pull request.

A

This is going to go ahead and run these tests. There's a few additional things here to kind of call out. One I've got this oidc role iron. So when GitHub actions run, it's going to use this configure AWS credentials and it's going to essentially assume the role that I pass. It there's also a job role iron. So when we run the EMR serverless job, we're going to go ahead and use this role to run the job and we have our application ID there too. This is a test application, that's running in EMR serverless.

A

So all this does it packages everything up uh copies it to S3 under this GitHub shop or the commit that comes in and then we're going to go ahead and start this integration test. We've got a script here that just runs an EMR. Serverless job runs this validation.pi script. If that job fails, it'll fail the entire job. So let's go ahead, check out a new branch.

A

Look what the changes are locally, we'll add those changes and we'll add our integration test. So let's go ahead. Push that up now, if I refresh, you can see, we've got this new feature: integration test Branch. Now that's not running the integration test, yet first we have to actually create a pull request. So let's go ahead. Compare a pull, request and we'll say: merge in my awesome code: let's go ahead. Click create pull request because we're adding that GitHub workflow as part of the um as part of the change in this pull request.

A

It's actually going to run now. So, if you see down here, we've got our spark job unit test that ran previously and then we've also got our new Pi spark integration test. This is the one that's going to go out, run the script over real data and make sure that it's actually working so we'll go here. Look at that really quickly!

A

There's our integration test. This will deploy to a test bucket and then run that validation code. It already copied the pi spark files up to S3, so you can see it's copying up using that GitHub hash and then it already went and started that Pi spark job. So what this did? It went and sent the uh the code to S3. It starts a new EMR serverless job using that entry point on S3 and we're going to see if that integration test actually works or not. So this goes ahead.

A

It starts and then it waits to see if it's actually going to finish so we'll give this a second and see what comes back from that job. Well, I already came back and guess what it failed. Let's take a look at the error message from that really quickly. So one nice thing here, you get the end of the log. So if you, if something does fail you, hopefully you can kind of dig in and see why it failed, and we see uh we got an import error in our python code.

A

So there's a GitHub views oops. uh Let me go ahead and see what happened there. I'll open up my validation script and um indeed, instead of importing GitHub views, uh we imported GitHub views oops. So let's go ahead and fix that I'll do a look at everything there get commits fix up. Our validation fix our integration test, push that back up and the way that we configured the pull request. Let's look at that again quickly. We have it on synchronize, so anytime, somebody pushes a new commit to this pull request.

A

It's going to rerun that integration test again. So, there's a whole bunch of different filters. You can kind of put on these GitHub actions to decide when these different things run.

A

So if we go back into our actions now we see that the integration test is running again, so I'll go ahead and check out that merge in my awesome code. This is the deploy and validate again already started that EMR serverless job. It's already running so we'll see if it goes ahead and completes that, and let's give that just a couple minutes to run aggregate that data and make sure it all fits within our assertions. Sometimes there are different Frameworks out there. You could use Great Expectations soda core Monte Carlo. These are all different.

A

You know testing, Frameworks and assertion Frameworks that you can use to make sure that your data is fitting in with you know what you expect right, and this is really important, as you're building your data pipelines to make sure that the data fits the shape that you expect that there's no spikes in the data or that there's no anomalies in the data, so these Frameworks can help a lot I'm.

A

Just doing some very basic assertions in my validation.pi script, you can of course, do a whole lot more here with all these different Frameworks all right, so that job finished. If I look at the last few lines of that Pi spark job, we can say see it went from running running to success, so it looks like our validation actually passed. We go back to our pull request now merge in my awesome code. I've got the nice green check mark next to it.

A

I can go down here, hit the big green button and I'm good to go. Let's delete that Branch go back to the main branch and you can see there. We've got our pull request in there. Now, let's go on to the next step.

A

So the next thing that I want to do is I want to show you how you can take this code and deploy it when you tag or create a new release. So let's go ahead and get that set up.

A

All I'm going to do now is create a couple new GitHub actions. The first one is going to be this deploy one so in here anytime, you push a tag. That's semantically versioned we're going to go ahead package up these artifacts and ship them out to S3 with that tag, so you can use semantic versioning and your production data engineering pipelines all we need here is this prod S3 bucket name. So it's going to take this data, push it out to that prod, S3 bucket.

A

This is similar to the integration test. It you know, checks out your credentials and then all we're going to do is copy those Pi spark files. The other thing that I want to show in a little bit is how you can run this job so I'm.

A

Also, creating this new run job GitHub action, and in here you can see this is going to happen on a schedule, so at 2 30 UTC every day, it's going to run this ETL job and I added this workflow dispatch too, because I kind of want to kick this off manually, maybe to do a backfill or something, but this is pretty similar to the integration test. It's going to get my AWS credentials, it's going to start a pi spark job and it's going to wait until the job is done.

A

If the job fails, it'll fail that GitHub action and I'll get notified about it pretty quickly. Of course, we always get notified about it when you know the data's bad somebody always notices before the uh the machines notice right. Finally, this is my production job. Now I've got a main.pi script. This is pretty similar to the validation job. It just extracts. My data uh gets the latest daily metrics and, in this case, we're actually just going to write the parquet data back out to S3.

A

So, let's see what happens when we commit this and push it up, I'm gonna do another. Get add.

A

Add deploy and scheduled actions now this won't do anything quite yet. So what we have to do in our repository we're going to create a new release. So let's go over here. Click this create new release button and what we want to do is we want to create a tag when we create this release, so we'll do v001, I love initial revisions. If we want, we can click. The nice generate release, notes button and we get these uh pretty release notes automatically generated from all the different pull requests.

A

I'm just going to go ahead, hit, publish release. Now, we've got a new release out here. If I go to actions and look at there, we've got our package and deploy spark job. That's going to take all the spark spark code, push it to S3 and that v001 tag I'll. Take a quick look at that this job runs pretty quickly. All it's doing is you know an S3 copy, it's already done so we've got our PI spark files copied up to S3 and this v001 location awesome. Let's run our job.

A

Let's see if this actually works so overhearing the actions we've got. This fetch data workflow that we added so in here I, can go ahead because I added that workflow dispatch trigger I can run this workflow manually and you can even add inputs there too. So if I wanted to run a different version or something like that or put it up to a different test bucket I could do that too. But I'm going to go ahead, hit, run workflow and now this is actually going to go, run our ETL job.

A

Let's do a quick refresh here. It's going to fetch data, transform it drop it back into a lovely parquet format in S3 and there you go. You know a little bit over 20 minutes, I think, but uh we just went from a completely bare repository to automatically running serverless spark jobs and automatically deploying them and running unit tests in a matter of minutes. So this job is running now. It should finish up in a few minutes and then we'll have our parquet data and that's about it.

A

So I hope that that was pretty useful for you, um I think the real power here is just there's no infrastructure right. I did all this without creating a single server. I could have done all this. Just uh you know using Code spaces if I wanted to, but this is all with GitHub actions with EMR serverless those jobs just go ahead and run I, don't have to worry about the infrastructure or scaling them up or down and with GitHub actions.

A

It's really easy to automate all sorts of different workflows, depending on the type of events that are coming to the GitHub repository all right. So that's it just to do a quick recap. What we saw there was. We saw how we can use a GitHub action with a pull request. Even when a bad pull request comes in and it fails, we can go back and fix that make sure the code is operating as we expect it to. We can use GitHub actions again to implement some continuous deployment.

A

So when we tag those releases, they immediately get shipped out to S3 and you can immediately use them in your data engineering pipelines and then finally, we saw how we can also use scheduled, workflows and manual workflows as well. We can run our ETL jobs directly from GitHub actions, so anything you ever wanted to do is possible. If you want to come, find me I'm. The court on GitHub I have a whole bunch of sample code there.

A

If you do check out EMR serverless we've got an EMR serverless samples, repo on the AWS samples organization, on GitHub and I'm to court on Twitter and Linkedin and YouTube so come check it out. Thanks again hope you have a great one.