GitLab PubSec & Regulated Industries, 25 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Commit Virtual 2020: Running DevSecOps at the Brazilian Fed. Public Ministry with Open Source Tools

Description

Speaker: Leonn Paiva

The Brazilian Federal Public Ministry has evolved its deployment strategy from a basically manual approach with few automations to a fully automated, continuous delivery strategy based on GitLab, Docker, and Kubernetes.

Learn about their DevSecOps Flow, which includes test automation, source code quality assurance, deployment tests, automation of Docker container builds, and deployment in Kubernetes clusters. See how security requirements are validated during the continuous delivery process by validating vulnerabilities in the source code (static analysis) and security scanning of Docker images...all created exclusively with open source tools.

Get in touch with Sales: http://bit.ly/2IygR7z

A

Are you curious about how to integrate open source security and automation? Let's listen to leon piva of the brazilian federal public ministry. Talk about running devsecops at the brazilian federal public ministry exclusively with open source tools. He'll share how they evolved their deployment strategy from a basically manual approach with few automations to a fully automated continuous delivery strategy based on gitlab docker and.

A

B

Hi everyone, thank you all for showing up. First, I would like to thank gitlab for the opportunity of sharing this content with you today. It's a pleasure to be here.

B

My name is leon piver, and I work as a senior software engineer in the government and the private sectors too. My main expertise is in high performance software development with c plus, plus python and java. It's been a while that has been focused in cloud security and devops. Overall.

B

What I present now is how we've run devsecops at the brazilian federal public ministry, and we do it exclusively with open source tools. This talk is not only about how we do it, but it's also about our journey to get where we are now and how we evolve it to the current state.

B

So these are the topics that are recovered to pass on our devsecours flow. First, I will talk about our legacy stat strategy that sends us back 10 to 5 years, so you can better understand the scenario and pain points that pushed us to our new strategy.

B

This strategy provides us now high availability tools and services for all the country, millions of citizens, 9 000 uh employees in more than 1 000. It professionals today, then we'll talk about the challenge that we face it with with this strategy and how we mitigate it and solve them. After that, I will share our results in more detail. In summary, it took around one year and a half to achieve all that, I'm showing to you today, but with a very short but very capable team.

B

By the end of our time together, I will talk about which are our next steps, next challenges and and what respect in what to come for the future? Okay, so about our legacy strategy going back a couple of years, our infrastructure was in physical machines scattered around all the country and the development teams was also distributed.

B

This led to repetitive efforts lacks of general management. We couldn't get the things right, we did many things twice was bad for us. So later we, the projects were centralized and only the development state distributed with this stage. This. This change also came the virtualization, so we we get better things beside the the physical machines, but the way the machine was were handled was still the same for every critical service. A minimum set of vitro machines were established, one for development, one for homologation and two for production.

B

This was our setup for high availability, even in physical machines, and now in the latest days, was with virtual machine.

B

Almost every operation was manual, create setups, updates maintenance and deploys everything, but the info team was also able to automate some application servers. He starts like a posh web logic, but was this the most okay? The most of builds were manual too, but later some depth introduces jenkins with tests and building their pipelines was. This fact was isolated.

B

uh It was an exception. Most of the operations were manual too. The and even the deploys were done by developers sending in mail for the info team with the source code and the path to be overrated. In the servers, the dev teams used the svn as virtual control, but its use was very inefficient.

B

We had a lot of ranges, a lot of subdirectories with libraries independence inside it, so the process with svn was very slow and very faulty we didn't use a tag with merge was hard to to keep maintaining uh software. The way that we were doing in the past.

B

For for our most critical system, our build time was between 18 to 25 minutes with 40 minutes to deploy it with high availability in our our infrastructure and finally, about our security. It was very insufficient, insufficient and infrequent. Overall, we, it was only done once the service becomes available in production or in the internet. It only checked the main error url and the server in the current stat status, regardless any security alters system updates. Over the time like 10 years, we didn't have updates in some service.

B

So this was our scenario.

B

In summary, our legacy exposed problems that we couldn't ignore. Keeping ignoring the lack of automation in ci cd, the excessive computer resources spent, which reflects directly in the high costs. This is bad for our our government in general. In any sector, it is low development, slow deployment process and the security vulnerabilities that was saw in many contexts.

B

So our new strategy came up to solve everything, but as a government institution, we must keep always the expenses low. To do this, we focused our strategies in free in open source tools.

B

The main decision was to migrate from sva into git, using a self-hosted gitlab community edition that can handle up to 1000 users. This gave us all the tools that we needed to leverage the development in deploy process to reach out the next level with ci cd.

B

We migrated from jenkins to the gitlab ci runner for providing us a fast, smart and integrated way to do ci cd to all projects using less computing resources to use our resources more efficient, incentivize, the environments we came up to docker and later we level up again using kubernetes, uh true reasonable. We could get up and running a cluster with high availability in 14 minutes.

B

With these our kubernetes clusters, we chose to do with vanilla kubernetes instead of uh open shift, bks ibm or any other to keep it in the low cost and almost every functionality that they provide are done with open source. So we can improve and do what they do in a matter of time.

B

In terms of security, we edited sonar cube to perform a star static analysis during the pipelines, so we can better enforce code, equality in multiple languages and sport, new security issues before they award reach to production also to improve security. We use it hub or as container history as it ensures the artifacts are, are protected with policy and rule basic access control, ensuring that images are scanned and free from vulnerabilities before we deploy it to the coordinates, and even after that, we keep securing it securing our our images.

B

Okay, these were our strategy overall, okay, but only defining it for our organization. A lot of organization like us was not enough. We must. It must be easy to apply easy to evolve. So how can we do this for many projects? Okay, so automation came up to the picture again.

B

We developed two projects here to solve that one to only handle the the deploy on kubernetes, so we have a kubernetes deployed project and another called git flow that handles all the brain chain, automatically automatically versioning tests, pipelines and docker everything to get uh to our strategy in an easy way. Basically, the user or someone that will start a new project just has to import the template and use it and all will be available to to the user following our pattern.

B

So these projects, what we should do, what which it should follow uh to achieve all that devsecops flow sure. Let me change here, get back one second, so when our developers, our developers, commit and do new new features it only, it only goes to the development and test environment in the branches after the code passes in in the test and quality way. So we can assure that the code is working and everything is good in a minimum way for the development team testing and after that, for the obligation and production too.

B

So everything must work here. We automatically tag, build and deploy. After that we we have a uh an action to promote a development, tech to production and homologation, and all the process go again and we can deliver to internet uh and everything.

B

So what what happens when the new feature comes so uh a developer commits a code handle app version of here, so the the pipelines do a validation that checks the project status. If everything is all right with merge settings branch settings uh variables, everything if it's okay, the project can run and keep going. So we build after that we test here we have sonar cube here. We have unit test, people can add more tests here and to make the developer life easier.

B

We added a merge request to a job that people can just click play here and we provided a template that they can use it or using another plane plates that the merch requests will be created and then merge it if the pipeline succeeds.

B

Okay, so when the pipeline succeeds, the birds go to the depth branch and runs everything again with more tests or no depending on on the approach or situations, but assuring near the minimum quality that we have building with docker automatically tagging the the version we have the version for for development with better and then deploying it to our servers. Okay, what we have here is the same thing that we have for the development. We can promote a tag better for the master, so it can go to homologation and then production.

B

So we have the project here running here in our school, our kubernetes services, and then we can promote to production. We have here of the same flow that can have more tests more things to test in the production before target, but in the way that we deploy, we have a little change here that we enable to so. The tag here is not is not better. We handle this automatically too and then in the deploys thing we can deploy to kubernetes, but for mobile projects.

B

We we are now allowed to deploy to the storage, to we generate the apk in epa files for deployment in the source, and we can automatically automatically deploy to the store google play and and in app store too. So we have these automations already in place, so can be used for a mini project. You have a back-end template everything that can be used so, which are the risk that we saw in our setup. What we did. Okay, we can crash in production very easy, because the pipeline was fast.

B

We can do things faster that we can before. If the tests are not uh assuring that everything is working, we will crash in production. This effect, we will have security issues in the same way, but what we saw after running these many times if this happens is because we do not have that the process has some fail failure, so we need to improve that before to to keep going more time. We need to to solve this those things test and keep the security and the application working flawlessly. Okay, we saw another problem too.

B

Another risk is the misconception of delivery and deploy. Sometimes we we can't deploy, but we don't need, we don't want, or the high management don't want to they want to to to deploy in a certain time. So we separated two in this way, so you can deploy uh scheduled manual or automatically after using these strategies for around every year.

B

We saw many fail points, so we need to improve the monitoring metrics logs, so we can know when the issue happened and where it happened as we're running with docker and kubernetes, the containers uh go and come very fast. If you do not have logs outdating metrics, we cannot see a problem occurring, the the container will restart and we application we keep running, but we will not see any problem at all if only if the problem keeps uh occurring multiple times.

B

So those were the risks, but we we developed this strategy to solve that show so, which were the challenges that we faced. uh Implementing all these strategies. First was the team. We didn't didn't have a large thing. Actually, it started with only one person which was me starting this project with kubernetes and after that we we get another developer uh summing to to the team in actually, today, maintaining all these structure are only two people: okay, but uh to show you is that you don't need many people to implement a good solution.

B

You need only a capable team, k-pop's solution to so after that we started to solve any many more problems. Another one was, it was training we didn't have training, we didn't have people uh enough to train every every development thing around the the country, so what we did to solve that we set up lightning talks and small events into the organization to share what we were doing so every every two weeks.

B

We got lightning talks to our stocks, sharing our main strategy, our migration, to get using docker teaching the the teams, so they could have a smooth or smooth transition. Okay. So this was our strategy. Another thing was resources. We didn't have resources enough for our vitro machines for our storage to handle all these things, so we couldn't go to cloud which would be very costly, but we we needed to to do it in uh effective way.

B

What we did was grow in a organic way start with little cluster with little memory started, grow as much as needed. So when we grow in an application, we start to grow memory cpu, so we did didn't start small to to reach a higher level.

B

Another thing we have is cost, so we couldn't uh expend much money of the organization we didn't have we here in brazil we have a citation that we must lower the cost even more that we we're not what are now so to do that we choose it open source, uh choose it a new strategy that could could enforce our strategy in a better way. Okay, another challenge that we had was the process we couldn't enforce for all the dev teams to use docker, as they didn't know whether what it was some dev teams.

B

We couldn't enforce uh using kubernetes using git lab in git overall, so we started to show the higher management, how this works in small teams and growing over the organization showing results showing what we could could be doing and how effective it was, how it could be faster than fast uh get a faster development, faster results, so showing these before us give strength to to achieve a higher level for our organization.

B

Okay and the last one was standards. Okay, a team wants to use this, but they don't have enough knowledge in how to do it in python in java node.js. So, what's what we did to solve this? We created reference architectures for the people use. We have templates for okay. How to do is servers with python kubernetes in our strategy. Here is a demo how to do it with spring boot. Here is a demo, so we have these demos for java python, sprig boots uh nodes and serverless and many applications that we developed in our organization.

B

So people can see how it's done, use it improve and get a new project up and running in these patterns very easily.

B

So about all our results, what we achieved with that today, we have around 210 services in production, with this new strategy.

B

With these services, now we saved around 300 000, around 350, 000 brazilian rears, and this this low cost was only able, because we are using open sources- okay, open software- and this was because we didn't use it.

B

Provided kubernetes kubernetes uh distributed system, we didn't use a cloud cloud environment, you could use our own infrastructure, so this uh this saving money is what we need for the government in many companies too. Okay, so we decreased our deploy time by 95 percent.

B

Our main application was about 40 minutes, and now we can do it deploying two minutes around 2 million and 30, and we deployed many applications in history already we decreased it up to 50 percent of good time using caches and operating in this way, the build time was around 18 minutes, and now it is around nine and eight eight minutes using cash and other things. We improve the security using kubernetes and our setup with solar, cube and hardware.

B

Okay, and now we have a replicable process that everyone in our organization can use it on a small project, big project, old project and new ones. So this is uh our main main goal to keep going: keep evolving in many ways. Okay, our next steps that we saw that we need to improve. Okay is alarms. Okay, we we have everything running, but we miss alarms. We miss when something bad happen.

B

We need to know where it happened, how it happens so we can take an action, so alonzo would solve this for us what we have now. We have telegram integration, but this is a a passive way. We must look and see how the thing is going, so we must set alarms that are useful to us and log aggregations. As the containers go up and down very fast. We don't have a log aggregation. We can't have uh effective, uh find in discovery, the bugs and solving them.

B

We need to amplify and evolve our organizational use. So we have a like, I said, 1080 professionals, so we need to to amplify the user to all the development teams, so these need to be amplified even more because today, not all development teams are using this strategy, so they are coming up. We are, we are helping them but, as I said, the team is very small.

B

We must handle in step by step, so we need to amplify so we can evolve more and for the last thing we we need to and a good strategy to back up and restore uh operations overall in containers kubernetes. We have what we already have an idea: what to use in kubernetes, velero and kubernetes operators for for the containers, so we have an idea where to go, but we didn't start it yet. So these are the next steps for us.

B

So this is what I shared with you today. Thank you for your time. Thank you all have a great day you.

B