GitLab GitLab Tech Talks, 30 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Orchestration to Delivery: Integrating GitLab with HashiCorp Terraform, Packer, Vault, and Waypoint.

Description

GitLab enables the efficient implementation of GitOps and infrastructure and code strategies with tools like HashiCorp Terraform, allowing organizations to efficiently and continuously roll out updates to infrastructure with shorter TTRs, Time to Recovery. It does this by enabling teams to leverage several other integrations:

- HashCorp Vault to manage secrets during CI job execution securely
- HashiCorp Packer to build images
- HashiCorp Waypoint for application delivery

The joint solutions developed by HashiCorp and GitLab are helping organizations find a better way for application development and keeping delivery, and infrastructure management workflows in lockstep.
This session will show how GitLab integrates with HashiCorp Terraform, Packer, Vault and Waypoint.

Speaker: Abubakar Siddiq Ango

#Terraform #GitLab #CICD #HashiCorpVault #Waypoint #HashiCorpPacker

If you liked this video and want to see more from HashiCorp, subscribe to our channel: https://www.youtube.com/c/HashiCorp?sub_confirmation=1

To learn more, visit our hands-on interactive lab environment, HashiCorp Learn: https://learn.hashicorp.com/

HashiCorp is the leader in multi-cloud infrastructure automation software. The HashiCorp software suite enables organizations to adopt consistent workflows to provision, secure, connect, and run any infrastructure for any application. HashiCorp open source tools Vagrant, Packer, Terraform, Vault, Consul, Nomad, Boundary, and Waypoint are downloaded tens of millions of times each year and are broadly adopted by the Global 2000. Enterprise versions of these products enhance the open source tools with features that promote collaboration, operations, governance, and multi-data center functionality.

For more information, visit: www.hashicorp.com or follow us on social media:
Twitter: @hashicorp
LinkedIn: https://www.linkedin.com/company/hashicorp
Facebook: https://www.facebook.com/HashiCorp

A

Hi, I'm a worker and in this session, I'll be sharing with you, how gitlab the product with hashicorp technologies like terraform, vault and console enable enterprises to maintain secure and highly available declarative systems.

A

But the main focus of this talk will be how gitlab the company uses these same technologies that is terraform and console to deliver great service to users of gitlab.com our sas offering of the gitlab product. But first introductions.

A

I am bakr dick ango, a developer evangelism program manager at git, lab speaking to you from len I'm also a hashicorp ambassador gitlab is an all remote company with over 1 000 employees in 67 countries. We pride ourselves as the first order: well, probably not the first, but an all remote company even before the pandemic started, and we create an all-in-one solution for the entire developed life cycle, getting enterprises from idea to production and back all within a single product.

A

But we don't do this alone. We work with other partners through integrations with best-in-class technologies like the hashicorp suites of technologies. We ensure excellence for our enterprise, customers and users of our sas offerings. That is gitlab.com by making sure that they can leverage on the awesome technologies like console, terraform, vaults, parker and waypoint, to make sure they are able to deliver on their product enterprise. Customers of gitlab who deploy gitlab in-house are able to ensure high availability with console, especially when their user account starts exceeding 5000.

A

Terraform is the main component of gitlab's infrastructure as code and github strategy enabling users to take advantage of the power it brings to the enterprise. Terraform is a huge, powerful platform that has been instrumental to almost all stories of infrastructure as good.

A

Ensuring secure and efficient management is extremely important to the enterprise, allowing companies to be able to prevent privileged escalations and leakage of secrets. Also, securing your cr secrets from being linked within ci jobs is also instrumental.

A

Instrumental gitlab with vault allows you to ensure that your secrets are safe now, but I'm not here to talk about how all these technologies just integrates with gitlab, but how we, as a company and kit, lab use, terra format console, but first a little background.

A

With over 8 million users, there are lots of activities happening every second, that need to happen efficiently and reliably fast, but that wouldn't be possible without a reliable infrastructure. Gitlab is hosted on google.com, with a pool of virtual machines for each of the components that make up the application. That's registry to gitlab api pages, gits and others.

A

Each group has a load balancer in front of it. One of the largest challenges when growing fast is keeping up with demand and we've been having quite a lot of demand when the infrastructure is static, which can be the case when your infrastructure is based on virtual machines. It is not trivial to dynamically grow and reduce the size of the infrastructure as a reaction to an external factor which can be anything which can be loaded due to a new, an announcement or an issue within the community.

A

This is a two-fold problem serving our customers is not fast enough and infrastructure cost can be high because of the infrastructure. Size is the same regardless of demand. It doesn't scale up or down efficiently.

A

At gitlab, we use teraform to manage vm lifecycle in google cloud platform and with chef for ensuring consistent configuration across all of our infrastructure. Auto scaling vms with these tools is of course possible. However, in order to accomplish that, we need to write additional tooling, which is often unnecessary.

A

That will allow us to scale vms on demand, even with the vm auto scaling, automation in place, the response times of creating new machines and installing and configuring them and everything else that is necessary to run our application, will lead to git lab being deployed or installed in a matter of minutes which can cause a lot of issues or delays, but with kubernetes.

A

Both of these problems are addressed on the go. Kubernetes allows us to be able to scale our infrastructure based on demand at the moment, rather than based on the best guest projections, when using auto scaling with vms, when the demand is low, kubernetes allows us to automatically scale down infrastructure, reducing cost instead of with vms, where you simply just throw servers at the problem and when you notice load is going down, you remove the servers. Kubernetes is also made to be an orchestration backed by a vast community.

A

This allows us to leverage industry standard tool instead of we having to write and maintain our own costume solution in-house.

A

One of the initial reasons of moving to the google cloud platform from our initial provider is to take advantage of the google kubernetes engine, which was chosen for its maturity among the other offerings that are available within the community. After all, kubernetes came out as a project from google. I think it was initially called bug and has been evolved within google.

A

We also provide home, chats official hem chat to our customers, be able to deploy gitlab at scale on kubernetes and in true gitlab style. We believe dog food in our own helm chat will allow us to proactively improve the charts our communities, migration isn't complete. Yet all the stateless components of the application have been moved to kubernetes, while the remaining components that handle all our persistent storage have been gradually moved. You will notice in the image on the slide that we still maintain some legacy via infrastructure.

A

The best part of we dog food in our own chat is we get to learn new things and experiment even before our customers hit those problems, and we share this improvement with our customers, so this allows them to get best in class solutions.

A

Now our ci infrastructure is another component of our cia of our assess service, where users get free, paid, free or paid minutes to run their ci job. We maintain a fleet of vms. We call runner managers, as you can see in the slide they spin new servers that run all of our ci jobs and those of our users by creating virtual machines on gcp and take them off later.

A

Everything I have described for of our entire infrastructure is maintained in a single monorail and organized in directors that we call environments. This allows our infrastructure team to be able to collaborate using git workflows.

A

All terraform workflows are done within ci, with terraform applied manually triggered. It is ensures that no misconfiguration that's been introduced by any team member gets past the reviews it needs to be reviewed. This also allows uh the whole of our team to see how and when changes are made.

A

The perform code, reviews and review all the logs and those logs are persistent over time, so that we can always know why. What changes what's made at what time and for what reason, with a monorail power and a single ci configuration file, changes are only applied to specific environment in which they are made.

A

That is when the ci script runs. It checks for the environment configuration that has changed with a new merge and runs the job for that specific environment.

A

The terraform files are public for infrastructure in the link provided on this slide for you, you can review it and be able to even recommend changes. It's an internal, it's a mirror of an internal project that maintains the file, so there's no harm that we expect needs to be done. They are just a bunch of terraform files that anyone can review and make recommendations now.

A

Environment in the context of our infrastructure can be provider specific, that is, it can be a bunch of cloudflare rules or aws configuration or any other configuration specific to a specific provider. Then we have infrastructure environments.

A

Which can be our production staging so on and also what we call env project? These are terraform configuration files to manage our gcp projects, along with the iem permissions service accounts and other settings needed to maintain those projects.

A

Aside from the resources created in them all, this makes it easier for our engineers to quickly create ephemera projects, an environment, to test out different scenarios or different uh issues that might have occurred, and they need an environment to fully test it out now, with a monorail pool, drifts are bound to happen due to a number of reasons. It can be. Probably uh much was done, but terraform apply was not executed.

A

So what we do is to run terraform plan with the detailed exit codes, flag enabled, which checks to see if there is any difference between the production and the new merge and sends a slack message anytime, a change has been made. A change has been made and needs to be reviewed.

A

Now, like I said earlier, all of our entire infrastructure is managing a single model apple, including our database. However, for our database, we leverage on console to ensure high availability service level objectives for our progress database include a maximum of 45 minutes downtime budget per month for maintenance.

A

That is if, in a moment, we have used up all those 45 minutes. We move our focus up to availability instead of making new changes or pushing new updates to our database infrastructure.

A

Also, a maximum of 200 milliseconds is our threshold for any query that needs to run into our database. Now, all this is possible with console ensuring that our database is in sync with the secondaries now. Lastly, for our enterprise customers, we have who, who have grown beyond just having a single instance of git lab and have more than 3 000 users.

A

Some of them have up to 10, 000 or 25 000, and so on, especially in huge corporations. It can be a challenge to be able to keep up with the resource needs and the issues that they need tend to work. With this, we did an experiment with our support team, where they reviewed different issues that our customers are facing, and we came up with reference architectures for okay, if you have 5 000 users. This is what we think is the recommended way to be able to deploy your infrastructure.

A

If you have uh 25 000 users, this is recommended and the core part of those reference architectures is console with a recommended minimum notes of three. They are able to ensure that they have high availability of the infrastructure within their team, because you definitely need to handle a lot of requests from git pools, git push registry and a whole lot of features that gitlab ships.

A

This is the end of my presentation, I hope you've learned or picked quite a number of things from how gitlab uses terraform and console. Like I mentioned earlier. You can definitely go through our infrastructure project, see all the ways we've been using.

A

We've been using we've designed our architecture and you can even create issues or create much requests I'll make recommendations. Probably there are things you think. Okay, we need to improve on that or we need to uh do differently.

A

We cherish our value, which increased transparency, and that is one reason why we decided to make that project public and accepting contributions from the public. Thank you very much for your time and if you have any questions, you can always check me on my website or banggood.com. If you want to learn more about git lab, you can check us on about.

A

Studentkidlab.Com.

A