GitLab PubSec & Regulated Industries, 16 Jan 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Commit San Francisco 2020: Driving Cultural Change to Improve Software Development and CI/CD

Description

Have disparate, distributed teams? A plethora of tools, processes, and development practices? Silos of people and data with obstacles to communication and knowledge transfer?

Learn how SRI rapidly transitioned over 2,500 projects across 350+ groups to radically improve their development practices, CI, artifact delivery, and client collaboration.

See how they:
- Lead by example to increase GitLab adoption
- Educate internal teams and clients on best development practices
- Continually improve operational efficiencies and delivery through cultural and organizational transformation
- Meet compliances for heavily regulated environments

In this talk, SRI International Software Engineer Roland Heusser offers valuable practices for undergoing rapid transition. Take away tips to improve delivery practices and visibility throughout your SDLC.

Slide deck: https://drive.google.com/file/d/1kVE7uXORokJtDq-9Q5us2iKGcXSI4fGm/view

Read more about our product vision: http://bit.ly/2IyXDOX

Learn about FOSS & GitLab: http://bit.ly/2KegFjx

Get in touch with Sales: http://bit.ly/2IygR7z

A

Thank you very much hello, everyone I hope, you're all staying warm and this silent disco setting is very interesting. So, as you probably already know like my talk, is going to be about driving cultural change to improve software software environment and see ICD. My name is Roland hi, sir I work for SSRI international I'm, a software engineer, and just two. So my the field that I work in is micro services, DevOps and I.

A

Work with the Sora ventures seem to transfer technology from our researchers, to our venture savvy or to our venture ventures and our clients. If you haven't heard about Nestor, is our I international. Before we are a research company in in Menlo, Park we've been there for over 70 years. Our main clients are the US government and a lot of commercial clients.

A

We've been around for over 70 years, so this was basically me when I started this RI 3 years ago. So, like you're, ready to go, you want to see all the problems you want to design solutions for them. You know implement code, and you may have found yourself in a similar set to a situation where you want to get going. You want to yeah, you want to like start, collaborating and figure think things out.

A

There are process and, as you start looking left and right, you see more and more people and you see mud going all around you and you may or may not start feeling a little bit stuck in the dirt. So what happened so in 2017 when I started SR I and some of these things may are still true today, but we we have a lot of different groups and teams, so we have five divisions and inside each division we have five to ten like groups and they all operate very independently.

A

Some of them don't even use our carpet. It services like they have their own IT services. They use. They use like different repository systems based on their current needs and depending on if they work on government contracts or if they work on commercial contracts, they have different policy and compliance requirements.

A

Many of these tools were in self service, so you either had to reach our corporate helpdesk or of that particulars group, their help desk to help you and one of the challenges for my team that I joined with was we were tasked to work with all of them, because we were trying to introduce them to micro services like how to implement them and so we're like. Well now we have to learn how to interface with 20 different systems and it's it became a big burden for us.

A

So we we felt like this biker or cyclists just like, and we see this ball in front of you and we're like well. I just see problems like how can I, how can I continue like we need a better way to work together, so we knew something had to change, but we were very small team, so we had no, we didn't have enough backing from management or we didn't have any anything that would allow us to make the entire organization change so spoiler alert.

A

We made it somehow and I'm going to talk about how we did this so from the status code. To now about two and a half thousand projects for we have in our gitlab instance, you have a little over 500 people, so we have five projects per person.

A

How did we get there? So, when I started in 2017, as I said, we needed to find a way for us to work with the other teams. Fortunately, we didn't have a lot of compliance or policies imposed on us, so we went ahead and just spun up our little rogue instance of gitlab sieve yeah.

A

We started hosting all our projects on it and started pointing P start and tell told people well just go look at our instance. This is our code and we started collaborating with them. In this way, a lot of people got really excited about it, they're like well. We want to put our projects on there too, and so we started adding them by a case-by-case basis, but fairly quickly. This case-by-case turned into a lot of projects as we as we as we grew.

A

We realized based on feedback that we needed more capabilities, so we started adding features like more CI CD, more privileged runners.

A

You may or may not have heard about problems with privileged owners for s RI, since we have a lot of government work, we want to make sure that if two projects run on a shared runner that they can't modify, the host and docker built, for example, cannot run in non-privileged system. So we needed to figure out a way. How can we? How can we do this without running into into compliance issues where we accidentally modify our hosts? One solution for the day, for this would be an alternative build system like Kanaka, but back in 2017.

A

It wasn't that well established. Yet we later realized stuff, we needed backups. So we towards the second half of 2017. We started backing off our system and towards the end of seven in corporate IT and our Center for software engineering. That provides a lot of services that we consume internally approached us and said so, who is running this kid'll FCE like we don't know who, in the neither corporate IT or our IT services, are providing it and our team were like well, we can't do and they they ask us all these questions.

A

Well, so how compliant is it? Can we put it-- our project on it and I'm not a citizen so pretty quickly? We realized that we needed to change this. Fortunately, they didn't tell us to stop they. They recognized that there is a huge benefit in using gate lab, so they did a threat assessment. They worked on creating multi-factor authentication using the gate CLI, which I haven't seen another tool providing this kind of capability and by the end of 2017 we had a gait lab EE instance spun up in 2018.

A

This was our kind of migration year, where we migrated almost all our systems. We also discontinued all of the redundant systems that we had, so that was a big year of like getting rid of licenses that we don't no longer have to pay and in 2019 we made if we basically just worked on increasing the usage of our current gate, lab instance and even upgrading it.

A

So this is how, in just a few minutes, how we made this entire migration process. So what helped us transitioning from today from the status quo to where we are now I identified six factors that I think we did right that helped us for this migration and the the first one would be leading by example. So we we started finding a system that fits the needs that we had and we we started using it showing other people how we can be successful.

A

Then the second factor is templates so like we, we don't just like lead and show of crazy tricks. We also showed them how we did it and so that others can learn from from yeah how we, how we achieved so the others can learn from from us. How that we take it, they can achieve what we did.

A

Another important piece is education, so, together with the center of software engineering, we started setting up like a helpdesk email specifically for get lab. We started adding channels to matter most so that they can collaborate or ask questions. We provided hackathons and demos where we you skate lab, so more people got exposed to it and saw how it can be used.

A

Our fourth factor is evangelism. So we at the beginning, when we started using gate lab, we didn't I, didn't expect it would cause such a drastic change, but like just the people, seeing us using it and getting excited about it kind of snowballed and helped getting gate lab like get corporate IT and everyone else excited about it. So I think eventually, ISM is a very powerful tool that we had and then oops I went one step too far.

A

Feedback I think our Center for software engineering and all the people that were involved setting up the the gate, Lab instance did a great job at listening to the feedback that people brought to the table. So when people had issues with kid lab runners, if they need help integrating in existing infrastructures and said, how can we use it with kubernetes? How can we use it with our artefact delivery system like our people?

A

They are very, very responsive and they set up documentation and they helped everyone to like be successful, and then the last piece is compliance. So, while our first instance of kit lab was not compliant, our current instance makes it very easy to be in compliance with all the restrictions and requirements that we have. They.

A

We put a lot of energy in automating our compliance process where, before you had to send emails to people and they they had to add you to a project or even set up the project, and they had asked your manager or other people to get approval. So it was a very slow process that could take days now with our current compliance process.

A

How it works is we can set up the project, add a configuration file and that will in will be picked up by an automated system that then actually verifies who can have access or not I'm, going to talk about this on this next slide a little bit more because, as I just started, mentioning compliance can get a little dirty and sometimes it kind of hits you in the face with a lot of third there we have, as as a government contractor, we have to follow like cui and dietary restrictions and as a commercial contractor.

A

We also have to follow their contract requirements so.

A

Which is to say so we in one of the things we did is we decided that hosting on premise was the right choice for us so that we have more control, then the biggie? What I mentioned earlier is for gate. We added a plug-in that we can do multi-factor authentication for it, especially for external collaborators. On our system, then the groups, for example, top-level groups on gate lab, are managed by our IT services, but self-service is very important, so all subgroups are managed. Our self-service and projects are self-service by default.

A

So whenever I need a project, I can just create it and, as I mentioned, I can add a meta file that says which contracts this project is associated with, and then our bot building they will read this file and decide if that project is one that is access restricted or if it is access, angry, unrestricted and if it is restricted the how we manage or how we ensure our compliance is that nobody can let in psych it lab, can add or remove users.

A

At that point, then, we just use an LDAP, a process where we just use them through our very add and remove users through LDAP and an admin will do those changes.

A

Yes, so what is the result? The result of this process is that we have self-service infrastructure. I think this is a very important piece that that we initially didn't have and that we were able to achieve with get labs so that people can self provision things like we do if they have to ask permission for everything it takes a lot of energy and time.

A

We greatly facilitated our compliance process. We made it after the fact that I can create the project and then I go through the compliance project process and it's also more transparent. I know how they are how these, how our compliance process works, because I know how the company decides on what information these decisions are being made. We reduced a lot of cost because we there is less waiting, there's a lot less communication for us to to get repository set up and configured the maintainability is has greatly improved.

A

Our our our Center for software engineering always tells me that they, the Freak the frequent updates updates on every 22nd or in the curse. It's they always get. The latest and greatest features that everyone wants to use and they regularly update the system and we improved collaboration.

A

So our the takeaway that you I hope I hope you take from this talk. Is that start small, with with a small instance, make sure that it works for your environments? Try out all the different settings that you have try to lead by example. So yeah.

A

If you try out everything that you use at your company, you can see what breaks and what doesn't and make sure you automate your compliance processes or make them make sure that they have as little friction as possible and we invested in energy a lot of energy in setting up documentation, especially very deviate from like standard documentation. That gitlab, for example, has so like the outline. This is what we do differently, but you should pay attention to.

A

But if you just want to know how a merge request works, go look, go, look at a get lefty a standard documentation or if you want to know how pipelines or the or the Yambol configuration works, go to gate lab and yeah I actually went way faster through this presentation and I plan to but I think this would leave some time for questions and so you're running around with the microphone and yeah.

B

We do have listener questions, so anyone have any way yeah.

C

B

B

I'm particularly interested in in more details on the automation you've done, particularly around the ITAR compliance. If you want to have an AI car or some other sensitive, you know distro a 2d distro deep project. Now, what kind of things do you do in your compliance process to help that out.

A

That's a good question so.

A

I'm trying to think how I can best answer your question, so what we? What we do is we have I'm gonna circle around your question for a second, so what we do have so when, when we create a new project, we had this made-up file. That says which contract is it is associated to and that will be picked up by our automated system, which then will notify like one of our admins.

A

So this is a project that has been labeled as like I to restricted, and then we have a group that will will like go through like the export control. If we want like non-citizens to work on that particular project or and yeah like verify, who can have access, then what? What then will happen is that they will reach back to the person who created that project and ask like who, like who else they want to add to it and then they'll.

A

Those people will then respond and- and they will like check the restrictions on that project and add them to the LDAP group. Now I think your question is more around like how they then ensure the ITAR compliance.

A

A

Well, our our meta filing crew includes a list of all the people that you have correctly. We add all the people that we want to have access to. So it's basically a yam L file that we customized that the bot will pick up and yeah I think that basically solves that problem. Does that somehow answer your question? Okay,.

D

Okay, just one moment, I could.

C

Gradually, in on the radical transformation now, I have two questions, see any challenges. Have you seen with import that you'd import from native get? Oh? Is it from another provider? We.

A

We have like a mix of multiple, like we had a lot of different on-prem, the on-premise version from all the gait lab competitors that you know, and we had also SVN repositories, and we had mercurial repositories that we had to migrate. So we yeah, we went through all of those any.

C

Specific challenges you would like to give.

A

Us I think the biggest challenge we had is that a lot of teams laughter autonomy, so they're like why should I migrate to a copper corporate IT provided service? If we can just do it ourselves, and why now do we have to add this meta information to each repository, which is just one file, but there is still a challenge around.

A

They feel like they lost control over their their baby and I can I can understand that, but at the same time Gettler bring so much more benefits that that is a naturally it's. It's such a small drawback. It's yeah great.

C

D

Hey there so I'm curious. How did you guys go through that migration process of migrating from SPN to get up? How did you guys handle that period? Were you maybe working in both environments and dealing with dependencies that make this in both ecosystems, yeah.

A

That's a that's a in I think in most of the SME and repositories. We were fortunate enough that we could, like just the I, mean, use the migration script that retains the changelog and just like migrate those repositories. We still have some huge SVN repositories remaining that we we can't migrate at once because they're gigantic, so we have to chunk it chop them into pieces, but in general we were able to like just migrate them in one step.