GitLab / Incubation Engineering Server Runtime and Remote Development

Added the ability to update workspaces from the glab CLI.
Added the support for --output=json/--watch flags for workspace list and view commands

Added the ability to list, view, modify and create workspaces from the glab CLI.
The following commands have been added

glab workspace list --group=gitlab-org
glab workspace create --group=gitlab-org --agent=1 -f devfile.yaml --editor=ttyd
glab workspace view --group=gitlab-org 1

This is a summary of what I've been working on in the Server Runtime SEG in the last 4 months.

To learn more about server runtime:
https://about.gitlab.com/handbook/engineering/incubation/server-runtime/

To see more videos on Server Runtime:
https://www.youtube.com/watch?v=59k2b43I4oI&list=PL05JrBw4t0Kr6sicAh753uzvoPqrGZuLB&index=1

Chapters
0:00 - Introduction
0:11 - What is server runtime
0:53 - Where are we today?
2:38 - Architecture
4:43 - Demo
10:05 - Whats next

I wanted to do something new this week. I wanted to take the existing Server Runtime / Remote Dev architecture and see what we can do with it. I will mostly be making changes to Devfiles in these updates and therefore these changes should work both on DWO as well as Server Runtime.

This is a continuation of the "What can you do with Server Runtime Series?". This is **Part 4 - Airflow**. Airflow is one of the most common tools used by Data Engineers. Typically you orchestrate ETL/ELT pipelines via Airflow. Airflow can get hard to work with when you're working in large teams. There is always the fear of overwriting each others changes. Server Runtime can create a dedicated Airflow instance from a branch or a MR that can then be used to test your changes so that you don't accidentally interfere with the work that someone else might be doing at the same time.

This idea needs to be credited to Fred who is working on making Airflow better on GitLab in the Airflow SEG.
You can find out more about the SEG at:
https://about.gitlab.com/handbook/engineering/incubation/airflow/

0:00 - Introduction
0:57 - Demo

I wanted to do something new this week. I wanted to take the existing Server Runtime / Remote Dev architecture and see what we can do with it. I will mostly be making changes to Devfiles in these updates and therefore these changes should work both on DWO as well as Server Runtime.

This is a continuation of the "What can you do with Server Runtime Series?". This is **Part 3 - Workspaces for Designers**. In this example I create a workspace that runs [PenPot](https://penpot.app/). PenPot is an excellent OpenSource collaborative tool for designers. The PenPot architecture consists of several components including the frontend, backend and exporter (written in Clojure) and database (postgres) and a broker (redis).

The Devfile for this project can be found [here](https://gitlab.com/gitlab-org/incubation-engineering/server-runtime/penpot/-/blob/main/devfile.yaml).

0:00 - Introduction
1:05 - Devfile overview and creating a workspace
2:25 - Authentication
2:49 - Penpot

I wanted to do something a bit new this week. I wanted to take the existing Server Runtime / Remote Dev architecture and see what we can do with it. This is Part 2 of this series in which I create an isolated workspace using Sysbox. The workspace is "rootless" and can run insecure workloads such as docker and kubernetes

0:00 - Introduction
1:39 - Creating a workspace
3:08 - Building a docker container in the workspace
4:32 - Docker compose example
5:01 - Running k8s in the workspace
6:32 - Workception

I wanted to do something a bit new this week. I wanted to take the existing Server Runtime / Remote Dev architecture and see what we can do with it. This is Part 1 of this series in which I create a Devfile so that I can create a dev workspace where I can make handbook updates.

Chapters
0:00 Introduction
0:41 Demo

This is the weekly update for the Server Runtime SEG for the week of the 23rd of December 2022

This week I've continued on build on the excellent GitLab agent for Kubernetes. Updates include:

1. Ability to change workspace state
Ability to start, stop and terminate a workspace

2. Reflect state from Kubernetes
Server Runtime will reflect the state of the workspace in Kubernetes

3. Retroactively fix state
Server Runtime will now refresh the state of the workspaces it is managing when the agent first starts up.

Chapters
0:00 Overview
0:49 Demo - Changing workspace state
1:49 Demo - Reflect state from Kubernetes
2:42 Demo - Retroactively fix state

A demo of using Server Runtime + GA4K to spin up a JupyterHub environment, make changes and push code.

This is the weekly update for the Server Runtime SEG for the week of the 12th of December 2022

This week I've continued on the new technical path of using GA4K. Updates include:

1. Rails UI Updates
POC to start thinking about what the UI could look like

2. Authentication Sidecar Injection
A forked version OAuth2-Proxy is being injected that is responsible for both authentication and authorization

3. Repository Cloning
Able to clone the workspace with the oauth2 token and refresh the token when it expires

Chapters
0:00 Overview
1:23 Demo - New Workspaces Page
2:05 Demo - Workspace updates from agentk
2:42 Demo - Authentication and Authorisation
3:14 Demo - Repository Cloning
4:06 Whats next

This is the weekly update for the Server Runtime SEG for the week of the 29th of Nov.

This week I've changed technical direction and I'm trying to align my work with the work being done by the Remote Development team. This includes:

1. Making Server Runtime a part of GA4K
Added provisioning of a basic IDE using Rails, KAS and Agentk

2. HTTPS and Secret support
Added ability to support HTTPS on the IDEs by injecting TLS support

Chapters
0:00 Overview
1:00 Integration Flow
2:53 Demo
3:50 Whats next

This is the weekly update for the Server Runtime SEG for the week of the 16th of Nov 2022.

This week I continued to work on the custom control plane and added the following features:

1. Deploying locally and connecting to GDK
Able to deploy the custom control plane in a disconnected local kubernetes cluster

2. UI Change - Glide Settings
Enable Glide and set the URL for your instance

3. UI Change - Open with Glide Button
Can provision an IDE from the repo, an MR or from the file / blob view

4. UI Change - Dev Environments Tab
Can view the list of developer environments, see the status and open a provisioned

Chapters
0:00 Overview
0:32 Glide Settings
1:01 Open in Glide button
1:54 Dev Environments Tab
2:43 Whats next

This is the weekly update for the Server Runtime SEG for the week of the 2nd of Nov.

This week I continued to work on the custom control plane and added the following features:

1. Devfile Support
Added the ability to use the Devfile standard to spin up an environment

2. SSH Authn and Authz
CCP now supports the ability to connect to the containers via SSH via a custom SSH Proxy that authenticates via certificates and checks whether the user has access to the IDE

Chapters
0:00 Overview
1:05 Devfile Support Demo
4:32 SSH
8:24 Whats next

In this update Shekhar walks you through the features added to the custom control plane, including:
1. Support for Google as a DNS Provider
2. Architecture Documentation
3. Ability to detect whether an existing IDE is present
4. Support for multiple IDE images
5. Improved unit and integration tests

Chapters
0:00 Overview
0:56 Documentation updates
1:40 Opening an existing IDE
2:22 Selecting from multiple IDEs
3:03 Attaching to the web IDE
3:25 Google as a DNS provider
3:38 Whats next

In this update Shekhar walks you through the features added to the custom control plane, including:
1. Support for authentication and authorisation
2. Containers now have attached volumes
3. Ability to open ports to aid in web development
4. Orchestrate backends for the new VS Code based Web IDE.

Chapters
0:00 Overview
0:49 Authentication / authorisation
1:41 Attaching volumes
2:00 Open ports
3:03 Attaching to the web IDE
4:20 What is next

This is the first weekly update for the Incubation Engineering - Server Runtime SEG. In this video, Shekhar walks you through what is Server Runtime and gives a demo of a prototype control plane that can provision environments / IDEs for the user.

Chapters
0:00 Hello! and What is Server Runtime
1:13 First goal - Custom Control Plane
2:18 VS Code Demo
4:59 vim Demo