GitHub Universe 2017 - GitHub For Business, 13 Jan 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Quickly Build a Collaborative CMS Powered by the GitHub API - GitHub Universe 2017

Description

Presented by David Stolarsky, The New York Times.

At The New York Times we have built an entire CMS in which the only stateful layer is the GitHub API. This vastly simplifies our infrastructure setup and maintenance, and gives us powerful primitives on which to build a collaborative content editing system used by writers, designers, photographers, editors, and even web developers. In this talk you will learn: how The New York Times uses the GitHub API to store documents in the CMS's intermediate format, how we support collaborative editing via the GitHub API, how we catalog our posts via the GitHub API, how we are able to support legacy ("custom") development at the same time, and how the GitHub ecosystem, e.g. CI services empower our CMS even further.

About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.

For more information on GitHub Universe, check the website:
https://githubuniverse.com

A

Hi everyone hi, my name- is David I work at the New York Times I think we might be one of the oldest companies represented here at this conference. I'm gonna talk about quickly, building a collaborative CMS powered by the github API and how to let designers control the web with a professional WYSIWYG.

A

So basic story, I'm here to tell you, is: if you're gonna build a CMS, maybe don't set up your own database and maintain it? Maybe just use the github API. What does the CMS do it organizes documents, lets multiple people, edit those documents and publish those documents. Github lets you organize repos lets multiple people edit those repos, maybe a repo, can be a document in a CMS. So this is what we've done at the New York Times and how about a live demo?

A

It's a good idea right. What could go wrong.

A

So here's our WYSIWYG CMS.

A

I can put in text do some basic formatting stuff.

A

David is literally on stage at github universe. Right now, tempting fate by doing a live demo.

A

Lots of text we like words at the New York Times.

A

Let's add a 360 photo.

A

I'm sure this will work.

A

A

Move it up here there. It is.

A

And we can publish this to the World Wide Web.

A

There it is oh.

A

I forgot wanted to add something at the end to really drive this home.

A

A

Make it big okay, what's what's your favorite Google Fonts.

A

Okay, we'll go with lobster.

A

A

So rest of this talk whoops yeah. That was my backup video. The rest of this talk will go like this. Why did we build our own WYSIWYG CMS? How did we build it? How was it received?

A

How can we improve it and if there's time at the end, we've got a bonus for you, a motion, graphics designer on github.

A

So why did we build our own WYSIWYG CMS, there's some of them out there I work in a part of the times called T brand studio, and we create native New York Times native content for advertisers.

A

This is one example piece of content we make. This is called a paid post. It looks kind of like an article, but not so much that you would confuse it for real news. We make a lot of these. Each one is lovingly designed by our graphic design team, but all of them share a certain structure, kind of a multimedia linear narrative, with text image text, video text interactive, something like that. We make a lot of them. We've recently published our 300th in about three years of our team's existence, so we're mass producing microsites.

A

This is how we do it. As of 2015, we've got a design team. They mock up the page. They like to use adobe indesign, they hand off the mock-up to the dev team. We build it out into our boilerplate and we deploy it to the world wide web, so standard web design, development workflow, but this is a slow, relatively slow workflow. Once you get to the web, you got to do technical, QA and visual QA, and this we're not lips. We're not so happy about this.

A

We feel like we're going more slowly than we could be so 2016 we decide to build a library of off-the-shelf components will QA before we drop them into the page. Then we just put the content into those components, and things will be better. We build that component library inside InDesign as an object library and as react components, but the correlation between the InDesign library and the react library is tenuous. They drift apart. Obviously InDesign and reactor totally different systems. They and these designers are in InDesign.

A

They don't know exactly what the boundaries of the off-the-shelf component is like how it's supposed to be configured, and so they go off the off the reservation and so we're still doing more QA. Then we'd like to be doing so.

A

We see this year an opportunity to put some tooling between the humans and the react components so that the humans can control the components in just the narrow ways they're supposed to be controlled, and we think we can speed up the the developers work with this tool and we also hope we can get the designers to use this tool as well. So easy wins here, our automates, some of the like paperwork. You have to do when you do web development like compressing images and putting them in the filesystem wiring them into the HTML.

A

Writing simple HTML. But writing it correctly.

A

So that's an easy win. The harder win here, a bigger challenge, bigger win, is getting the design team to use this tool cut out that whole InDesign step have the develop. Sorry, have the designers do what the developers we're doing so? Obviously we're we're doing the same work with fewer people. This will be better.

A

But this is going to be a real challenge, because we actually previously tried to force the design team to use tiny MCE, which is an older, mature, HTML WYSIWYG, and this tiny MCE was just too from their design aesthetic and they were not into it, and we realized that what we're selling advertisers in these paid posts is largely our design teams work.

A

So this is where we want our tool to be, and it's called paid post Composer somewhere between medium and InDesign. We can't hope to offer our design team as much power and control as in design, but we can do. We can give them more control than medium gives. Them medium is great. It lets people write words and put those words on the internet and they've picked a nice default fonts, but graphic designers cannot express themselves in a tool like medium or in tools like Twitter or Facebook.

A

These are very lively communities, but in terms of graphic design, they're pretty boring whoops, so I want professional designers to have more control over web content, because hTML is actually an amazing layout engine and you can do great things with type and color and layout in HTML, but we're not I, don't see that really happening on most of the web. I also want amateur designers to have these powerful tools, because when they have these tools they make geo cities and we need geo cities now, of course.

A

So we don't know whether the design team is going to even bite on this tool, so we're not willing to make a huge bet on it. So here's our challenge is give designers powerful control over the web, but don't spend too much time on it so part two. How can we quickly give designers powerful control by making smart choices about the user interface and the application infrastructure right?

A

Yes, right, okay, so user interface! This is in my mind what 3d modeling tools look like it's kind of overwhelming it's scary to jump into for the first time. But what is this interface good at? It's good at allowing the develop this tool to move lots of controls into the hands of the user of this tool. They didn't think too much about how to layout this interface.

A

They just made a list of data types and labeled them, and then the runtime of the of the 3d modeling tool just throws them all on the screen, and you know it's hard to get through all these controls and find what you want, but once you figure it out, you have all that control.

A

So that's how we design the user interface of our WYSIWYG, most WYSIWYG x'. I think you manipulate the content directly where the content is being previewed, but in our tool we completely separate that and on the Left we have all the controls. You have to scroll quite a bit to get through them, but there's a lot of control there and then on the right. We have our preview and again you can't manipulate the content there.

A

You can just view it, but we sort of got lucky here and that we just dropped our existing boilerplate into an iframe inside this editor. So this is our real production rendering. So this is very. This is really what you see. It's really what you get, and then we have a little interface in there to allow the editor to pass in a document from outside the iframe and change the document.

A

One advantage of using our actual production runtime inside the editor is we can resize the editor and preview, our mobile, the mobile version of our site, because our front-end is fully responsive without without requiring a back-end.

A

So this is how we preview on mobile today. Goldstar means opportunity for future improvement. So talk about this in the last section, so as I alluded to the control panel is automatically derived from, in our case, our document model and what is our document model? Basically an instance of a metadata component and an array of instances of component that make up the body component is an abstract base class and some concrete examples are split. Header body copy figure, video player, image, 360, and we have lots more now.

A

These components here are CMS controller components, distinct from the off-the-shelf content components we made last year, but they're in roughly one-to-one correspondence.

A

How do we manipulate that array of components in our user interface shoutout to github comm cloud? Eric react, sortable, higher-order components, easy to build a draggable resore to bowl list.

A

So what is a components? It's an array of labeled parameters and it's a function rendered JSX that spits out the JSX that calls into our off-the-shelf components.

A

Some example parameters are text, image, responsive image, boolean, google, fonts and a parameter is just a user interface that text box or that image drop well and then a function render value JavaScript, which aids, the parent components render JSX function.

A

So, to give designers more control, we just have to create or extend subclasses of components.

A

Here's an example for 360 photo component, an equi rectangular image, parameter a caption and a credit text parameters and our render JSX function. It's really fun writing es6 javascript that renders JSX by the way the escaping is awesome, make sure you use a color highlighting syntax color highlighting text editor when you do this.

A

So that's easy. This is this is how we're able to ship lots of control into our designers hands, because we just have to manipulate this array of labeled parameters and this render function, but actually there's still a bit of a barrier here and we'll talk about improving that at the end.

A

So here's one component, a simple one body copy, takes some text. It's got a boolean for drop cap, a couple other things and the content rendering on the right.

A

This is a more complicated component, the split header. It's got a lot of typeface control, you can drop in a mobile image and a desktop image and actually drop in videos as well to use on that right side.

A

And how do we get from our document to our preview in the iframe? The document renders JSX by calling all the component render JSX methods would go through babel. We get old-school javascript, we pass the old-school javascript into the iframe, and then we evaluate which is such a rush. Now.

A

All the time, while we're editing these parameters were going through babel over and over we're rendering that preview over and over and we're auto-saving. So let's talk about the application infrastructure.

A

Now we want to use the the least infrastructure possible because we want to build this quickly and we don't want to have to do a lot of maintenance, work and watch it be. Of course it has to be reliable.

A

We considered only building a front-end and interacting with the github api, but we decided that, due to our specific authentication environment, we should probably have a back-end we're running our back-end and App Engine standard, which is cool for a couple of reasons. One is that all new york times employees use google for email so to authenticate them. We just put log in required in our app demo, do a little configuration in the cloud console and at $0 an hour during idle time, and this tool actually is not going to be used 24/7.

A

So this is cool.

A

And we save the documents in to get up, and this is how we do it. Of course it takes a couple: API calls in the v3 API, at least to save into a repo we're saving three files.

A

So you create a blob for each file. Then you create a tree from those blobs. Then you create a commit from that tree. Then you have to update the ref, the head, the master head ref to point to that commit.

A

And we saved these three files: composer doc, JSON, that's the abstract document. It's the which components make up this post. What are the values of the parameters in that component and then we also save package dot JSON and index JSX, which is fitting into our legacy manual development format? Why do we do that?

A

Well, one reason is we're not building a new production deployment workflow here we're relying on our pre-existing one, so we need to just package that JSON and index type JSX for that, but another reason we save this is because we can't hope to give the designers absolutely all the controls they could possibly dream of wanting. But what we can say is that look this this CMS.

A

This WYSIWYG is going to write into our existing manual development format and at any time, if push comes to shove, we can clone that and start doing manual development, so everything you can get from us today, you can get from us with this tool. Also.

A

Now, when you create a commit in github, you have to specify at least one previous version. Apparently you can specify multiple previous versions, which I don't know what's up with that, but gets tough and what can happen here.

A

What may be well, we always create a parent. What can happen is someone else might have updated the document before you did and we get like a merge conflict. So what do we do in that situation? We do the absolute simplest thing possible that lets us ship this tool as early as possible. We show an error message, but we spin it.

A

We show office, co-workers, jumping in unison and joy and we say we're more productive when we work together as a team, it appears your teammate or you and a different browser tab made changes before you did.

A

So we're a small team at the times we, you know, we didn't think we didn't know how much people would even would use this tool at all. So this is where we started obviously opportunity for improvement.

A

Our CMS has a home screen where you can browse all the documents available in the system notice the try, the laser pointer notice, the github links there, so we can click, go to the repo and clone it and start doing manual development if necessary.

A

This is one simple call to the repository Search API.

A

And what we can do here is duplicate an existing post. You guys understand the magic of working.

A

This actually solves a pet peeve of mine, which is you can't fork a repository into its own organization. So that's what we're doing here with the by the API.

A

Now when we built this feature, we noticed that some repos were taking a really long time to duplicate and those are the repos that had a lot of images in them. It would we in our manual development, we stored all our images and github. So that's how we started with this tool, but this was blocking us from basically building the duplication feature. So we switched to deploying our images straight to the s3 backing of our CDN, which I think is a better way to do it anyways, it's the images, an immutable resource.

A

You can't really track changes on it, I mean there's. No, maybe you can't, but anyway we wanted to duplicate repos, but it still takes a little while to duplicate that repo. So it's a good opportunity for some dark humor about the death of print media.

A

It says anti Gutenberg 3000 and it's at 451 degrees, Fahrenheit.

A

Okay, so that's how it works. How was it received whoops?

A

This is real photojournalism from early in the life of paid post composer, when I was the only user of it and this graphic designer and photo editor have actually commandeered my keyboard because they see how easily I am manipulating the layout of a photo and editing the caption and they figure instead of trying to say words into my ears and then wait for them to get to my fingers. They could just put their hands on the keyboard. Do it themselves.

A

In a few months of composer being out there, we've published about 15 paid posts with it. That's a number needs context. We published about 300 paid posts in a few years, so this is not terrible.

A

How does our workflow look when we use this tool? It looks more like this. The designers are manipulating the web directly. The feedback loop with them is like milliseconds and there's like one developer, supporting all of them, helping them figure out how to do things in the tool and in some cases, extending the tool and the other two developers are just drinking cocktails, also known as improving our platform, which is what we'd rather spend our time doing.

A

The design team is sending us lots of feature requests, so seems they are biting and they're starting to complain more about this error message. So we've got graphic designers and we have text like text. Editors, English writers photo editors, they're all in there and and we're getting this conflict. So that's good because they're using it, but it's getting to be an annoyance. So what's next, how can we improve this?

A

So right? Now it's pretty easy to get this merge conflict. It can happen over the course of hours, and this is not you know, compatible with the modern fast-paced working environment.

A

But to fix this issue, we don't need to actually do any fancy merging. All we have to do is push that remote edit into all open instances, and then we can edit away, and once we do that whoops to do that, I think we'll need a little bit of more infrastructure, but it's just messaging. These things are readily available. You can pay per message. So it's not a big deal now to get the merge conflict.

A

Two people will have to do an edit within about five seconds of each other, which is the time it takes us to debounce the autosave and do a round trip to the back end.

A

This will be, people will see this people will be much less likely to see this I think, although I'm guessing. If we remove that earlier error, then they'll start working more quickly and they'll start seeing this error some of the time, maybe a lot of the time. So in that case we can do a merge. This is not actually too terribly difficult.

A

It's a well-understood problem, the important thing there is just if, if one person edits some text and the other person deletes that entire component, we want to show what's happening by animating the user interface to just show what the other person is doing. Of course, this is what Google Docs does so we'll just follow their lead on that.

A

So we said it was easy to ship new controls into the hands of the designers. We just have to manipulate these parameters and the render function well. Actually, after we edit the source code, then we have to ship a new version of the tool and right now whoops right now.

A

We we ship updates to the control through the same mechanism as updating the entire CMS, and if we've done other changes to the CMS, we're gonna be more slow to deploy that, because it's a risky thing to do, we want to make sure it's all tested and working before we push it out. So I think if we move the deployment of new controls out of that main pathway into its own pathway, we can speed up that delivery of control to designers even further.

A

This could take the form of like a library that you can switch out yourself at runtime, maybe even a library of controllers that are available in an online community.

A

Mobile preview: this is how we do it right now,.

A

Some of the users of composer have realized that they can hit that publish button, get a standalone URL with the content and then they'll look at it on their actual phone, and then they get a better, more accurate preview of how it's gonna look on a phone.

A

And- and that's really what you have to do to see what content will look like for the majority of internet consumption scenarios where people are on the phone? You really have to look at the real device, so I'm glad we built that edit push system earlier, because we're gonna attach that to our our WYSIWYG desk. Here we're gonna have a phone next to our main monitor and it's going to be wired into that preview. Pane it's going to get those at it's pushed into it immediately.

A

So as we're typing away, dropping images, it's all gonna update in real time on the phone- and this is great because you really have to look on a real phone and the more we can cut down the the loop for design and for QA the better.

A

But we're also gonna put an iPad here, because some people browse the web on iPad and we're also going to attach the Internet Explorer 11 into this live editing system because we're responsible web developers we support Internet Explorer 11. Do you guys support inter Explorer 11, a few okay I'm? Glad we're not the only ones.

A

We're also going to attach an iPhone se, because some people have the small iPhone. Some people have the plus things look different. All these devices have a Galaxy s8 and a Google pixel and a Firefox desktop and a safari desktop and Microsoft edge, which I understand, is a web browser.

A

And then lots more devices, I, don't really know what all these are. That's a big 4k screen because our content is going to look great there. It's going to keep us inspired, we're gonna, have landscape mode, iPhone and iPhone SC landscape, iPhone, plus everything's different there, ipad mini ipad, mini landscape and well I. Don't know what that is: iPhone 10, that's gonna, be crazy and there's probably some watch out there with a web browser on it.

A

So that's our awesome imposing mega integrated design, QA desk of the future. Who wants one of these all right? Okay? So that's the WYSIWYG CMS we've got some time left. So, let's talk about a motion, graphics designer on github.

A

This is in the same spirit as our WYSIWYG CMS, where we want to put lots of design control in the hands of designers, but in this case it's about the it's about the type of paid posts we make when we have a bigger budget and we're really going all out.

A

So this is a professional motion, graphics designer which we hired for one week and he's he's doing animation in After Effects for a immersive, animated scroll, narrative, visual experience, which has been blurred out because it's hopefully going to launch in a week or two so he's doing animation in After Effects and he's exporting from After Effects to a vector web animation format called body movin and then he's we've instructed him how to do this ad hoc, automated workflow, where he drops the body move in JSON and related images into github.com in a web browser.

A

Then he clicks that nice, big fat green, commit button. He even writes a commit message. Sometimes- and this is this- is a he's he's an animator he's, not a developer. He's one day into working with us he's already dropping stuff into github. Then Travis builds the page, deploys it to a URL, and he knows what that URL. Is. He he's actually watching Travis do the build as well.

A

So when Travis is done, he looks at the page in a real web browser and on his phone, and he can see his animation how it works in the actual web browser and on day two well after day, one of his work I looked over. It I saw that wow there's, there's some serious graphics performance issues in certain parts of this animation. Oh man, we only have four days left. How are we gonna get the QA done, while he's still here so I asked him well I request. Mr.

A

animator I need you to take ownership over the graphics performance of the animation in the web browser he was like whoa I, don't know anything about that I'm, just an animator I'm, not a developer.

A

My response was yeah I'm a developer, but I didn't make body moving, so I, don't really know exactly what's gonna work and what's not and my strategy over this short timeframe is going to be trial and error when I see part of the animation not performing well I'm gonna, look in after-effects.

A

What's going on in After Effects right, there I'm just gonna, remove it and try something else, and he was able to do that and actually just enjoyed the opportunity to create interactive animation, whereas he's used to doing static stuff for film and television and he's a he's, a great graphics, performance, QA, analyst and implementer after a few hours. So what's the moral of the story? Well, the story is web. Development is hard because you have to do all this meta.

A

Work, paperwork right, syntax, correctly, put images into the right place, transmit files to remote servers and if you automate all that stuff with tooling- and you just expose the the design surface to non developers, they can make great stuff on the web.

A

So thank you. This is my github Twitter.

A