21 Dec 2016
This presentation was recorded at GOTO Berlin 2016
http://gotober.com
Martin Kleppmann - Researcher at University of Cambridge
ABSTRACT
What do collaborative editors like Google Docs, the calendar app on your phone, and multi-datacenter database clusters have in common?
Answer: They all need to cope with network interruptions, and still work offline. They all allow state to be updated [...]
Download slides and read the full abstract here:
https://gotocon.com/berlin-2016/presentations/show_talk.jsp?oid=7910
RECOMMENDED BOOKS
Martin Kleppmann • Designing Data-Intensive Applications • https://amzn.to/3mk2Roj
https://twitter.com/gotober
https://www.facebook.com/GOTOConference
http://gotocon.com
http://gotober.com
Martin Kleppmann - Researcher at University of Cambridge
ABSTRACT
What do collaborative editors like Google Docs, the calendar app on your phone, and multi-datacenter database clusters have in common?
Answer: They all need to cope with network interruptions, and still work offline. They all allow state to be updated [...]
Download slides and read the full abstract here:
https://gotocon.com/berlin-2016/presentations/show_talk.jsp?oid=7910
RECOMMENDED BOOKS
Martin Kleppmann • Designing Data-Intensive Applications • https://amzn.to/3mk2Roj
https://twitter.com/gotober
https://www.facebook.com/GOTOConference
http://gotocon.com
- 1 participant
- 52 minutes
15 Mar 2016
Slides and more info: http://www.lambdadays.org/lambdadays2016/dmitry-ivanov
In a connected world, synchronising mutable information between different devices with different clock precision can be a difficult problem. A piece of data may have many out-of-sync replicas but all of those should eventually be in a consistent state. For example, TomTom users, having personal navigation devices, smartphones, MyDrive website accounts, expect their navigation information be synchronised properly even in the occasional absence of network connection.
Conflict-free Replicated Data Types (CRDTs) provide robust data structures to achieve proper synchronisation in an unreliable network of devices. They enable the conflict resolution being done locally at the data type level while guaranteeing the eventual consistency between replicas.
In this talk, in addition to an introduction to CRDT, our main focus will be on a special type of CRDT-set called OUR-set (Observed, Updated, Removed) which we created to extend known CRDT-sets with update functionality. We will explain the advantages of this data structure to solve many synchronisation problems as well as its limitations. We also show how a basic implementation of OUR-set CRDT in Scala and its counterpart in Java looks like and enumerate a set of subtle considerations which should be taken into account.
In a connected world, synchronising mutable information between different devices with different clock precision can be a difficult problem. A piece of data may have many out-of-sync replicas but all of those should eventually be in a consistent state. For example, TomTom users, having personal navigation devices, smartphones, MyDrive website accounts, expect their navigation information be synchronised properly even in the occasional absence of network connection.
Conflict-free Replicated Data Types (CRDTs) provide robust data structures to achieve proper synchronisation in an unreliable network of devices. They enable the conflict resolution being done locally at the data type level while guaranteeing the eventual consistency between replicas.
In this talk, in addition to an introduction to CRDT, our main focus will be on a special type of CRDT-set called OUR-set (Observed, Updated, Removed) which we created to extend known CRDT-sets with update functionality. We will explain the advantages of this data structure to solve many synchronisation problems as well as its limitations. We also show how a basic implementation of OUR-set CRDT in Scala and its counterpart in Java looks like and enumerate a set of subtle considerations which should be taken into account.
- 2 participants
- 40 minutes
6 Nov 2015
Slides and more info: http://www.codemesh.io/codemesh2015/marc-shapiro
The CRDT abstraction (Conflct-free Replicated Data Type) is a useful tool for building highly scalable and available applications. CRDTs combine three key intuitions: (i) Encapsulate distribution inside the boundaries of a data type, so that ordinary programmers can build applications by combining CRDTs from a library; (ii) support "write now, propagate later" concurrent updates to ensure scalability, availability and performance; (iii) some simple of mathematical properties that ensure that data replicas converge safely. This talk presents the principles and design of some useful CRDTs. We discuss the CRDT approach in the context of some real applications of our industrial partners. We present some of the difficulties and limitations of CRDTs, in the more general perspective of consistency and scalability. We present recent extensions that address some of these limitations, in particular formaintaining application invariants.
Marc Shapiro does his research on distributed computer systems, data replication and consistency algorithms, and distributed garbage collection. He invented the proxy concept, which is now universal on the Internet. He published at SOSP and OSDI, the two most prestigious venues of the area (one of the only two French papers at both venues). He was instrumental in the creation of EuroSys, the main European venue in the area. He authored 64 international publications, 17 recognised software systems, and four patents. Dr Shapiro's research started with a PhD from Université Paul Sabatier for research performed at atLAAS in Toulouse, France (1980), followed by a post-doc at MIT, and a researcher position at CMIRH. He is a researcher at INRIA since 1984. He spent a one-year sabbatical at Cornell (1993—1994), and he led the Cambridge Distributed Systems group at Microsoft Research Cambridge (UK) from 1999 to 2005. He is currently a Senior Researcher for INRIA Paris-Rocquencourt, in the Regal group, located at LIP6.
Nuno Preguiça is curently Associate Professor at FCT - Univ. NOVA de Lisboa and leads the Computer Systems group at NOVA LINCS. He does research in the area of distributed systems, mobile computing and data management. He co-proposed the concept of CRDTs and a number of extensions to CRDTs, including variants for enforcing global invariants and for efficient execution of distributed computations.
The CRDT abstraction (Conflct-free Replicated Data Type) is a useful tool for building highly scalable and available applications. CRDTs combine three key intuitions: (i) Encapsulate distribution inside the boundaries of a data type, so that ordinary programmers can build applications by combining CRDTs from a library; (ii) support "write now, propagate later" concurrent updates to ensure scalability, availability and performance; (iii) some simple of mathematical properties that ensure that data replicas converge safely. This talk presents the principles and design of some useful CRDTs. We discuss the CRDT approach in the context of some real applications of our industrial partners. We present some of the difficulties and limitations of CRDTs, in the more general perspective of consistency and scalability. We present recent extensions that address some of these limitations, in particular formaintaining application invariants.
Marc Shapiro does his research on distributed computer systems, data replication and consistency algorithms, and distributed garbage collection. He invented the proxy concept, which is now universal on the Internet. He published at SOSP and OSDI, the two most prestigious venues of the area (one of the only two French papers at both venues). He was instrumental in the creation of EuroSys, the main European venue in the area. He authored 64 international publications, 17 recognised software systems, and four patents. Dr Shapiro's research started with a PhD from Université Paul Sabatier for research performed at atLAAS in Toulouse, France (1980), followed by a post-doc at MIT, and a researcher position at CMIRH. He is a researcher at INRIA since 1984. He spent a one-year sabbatical at Cornell (1993—1994), and he led the Cambridge Distributed Systems group at Microsoft Research Cambridge (UK) from 1999 to 2005. He is currently a Senior Researcher for INRIA Paris-Rocquencourt, in the Regal group, located at LIP6.
Nuno Preguiça is curently Associate Professor at FCT - Univ. NOVA de Lisboa and leads the Computer Systems group at NOVA LINCS. He does research in the area of distributed systems, mobile computing and data management. He co-proposed the concept of CRDTs and a number of extensions to CRDTs, including variants for enforcing global invariants and for efficient execution of distributed computations.
- 3 participants
- 50 minutes
14 Oct 2015
Conflict-free Replicated Data Types (CRDTs) are a hot new datatype in distributed programming that promise coordination-free and always eventually-consistent writes. Most of the stuff you will read about them is limited to the context of high-end distributed databases, but CRDT's are more flexible and shouldn't be limited to this field. In the talk, I will demonstrate how CRDT's are great for applications that have partial connectivity (including websites): updates can be applied locally, and when communication is possible, you can send the data back up, and the data will remain consistent even in the most byzantine (or apocalyptic) scenarios. There are even scenarios that can support multiple simultaneous editors.
Beyond that, I will also demonstrate how Elixir's metaprogramming can be used to compose complex models out of CRDT's that themselves exhibit the same exact features. I will also exhibit some newer CRDT features, such as shared-context and delta-operation CRDT's to overcome some of the shortcomings of older CRDT's.
I plan to keep the talk light on theory (the academic literature is sufficient for that).
Help us caption & translate this video!
http://amara.org/v/HO21/
Beyond that, I will also demonstrate how Elixir's metaprogramming can be used to compose complex models out of CRDT's that themselves exhibit the same exact features. I will also exhibit some newer CRDT features, such as shared-context and delta-operation CRDT's to overcome some of the shortcomings of older CRDT's.
I plan to keep the talk light on theory (the academic literature is sufficient for that).
Help us caption & translate this video!
http://amara.org/v/HO21/
- 1 participant
- 32 minutes
12 Oct 2015
Idris is a general purpose pure functional programming language with dependent types. In Idris, types are a first class language construct, meaning that they can be manipulated and computed like any other language construct. It encourages a type-driven style of development, in which programmers give types first and use interactive editing tools to derive programs. Introductory examples typically involve length-preserving operations on lists, or ordering invariants on sorting.
Realistically, though, programming is not so simple: programs interact with users, communicate over networks, manipulate state, deal with erroneous input, and so on. In this talk I will show how advanced type systems allow us to express such interactions precisely, and how they support verification of stateful systems as a result.
The talk will include several examples, leading to a verified implementation of a word game (Hangman). I will show how Type-driven Development allows programmers to specify the game rules in a direct and concise style, and how it leads to an implementation, guaranteed to correctly follow the rules by typechecking.
Realistically, though, programming is not so simple: programs interact with users, communicate over networks, manipulate state, deal with erroneous input, and so on. In this talk I will show how advanced type systems allow us to express such interactions precisely, and how they support verification of stateful systems as a result.
The talk will include several examples, leading to a verified implementation of a word game (Hangman). I will show how Type-driven Development allows programmers to specify the game rules in a direct and concise style, and how it leads to an implementation, guaranteed to correctly follow the rules by typechecking.
- 2 participants
- 47 minutes
28 Sep 2015
CRDTs (Conflict-Free Replicated Data Types) are a great way to implement applications that gracefully recover from partitions - such as clusters or (mobile) applications that can work offline.
This talk will:
Introduce the concept of CRDTs and their use cases
Describe the 2 main ways to implement a CRDT (convergent and commutative), with visual examples to illustrate their behavior
Show 2 examples of CRDTs used in practice (RIAK 2 Data Types (distributed storage) and swarm.js (offline-friendly web apps))
Arnout Engelen
XEBIA
@raboofje
With over a decade of experience in professional software development, Arnout is constantly on the lookout for tools to smoothen and simplify that process. He currently consults for Xebia, injecting the startup mentality into larger corporations in the Netherlands.
This talk will:
Introduce the concept of CRDTs and their use cases
Describe the 2 main ways to implement a CRDT (convergent and commutative), with visual examples to illustrate their behavior
Show 2 examples of CRDTs used in practice (RIAK 2 Data Types (distributed storage) and swarm.js (offline-friendly web apps))
Arnout Engelen
XEBIA
@raboofje
With over a decade of experience in professional software development, Arnout is constantly on the lookout for tools to smoothen and simplify that process. He currently consults for Xebia, injecting the startup mentality into larger corporations in the Netherlands.
- 2 participants
- 36 minutes
27 Sep 2015
Consistency is hard and coordination is expensive. As we move into the world of connected 'Internet of Things' style applications, or large-scale mobile applications, devices have less power, periods of limited connectivity, and operate over unreliable asynchronous networks. This poses a problem with shared state: how do we handle concurrent operations over shared state, while clients are offline, and ensure that values converge to a desirable result without making the system unavailable?
We look at a new programming model, called Lasp. This programming model combines distributed convergent data structures with a dataflow execution model designed for distribution over large-scale applications. This model supports arbitrary placement of processing node: this enables the user to author applications that can be distributed across data centers and pushed to the edge. In this talk, we will focus on the design of the language and show a series of sample applications.
Christopher Meiklejohn
MACHINE ZONE, INC.
@cmeik
Christopher Meiklejohn is a Senior Software Engineer with Machine Zone, Inc. working on distributed systems. Previously, Christopher worked at Basho Technologies, Inc. on the distributed key-value store, Riak. In his spare time, Christopher develops a programming language for distributed computation, called Lasp. Christopher is starting his Ph.D. studies at the Université catholique de Louvain in Belgium in 2016.
We look at a new programming model, called Lasp. This programming model combines distributed convergent data structures with a dataflow execution model designed for distribution over large-scale applications. This model supports arbitrary placement of processing node: this enables the user to author applications that can be distributed across data centers and pushed to the edge. In this talk, we will focus on the design of the language and show a series of sample applications.
Christopher Meiklejohn
MACHINE ZONE, INC.
@cmeik
Christopher Meiklejohn is a Senior Software Engineer with Machine Zone, Inc. working on distributed systems. Previously, Christopher worked at Basho Technologies, Inc. on the distributed key-value store, Riak. In his spare time, Christopher develops a programming language for distributed computation, called Lasp. Christopher is starting his Ph.D. studies at the Université catholique de Louvain in Belgium in 2016.
- 1 participant
- 40 minutes