Ceph / Cloud Platforms

Chris Jones, Technical Lead for Bloomberg Cloud Storage and maintainer for Ceph's Chef cookbooks on github, will provide a technical overview of how Ceph Object and Block storage are used at Bloomberg with and without OpenStack. Learn how-to build your own multi-petabyte hyper-converged cloud storage system using the same automation tools used at Bloomberg on commodity hardware. Topics will include: -Multi-Cluster architecture with index sharding for Object Store (RADOS Gateway) -Architecture to

#vBrownBag 2014 OpenStack Paris - Boyan Krosnov – Block Storage: When Ceph is not enough

Ceph is a mature platform for software-defined storage environments scaling to dozens or hundreds of PetyBytes. However real life implementation, operations and maintenance are complex tasks. Full session details here: http://awe.sm/t7gR6

Richard Morrell talks to Ross Turk about the latest greatest news around Ceph and how Red Hat Cloud marches on storage, why Ceph and Gluster are complimentary. We talk Firefly and a look forward to Giant and we discuss why Inktank was such a no brainer addition to Red Hat and why it adds to the strength of Red Hat Open Hybrid Cloud.

This video shows how I built my Openstack Icehouse test lab under ESXi 5.1 and the way it works connected to a Ceph cluster for storing Glance images and Cinder block devices.

All the machines are running CentOS seven, which is not fully supported at the moment

By Ian Colle
As the size and performance requirements of cloud deployments increases, storage architects are increasingly seeking new architectures that scale.

Ceph is a fully open source distributed object store, network block device, and file system designed for reliability, performance, and scalability from terabytes to exabytes. Ceph utilizes a novel placement algorithm (CRUSH), active storage nodes, and peer-to-peer gossip protocols to avoid the scalability and reliability problems associated with centralized controllers and lookup tables.

Ceph's architecture is based on RADOS, an object store with support for snapshots and distributed computation. Ceph offers a Swift and S3-compatible REST API for seamless data access to RADOS. Ceph's network block device can be used to store large images and volumes in RADOS, supporting thin provisioning and snapshots. Ceph has native support for Qemu/KVM, libvirt, CloudStack and OpenStack which makes it an attractive storage option for cloud deployments.

As a result of participating in this session, attendees will gain a full understanding of the Ceph architecture, its current status, and plans for future development. Attendees will also learn how Ceph integrates with OpenStack to provide a unified object and block storage service.

Help us caption & translate this video!

http://amara.org/v/FG0G/

Open Storage: Intel's Investments in Object Storage features Intel's Rob Armstrong, Tushar Gohad and Paul Luse.

Come hear about all the existing things Intel is doing in Swift and with Object Storage in general. The Juno release will introduce the notion of Storage Policies to Swift and laid the groundwork for a whole host of new usage models. Also with Juno we'll be introducing Erasure Codes as the first significant Storage Policy. We'll also talk about a new test/QA cluster that has been donated by SwiftStack, Intel, and HGST, which uses the latest drive and processor technology to provide automated tests and a public baseline for performance, a key aspect of the release process not previously available. We'll round this out with an update and demo on COSBench, the benchmarking tool, which now has more storage types like Ceph, S3 verified, and more test scenarios supported.

There has been a lot of buzz surrounding Ceph and OpenStack lately, and a lot of new developments. This talk will provide a quick overview of Ceph's architecture and the integration between Ceph and OpenStack, examine recent use cases, and discuss what is coming up next. We'll take a look at how to keep fans of both platforms happy and discuss how the less-oft-used pieces of the Ceph platform can help augment your OpenStack setup. We'll also talk about how to get started and what the community can do to get involved.

The University of Alabama - Birmingham case study features John-Paul Robinson and Dell's Kamesh Pemmaraju.

The University of Alabama at Birmingham gives scientists and researchers a massive, on-demand, virtual storage cloud using OpenStack and Ceph for less than $0.41 per gigabyte. This session will detail how the university IT staff deployed a private storage cloud infrastructure using the Dell OpenStack cloud solution with Dell servers, storage, networking and OpenStack, and Inktank Ceph.

After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that was capable of scaling seamlessly and rapidly, was cost-effective, and that could leverage a single hardware infrastructure for the OpenStack compute and storage environment.

Ceph Update: Erasure Coding and Tiering features Ross Turk from Inktank.

CloudStack Collaboration Conference North America 2014

Unified Cloud Storage with Synnefo + Ganeti + Archipelago + Ceph

Speaker: Vangelis Koukis

This talk presents Synnefo's evolution since FOSDEM '13, focusing on its integration with Ganeti, Archipelago, and Ceph to deliver a unified, scalable storage substrate for IaaS clouds.

The talk will begin with an overview of the Synnefo architecture (Python, Django, Ganeti, and Xen or KVM). Then, it will introduce Archipelago, a software-defined, distributed storage layer that decouples Volume and File operations/logic from the underlying storage technology, used to store the actual data. Archipelago provides a unified way to provision, handle and present Files, Images, Volumes and Snapshots independently of the storage backend. We will focus on using Ceph/RADOS as a backend for Archipelago.

With Archipelago and its integration with Synnefo and Ganeti one can: * maintain commonly-used data sets as snapshots and attach them as read-only disks or writeable clones in VMs running the application, * begin with a base OS image, bundle all application and supporting library code inside it, and upload it to the Archipelago backend in a syncing, Dropbox-like manner, * start a parallel HPC application in hundreds of VMs, thinly provisioned from this Image, * modify the I/O processing pipeline to enable aggressive client-side caching for improved number of IOPS, * create point-in-time snapshots of running VM disks, * share snapshots with other users, with fine-grained Access Control Lists, * and sync them back to their PC for further processing.

The talk will include a live demonstration of a workflow including this functionality in the context of a large-scale public cloud.

The intended audience spans from enterprise users comparing cloud platforms, to developers who wish to discover a different design approach to IaaS clouds and storage virtualization.

Please see the abstract above for a rough sketch of the proposed presentation. The presentation will mostly consist of live demonstration and a small deck of slides, meant to describe the main features of Archipelago and its integration with Synnefo, as well as provoke discussion in the Q&A session.

The main workflow in the demonstration will be [copied and pasted from the Abstract]: * begin with a base OS image, bundle all application and supporting library code inside it, and upload it to the Archipelago backend in a syncing, Dropbox-like manner, * start a parallel HPC application in hundreds of VMs, thinly provisioned from this Image, * modify the I/O processing pipeline to enable aggressive client-side caching for improved number of IOPS, * create point-in-time snapshots of running VM disks, * share snapshots with other users, with fine-grained Access Control Lists, * and sync them back to their PC for further processing

Presenter(s): Sage Weil
URL: https://lca2014.linux.org.au/schedule/30079/view_talk

The Ceph distributed storage system sports object, block, and file interfaces to a single storage cluster. The core infrastructure supporting these use cases is a distributed object storage and compute platform called RADOS (reliable autonomic distributed object store). The low-level librados API exports a conceptually simple yet powerful interface for storing and processing large amounts of data and is well-suited for backing web-scale applications and data analytics. In features a rich object model (bytes, attributes, and sorted keys/values), efficient key/value storage (using leveldb on the backend), atomic transactions (including efficient compare-and-swap semantics), object cloning and other primitives for supporting snapshots, simple inter-client communication and coordination (ala Zookeeper), and the ability to extend the object interface using arbitrary code executed on the storage node as part of the IO pipeline. Librados classes are a plugin API that allows computation to be performed on the storage node, avoiding expensive read/modify/write cycles, while still leveraging the replication, atomicity, and high-availability provided by RADOS.

This talk will include a brief overview of the Ceph architecture, and then focus on librados, the API and how it is used, the security model, and some examples of RADOS classes implementing interesting functionality (like cls_lua, which embeds a LUA interpreter for arbitrary code execution).

http://lca2014.linux.org.au - http://www.linux.org.au
CC BY-SA - http://creativecommons.org/licenses/by-sa/4.0/legalcode.txt

Vangelis Koukis will introduce Synnefo [1]: a complete open source cloud stack written in Python/Django, providing Compute, Network, Image, Volume and Storage services, similar to the ones offered by AWS. Synnefo manages multiple Ganeti clusters at the backend for the handling of low-level VM operations and exposes the OpenStack APIs to users. Vangelis will talk about how decoupling the cloud layer from the cluster layer boosts stability and scalability. He will describe Synnefo_s unified, content-addressable storage layer for files, images, and volumes, and how it enables thin, zero-copy cloning and snapshotting independently of the underlying storage technology, with support for live migrations. Finally, the talk will include a sneak peak at a large-scale deployment of Synnefo [2]. [1] http://goo.gl/BFaCB [2] http://goo.gl/aKUEb http://events.linuxfoundation.org/

For more than a year, Ceph has become increasingly popular and saw several deployments inside and outside OpenStack. The community and Ceph itself has greatly matured. Ceph is a fully open source distributed object store, network block device, and file system designed for reliability, performance, and scalability from terabytes to exabytes. Ceph utilizes a novel placement algorithm (CRUSH), active storage nodes, and peer-to-peer gossip protocols to avoid the scalability and reliability problems associated with centralized controllers and lookup tables. Since Grizzly, the Ceph integration gained some good additions: Havana definitely brought tons of awesome features. It also made the integration easier and definitely removed all the tiny hacks. All these things, will certainly encourage people to use Ceph in OpenStack. Ceph is excellent to back OpenStack platforms, no matter how big and complex the platform.. The main goal of the talk is to convince those of you who aren't already using Ceph as a storage backend for OpenStack to do so. I consider the Ceph technology to be the de facto storage backend for OpenStack for a lot of good reasons that I'll expose during the talk. In this session, Sebastien Han from eNovance will go through several subjects such as (depends on the time that I'll get): Quick Ceph overview (for those of you who are not familiar with it) Quick state of the integration with OpenStack (general state and Havana's best additions) Quick cinder drivers overview and comparisons Building a Ceph cluster - general considerations Use cases and design examples (hardware harware hardware) Achieve HA with Ceph Operations: backups, monitoring, upgrades Tips and best practices

With the rising of cloud computing and big data, more and more companies start to build their own cloud storage system. Swift and Ceph are two popular open source software stacks, which are well deployed in today's OpenStack based cloud environment to implement object and virtual block service. In this presentation, we do a deep study on both swift and Ceph performance on Commodity x86 platform, including testing environment, methodolgy and thorough analysis. Tuning guide and optimization BKM (best known method) will also be shared for reference.

Hear detail around a unique deployment of the Dell OpenStack-Powered Cloud Solution with Inktank Ceph installed at a large nationally recognized American University that specializes in cancer and genomic research. The University had a need to provide a scalable, secure, centralized data repository to support approximately 900 researchers and an ever-expanding number of research projects and rapidly expanding universe of data. The Dell and Inktank cloud storage solution addresses these storage challenges with an open source solution that leverages the Dell Crowbar Framework and Reference Architecture. After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that is capable of scaling seamlessly and rapidly, is cost-effective, and that can leverage a single hardware infrastructure, with Dell Power Edge R-720XD servers and the Dell Reference Architecture for their OpenStack compute and storage environment.

Ceph is a massively scalable, open source, software defined storage system that runs on commodity hardware.  Integrated with OpenStack since the Folsom release, Ceph provides cost-effective block and object storage for the cloud.   This demonstration will showcase tools for enterprises who operate Ceph clusters, and will show how they can be used to detect issues, troubleshoot, and report on the health of the entire storage cluster.&nbsp. https://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html

Speaker(s):

Connect with us:
OpenInfra Twitter: https://twitter.com/openinfradev
OpenInfra LinkedIn: https://www.linkedin.com/company/open-infrastructure-foundation/mycompany/
OpenInfra Facebook: https://www.facebook.com/openinfradev
OpenInfra Website: https://openinfra.dev/

#OpenStack #OpenInfra #OpenStackDemo #OpenSource

This presentation covers practical experience from building an OpenStack private cloud with a specific emphasis on high availability. Based on a production deployment of OpenStack Grizzly at Pixelpark (a hosting company based in Berlin, Germany), we explore and explain highly-available data storage with Ceph, OpenStack service high availability with Pacemaker, virtual machine high availability in OpenStack Compute, and high availability considerations for OpenStack Networking. We also outline the lessons a previously conventional hosting company has learned from moving to a private cloud deployment, both from a technical and organizational perspective. About the speakers: Sebastian Kachel is an all-around devops guy at Pixelpark, and has been involved in the OpenStack deployment project since its inception. Florian Haas has provided consulting, training and guidance to Pixelpark over the course of the project. https://www.openstack.org/

Speaker(s):
Sebastian Kachel
Florian Haas https://twitter.com/xahteiwi

Connect with us:
OpenInfra Twitter: https://twitter.com/openinfradev
OpenInfra LinkedIn: https://www.linkedin.com/company/open-infrastructure-foundation/mycompany/
OpenInfra Facebook: https://www.facebook.com/openinfradev
OpenInfra Website: https://openinfra.dev/

#OpenStack #OpenInfra #OpenSource #PrivateHybridCloud #OpenStackHongKongSummit2013

Abstract:
BBC Research & Development are in the process of deploying a department wide virtualization solution, catering for use cases including web development, machine learning, transcoding, media ingress and system testing. This talk discusses the implementation of a high performance Ceph storage backend and the challenges of virtualization in a broadcast research and development environment.

Bio:
Joel Merrick has been involved in system administration and engineering for well over a decade. He is the project lead for an internal VM platform for BBC Research and Development. First becoming involved in virtualisation more than 5 years ago, both professionally and working for the non-profit Sahana Foundation, whereby live production deployments of the software have been running under KVM 'in the field'.

RBD, the RADOS Block Device in Ceph, gives you virtually unlimited scalability (without downtime), high performance, intelligent balancing and self-healing capabilities that traditional SANs can't provide. Ceph achieves this higher throughput through a unique system of placing objects across multiple nodes, and adaptive load balancing that replicates frequently accessed objects over more nodes. This talk will give a brief overview of the Ceph architecture, current integration with Apache CloudStack, and recent advancements with Xen and blktap2.

http://www.twitter.com/user/LinuxConference https://twitter.com/LinuxConference

Speaker: Josh Durgin, Inktank

Ceph is an open source distributed object store, network block device, and file system. Ceph can be used for object storage through its S3 and Swift APIs. It can also provide storage for network block devices, with the thin provisioning and copy-on-write cloning features necessary to support large-scale virtualization.

Since the Folsom release, Cinder makes block storage for backing VMs a first class feature in OpenStack. Block devices can be created from images stored in Glance, and with Ceph's RBD behind both, new VMs can be created faster while using less space. In the latest Ceph 'Bobtail' release, you can start many VMs instantly by cloning from templates. Also, on the storage backend side, you will see increased I/O performance due to improved threading.

This session will cover an intro to Ceph, the current status of Ceph and Grizzly, the latest features of the Ceph Bobtail release and also the technical implications and the advantages of block storage within OpenStack.

For more summit videos, visit: http://www.openstack.org/summit/portland-2013/session-videos/

Speakers: Neil Levine (Inktank), Kamesh Pemmaraju (Dell)

Inktank Ceph is a transformational open source storage solution fully integrated into OpenStack providing scalable object and block storage (via Cinder) using commodity servers. The Ceph solution is resilient to failures, uses storage efficiently, and performs well under a variety of VM Workloads. Dell Crowbar is an open source software framework that can automatically deploy Ceph and OpenStack on bare metal servers in a matter of hours. The Ceph team worked with Dell to create a Ceph barclamp (a crowbar extention) that integrates Glance, Cinder, and Nova-Volume. As a result, it is lot faster and easier to install, configure, and manage a sizable OpenStack and Ceph cluster that is tightly integrated and cost- optimized.

Hear how OpenStack users can address their storage deployment challenges:

- Overview of the Ceph architecture with unique features and benefits
- Overview of Dell Crowbar and how it can automate and simplify Ceph/OpenStack deployments
- Best practices in deploying cloud storage with Ceph and OpenStack.

For more summit videos, visit: http://www.openstack.org/summit/portland-2013/session-videos/

Ross Turk, InkTank, at OpenStack Summit 2013 with John Furrier and Jeff Frick.
Ross Turk, Vice President of Community at Inktank, visited SiliconAngle’s theCube during the OpenStack Summit taking place in Portland this week to discuss Inktank’s open source project, Ceph, and the current community environment with John Furrier and Jeff Frick.

The Ceph Project has been in development since 2004, and is a commodity storage software layer that exposes the storage resources of the hardware that its sitting on top of. Furrier asked Turk what kind of traction Ceph is getting with OpenStack, and what sets it apart from other storage software.

The first reason is the lowest-cost-per-gig, Turk readily answered. Keeping storage costs down is key, especially when scale is becoming more and more of an issue, and, because of Ceph’s self-managing and self-healing capabilities, it is designed for just that. The other reason is Ceph’s focus on integration with cloud, especially with OpenStack.

In the Ceph community, there has been overwhelming growth in participation. While proceeding in an orderly fashion with everyone who wants to contribute to the project, Ceph’s community management team also has to focus on who will maintain the project’s quality and its architectural integrity, making governance a main challenge.

Turk also discussed what trends he’s seen in the open source community. Everyone who, in the early days, was saying “I’m an open source person,” is now saying “I’m a cloud person,” Turk describes, it’s an interesting movement. The open source and cloud communities are really intermingling. Additionally, it’s great to see that everyone has now realized the importance and value of the community. All kinds of companies that didn’t have community intelligence before, definitely have it now.


#theCUBE #Ceph #OpenStack #SiliconANGLE #IntTank

"Scaling Storage with Ceph", Ross Turk, VP of Community, Inktank
Ceph is an open source distributed object store, network block device, and file system designed for reliability, performance, and scalability. It runs on commodity hardware, has no single point of failure, and is supported by the Linux kernel. This talk will describe the Ceph architecture, share its design principles, and discuss how it can be part of a cost-effective, reliable cloud stack.

http://bestdreamhostcoupons.com




RCE 62: Ceph Petabyte Scale Storage Download the entire show and subscribe at: Brock Palen and Jeff Squyres speak with Sage Weil about Ceph, a next generation distributed storage and filesystem for Linux. Sage Weil designed Ceph as part of his PhD research in Storage Systems at the University of California, Santa Cruz. Since graduating, he has continued to refine the system with the goal of providing a stable next generation distributed storage and file system for Linux. Prior to his graduate work, Sage helped found New Dream Network, the company behind DreamHost.com, who now supports a small team of Ceph developers. RCE 62: Ceph Petabyte Scale Storage