Mirantis Official Blog

Under the hood of Swift. The Ring

2012-02-14T05:51:00.000-08:00

This is the first post in series that summarizes our analysis of Swift architecture. We've tried to highlight some points that are not clear enough in the official documentation. Our primary base was an in-depth look into the source code. The Ring is the vital part of Swift architecture. This half database, half configuration file keeps track of where all data resides in the cluster. For each possible path to any stored entity in the cluster, the Ring points to the particular device on the particular physical node.

There are three types of entities that Swift recognizes: accounts, containers and objects. Each type has the ring of its own, but all three rings are put up the same way. Swift services use the same source code to create and query all three rings. Two Swift classes are responsible for this tasks: RingBuilder and Ring respectively.

Ring data structure

Every Ring of three in Swift is the structure that consists of 3 elements:

a list of devices in the cluster, also known as devs in the Ring class;
a list of lists of devices ids indicating partition to data assignments, stored in variable named _replica2part2dev_id;
an integer number of bits to shift an MD5-hashed path to the account/container/object to calculate the partition index for the hash (partition shift value, part_shift).

List of devices

A list of devices includes all storage devices (disks) known to the ring. Each element of this list is a dictionary of the following structure:

Key	Type	Value
id	integer	Index of the devices list
zone	integer	Zone the device resides in
weight	float	The relative weight of the device to the other devices in the ring
ip	string	IP address of server containing the device
port	integer	TCP port the server uses to serve requests for the device
device	string	Disk name of the device in the host system, e.g. `sda1`. It is used to identify disk mount point under `/srv/node` on the host system
meta	string	General-use field for storing arbitrary information about the device. Not used by servers directly

Some device management can be performed using values in the list. First, for the removed devices, the 'id' value is set to 'None'. Device IDs are generally not reused. Second, setting 'weight' to 0.0 disables the device temporarily, as no partitions will be assigned to that device.

Partitions assignment list

This data structure is a list of N elements, where N is the replica count for the cluster. The default replica count is 3. Each element of partitions assignment list is an array('H'), or Python compact efficient array of short unsigned integer values. These values are actually index into a list of devices (see previous section). So, each array('H') in the partitions assignment list represents mapping partitions to devices ID.

The ring takes a configurable number of bits from a path's MD5 hash and converts it to the long integer number. This number is used as an index into the array('H'). This index points to the array element that designates an ID of the device to which the partition is mapped. Number of bits kept from the hash is known as the partition power, and 2 to the partition power indicates the partition count.

For a given partition number, each replica's device will not be in the same zone as any other replica's device. Zones can be used to group devices based on physical locations, power separations, network separations, or any other attribute that could make multiple replicas unavailable at the same time.

Partition Shift Value

This is the number of bits taken from MD5 hash of '/account/[container/[object]]' path to calculate partition index for the path. Partition index is calculated by translating binary portion of hash into integer number.

Ring operation

The structure described above is stored as pickled (see Python pickle) and gzipped (see Python gzip.GzipFile) file. There are three files, one per ring, and usually their names are:

account.ring.gz
container.ring.gz
object.ring.gz

These files must exist in /etc/swift directory on every Swift cluster node, both Proxy and Storage, as services on all these nodes use it to locate entities in cluster. Moreover, ring files on all nodes in the cluster must have the same contents, so cluster can function properly.

There are no internal Swift mechanisms that can guarantee that the ring is consistent, i.e. gzip file is not corrupt and can be read. Swift services have no way to tell if all nodes have the same version of rings. Maintenance of ring files is administrator's responsibility. These tasks can be automated by means external to the Swift itself, of course.

The Ring allows any Swift service to identify which Storage node to query for the particular storage entity. Method Ring.get_nodes(account, container=None, obj=None) is used for identification of target Storage node for the given path (/account[/container[/object]]). It returns the tuple of partition and dictionary of nodes. The partition is used for constructing the local path to object file or account/container database. Nodes dictionary elements have the same structure as the devices in list of devices (see above).

Ring management

Swift services can not change the Ring. Ring is managed by swift-ring-builder script. When new Ring is created, first administrator should specify builder file and main parameter of the Ring: partition power (or partition shift value), number of replicas of each partition in cluster, and the time in hours before a specific partition can be moved in succession:

When the temporary builder file structure is created, administrator should add devices to the Ring. For each device, required values are zone number, IP address of the Storage node, port on which server is listening, device name (e.g. sdb1), optional device meta-data (e.g., model name, installation date or anything else) and device weight:

Device weight is used to distribute partitions between the devices. More the device weight, more partitions are going to be assigned to that device. Recommended initial approach is to use the same size devices across the cluster and set weight 100.0 to each device. For devices added later, weight should be proportional to the capacity. At this point, all devices that will initially be in the cluster, should be added to the Ring. Consistency of the builder file can be verified before creating actual Ring file:

In case of successful verification, the next step is to distribute partitions between devices and create actual Ring file. It is called 'rebalance' the Ring. This process is designed to move as few partitions as possible to minimize the data exchange between nodes, so it is important that all necessary changes to the Ring are made before rebalancing it:

The whole procedure must be repeated for all three rings: account, container and object. The resulting .ring.gz files should be pushed to all nodes in cluster. Builder files are also needed for the future changes to rings, so they should be backed up and kept in safe place. One of approaches is to put them to the Swift storage as ordinary objects.

Physical disk usage

Partition is essentially the block of data stored in the cluster. This does not mean, however, that disk usage is constant for all partitions. Distribution of objects between the partitions is based on the object path hash, not the object size or other parameters. Objects are not partitioned, which means that an object is kept as a single file in storage node file system (except very large objects, greater than 5Gb, which can be uploaded in segments - see the Swift documentation).

The partition mapped to the storage device is actually a directory in structure under /srv/node/<dev_name>. The disk space used by this directory may vary from partition to partition, depending on size of objects that have been placed to this partition by mapping hash of object path to the Ring.

In conclusion it should be said that the Swift Ring is a beautiful structure, though it lacks a degree of automation and synchronization between nodes. I'm going to write about how to solve these problems in one of the following posts.

More information

More information about Swift Ring can be found in following sources:
Official Swift documentation - base source for description of data structure
Swift Ring source code on Github - code base of Ring and RingBuilder Swift classes.
Blog of Chmouel Boudjnah - contains useful Swift hints

Introducing OpenStackAgent for Xen-based Clouds. What?

2012-01-30T05:28:00.000-08:00

What it is all about

Not long ago we’ve been working on deployment of OpenStack Cactus-based public cloud using Xen as an underlying hypervisor. One of the problems we’ve faced were Windows guest instances failing to set up their administrator password to those generated by nova on instance creation. As it turned out the overall process of compute-guest instance communication in OpenStack-Xen environment is rather tricky (see the illustration). One of the core components of the process is so called guest agent - a special user space service which runs within a guest OS and executes commands provided from outside. Originally we’ve used the guest agent implementation provided by Rackspace. One can find the source code both for *nix and Windows OS on the Launchpad page. Although the project seemed to be quite stable at the moment the service built from C# code and combined with Cactus version of nova plugin for Xen was unable to set the password for the Windows instances. Deep log analysis revealed the problem at the stage of cryptography engine initialization. It should be noted that the procedure of resetting administrator’s password itself is complex. It first includes Diffie-Hellman key exchange between compute and guest agent. Next the password is encrypted for the sake of security and sent via the public channel i.e. Xen Store to the agent. For the deadline was coming in several hours we had no time to set up a proper environment for debugging and therefore we decided to perform a rather immature step which turned out to be a success afterwards. Hastily we implemented our own guest agent service using pywin32 library. Later on, it acquired several additional features including MSI installer and grew up into a separate project named OpenStackAgent. And now we would like to introduce it to the community.

What it is currently capable of

Instance spawn time and run time password changing.
Updating itself from network on “update” command retrieval.
Running and logging :)

The capabilities look quite limited, don’t they? However it is yet of version 0.0.5 and the development is on the way. Thus visit the project homepage soon to find the new features.

What else is going to be implemented

Support *nix OS as well. We’re planning to share the same code base for all the types of guest OSes.
Switch to pyInstaller extension for distribution package creation. Get rid of py2exe to make building really cross-platform.
Support guest network adapter configuration commands and file injection as well.
Tests, tests and once again tests.

What does one need to use it

Building
In order to build the agent one will need the following software to be installed

Python interpreter of version 2.7 or higher
PyCrypto version 2.x. Pre-compiled binary distribution for Windows.
py2exe extension for Python
py2exe2msi extension. Easily installable from PyPi. with easy_install py2exe2msi command

After everything is ready to build run python guest_agent/setup.py py2exe2msi and find the compiled MSI package in the current working directory.

Running
In order to run the compiled service the following requirements has to be satisfied on the target machine

Microsoft Visual C 2008 SP1 Runtime.
The latest version of Xen Guest Utilities installed

Just install the MSI package from "Building" step and the service will be started automatically. In order to troubleshoot look up the system application event log or log file located at %WINDIR%\Logs\OpenStackAgent.log

Updating
Just install the MSI package of a newer version into the system. It will automatically replace all the required components and restart services.

What should one do to contribute

Fork it, update it, merge it using the GitHub repository but make sure you follow the Apache 2.0 license.

Diablo RPM repository

2011-12-29T03:41:00.000-08:00

Recently we've deployed OpenStack Diablo release for one of our customers. The target operating system happened to be CentOS 6.0. During deployment testing we've stumbled upon a number of bugs in OpenStack RPMs that we've tried to use.

All existing RPMs of OpenStack that we've found contained problems that prevented components from operating correctly with each other:
1. Incompatible protocol in packaged version of Keystone (already fixed): https://lists.launchpad.net/openstack/msg04876.html
2. Json template bug (already fixed): https://bugs.launchpad.net/keystone/+bug/865448/
3. ISCSI target management troubles: https://bugzilla.redhat.com/show_bug.cgi?id=737046

In addition, there was no packaged nova-vnc in CentOS repositories.
So we've fixed these bugs and established our own repository for OpenStack Diablo. Packages added there have been tested in a real-world deployment.

You can easily install the repository on your CentOS system using wget:

$ sudo wget -O /etc/yum.repos.d/epel-mirantis.repo http://download.mirantis.com/epel-el6-mirantis/epel-mirantis.repo

You can browse the repository here: Mirantis OpenStack Diablo

Meet & Drink: OpenStack in Production – Event Highlights

2011-12-20T15:06:00.001-08:00

As a matter of tradition at this point, we offer a photo report, covering OpenStack Meetup event series hosted by Mirantis and Silicon Valley Cloud Center. Our December 14th event focused on sharing experience around running OpenStack in production. I moderated a panel consisting of Ken Pepple – director of cloud development at Internap, Ray O’Brian – CTO of IT at NASA and Rodrigo Benzaquen – R&D director at MercadoLibre.

This time we went all out and even recorded the video of the event:

For those that are not in the mood to watch this 50 minute panel video, here is a quick photo report:

We served wine and beer with pizza, salad and deserts...

...While people ate, drank, and mingled...

…and then they drank some more…

We started the panel with myself saying smart stuff about OpenStack. After the intro we kicked off with questions to the panel.

The panelists talked...

...and talked...

...and then talked some more.

Meanwhile, the audience listened...

...and listened.

Everyone in our US team was sporting these OpenStack shirts.

At the end we gave out 5 signed copies of "Deploying OpenStack" books, written by one of our panelists - Ken Pepple. Roman (pictured above) did not get a copy.

Converging OpenStack with Nexenta

2011-11-24T15:10:00.000-08:00

For those folks that have missed our webcast on using OpenStack Compute with NexentaStor for managing VM volumes, recording is below.

Please note, you can download the NexentaStor driver for OpenStack here: http://www.nexentastor.org/projects/osvd/files.

You can also read additional information about this project here: http://wiki.openstack.org/NexentaVolumeDriver

If you need help installing / troubleshooting the Nexenta driver for OpenStack, please do contact us.

OpenStack Meet & Drink: Toast to Diablo – Event Highlights

2011-09-29T15:01:00.000-07:00

As usual, here are the highlights from the last Bay Area OpenStack Meet & Drink: Toast to Diablo – September 28th, 2011. Thanks to WireRE for hosting us, Dave Nielsen – for helping to organize, and all the attendees – for coming. Once again, this was the biggest MeetUp thus far with 150 in attendance. For those of you that didn’t come – here is what you missed:

We started our Diablo release celebration with wine, beer and pizza. Fun mingling with fellow stackers. As people kept arriving it got almost too crowded.

Mirantis founder – Alex Freedland – passionately explaining something to David Allen.

Mike Scherbakov from Mirantis, Josh McKenty from Pison and Eric from CloudScaling debating OpenStack with noticeable vigor.

Eric Windisch proudly sporting his uber cool CloudScaling shirt, listing to Mike Scherbakov from Mirantis.

While the crowd was mingling, Dave Nielsen took people on datacenter tours. The datacenter basically looked like this.

As usual, I opened with some thank you's and acknowledgements to our sponsors and organizers. Marc Padovani of HP Cloud Services – clapping and anxiously waiting his turn to tell the crowd about OpenStack based hpcloud.com.

With 150 stackers in attendance, we didn’t have quite enough chairs to accommodate everyone.

Dave Nielsen talking about our venue host – WiredRE.

Chris Kemp – CEO and Founder of Nebula announced the OpenStack Silicon Valley LinkedIn group that Nebula recently started.

…meanwhile, Josh McKenty was waiting for his turn to speak…

Don’t remember why, but for some reason Josh’s presentation involved talking about O-Ren Ishi from Kill Bill. Whatever it was, Chris Kemp got a kick out of it.

Everybody likes Kill Bill, so the crowd was cheering.

Geva Perry shared his perspective on why OpenStack’s strength is in its ecosystem of developers and partners.

Jason Venner of X.com talked about OpenStack and CloudFoundry. He was careful not to reveal anything with respect to the upcoming “October 13th” announcement of X.commerce platform.

In closing we had Marc Padovani from HP talk about hpcloud and HP’s commitment to OpenStack. The presentation quickly turned into a Q&A grilling session, with stackers expressing their suspicions over hpcloud.com being a smoke screen, rather than real offering. Marc did his best to address the questions without incriminating his big corporation… My wife got too tired of taking pictures at that point, so there are none of Marc… sorry Marc.

Hungry stackers drank most of the wine and ate most of the food. Whatever was left over, people took home. We kept one last bottle of Cloud Wine. I intend to give it as a gift to our 500th MeetUp member – Ilan Rabinovich. Ilan – if you read this, ping me on twitter @zer0tweets to claim your prize!

Thank you to everyone and we’ll do it again in 3 months.

What is this Keystone anyway?

2011-09-23T01:16:00.000-07:00

The simplest way to authenticate a user is to ask for credentials (login+password, login+keys, etc.) and check them over some database. But when it comes to lots of separate services as it is in the OpenStack world, we have to rethink that. The main problem is an inability to use one user entity to be authorized everywhere. For example, a user expects Nova to get one's credentials and create or fetch some images in Glance or set up networks in Quantum. This cannot be done without a central authentication and authorization system.

So now we have one more OpenStack project - Keystone. It is intended to incorporate all common information about users and their capabilities across other services, along with a list of these services themselves. We have spent some time explaining to our friends what, why, and how it is and now we decided to blog about it. What follows is an explanation of every entity that drives Keystone’s life. Of course, this explanation can become outdated in no time since the Keystone project is very young and it has developed very fast.

The first basis is the user. Users are users; they represent someone or something that can gain access through Keystone. Users come with credentials that can be checked like passwords or API keys.

The second one is tenant. It represents what is called the project in Nova, meaning something that aggregates the number of resources in each service. For example, a tenant can have some machines in Nova, a number of images in Swift/Glance, and couple of networks in Quantum. Users are always bound to some tenant by default.

The third and last authorization-related kinds of objects are roles. They represent a group of users that is assumed to have some access to resources, e.g. some VMs in Nova and a number of images in Glance. Users can be added to any role either globally or in a tenant. In the first case, the user gains access implied by the role to the resources in all tenants; in the second case, one's access is limited to resources of the corresponding tenant. For example, the user can be an operator of all tenants and an admin of his own playground.

Now let’s talk about service discovery capabilities. With the first three primitives, any service (Nova, Glance, Swift) can check whether or not the user has access to resources. But to try to access some service in the tenant, the user has to know that the service exists and to find a way to access it. So the basic objects here are services. They are actually just some distinguished names. The roles we've talked about recently can be not only general but also bound to a service. For example, when Swift requires administrator access to create some object, it should not require the user to have administrator access to Nova too. To achieve that, we should create two separate Admin roles - one bound to Swift and another bound to Nova. After that admin access to Swift can be given to user with no impact on Nova and vice versa.

To access a service, we have to know its endpoint. So there are endpoint templates in Keystone that provide information about all existing endpoints of all existing services. One endpoint template provides a list of URLs to access an instance of service. These URLs are public, private and admin ones. The public one is intended to be accessible from the global world (like http://compute.example.com), the private one can be used to access from a local network (like http://compute.example.local), and the admin one is used in case admin access to service is separated from the common access (like it is in Keystone).

Now we have the global list of services that exist in our farm and we can bind tenants to them. Every tenant can have its own list of service instances and this binding entity is named the endpoint, which “plugs” the tenant to one service instance. It makes it possible, for example, to have two tenants that share a common image store but use distinct compute servers.

This is a long list of entities that are involved in the process but how does it actually work?

To access some service, users provide their credentials to Keystone and receive a token. The token is just a string that is connected to the user and tenant internally by Keystone. This token travels between services with every user request or requests generated by a service to another service to process the user's request.
The users find a URL of a service that they need. If the user, for example, wants to spawn a new VM instance in Nova, one can find an URL to Nova in the list of endpoints provided by Keystone and send an appropriate request.
After that, Nova verifies the validity of the token in Keystone and should create an instance from some image by the provided image ID and plug it into some network.
- At first Nova passes this token to Glance to get the image stored somewhere in there.
- After that, it asks Quantum to plug this new instance into a network; Quantum verifies whether the user has access to the network in its own database and to the interface of VM by requesting info in Nova.
All the way this token travels between services so that they can ask Keystone or each other for additional information or some actions.

Here is a rough diagram of this process:

Cloudpipe Image Creation Automation

2011-09-16T12:06:00.000-07:00

Cloudpipe is used in OpenStack to provide access to project’s instances when using VLAN networking mode. It is just a custom Virtual Machine (VM) prepared in a special way, i.e. coming with an accordingly configuredopenvpn and startup scripts. More details on what cloudpipe is and why it is needed are available in OpenStack documentation.
The process of creating an image involves a lot of manual steps which crave to be automated.To simplify these steps, I wrote a simple script that uses some libvirt features to provide fully automated solution, in a way that you don't even have to bother with preparing base VM manually.
The solution can be found on a github and consists of 3 parts:

The first ubuntukickstart.sh is the main part. Only this part should be executed.When you run it, it will configure the virtual network and PXE.Then it will start a new VM to install a minimal server Ubuntu bykickstart, so the installation is fully automated and unattended.
The second cloudpipeconf.sh is used to turn minimal server Ubuntu to cloudpipe. It is being executed when the VM is ready to make this turning.
The last ssh.fs is used to ssh into the VM and shutdown it.

So, if you need the cloudpipe image, just run ubuntukickstart.sh andwait. You'll get the cloudpipe image without any mouse clickings andkeyboard pressings!
More detailed information about how it works can be found in README file.
Don’t hesitate to leave a comment If you have any questions or concerns.

Cloud Accelerates Open Source Adoption

2011-09-08T16:05:00.001-07:00

Historically, commercial software provided enterprises with reliability and scalability, especially for mission-critical tasks. No one wanted to risk failure in finance, operations, or any essential or enterprise-wide areas. So, enterprises considered open source technology only for less important, more tactical purposes.

Recently, however, many large IT organizations have developed significant open source strategies. Cisco, Dell, NASA, and Rackspace came together to give birth to OpenStack. VMWare acquired SpringSource and shortly thereafter, announced Cloud Foundry, their open source PaaS. Amazon, salesforce.com, and others built solutions entirely on an open source stack. Whole categories of technologies, such as noSQL databases, made their way to mass adoption shortly after being open sourced by Google and Facebook. There has been more activity in open source during the last two years than in the preceding decade. So what’s going on here?

Without a doubt, cloud is the IT topic that’s been grabbing headlines and investment dollars in the past few years. The recent high level of activity in open source noticeably correlates with the cloud movement, because there is a deep, synergetic relationship between the two. In fact, cloud is the primary driver for the increased adoption of open source.

In general, open source projects typically require two components to get community uptake. First, the nature of the project itself has to be technologically challenging. Successful open source projects are largely about solving a set of complex technological tasks vs. just writing a lot of code to support complex business process, such as the case with building enterprise software. Linux, MySQL and BitTorrent are all good examples here. Second, it requires a high rate of end user adoption. The more people and organizations that start using the open source technology at hand, the more mature the community and the technology itself becomes.

Cloud has created an enormous amount of technologically challenging fodder for the open source community. The adoption of cloud translates to greater scale at the application infrastructure layer. Consequently, all cloud vendors, from infrastructure to application, are forced to innovate and build proprietary application infrastructure solutions aimed at tackling scale-driven complexity. Facebook’s Cassandra and Google’s Google File System/Hadoop/BigTable stack are prime examples of this innovation.

However, it is important to note that neither Facebook, nor Google are in the business of selling middleware. Both make money on advertising. Their middleware stack may be a competitive advantage, but it is by no means THE competitive advantage. Because companies want to keep IT investments as low as possible, a the natural way to drive down costs associated with scale-driven complexity is to have the open developer community help address at least some of the issues to support and growing the stack. The result? Instances like Facebook’s open sourcing of Cassandra and Rackspace contributing its object storage code to OpenStack. Ultimately, cloud drives complexity while cloud vendors channel that complexity down to the open developer community.

What about end user adoption? Historically, enterprises were slow to adopt open source. Decades of lobbying by vendors of proprietary software have drilled the idea of commercial software superiority deep into the bureaucracy of enterprise IT. Until recently, the biggest selling point for commercial enterprise software was reliability and scalability for mission-critical tasks; open source was “OK” for less important, more tactical purposes. Today, after leading cloud vendors like Amazon, Rackspace, and Google built solutions on top of an open source stack, the case against open source for mission-critical operations or incapable of supporting the required scale is no longer valid.

But the wave of open source adoption is not just about the credibility boost it received in recent years. It is largely about the consumption model. Cloud essentially refers to the new paradigm for delivery of IT services. It is an economic model that revolves around “pay for what you get, when you get it.” Surprisingly, it took enterprises a very long time to accept this approach, but last year was pivotal in showing that it is tracking and is the way of the future. Open source historically has been monetized leveraging a model that is much closer to “cloud” than that of commercial software. In the case of commercial software, you buy the license and pay for implementation upfront. If you are lucky to implement, you continue to pay for a subscription that is sold in various forms – support, service assurance, etc. With open source, you are free to implement first, and if it works, you may (or may not) buy commercial support, which is also frequently sold as a subscription to a particular SLA. The cloud hype has helped initiate the shift in the standard for the IT services consumption model. As enterprises wrap their minds around cloud, they shy further away from the traditional commercial software model and move closer to the open source / services-focused model.

It is also important to note that the consumption model issue is not simply a matter of perception. There are concrete, tactical drivers behind it. As the world embraces the services model, it is becoming increasingly dominated by service-level agreements (SLAs). People are no longer interested in licensing software products that are just a means to an end. Today, they look for meaningful guarantees where vendors (external providers or internal IT) assure a promised end result. This shift away from end user licensing agreements (EULAs) and toward SLAs is important. If you are a cloud vendor such as Salesforce.com, you are in the business of selling SLA-backed subscription services to your customer. If, at the same time, you rely on a third party vendor for a component of your stack, the SLA of your vendor has to provide the same or better guarantees that you pass on to your client. If your vendor doesn’t offer an SLA or only offers an end user license agreement, you end up having to bridge the gap. These gaps that an organization is forced to bridge ultimately affects its enterprise value. As we move away from the EULA-driven economy and more towards SLAs, open source stands to benefit.

Ultimately, as cloud continues to mature, we will continue to see more and faster growth in open source. While the largest impact so far has been in the infrastructure space, open source popularity will eventually start spreading up the stack towards the application layer.

Tracing the IT Evolution from the Big Bang to the Big Crunch

2011-08-25T14:13:00.000-07:00

How enterprises are progressing from overgrown, difficult-to-manage IT systems to high performance open source infrastructure

Over the history of computing, we can trace a pattern of continuous decomposition, from a single system into disparate components. Early on, these individual parts made it easier to design, program and maintain systems, and meet the fast-growing demand for more power and more capacity.

The industry began with the mainframe, where the entire stack from hardware to application logic was contained in a single box. The next phase was the move from mainframe to client-server. This was followed by SOA (service-oriented architecture). This process of decomposition is a natural byproduct of growth in scale. As we consume increasingly more computing and storage, efficiency gains are achieved through specialization.

Such continuous decomposition is a typical pattern of many industries. Several centuries ago, the model was subsistence farming, where every family as a single unit grew all of their own crops. Today, food production has decomposed into a collection of highly specialized industries.

However, this process of decomposition in IT injects complexity. At a certain scale, highly decomposed systems become extremely challenging to manage. This then drives a pressing need to abstract away from some of the individual components to a higher level. This is largely what we are observing today with infrastructure computing. The complex mammoth of enterprise IT, today comprised of a spaghetti mix of application servers, relational and noSQL databases, messaging queues, caching and search services, etc., is no longer manageable.

Gartner labeled 2011 as the year of cloud platforms or PaaS. Thinking of PaaS, we intuitively think Heroku, Force.com, and Google App Engine, all off-premise cloud platforms. But the cloud movement is not just about on-premise versus off-premise. It's about creating an effective means to abstract away from application infrastructure complexity. As mainframes exploded into myriad sub-components, we experienced sort of a Big Bang in enterprise IT. What we are starting to observe now is the Big Crunch, turning application infrastructure back into a more unified, manageable artifact.

OpenStack

OpenStack is one of the most interesting initiatives topping the headlines during the last several months, and it's directly related to the Big Crunch. An open source project with the promise to help consolidate the many disparate components of application infrastructure, OpenStack is only a year old and is far from fulfilling this promise today. However, I believe that OpenStack for application infrastructure will eventually become what Linux became to application logic many years ago - a single interface unifying all application infrastructure components and exposing a standardized set of APIs to applications running on top of it.

Open Source Cloud Projects and How They Differ

OpenStack is not the first open source cloud project. Eucalyptus, OpenNebula, and Cloud.com all emerged before OpenStack and all of them are still very much alive. However, OpenStack is different from these others because it's the only one that has gained enough critical mass to get on a steady course to mass adoption.

What enabled OpenStack to reach this point was not an accident, but a clever strategy by RackSpace and other founding members. Rather than following a more common, vendor-centric approach to building an open source community, like Eucalyptus and Cloud.com did, RackSpace quickly figured out that getting a "cloud operating system" to mass adoption would require more marketing muscle then any single vendor has. So it positioned OpenStack as a decentralized, community-driven project from the very beginning and set out to get the support of big players in the application infrastructure space, namely Dell, Cisco, and Citrix. It didn't go after just any infrastructure player, but specifically focused on those who were arguably late to the cloud game and aching to make up the distance they lost to the likes of VMware and IBM. Ultimately, OpenStack's blitz to success is a result of unleashing an enormous amount of marketing energy in a short period of time, carefully coordinated between a number of application infrastructure power houses.

Following Amazon to Open Source Infrastructure

Today, OpenStack is focused on low level infrastructure services - compute, storage, image service, etc., and much work still remains to be done by the community in that area. However, we know the trend and have already seen it with Amazon Web Services (AWS). AWS initially started as Infrastructure as a Service (IaaS) with EC2 and S3 offerings; it then evolved into a fully blown Platform as a Service (PaaS). The value in solving application infrastructure complexity in a broader sense, by embedding higher level services like automated deployment, message queues, map reduce, and monitoring, is simply too compelling. At some point, we expect to see OpenStack creeping into the PaaS space, the same way AWS is doing today.

This gradual transition from simply being a compute and storage infrastructure orchestrator into a complete cloud operating system will happen naturally for OpenStack. It will be driven by infrastructure vendors of all sizes that are looking to plug their solutions into the OpenStack ecosystem. With more than 100 member companies on board already today, we see various announcements to this effect right and left: Gluster contributes its file system, Dell builds a deployment services, CloudCruiser builds a cost management solution, etc.

What's Ahead for OpenStack

The openness and decentralized nature of OpenStack is central to the realization of its vision of the cloud operating system. Instead of trying to solve all application infrastructure complexity inside one monolithic system, such as with the VMware stack, OpenStack harnesses the naturally occurring decomposition in the infrastructure space. This is the Big Bang in infrastructure we've all experienced. Individual vendors with competence in one particular area of application infrastructure can plug their solutions (storage, caching, monitoring, etc.) into OpenStack. As OpenStack continues to gain adoption, it will become a channel for infrastructure vendors to sell their offerings in the same way that the Apple app store is a channel for mobile app developers. At the same time, OpenStack will help abstract end users and resident applications away from the complexity of disparate infrastructure solutions.

Today we are still in the early days of OpenStack. It's far from being the ultimate platform. It may also be less feature-rich than competing offerings from Microsoft or VMware. However, this is unimportant today. What's important is that the need for the Big Crunch that will decrease application infrastructure complexity is obvious. The magnitude of effort required to make this happen is not something any single vendor could credibly pull off. Ultimately, it's not OpenStack features that matter, but the "idea" behind this project and the degree of uptake it has already received in the community. When many people come together to realize a sensible vision, that vision inevitably becomes a reality.

Our Contribution to the Vegas Economy

2011-08-16T16:15:00.000-07:00

Here are the highlights on our corporate team-building in Vegas last week. Special thanks to Rachel and Athena for making this party happen. Thank you to all who participated and helped make it fun.

We started by warming up with some drinks in the airport bar on the way over to Vegas.

The luggage belt broke upon our arrival and it took over an hour to get our luggage. By then, the buzz from the airport bar session started to wear off… =(.

The taxi line outside the airport was loooong… so we decided to embellish our Vegas experience immediately by taking a limo to the hotel.

Finally arrived; herding around the Aria hotel entrance.

After a brief bite to eat in Vettro café in Aria (which, by the way, is a horrible restaurant… don’t go there), we split up into two groups - the strong and the weak. The weak went to sleep or gamble. The strong went clubbing. Came back to the hotel room only at 4am.

The next morning, we woke up to this view; 51st floor in Aria. Don’t get too excited – as with many Vegas hotels, they don’t have floors 40-50 in Aria.

Breakfast… some people slept in late, so our ranks were slim at breakfast.

Ilya enjoyed his fries enormously!

Next stop – quintessential Vegas pool party at Liquid Lounge. $5 to anyone who can spot Mike Scherbakov and Julia Varigina in the crowd.

Why would anyone herd in the pool with 100 people in it, music blasting and no seating space, when you can quietly lounge next to one of 10 other pools in the hotel? The point of the pool party only comes to you after a few drinks… as you can see from the stampede by the bar, we were not the only ones to feel that way.

Once you get a drink in your hand – it’s BLAST OFF!

No comment.

Winding down at the pool… next stop: corporate dinner.

It was dark and all we had was a point and shoot… so not so many pictures at the dinner. But basically this is what it looked like.

After the dinner we went to watch a show – Absinthe. This group picture was taken immediately after.

11:48pm – time to split up again into gamblers and partiers. Since I belonged to the party group, you don’t get to see the pictures of the gamblers… sorry.

Second night of clubbing looked like this. 1:45am and Mike is asleep on a couch at Tryst. This is called a SHUT DOWN!

And my SHUT DOWN happened in the airport on the way back.

LDAP identity store for OpenStack Keystone

2011-08-12T06:19:00.000-07:00

After some time working with OpenStack installation using existing LDAP installation for authentication, we encountered one big problem. The latest Dashboard code dropped support of old bare authentication in favor of Keystone-based one. That time Keystone had no support for multiple authentication backends, so we had to develop this feature.
Now we have a basic support of LDAP authentication in Keystone which provides subset of functionality that was present in Nova. Currently, the main limitation is inability to actually integrate with the existing LDAP tree due to limitations in backend, but it works fine in isolated corner of LDAP.
So, after a long time of coding and fighting with new upstream workflows, we can give you a chance to try it out.
To do it, one should:

Make sure that all necessary components are installed. They are Nova, Glance, Keystone and Dashboard.

Since the latter pair is still in incubator, you’ll have to download them from the source repository:
Set up Nova to authorize requests in Keystone:

It assumes that you’re in the same dir where you’ve downloaded Keystone sources. Replace nova.conf path if it differs in your Nova installation.
Add schema information to your LDAP installation.

It heavily depends on your LDAP server. There is a common .schema file and .ldif for the latest version of OpenLDAP in keystone/keystone/backends/ldap/ dir. For local OpenLDAP installation, this will do the trick (if you haven’t change the dir after previous steps):
Modify Keystone configuration at keystone/etc/keystone.conf to use ldap backend:
- add keystone.backends.ldap to the backends list in [DEFAULT] section;
- remove Tenant, User, UserRoleAssociation and Token from the backend_entities list in [keystone.backends.sqlalchemy] section;
- add new section (don’t forget to change URL, user and password to match your installation):
Make sure that ou=Groups,dc=example,dc=com and ou=Users,dc=example,dc=com subtree exists or set LDAP backend to use any other ones by adding tenant_tree_dn, role_tree_dn and user_tree_dn parameters into [keystone.backends.ldap] section in config file.
Run Nova, Keystone and Dashboard as usual.
Create some users, tenants, endpoints, etc. in Keystone by using keystone/bin/keystone-manage command or just run keystone/bin/sample-data.sh to add the test ones.

Bay Area OpenStack Meet & Drink Highlights

2011-06-30T15:06:00.000-07:00

For those of you that weren’t able to make it yesterday and maybe for those of you who want to reminisce about the events of last night, Bay Area OpenStack Meet & Drink was probably the most well-attended OpenStack meetup in the valley to date, outside of the OpenStack summit this spring. A diverse crowd of over 120 stackers showed up – ranging from folks just learning the basics of OpenStack to hardcore code committers.

We originally planned on hosting a 30-40 person tech meetup session in a small cozy space at the Computer History Museum. However, with over 100 RSVPs we had to go all out and rent out Hahn Auditorium, making space for all of those wanting to participate.

First 40 minutes – people eating drinking and mingling. The food line was a bit overwhelming.

Cloud wine was served with dinner.

Joe Arnold from Cloudscaling brought a demo server, running SWIFT for people to play around with.

I opened the ceremony with a 5-minute intro – polling the audience on their experience with OpenStack, saying a few words about Mirantis and upcoming events, as well as introducing Mirantis team members.

Meanwhile, Joe was getting all too excited to do his pitch of SWIFT.

Joe did his 10-minute talk on “Swift in the Small.” You can read up on the content that was presented in Joe’s blog: http://joearnold.com/2011/06/27/swift-in-the-small/. You can also view the slides here: http://bit.ly/mMRcpt. And the live recording of the presentation can be found here: http://bit.ly/mJOr2R

We gave out Russian Standard vodka bottles at the meetup as favors. To complete the theme and give the audience a taste of Russian hospitality, we had an accordionist perform a 5-minute stunt immediately after Joe’s pitch on Swift (see his performance here: http://bit.ly/iiYveN).

Party time…

Mike Scherbakov from our team of stackers talked about implementing Nova in Mirantis’ internal IT department, taking quite a few questions from the audience. The deck of his presentation is here: http://slidesha.re/jyS4WL. The recording of the talk can be found here: part 1; part 2; part 3; and part 4.

I’d like to thank everyone for coming and we’ll appreciate any comments or suggestions on the event. We plan to have our next meetup at the end of September. If you would like to help organize, present your OpenStack story, or offer any ideas on how to make the experience better, please ping me on twitter @zer0tweets or send me an email – borisr at mirantis dot com.

vCider Virtual Switch Overview

2011-06-30T03:47:00.000-07:00

A couple of months ago, Chris Marino, CEO at vCider, stopped by the Mirantis office and gave a very interesting presentation on the vCider networking solution for clouds. A few days later, he kindly provided me with beta access to their product.

A few days ago, vCider announced public availability of the product. So now it's a good time to blog about my experience concerning it.

About vCider Virtual Switch

To make a long story short, vCider Virtual Switch allows you to build a virtual Layer 2 network across several Linux boxes; these boxes might be Virtual Machines (VMs) on a cloud (or even in different clouds), or it might be a physical server.

The flow is pretty simple: you download a package (DEBs and RPMs are available on the site) and install it to all of the boxes for which you will create a network. No configuration is required except for creating a file with an account token.

After that, all you have to do is to visit the vCider Dashboard and create networks and assign nodes to them.

So to start playing with that, I created two nodes on Rackspace and created a virtual network for them for which I used 192.168.87.0/24 address space.

On both boxes two new network interfaces appeared:

On the first box:

5: vcider-net0:  mtu 1442 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether ee:cb:0b:93:34:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.87.1/24 brd 192.168.87.255 scope global vcider-net0
inet6 fe80::eccb:bff:fe93:3445/64 scope link
 valid_lft forever preferred_lft forever

and on the second one:

7: vcider-net0:  mtu 1442 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 6e:8e:a0:e9:a0:72 brd ff:ff:ff:ff:ff:ff
inet 192.168.87.4/24 brd 192.168.87.255 scope global vcider-net0
inet6 fe80::6c8e:a0ff:fee9:a072/64 scope link
 valid_lft forever preferred_lft forever

tracepath output looks like this:

root@alice:~# tracepath 192.168.87.4
1: 192.168.87.1 (192.168.87.1) 0.169ms pmtu 1442
1: 192.168.87.4 (192.168.87.4) 6.677ms reached
1: 192.168.87.4 (192.168.87.4) 0.338ms reached
Resume: pmtu 1442 hops 1 back 64
root@alice:~#

arping also works fine:

novel@bob:~ %> sudo arping -I vcider-net0 192.168.87.1
ARPING 192.168.87.1 from 192.168.87.4 vcider-net0
Unicast reply from 192.168.87.1 [EE:CB:0B:93:34:45] 0.866ms
Unicast reply from 192.168.87.1 [EE:CB:0B:93:34:45] 1.030ms
Unicast reply from 192.168.87.1 [EE:CB:0B:93:34:45] 0.901ms
^CSent 3 probes (1 broadcast(s))
Received 3 response(s)
novel@bob:~ %>

Performance

One of the most important questions is performance. First, I used iperf to measure bandwidth on the public interfaces:

novel@bob:~ %> iperf -s -B xx.yy.94.250
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address xx.yy.94.250
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34231
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.3 sec 12.3 MBytes 9.94 Mbits/sec
[ 5] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34232
[ 5] 0.0-20.9 sec 12.5 MBytes 5.02 Mbits/sec
[SUM] 0.0-20.9 sec 24.8 MBytes 9.93 Mbits/sec
[ 6] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34233
[ 6] 0.0-10.6 sec 12.5 MBytes 9.92 Mbits/sec
[ 4] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34234
[ 4] 0.0-10.6 sec 12.5 MBytes 9.94 Mbits/sec
[ 5] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34235
[ 5] 0.0-10.5 sec 12.4 MBytes 9.94 Mbits/sec
[ 6] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34236
[ 6] 0.0-10.6 sec 12.6 MBytes 9.94 Mbits/sec
[ 4] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34237
[ 4] 0.0-10.7 sec 12.6 MBytes 9.94 Mbits/sec
[ 5] local xx.yy.94.250 port 5001 connected with xx.yy.84.110 port 34238
[ 5] 0.0-10.6 sec 12.6 MBytes 9.93 Mbits/sec

So it gives average bandwidth ~9.3Mbit/sec.

And here's the same test via vCider network:

novel@bob:~ %> iperf -s -B 192.168.87.4
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 192.168.87.4
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60977
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.5 sec 11.4 MBytes 9.10 Mbits/sec
[ 5] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60978
[ 5] 0.0-10.5 sec 11.4 MBytes 9.05 Mbits/sec
[ 6] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60979
[ 6] 0.0-10.6 sec 11.4 MBytes 9.03 Mbits/sec
[ 4] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60980
[ 4] 0.0-10.4 sec 11.2 MBytes 9.03 Mbits/sec
[ 5] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60981
[ 5] 0.0-10.5 sec 11.4 MBytes 9.06 Mbits/sec
[ 6] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60982
[ 6] 0.0-10.4 sec 11.3 MBytes 9.05 Mbits/sec
[ 4] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60983
[ 4] 0.0-20.8 sec 11.2 MBytes 4.51 Mbits/sec
[SUM] 0.0-20.8 sec 22.4 MBytes 9.05 Mbits/sec
[ 5] local 192.168.87.4 port 5001 connected with 192.168.87.1 port 60984
[ 5] 0.0-10.5 sec 11.3 MBytes 9.03 Mbits/sec

It gives an average bandwidth of 8.5Mbit/sec, and it's about 91% of the original bandwidth, which is not bad I believe.

For the sake of experimenting, I tried to emulate TAP networking using openvpn. I chose the quickest configuration possible and just ran openvpn on the server this way:

# openvpn --dev tap0

and on the client:

# openvpn --remote xx.yy.94.250 --dev tap0

As you might guess, openvpn runs in user space and it tunnels traffic over the public
interfaces on the boxes I use for tests.

And I conducted another iperf test:

novel@bob:~ %> iperf -s -B 192.168.37.4
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 192.168.37.4
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53923
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.5 sec 11.2 MBytes 8.97 Mbits/sec
[ 5] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53924
[ 5] 0.0-10.5 sec 11.1 MBytes 8.88 Mbits/sec
[ 6] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53925
[ 4] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53926
[ 6] 0.0-10.4 sec 11.1 MBytes 8.90 Mbits/sec
[ 4] 0.0-20.6 sec 10.8 MBytes 4.38 Mbits/sec
[SUM] 0.0-20.6 sec 21.8 MBytes 8.90 Mbits/sec
[ 5] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53927
[ 5] 0.0-10.4 sec 11.0 MBytes 8.87 Mbits/sec
[ 6] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53928
[ 6] 0.0-10.3 sec 10.9 MBytes 8.90 Mbits/sec
[ 4] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53929
[ 4] 0.0-10.5 sec 11.1 MBytes 8.88 Mbits/sec
[ 5] local 192.168.37.4 port 5001 connected with 192.168.37.1 port 53930
[ 5] 0.0-10.3 sec 10.9 MBytes 8.88 Mbits/sec

It gives an average bandwidth of 8.3Mbit/sec, and it's 89% of the original bandwidth. It's just a little slower than vCider Virtual Switch which is very good for openvpn, but I have to note it's not quite a fair comparison:

I don't use encryption in my openvpn setup
Real-world openvpn configuration will be much more complex
I believe openvpn will scale significantly worse with the growth of the number of machines in the network, as openvpn works in client/server mode while vCider works in p2p mode and uses central service to grab metadata such as routing information etc.

Also, it seems to me that the vCider team's comparison to openvpm is helpful, as they have a note on it in the FAQ -- be sure to check it out.

Support

It's a pleasure to note that the vСider team is very responsive. As I started testing the product at quite an early stage, I spotted some issues, and even they were not critical. It's a great pleasure to see they are all fixed in the next version.

Conclusion

vCider Virtual Switch is a product with expected behavior, good performance, complete documentation, and it's easy to use. The vCider team provides good support as well.

It seems that for relatively small setups within a single trusted environment, e.g. about 5-8 VMs within a single cloud provider, where traffic encryption and performance are not that critical, one could go with a openvpn setup. However, when either security or performance becomes important or the size of the setup increases, vCider Virtual Switch would be a good choice.

I am looking forward to new releases and specifically I'm very curious about multicast support and exposed API which manages networks.

Clustered LVM on DRBD resource in Fedora Linux

2011-06-09T01:10:00.000-07:00

As Florian Haas has pointed out in my previous post's comment, our shared storage configuration requires special precautions to avoid corruption of data when two hosts connected via DRBD try to manage LVM volumes simultaneously. Generally, these precautions concern locking LVM metadata operations while running DRBD in 'dual-primary' mode.

Let's examine it in detail. The LVM locking mechanism is configured in the [global] section of /etc/lvm/lvm.conf. The 'locking_type' parameter is the most important here. It defines which locking LVM is used while changing metadata. It can be equal to:

'0': disables locking completely - it's dangerous to use;
'1': default, local file-based locking. It knows nothing about the cluster and possible conflicting metadata changes;
'2': uses an external shared library and is defined by the 'locking_library' parameter;
'3': uses built-in LVM clustered locking;
'4': read-only locking which forbids any changes of metadata.

The simplest way is to use local locking on one of the drbd peers and to disable metadata operations on another one. This has a serious drawback though: we won't have our Volume Groups and Logical Volumes activated automatically upon creation on the other, 'passive' peer. The thing is that it's not good for the production environment and cannot be automated easily.

But there is another, more sophisticated way. We can use the Linux-HA (Heartbeat) coupled with the LVM Resource Agent. It automates activation of the newly created LVM resources on the shared storage, but still provides no locking mechanism suitable for a 'dual-primary' DRBD operation.

It should be noted that full support of clustered locking for the LVM can be achieved by the lvm2-cluster Fedora RPM package stored in the repository. It contains the clvmd service which runs on all hosts in the cluster and controls LVM locking on shared storage. In this case, we have only 2 drbd-peers in the cluster.

clvmd requires a cluster engine in order to function properly. It's provided by the cman service, installed as a dependency of the lvm2-cluster (other dependencies may vary from installation to installation):

The only thing we need the cluster for is the use of clvmd; the configuration of cluster itself is pretty basic. Since we don't need advanced features like automated fencing yet, we specify manual handling. As we have only 2 nodes in the cluster, we can tell cman about it. Configuration for cman resides in the /etc/cluster/cluster.conf file:

clusternode name should be a fully qualified domain name and should be resolved by DNS or be present in /etc/hosts. Number of votes is used to determine quorum of the cluster. In this case, we have two nodes, one vote per node, and expect one vote to make the cluster run (to have a quorum), as configured by cman expected attribute.

The second thing we need to configure is the cluster engine (corosync). Its configuration goes to /etc/corosync/corosync.conf:

The bindinetaddr parameter must contain a network address. We configure corosync to work on eth1 interfaces, connecting our nodes back-to-back on 1Gbps network. Also, we should configure iptables to accept multicast traffic on both hosts.

It's noteworthy that these configurations should be identical on both cluster nodes.

After the cluster has been prepared, we can change the LVM locking type in /etc/lvm/lvm.conf on both drbd-connected nodes:

Start cman and clvmd services on drbd-peers and get our cluster ready for the action:

Now, as we already have a Volume Group on the shared storage, we can easily make it cluster-aware:

Now we see the 'c' flag in VG Attributes:

As a result, Logical Volumes created in the vg_shared volume group will be active on both nodes, and clustered locking is enabled for operations with volumes in this group. LVM commands can be issued on both hosts and clvmd takes care of possible concurrent metadata changes.

OpenStack Nova: basic disaster recovery

2011-06-06T03:09:00.000-07:00

Today, I want to take a look at some possible issues that may be encountered while using OpenStack. The purpose of this topic is to share our experience dealing with the hardware or software failures which definitely would be faced by anyone who attempts to run OpenStack in production.

Software issue

Let's look at the simplest, but possibly the most frequent issue. For example, if we need to upgrade the kernel or software that will require a host reboot on one of the compute nodes, the best decision in this case is to migrate all virtual machines running on this server to other compute nodes. Unfortunately, sometimes it may be impossible due to several reasons, such as lack of shared storage to perform migration or cpu/memory resources to allocate all VMs. The only option is to shut down virtual machines for the maintenance period. But how should they be started correctly after being rebooted? Of course, you may set the special flag in nova.conf and instances will start automatically on the host system boot:

However, you may want to disable it (in fact, setting this flag is a bad idea if you use nova-volume service).

There are many ways to start virtual machines. Probably the simplest one is to run:

It will recreate and start the libvirt domain using instance XML. This method works good if you don't have remote attached volume; otherwise, nova boot will fail with an error. In this case, you'll need to start the domain manually using the virsh tool, connect the iscsi device, create an XML file and attach it to the instance, which is a nightmare if you have lots of instances with volumes.

Hardware issue

Imagine another situation. Assume our server with a compute node experiences a hardware issue that we can't eliminate in a short time. The bad thing is that it often happens unpredictably, without the ability to transfer virtual machines to a safe place. Yet, if you have shared storage, you won't lose instances data; however, the way to recover may be pretty vague. Going into technical details, the procedure can be described by following steps:

update host information in DB for recovered instance

spawn instance on compute node

search for any attached volumes in database

look for volume device path, connect to it by iscsi or some other driver if necessary

attach it to the guest system

Solution

For this and previous situations we developed python script that would run a virtual machine on the host where this script is executed. You can find it on our git repository: openstack-utils. All you need is to copy the script on the compute node where you want to recover the virtual machine and execute:

You can look for instance_id using the nova list command. The only limitation is that the virtual machine should be available on the host system.

Of course, in everyday OpenStack usage, you will be faced with lots of troubles that couldn't be solved by this script. For example, you may have storage configuration that provides the mirroring of data between two compute nodes and you need to recover the virtual machine on the third node that doesn't contain it on local hard drives. The more complex issues require more sophisticated solutions and we are working to cover most of them.

OpenStack Nova and Dashboard authorization using existing LDAP

2011-05-27T02:04:00.000-07:00

Our current integration task involves using goSA as the central management utility. goSA internally uses the LDAP repository for all of its data. So we had to find a solution to make both OpenStack Nova and Dashboard authenticate and authorize users using goSA's LDAP structures.

LDAP in Nova

Nova stores its users, projects and roles (global and per-project) in LDAP. Necessary schema files are in /nova/auth dir in the Nova source distribution. The following describes how Nova stores each of these object types.

Users are stored as objects with a novaUser class. They have mandatory accessKey, secretKey and isNovaAdmin (self-explanatory) attributes along with customizable attributes set by flags ldap_user_id_attribute (uid by default) and ldap_user_name_attribute (cn). To use the latter ones, it assigns person, organizationalPerson and inetOrgPerson to all newly created users. All users are stored and searched for in the LDAP subtree defined by ldap_user_subtree and ldap_user_unit.

If you want to manage user creation and deletion from some other place (such as goSA in our case), you can set the ldap_user_modify_only flag to True.

Projects are objects with the widely used groupOfNames class in the subtree defined by the ldap_project_subtree flag. Nova uses the cn attribute for the project name, description for description, member for the list of members' DNs, owner for the project manager's DN. All of these attributes are common for user (and any object) groups management, so it's easy to integrate Nova projects with an existing user groups management system (e.g. goSA).

Roles are also stored as groupOfNames, with similar cn, description and member attributes. Nova has hard-coded roles: cloudadmin, itsec, sysadmin, netadmin, developer. Global roles are stored in a subtree defined by role_project_subtree, cn's are defined by the ldap_cloudadmin, ldap_itsec, ldap_sysadmin, ldap_netadmin and ldap_developer flags respectively. Per-project roles are stored right under the project's DN with cn set to the role's name.

LDAP in Dashboard

To make Dashboard authorize users in LDAP, I use the django-ldap-auth module.
First, you need to install it using your preferred package manager (easy_install django-auth-ldap is sufficient). Second, you need to add it to Dashboard's local_settings.py in AUTHENTICATION_BACKENDS and set up AUTH_LDAP_SERVER_URI to your LDAP URI and AUTH_LDAP_USER_DN_TEMPLATE to Python's template of users' DN; in our case, it should be "ldap_user_id_attribute=%(user)s,ldap_user_subtree".

Note that in local_settings.py you override default settings, so if you want to just add a backend to AUTHENTICATION_BACKENDS, you should use +=. Also if you want to totally disable ModelBackend like we did, you can use = as well.

Also note that to make Dashboard work, you'll have to create an account in Nova with admin privileges and a project with the same name as the account. You can either set all parameters in LDAP by hand or add it using nova-manage user admin using one of usernames from LDAP.

Configuration examples

Let's say goSA is managing the organization exampleorg in the domain example.com on LDAP at ldap://ldap.example.com. To make use of its users and groups for Nova's user, projects and roles, we wrote configs like this:

By the way, to make goSA the central user management utility, we created a special plugin that manages Nova users. The plugin can be found here. It looks like this:

Shared storage for OpenStack based on DRBD

2011-05-19T05:22:00.001-07:00

Storage is a tricky part of the cloud environment. We want it to be fast, to be network-accessible and to be as reliable as possible. One way is to go to the shop and buy yourself a SAN solution from a prominent vendor for solid money. Another way is to take commodity hardware and use open source magic to turn it into distributed network storage. Guess what we did?

We have several primary goals ahead. First, our storage has to be reliable. We want to survive both minor and major hardware crashes - from HDD failure to host power loss. Second, it must be flexible enough to slice it fast and easily and resize slices as we like. Third, we will manage and mount our storage from cloud nodes over the network. And, last but not the least, we want decent performance from it.

For now, we have decided on the DRBD driver for our storage. DRBD® refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network-based RAID-1. It has lots of features, has been tested and is reasonably stable.

DRBD has been supported by the Linux kernel since version 2.6.33. It is implemented as a kernel module and included in the mainline. We can install the DRBD driver and command line interface tools using a standard package distribution mechanism; in our case it is Fedora 14:

The DRBD configuration file is /etc/drbd.conf, but usually it contains only 'include' statements. The configuration itself resides in global_common.conf and *.res files inside /etc/drbd.d/. An important parameter in global_common.conf is 'protocol'. It defines the sync level of the replication:

A (async). Local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has been placed in the local TCP send buffer. Data loss is possible in case of fail-over.

B (semi-sync or memory-sync). Local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has reached the peer node. Data loss is unlikely unless the primary node is irrevocably destroyed.

C (sync). Local write operations on the primary node are considered completed only after both the local and the remote disk write have been confirmed. As a result, loss of a single node is guaranteed not to lead to any data loss. This is the default replication mode.

Other sections of the common configuration are usually left blank and can be redefined in per-resource configuration files. To create a usable resource, we must create a configuration file for our resource in /etc/drbd.d/drbd0.res. Basic parameters for the resource are:

Name of the resource. Defined with 'resource' parameter, open main configuration section.

'on' directive opens the host configuration section. Only 2 'on' host sections are allowed per resource. Common parameters for both hosts can be defined once in the main resource configuration section.

'address' directive is unique to each host and must contain the IP-address and port number to which the DRBD driver listens.

'device' directive defines the path to the device created on the host for the DRBD resource.

'disk' is the path to the back-end device for the resource. This can be a hard drive partition (i.e. /dev/sda1), soft- or hardware RAID device, LVM Logical Volume or any other block device, configured by the Linux device-mapper infrastructure.

'meta-disk' defines how DRBD stores meta-data. It can be 'internal' when meta-data resides on the same back-end device as user data, or 'external' on a separate device.

Configuration Walkthrough

We are creating a relatively simple configuration: one DRBD resource shared between two nodes. On each node, the back-end for the resource is the software RAID-0 (stripe) device /dev/md3 made of two disks. The hosts are connected back-to-back by GigabitEthernet interfaces with private addresses.

As we need write access to the resource on both nodes, we must make it 'primary' on both nodes. A DRBD device in the primary role can be used unrestrictedly for read and write operations. This mode is called 'dual-primary' mode. Dual-primary mode requires additional configuration. In the 'startup' section directive, 'become-primary-on' is set to 'both'. In the 'net' section, the following is recommended:

The 'allow-two-primaries' directive allows both ends to send data.
Next, three parameters define I/O errors handling.
The 'sndbuf-size' is set to 0 to allow dynamic adjustment of the TCP buffer size.

Resource configuration with all of these considerations applied will be as follows:

Enabling Resource For The First Time

To create the device /dev/drbd0 for later use, we use the drbdadm command:

After the front-end device is created, we bring the resource up:

This command set must be executed on both nodes. We may collapse the steps drbdadm attach, drbdadm syncer, and drbdadm connect into one, by using the shorthand command drbdadm up.
Now we can observe the /proc/drbd virtual status file and get the status of our resource:

We must now synchronize resources on both nodes. If we want to replicate data that are already on one of the drives, it's important to run the next command on the host which contains data. Otherwise, this can be issued on any of two hosts.

This command puts the node host1 in 'primary' mode and makes it the synchronization source. This is reflected in the status file /proc/drbd:

We can adjust the syncer rate to make initial and background synchronization faster. To speed up the initial sync drbdsetup command used:

This allows us to consume almost all bandwidth of Gigabit Ethernet. The background syncer rate is configured in the corresponding config file section:

The exact rate depends on available bandwidth and must be about 0.3 of the slowest I/O subsystem (network or disk). DRBD seems to make it slower if it interferes with data flow.

LVM Over DRBD Configuration

Configuration of LVM over DRBD requires changes to /etc/lvm/lvm.conf. First, physical volume is created:

This command writes LVM Physical Volume data on the drbd0 device and also on the underlying md3 device. This can pose a problem as LVM default behavior is to scan all block devices for the LVM PV signatures. This means two devices with the same UUID will be detected and an error issued. This can be avoided by excluding /mnt/md3 from scanning in the /etc/lvm/lvm.conf file by using the 'filter' parameter:

The vgscan command must be executed after the file is changed. It forces LVM to discard its configuration cache and re-scan the devices for PV signatures.
Different 'filter' configurations can be used, but it must ensure that: 1. DRBD devices used as PVs are accepted (included); 2. Corresponding lower-level devices are rejected (excluded).

It is also nessesary to disable the LVM write cache:

These steps must be repeated on the peer node. Now we can create a Volume Group using the configured PV /dev/drbd0 and Logical Volume in this VG. Execute these commands on one of nodes:

To make use of this VG and LV on the peer node, we must make it active on it:

When the new PV is configured, it is possible to proceed to adding it to the Volume Group or creating a new one from it. This VG can be used to create Logical Volumes as usual.

Conclusion
We are going to install Openstack on nodes with shared storage as a private cloud controller. The architecture of our system presumes that storage volumes will reside on the same nodes as nova-compute. This makes it very important to have some level of disaster survival on the cloud nodes.

With DRBD we can survive any I/O errors on one of nodes. DRBD internal error handling can be configured to mask any errors and go to diskless mode. In this mode, all I/O operations are transparently redirected from the failed node to the replicant. This gives us time to restore a faulty disk system.

If we have a major system crash, we still have all of the data on the second node. We can use them to restore or replace the failed system. Network failure can put us into a 'split brain' situation, when data differs between hosts. This is dangerous, but DRBD also has rather powerful mechanisms to deal with these kinds of problems.

OpenStack Deployment on Fedora using Kickstart

2011-05-18T08:33:00.000-07:00

Overview

In this article, we discuss our approach to performing an Openstack installation on Fedora using our RPM repository and Kickstart. When we first started working with OpenStack, we found that the most popular platform for deploying OpenStack was Ubuntu, which seemed like a viable option for us, as there are packages for it available, as well as plenty of documentation. However, because our internal infrastructure is running on Fedora, instead of migrating the full infrastructure to Ubuntu, we decided to make OpenStack Fedora-friendly. The challenge in using Fedora, however, is that there aren't any packages, nor is there much documentation available. Details of how we worked around these limitations are discussed below.

OpenStack RPM Repository

Of course, installing everything from sources and bypassing the system's package manager is always an option, but this approach has some limitations:

OpenStack has a lot of dependencies, so it's hard to track them all
Installations that bypass the system's package manager take quite some time (compared to executing a single Yum installation)
When some packages are installed from repositories, and some are installed from sources, managing upgrades can become quite tricky

Because of these limitations, we decided to create RPMs for Fedora. In order to avoid reinventing the wheel, we've based these RPMs on RHEL6 OpenStack Packages, as RHEL6 and Fedora are fairly similar. There are two sets of packages available for various OpenStack versions:

Cactus - click here for the latest official release
Hourly - click here for hourly builds from trunk

There are two key metapackages:

node-full: installing a complete cloud controller infrastructure, including RabbitMQ, dnsmasq, etc.
node-compute: installing only node-compute services

To use the repository, just install the RPM:

In addition to installing everything with a single "yum install" command, we also need to perform the configuration. For a bare metal installation, we've created a Kickstart script. Kickstart by itself is a set of answers for the automated installation of Fedora distributive. We use it for automated hosts provisioning with PXE. The post-installation part of the Kickstart script was extended to include the OpenStack installation and configuration procedures.

Cloud Controller

To begin with, you can find the post-installation part of the Kickstart file for deploying a cloud controller below.
There are basic settings you will need to change. In our case, we are using a MySQL database.

Your server must be accessible by hostname, because RabbitMQ uses "node@host" identification. Also, because OpenStack uses hostnames to register services, if you want to change the hostname, you must stop all nova services and RabbitMQ, and then start it again after making the change. So make sure you set a resolvable hostname.

Add required repos and install the cloud controller.

qemu 0.14+ is needed to support creating custom images.
(UPD: Fedora 15 release already has qemu 0.14.0 in repository)

If you're running nova under a non-privileged user ("nova" in this case), libvirt configs should be changed to provide access to the libvirtd unix socket for nova services. Access over TCP is required for live migration, so all of our nodes should have read/write access to the TCP socket.

Now we can apply our db credentials to the nova config and generate the root certificate.

And finally, we add services to "autostart", prepare the database, and run the migration. Don't forget the setup root password for the MySQL server.

Compute Node

Compute Node script is much easier:

The config section differs very little; there is a cloud controller IP variable, which points to full nova infrastructure and other support services, such as MySQL and rabbit.

That code is very similar to cloud controller, except that it installs the openstack-nova-node-compute package, instead of node-full.

It is required to change the Cloud Controller IP address (CC_IP variable) for Compute Node installation.

IMPORTANT NOTE: All of your compute nodes should have synchronized time with the cloud controller for heartbeat control.

Make your bet on open source infrastructure computing

2011-05-16T15:42:00.000-07:00

Today we are launching our company blog, focused on open source infrastructure computing. We plan to cover various emerging technologies and market paradigms related to this segment of IT. As you might imagine, we did not choose this topic by accident. Aside from being the focus for our blog, it is also the focus of Mirantis as a company. Employing Silicon Valley industry veterans backed by 150 open source hackers and programming champions from Russia we have built this company because we believe in a few basic principles. I felt there is no better way to open our blog, than to share these principles with the world. So here we go:

1. Cloud Drives Adoption of Open Source

Until recently the biggest selling point of commercial enterprise software was its reliability and scalability when it comes to mission-critical tasks. Open source was considered OK by enterprises for tactical purposes, but a no-no for mission-critical, enterprise wide stuff. Now after Amazon, Rackspace, salesforce.com etc. have built out their systems on top of what’s now largely available in open source the argument of OSS being unreliable no longer holds water.

Moreover, today, cloud essentially refers to the new paradigm for delivery of IT services… i.e. it is an economic model that revolves around “pay for what you get, when you get it.” Surprisingly, it took enterprises a very long time to accept this approach, but last year was pivotal in showing that it is tracking and is the way of the future. Open source historically, has been monetized, leveraging a model that is much closer to “cloud” than that of commercial software. I.e. in case of commercial software you buy the license and pay for implementation upfront. If you are lucky to implement, you continue to pay subscription which is sold in various forms – support, service assurance etc. With open source – you always implement first, if it works – you may (or may not) buy commercial support, which is also frequently sold as a subscription service. Therefore, as enterprises wrap their mindset around cloud, they shy further away from the traditional commercial software model and closer to the open source / services focused model.

2. OSS is The Future of Enterprise Infrastructure Computing

I expect that enterprise adoption of open source will be particularly concentrated in the infrastructure computing space. I.e. open source databases (NoSQL, MySQL instead of Oracle, DB2 etc.), application servers (SrpingDM, JBoss vs. WebSphere, WebLogic), messaging engines (RabbitMQ vs. Tibco), infrastructure monitoring and security tools etc. Adoption of OSS initiatives higher up the stack (Alfresco, Compiere ERP, Pentaho etc.) in my opinion will lag behind infrastructure projects. One of the reasons here being greater end user dependence on tools that are higher up the stack. If you have 100 employees that are used to getting their BI reports in Cognos, it is hard to get them to switch to Pentaho and get used to the new user interface and report formats. However, if your Cognos BI runs on Oracle, switching it to MySQL will likely only affect a few IT folks, while 100 users will not notice the difference.

More importantly, however, the lower down the stack you are, the more “techie” the consumer of your product is. The more techie your consumer, the more likely he is to a) prefer customizing the product to the process and not the other way around; b) ultimately contribute to the open source product. Lower level OSS products tend to be more popular and more in demand overall. The extreme example would be to look at operating system vs. end user apps. Linux powers more than half of enterprise servers, but how many people use open source text editing software?

3. Public PaaS is not for Everyone

An alternative to dealing with infrastructure computing is to not deal with it at all and use a platform like Google App Engine or Force.com to build your apps. Why deal with lower end of the stack at all if the guys that know how to do it best already today allow you to use their platform? I believe that PaaS will become the dominant answer in the SMB market, however, organizations that fall in the category of “technology creators” such as cloud service vendors themselves, financial services, large internet portals etc. will always want to keep control over their entire stack to be able to innovate ahead of the curve and remain vendor independent. Therefore, technology driven companies (those that differentiate with technology) will be the primary market for proprietary OSS based infrastructure computing.

4. Infrastructure Computing is Nobody’s Core Competency

Although infrastructure computing is a necessary component in every organization and most technology driven companies want to have full control over their entire stack, there are no technology companies out there that differentiate themselves based on the awesomeness of their infrastructure stack. Yes, everybody knows that Google’s application infrastructure is great and so is that of salesforce.com, but in the end, the customers don’t care if it takes 2K servers to power salesforce.com or 100K servers, as long as the features are there. In that context, it almost always makes sense to outsource infrastructure computing functions to some third party so as to enable the company to focus on those aspects of its technology that differentiate it from the competition.