Thursday, May 17, 2012

Our blog has moved!

To help simplify search and navigation in the context of our main website,, we've moved our  blog to You'll find a list of the posts we've moved below for your convenient reference. See you at

Friday, May 4, 2012

How to Monetize the OpenStack Wave

After OpenStack was announced at OSCON in the summer of 2010, the degree of momentum behind this new open source platform has been nothing short of spectacular. Startups and enterprises alike have placed their strategic bets to monetize the OpenStack wave in various ways. As an ecosystem insider and one of the founding sponsors of the OpenStack Foundation, I wanted to offer my views on how various organizations are looking to skin this cat.

I’d like to focus on three of the many efforts currently underway. These three, in particular,  happen to be  the most vocal about their position and  represent three distinct strategy camps. They are Nebula with its OpenStack appliance, Piston with its PentOS cloud operating system, and Dell’s Crowbar, an OpenStack installer.

While all these approaches appear radically different on the surface, all three ultimately offer the same exact final product to the customer: a cloud-enabled rack of servers.  Moreover, in all three cases, the customer is required to purchase a particular type of hardware for the solution to work. The Nebula appliance in the case of Nebula; an Arista Switch with Silicon Mechanics servers in the case of Piston, and Dell PowerEdge C servers,  in the case of Crowbar.    

Let’s take a closer look at these three approaches and the key differences between them. 

Nebula, founded by NASA employees that were originally behind the OpenStack compute (aka Nova) project, has opted to deliver an integrated OpenStack appliance. Advertised as an open solution (…OpenStack, Open Flow, Open Compute – you name it), in actuality, it is a proprietary network switch pre-loaded with software that will automatically deploy the OpenStack-like environment onto a set of servers that plug into it. While maybe not the most open option, Nebula’s offering is probably the most straightforward. Grab the appliance, plug in some Open Compute compliant servers and storage and, voilĂ ! You have a cloud (or so they claim).
Piston, also founded by NASA alums, went the software route. Instead of building a physical network switch, the company chose to deliver PentOS, the cloud operating system based on OpenStack that is hardware agnostic … kind of. At this point, there is only one certified reference architecture that specifically requires Arista switches and servers from Silicon Mechanics. Longer term, there is a vision to support most commodity hardware configurations.

Finally, there is Dell and Crowbar. Dell’s approach to riding the OpenStack wave is, perhaps, the most creative. Crowbar is neither a hardware appliance nor an enterprise version of OpenStack. It is a configuration management tool built around OpsCode’s Chef, designed specifically to deploy OpenStack on Dell servers. Crowbar effectively serves as the glue between the hardware and any distribution of OpenStack (and not only OpenStack).   To generalize these approaches, I would classify Nebula as belonging to the “hardware camp,” Piston to the “OpenStack distro camp,” and Dell to the “tools camp.” 

Aside from the three players I explicitly described above, there are other members in each of these three camps. Nebula’s hardware camp is shared by MorphLabs with their converged OpenStack infrastructure offering. The OpenStack distro camp, which Piston belongs to, has folks like CloudScaling and StackOps under its roof. Finally, the Dell Crowbar camp is shared by Canonical, with its Puppet- based Juju installer. I believe Rackspace CloudBuilders will eventually join the tools camp in some shape or form. 

The key difference between these three camps is in the degree of openness and rigidity of coupling between hardware and software.  Ultimately, it all comes down to the age old tradeoff between simplicity and flexibility. These three camps are generalizations of approaches where companies have placed their bets at different points of the simplicity vs. flexibility trade-off scale.  The hardware camp is betting on simplicity, at the cost of openness. Just like an iPad, Nebula’s appliance may be very simple to use and beautifully designed, but nobody has a clue about what’s inside except for the people that built it. The tools camp is betting on openness, but does so at the expense of ease-of-use. The OpenStack distro camp is somewhere in the middle.

I believe that the tools approach is more effective in the context of large scale deployments, such as enterprise-wide cloud rollouts, cloud enablement of service providers, or infrastructure refactoring for SaaS vendors. Traditionally, the scale of application infrastructure is inversely related to one’s propensity to use proprietary, closed products. Leveraging third-party solutions for massive infrastructure build-outs simply doesn’t yield the agility and economics required to be successful. For that exact reason, when one examines Facebook, Amazon, or Google infrastructure, virtually no third party, proprietary solutions can be found inside the stack.

The simplicity approach targeted by the OpenStack hardware camp and OpenStack distro camp is what appeals to smaller, department-level enterprise deployments, such as situations where IT is not the core competency of the organization and application infrastructure is fragmented into a series of diverse silos of smaller scale. These are the scenarios where organizations don’t care about lock-in or cost at scale; they need an out-of-the-box solution that works and a button to push for support when it doesn’t.

Now, if we looked at the three approaches from the pure business perspective, there is an interesting dynamic at play. (As a co-founder of an OpenStack systems integrator closely partnered with Dell, my opinion on the business aspect is completely biased.) Nevertheless, here is how I see it…

On the one hand, diverse enterprise IT silos is the bigger chunk of the overall application infrastructure pie, at least for now.  On the other hand, when it comes to simple, proprietary enterprise solutions aimed at these diverse IT silos, the line of vendors offering them is out the door and around the corner, with VMWare controlling the bulk of the market.

Historically, the philosophy of the OpenStack community in general was about the very notion of openness, scale, and flexibility. RackSpace, a company with the service provider mentality, has been dominating the project technical board since inception and drove the OpenStack brand forward. Their build-for-scale, build-for-flexibility approach has spilled out into the community and affected both technical and marketing decisions. Consequently, the majority of OpenStack deployments we hear about today – Internap, Korea Telecom, Mercado Libre, Yahoo etc. – are all in the service provider / Internet application vendor space. I don’t see OpenStack as something that was built for enterprise IT silos.

Monetizing OpenStack by betting on openness at the expense of simplicity, such as the companies in the tools camp have done, may not be about penetrating the biggest market, but at least it is in line with the current momentum of the OpenStack community. On the other hand, leveraging the still young and fairly immature OpenStack platform, architecturally aimed at Web-scale deployments to build proprietary offerings that target enterprise IT silos and rival VMware, Citrix, and Microsoft, is a risky business proposition.  

Thursday, April 12, 2012

The New Open Source Superpower

Today is yet another important day in the history of OpenStack. The initial list of founding organizations for the independent OpenStack foundation has been announced and we, at Mirantis, are proud to be on that list.

While there is a lot to talk about on what this means for the infrastructure cloud market, I’d like to focus on what this means as far as illustrating the sheer momentum of the OpenStack beast. The non-profit legal entity that will house OpenStack has not yet been formed, but 18 organizations have already pledged significant financial (and not only financial) support to the foundation. The current financing model calls for $500K/year commitment from a Platinum sponsor and $50-$200K/year – from a Gold sponsor. Judging by the current composition of the supporting organizations, it is clear that the new foundation will launch with the initial budget north of $5M.

So how does this measure up to the rest of the FLOSS ecosystem? Well, there is a reason why OpenStack has been repetitively tagged as the Linux of the cloud. With the $5M annual budget, the newly formed OpenStack foundation takes the second spot in the entire FLOSS world. And it is second only to… you guessed it… the Linux foundation itself. According to form 990 filed by the Linux foundation in 2010 its operating revenues were $9.6M. Yes, the Linux foundation budget is still double that the OpenStack…but…come on…Linux is close to 20% of the server market. It also happens to power the majority of all mobile devices. OpenStack = Linux was a vision… judging by these numbers, this vision may soon be realized.

Another interesting thing that these numbers portray is why OpenStack (unlike CloudStack) has opted to create its own foundation, rather than surrendering everything to the governance of the Apache Foundation. With the Apache Foundation budget under $1M, OpenStack eats it for breakfast.

Now many of you will argue that none of this matters. Apache foundation houses many great projects that are far more mature and popular than OpenStack… true. But can you tell me, how many of these are truly vendor agnostic? And I am not talking about developer tools like Ant, Maven, Beehive etc. All Apache projects fall into two categories – they are either developer tools or vendor centric enterprise products: Tomcat – VMWare, Hadoop – Cloudera, – will now be Citrix =).

In my opinion, there is a reason for it and it is somewhat tied to foundation budgets. Open source is heavily driven by marketing. The number one open source company – RedHat - spends 2-3x more on marketing relative to its revenue than any of its closed source competitors. Ultimately, it is the marketing spend on an open source project that heavily affects its vendor independence status. If the entire spend comes from a single pocket, there is a single vendor that dominates that product.

Unlike most Apache open source projects, OpenStack (while still under RackSpace) was backed by a significant marketing and PR budget. Consequently, when foundation plans were being discussed, it was the desire to continue this centralized marketing effort that precluded OpenStack from considering the Apache foundation as its home. A significant chunk of the $5M raised will be spent by the foundation to promote and protect the OpenStack brand and the projects that the foundation will house. In a sense, this implies that for anyone to derail the vendor independent status of OpenStack, one will need the marketing budget, comparable to the $5M the foundation has raised… I say this is a decent barrier to start with.

Thursday, April 5, 2012

Some Brutally Honest Thoughts on Citrix’s Defection

When I first heard the announcement about being spun off into the Apache foundation, my initial reaction was to interpret the event as a hostile move by one of the OpenStack community insiders. Citrix is one of the founding members of OpenStack, with representation on the project policy board; the company has been quite active evangelizing the community through various events and code contributions. So why, all of a sudden, a move that may appear to undermine the OpenStack momentum?

Let’s take a look at the history. When Citrix bought for more than $200 million in July, 2011, insider information suggested the company had revenue of only a several million. While high valuations were not uncommon in the cloud space, a 40x revenue multiple is quite unusual. Why did Citrix do it? The only answer that comes to mind was that it wanted to quickly gain credibility in the cloud market.

I believe that corporate politics and relationships also played a role in this deal. was backed by Redpoint Ventures, which had an existing track record of selling its portfolio companies to Citrix. But, more importantly, founder and CEO – Sheng Liang – was also the founder and CTO of Teros Networks, a Web security company that was acquired by the very same Citrix just a few years before was founded. In fact, I am pretty sure, that in some sense was Citrix’s skunk works project; acquisition by Citrix was the key part of the business plan. While there is nothing wrong with the approach and I can only complement the strategy, the early connection between Citrix and was key to its successful exit and the events that followed.

Just one year before the acquisition of, OpenStack was announced at OSCON and nobody knew what to think of it. It took the open source community by a storm and it soon became evident to all those competing for open cloud dominance, that simply ignoring the OpenStack phenomenon was not an option. “Open cloud strategy” soon became synonymous with the “OpenStack Strategy”. Citrix, a founding member of OpenStack itself, was in a bit of a tight spot. One choice was to abandon its project. Given the OpenStack momentum at the time, this could inevitably translate to the swift death of and $17 million in losses to the VCs backing it. Alternatively, Citrix could go all in, acquire the community to boast its credibility in the open source cloud space and take a stab at creating the dominant distribution of OpenStack, ultimately becoming to OpenStack what Red Hat has become to Linux. In the end, the scales tipped towards the latter option. In May, 2011 Citrix announced its distribution of OpenStack – project Olympus. Two months thereafter, the acquisition was announced.

However, when the dust settled, it became evident that Citrix’s involvement with and OpenStack (Project Olympus), instead of being complimentary as Citrix has anticipated, has been perceived as strange and surprising. CloudStack is Java based, whereas OpenStack is all Python. On the compute side, CloudStack focused on Xen, whereas the dominant hypervisor for OpenStack so far has been KVM. CloudStack was licensed under GPL, and OpenStack under Apache 2.0. Ultimately, Citrix’s acquisition was sending confusing messages to both communities and Citrix’s customer base. A few months after Citrix’s acquisition, the community had little momentum left. At the same time, the OpenStack community remained wary of Citrix due to its involvement with CloudStack. Consequently, not much has happened with Project Olympus since its announcement over a year ago until it was officially abandoned with the latest announcement.

Today, Citrix announced that will find a new home with the Apache foundation. Is it a hostile move that will undermine OpenStack? I see it more as an act of desperation. Clearly, that wasn’t the initial plan, when Citrix first acquired Consequently Citrix has failed to build the community around, miscalculated the synergies between the two communities, got trumped by OpenStack momentum, and dumped what’s left of to the Apache foundation. They have already announced CloudStack would be open source twice before, yet have received no outside contributions to date. The last commit to on GitHub by a non-Citrix employee is dated several months ago.

At this point, Citrix has a spotty history when it comes to open source. Open source is built on trust and they are hard to trust right now. Having burned bridges at their last two communities (Xen / Linux) and now OpenStack, it is going to be big challenge for them to revive CloudStack from its present semi-dead state.

Saturday, March 31, 2012

I hear the Essex Train a-coming

With Essex train in the wilds of testing, and the Essex release intended date less than 10 days away, we are pretty excited about everyone descending on San Francisco -- practically our home town -- for the Design Summit and conference.

Here at Mirantis, the company famous across OpenStack community for distributing vodka bottles at OpenStack meetups, we are gearing up in a big way for the summit and conference. If you haven't seen the agenda, here's what we've got teed up:

(1) We’ll start the frenzy with just-in-time-training: we have a few seats left at our 2-day OpenStack Boot Camp, crammed into the weekend of April 14-15, right before the summit and conference. REGISTER HERE and come to the event fully prepared to torment the presenters with insidious technical questions about OpenStack technology and its future.

(2) Our team will participate in / moderate a few exciting sessions during the conference: OpenStack and Block Storage, OpenStack and High Performance Computing, Expanding the Community. Please be sure to pay us a visit.

(3) …and just to show how happy we are to have you here, we invite the community at the conference to join Mirantis Summit Kick-Off Party. Click for a preview of what’s to come! Vodka bottles and fun times in the best traditions of all our events are guaranteed. Be sure not to miss.

Looking forward to receiving everyone at the 2012 OpenStack Design Summit and Conference.

Tuesday, March 13, 2012

Long Live Enterprise Clouds

One active arena of cloud blog battles lately was about open vs. enterprise clouds. Randy Bias of Cloudscaling is on the forefront of open cloud advocacy with a clear stance that anything that is not an open cloud - is a legacy cloud. Massimo, in his recent blog did a great job categorizing different types of clouds. He, too, gently implies that as organizations’ IT strategies mature, they will be making a shift away from “orchestrated clouds” to “design for fail clouds.” Much of the “cool cloud thinking” today is based on the notion that a) open clouds are good; b) writing applications for infrastructure (rather than tuning infrastructure for the apps) is the future of IT.

I disagree with both of these.

So let’s start with “open cloud is good, enterprise cloud is bad” stance. In my view, making this comparison is like saying a kitchen knife is better than a swiss-army knife. A kitchen knife is simple, sharp and has no moving parts. Just like an open cloud it is designed to be a simple solution to a single, concrete problem. Ever try to prepare a full meal with a swiss-army knife? Sure, when you go camping, it’s probably fine. But when your mother-in-law is coming to dinner?

The fundamental difference between enterprise and open clouds is in their approach to the problem. Open cloud mentality comes from a service provider view of the world where the cloud is not built to support the business, but rather IS THE BUSINESS. The approach is to build from the bottom up for a narrow problem, just like one would if you were a software company and the cloud was your product, aimed at capturing a chunk of some market.

In the open cloud world, apps do not dictate the infrastructure; you start with the infrastructure, and go up the stack. In other words, the information technology infrastructure dictates the way the application solves a business problem.

In the enterprise world – it’s precisely the other way around . IT exists to support the business. The applications are king and they dictate the IT underneath. Is this the best scenario when it comes to simplifying the infrastructure and containing its cost? Definitely not! Is there an alternative for the enterprise? Definitely not! Reason? There’s an irreducible domain knowledge gap between IT and business. In the case of AWS, Salesforce or Google – IT is the business. In enterprise – IT’s job is to salute the business and support it.

Let’s take a concrete example (real life story by the way). We are now enjoying the dawning of the age of big data. Say some entrepreneur decides to take advantage of Hadoop and Lucene and build a new engine for parsing and aggregating bioinformatics data, and can extract results that were never before possible. He then sells his marvelous, vertically focused innovation to Pharma companies. If I’m at Pfizer and I don’t buy it, but my rivals at Roche buy – I get left behind. But say my IT does not do Hadoop and Lucene and I can’t take the solution to run in a public cloud because of regulatory compliance. Now what do I do?

If you guessed that I call my CIO and tell him to stand up the environment that will support this, you’re right. IT has to follow the lead of the business, or the whole business fails. This happens over and over again. Over time, IT has to support an extremely diverse environment. Conceivably, the gap may shrink over time as IT becomes an ever increasingly dominant business process in any vertical, but don’t plan on it happening this month. Or even next month.

Now, there is a common view that it doesn’t have to be that way that stems from an elegant but very one-dimensional comparison between IT infrastructure and electricity. I.e. application infrastructure is a commodity, just like electricity. All apps should be built to run on top of this common, standardized infrastructure, and just like we all have the same shape of electrical outlets (except for, well, the Europeans) this is where we’ll all be soon.

It sounds great, but I’m sad to say that I have to call bullshit. Electricity and application infrastructure is not the same. Unlike with electricity, there is massive innovation at the bottom part of the application stack. We didn’t even use virtualization until recently. Yesterday it was all about disk, today it is all about SSD.

We don’t know what new paradigms will emerge in the coming years. This innovation shakes the entire stack. Going back to my example, Hadoop was not widely used just not too long ago. Had it not existed, the new app would not have been possible and IT would not have had to buy new infrastructure and deploy unknown middleware on it. But because it does, IT has to adjust. And tomorrow will be a new paradigm and IT will have to adjust again and again.

Commoditization and standardization can only happen in stagnant industries like electricity generation and distribution or growing potatoes, where the world has pretty much stopped. Until that kind of stable stagnation becomes a common theme in the application infrastructure space, there will always be expensive enterprise clouds and open, inexpensive, commodity clouds. The enterprise will be constantly configuring its swiss army knife, aimed at minimizing the pain of dealing with diversity in the stack.

Tuesday, February 14, 2012

Under the hood of Swift. The Ring

This is the first post in series that summarizes our analysis of Swift architecture. We've tried to highlight some points that are not clear enough in the official documentation. Our primary base was an in-depth look into the source code. The Ring is the vital part of Swift architecture. This half database, half configuration file keeps track of where all data resides in the cluster. For each possible path to any stored entity in the cluster, the Ring points to the particular device on the particular physical node.

There are three types of entities that Swift recognizes: accounts, containers and objects. Each type has the ring of its own, but all three rings are put up the same way. Swift services use the same source code to create and query all three rings. Two Swift classes are responsible for this tasks: RingBuilder and Ring respectively.

Ring data structure

Every Ring of three in Swift is the structure that consists of 3 elements:
  • a list of devices in the cluster, also known as devs in the Ring class;
  • a list of lists of devices ids indicating partition to data assignments, stored in variable named _replica2part2dev_id;
  • an integer number of bits to shift an MD5-hashed path to the account/container/object to calculate the partition index for the hash (partition shift value, part_shift).
List of devices
A list of devices includes all storage devices (disks) known to the ring. Each element of this list is a dictionary of the following structure:
idintegerIndex of the devices list
zoneintegerZone the device resides in
weightfloatThe relative weight of the device to the other devices in the ring
ipstringIP address of server containing the device
portintegerTCP port the server uses to serve requests for the device
devicestringDisk name of the device in the host system, e.g. sda1. It is used to identify disk mount point under /srv/node on the host system
metastringGeneral-use field for storing arbitrary information about the device. Not used by servers directly
Some device management can be performed using values in the list. First, for the removed devices, the 'id' value is set to 'None'. Device IDs are generally not reused. Second, setting 'weight' to 0.0 disables the device temporarily, as no partitions will be assigned to that device.
Partitions assignment list
This data structure is a list of N elements, where N is the replica count for the cluster. The default replica count is 3. Each element of partitions assignment list is an array('H'), or Python compact efficient array of short unsigned integer values. These values are actually index into a list of devices (see previous section). So, each array('H') in the partitions assignment list represents mapping partitions to devices ID.

The ring takes a configurable number of bits from a path's MD5 hash and converts it to the long integer number. This number is used as an index into the array('H'). This index points to the array element that designates an ID of the device to which the partition is mapped. Number of bits kept from the hash is known as the partition power, and 2 to the partition power indicates the partition count.

For a given partition number, each replica's device will not be in the same zone as any other replica's device. Zones can be used to group devices based on physical locations, power separations, network separations, or any other attribute that could make multiple replicas unavailable at the same time.
Partition Shift Value
This is the number of bits taken from MD5 hash of '/account/[container/[object]]' path to calculate partition index for the path. Partition index is calculated by translating binary portion of hash into integer number.

Ring operation

The structure described above is stored as pickled (see Python pickle) and gzipped (see Python gzip.GzipFile) file. There are three files, one per ring, and usually their names are:
These files must exist in /etc/swift directory on every Swift cluster node, both Proxy and Storage, as services on all these nodes use it to locate entities in cluster. Moreover, ring files on all nodes in the cluster must have the same contents, so cluster can function properly.

There are no internal Swift mechanisms that can guarantee that the ring is consistent, i.e. gzip file is not corrupt and can be read. Swift services have no way to tell if all nodes have the same version of rings. Maintenance of ring files is administrator's responsibility. These tasks can be automated by means external to the Swift itself, of course.

The Ring allows any Swift service to identify which Storage node to query for the particular storage entity. Method Ring.get_nodes(account, container=None, obj=None) is used for identification of target Storage node for the given path (/account[/container[/object]]). It returns the tuple of partition and dictionary of nodes. The partition is used for constructing the local path to object file or account/container database. Nodes dictionary elements have the same structure as the devices in list of devices (see above).

Ring management

Swift services can not change the Ring. Ring is managed by swift-ring-builder script. When new Ring is created, first administrator should specify builder file and main parameter of the Ring: partition power (or partition shift value), number of replicas of each partition in cluster, and the time in hours before a specific partition can be moved in succession:

When the temporary builder file structure is created, administrator should add devices to the Ring. For each device, required values are zone number, IP address of the Storage node, port on which server is listening, device name (e.g. sdb1), optional device meta-data (e.g., model name, installation date or anything else) and device weight:

Device weight is used to distribute partitions between the devices. More the device weight, more partitions are going to be assigned to that device. Recommended initial approach is to use the same size devices across the cluster and set weight 100.0 to each device. For devices added later, weight should be proportional to the capacity. At this point, all devices that will initially be in the cluster, should be added to the Ring. Consistency of the builder file can be verified before creating actual Ring file:

In case of successful verification, the next step is to distribute partitions between devices and create actual Ring file. It is called 'rebalance' the Ring. This process is designed to move as few partitions as possible to minimize the data exchange between nodes, so it is important that all necessary changes to the Ring are made before rebalancing it:

The whole procedure must be repeated for all three rings: account, container and object. The resulting .ring.gz files should be pushed to all nodes in cluster. Builder files are also needed for the future changes to rings, so they should be backed up and kept in safe place. One of approaches is to put them to the Swift storage as ordinary objects.

Physical disk usage

Partition is essentially the block of data stored in the cluster. This does not mean, however, that disk usage is constant for all partitions. Distribution of objects between the partitions is based on the object path hash, not the object size or other parameters. Objects are not partitioned, which means that an object is kept as a single file in storage node file system (except very large objects, greater than 5Gb, which can be uploaded in segments - see the Swift documentation).

The partition mapped to the storage device is actually a directory in structure under /srv/node/<dev_name>. The disk space used by this directory may vary from partition to partition, depending on size of objects that have been placed to this partition by mapping hash of object path to the Ring.

In conclusion it should be said that the Swift Ring is a beautiful structure, though it lacks a degree of automation and synchronization between nodes. I'm going to write about how to solve these problems in one of the following posts.

More information

More information about Swift Ring can be found in following sources:
Official Swift documentation - base source for description of data structure
Swift Ring source code on Github - code base of Ring and RingBuilder Swift classes.
Blog of Chmouel Boudjnah - contains useful Swift hints