Wow, great week last week at the Cloud Connect event in Santa Clara. Alistair and Dave set out to create the “definitive event” for the cloud computing community, and definitely succeeded in my book — except of course for CloudCamp!

Most of my time last week was consumed by Appistry’s launch of CloudIQ Storage, so I didn’t get to check out as many sessions as I otherwise would have liked. The ones I did attend were great though, especially Randy Bias’ panel on migrating to the cloud (featuring Tom Gregory from Appistry customer Presidio Health) the ‘Ubicomp‘ panel that I was on, which focused on the broader implications of ubiquitous computing and the cloud.

Another highlight was Tuesday evening’s SF Cloud Club meeting, organized by James Watters and Shanley Kane. The event was a bit over-scheduled, with about 8 companies due to present on the evening’s theme — platform-as-a-service (PaaS) — but other than that it was a great event for both networking and learning.

The evening’s “entertainment” was provided by Shlomo Swidler, who demonstrated the creation of a physical cloud — yes, you know, the water kind — for the group. Here’s the video I shot. (Don’t worry, it steadies out after the first few seconds.)

Back in 2004, Wired News proclaimed the death of capitalization for the internet, the web and the net.

Why? The simple answer is because there is no earthly reason to capitalize any of these words. Actually, there never was.

True believers are fond of capitalizing words, whether they be marketers or political junkies or, in this case, techies. If It’s Capitalized, It Must Be Important. In German, where all nouns are capitalized, it makes sense. It makes no sense in English. So until we become Die Wired Nachrichten, we’ll just follow customary English-language usage. (Web will continue to be capitalized when part of the more official entity, World Wide Web.)

The same is certainly true for cloud computing, the subject of much unnecessary capitalization. Let’s make 2010 the year we put an end to Big Cs when talking about cloud.

At CloudCamp St. Louis, Michael Groner, my colleague at Appistry, gave a talk on the MapReduce framework. The 5-minute Lightening Talk was a condensed version of the 45-minute presentation he delivered originally at the St. Louis Lambda Lounge (a local group focused on dynamic and functional programming). You can check out his slides below.

Michael received great feedback on his presentation and generated a lot of interest in MapReduce. As a follow-up, I thought it would be nice to feature a brief Q&A with him, which I present below:

Sam: What is MapReduce?

Michael: MapReduce is a software framework to support processing of massive data sets across distributed computers. MapReduce has its origins from functional programming where the  “map” and “reduce” operations assist developers in processing data sets easily. For applications requiring the translation of  gigabytes, terabytes, or even petabytes of data into usable data structures and then collecting to end results, MapReduce is a good candidate.

Sam: Where did it come from? Is it new?

Michael: MapReduce’s popularity surge started in 2004 with the publication of “MapReduce: Simplified Data Processing on Large Clusters” by Google. In this paper Google illustrated the framework they use to construct an index of the internet through a simple programming paradigm.

Sam: Why should people care? What does MapReduce have to do with cloud computing?

Michael: This framework contains many of the key architecture principles promoted for cloud computing:

• Scale – The framework is able to grow and expand in direct proportion to the number of machine applied to the system.
• Reliable – The framework is able to handle the loss of a node and restart the work somewhere else.
• Affordable – As reliability is accounted for in the framework, commodity hardware can be used. As scale is accounted for, a user can start small and add additional hardware as necessary.
• Simple – A user simply provides the implementation of a map and reduce function in order to process large quantities of data.

Due to these principles, the MapReduce framework has become one of the most popular platform tools for building cloud applications.

Sam: Can you give us some examples of who is using it?

Michael: MapReduce applications are used in by companies like Google, Yahoo, Visa, and the intelligence community, among others, to produce results from the large data sets they produce.

Sam: How should someone go about learning more?

Michael: To learn more I suggest reading the original Google paper and looking into a MapReduce framework such as the Apache Hadoop project.

Thanks, Michael!

Are you using MapReduce? We would love to hear your experiences with it!

In yesterday’s batch of links I commented on James Urquhart’s video “Clarifying Internal Cloud versus Private Cloud.”

James Urquhart on Internal/External vs Private/Public clouds. While I agree that Private is not always equal to Internal, James’ [re]definition of increasingly accepted terminology just serves to muddy the waters by introducing the existence of unified control systems as a defining characteristic.

James called me on my comment, asking what I see the difference being.

Just as I finished typing the above words, Savvis’ Benson Schliesser jumped into the mix with his post “Cloud: Private vs Public, Internal vs External, Oh My!” adding an interesting dimension to the dialogue.

First off, my thoughts on James’ video. In the video he is making two points:

• Internal/External is an issue of “ownership”
• Private/Public is an issue of “control”

To illustrate the latter point, he provides an example of a unifying control system creating a “private cloud” from internal and external resources. This example is what I was primarily responding to, as it seems to label as “private cloud” an idea more commonly called “hybrid cloud,” primarily by virtue of the unified control system. This masks the “less private” (e.g. hybrid) aspects of the cloud and as a result (IMO) “muddies the waters.”

The notion of private vs public cloud remains central to the broader discussion around cloud computing, at least from an enterprise perspective. Right now, there a number of terms that seem to be used more or less synonymously to identify the same concept: private vs public, internal vs external, on-premise vs off-premise.

Most of the time, when I hear folks trying to subdivide this world even further, it comes as part of an attempt to sell “public” cloud offerings as “private” ones, to put it bluntly.

To extend James’ example, if I deploy a tool that allows me to “control” virtual resources from Amazon EC2 and GoGrid, either directly or via a broker, without having to utilize either offerings’ native tools, does that mean I’ve created a “private” cloud based on these “public” cloud services? Amazon would like you to think so. Most enterprises would not.

I define “private cloud” as a system exhibiting cloud characteristics, but operated for/owned by/contolled by a single organization. As mentioned in previous posts, I believe tenancy (an attribute that didn’t even figure into your list, Benson) is a key determinant of “privacy.” By tenancy I’m referring to who the cloud is “operated for.”

Ownership seems less important — if the system is physically owned by a third party but operated for my exclusive use (maybe even by me), that’s private. And control seems too hard to pin down — do I control EC2 since they give me an API? If you control a system for my exclusive use, but do so on my behalf (i.e. I control you, e.g. contractually), who is in control? (In other words, is “control” a transitive property?)

So Benson, to the point of Bryan’s quote, off-premise private cloud works fine for me. Shared private cloud, not so much. (The private cloud is shared among its users, yes, but not with other customers.) Shared clouds with better security measures and “enterprise grade” QoS are  perhaps “enterprise-grade public clouds” in my book (or maybe “virtually private” clouds).

In the trenches, of course, you’re absolutely right though. There are more than just a couple of defining attributes of cloud computing. And, in the same vein, we will measure each of the attributes with more than a single bit.

In the end, customers are smart, and the labels we put on our offerings will mean less and less as the market matures and educates itself.

Everyone seems to be comfortable with the idea that Amazon is the most popular cloud infrastructure provider. But I’m often asked who is second.

I’ll typically respond by mentioning Rackspace as a possible second, but will comment that there seems to be a huge gulf between Amazon and whoever number two actually is. Based on data recently published in the Jack of all Clouds “State of the Cloud” survey, Rackspace is clearly number two and the gap isn’t actually all that large.

The survey’s methodology doesn’t measure non-Web uses of IaaS, which I suspect would increase Amazon’s lead by quite a bit.

“We interrupt your regularly scheduled arguments about the legitimacy of private clouds to legitimize private cloud.”

Last night Amazon announced its new Virtual Private Cloud offering, which makes it easier for enterprises to connect EC2-provided resources with intra-enterprise resources.

This announcement comes right in the midst of a flare up in the ongoing discussion about the role and reality of private clouds; what I characterized as the battle between the “cloud purists and pragmatists” at the CloudWorld keynote panel the other day.

For those advocating for private clouds on behalf of enterprise users, the announcement lent an element of legitimacy to the cause. By acknowledging that the transition to public cloud is challenging and won’t happen over night, that some applications will never move to the public cloud, and that enterprise IT executives are demanding private cloud solutions, Amazon CTO Werner Vogels tossed the private cloud crowd a fat pitch in his blog post announcing the service. Chris Hoff (@beaker) jumped on this opening with his post “Calling All Private Cloud Haters: Amazon Just Peed On Your Fire Hydrant…“.

Yet, there’s more than just legitimizing happening here. With this announcement, Amazon is attempting, intentionally or not, to co-opt the notion of private clouds by adopting confusing and misleading terminology. In effect, they are peeing in the pool, not just on fire hydrants.

What is a virtual private cloud?

Common use of the term “virtual private cloud” refers to a private cloud that is somehow virtual or abstracted from the end-user — in other words, a hosted private cloud. This traditional virtual private cloud offers the full privacy of a private cloud, but pushes hardware ownership to someone else. It offers the full benefit of cloud, such as elasticity, metering, etc., and is backed by the cloud provider’s infinite access to hardware (wink wink). Virtual private cloud presumably comes at a steeper price, because the cloud provider can’t oversubscribe the servers that your VMs are running on to a bunch of other customers. But it’s still legitimately cloud computing. (To the end-user, the multitenancy of the cloud isn’t a feature, it’s a compromise.)

Here is how real “virtual private cloud” relates to other types of cloud:

(Diving deep into this quadrant is a topic for a future post.)

Amazon VPC is not virtual private cloud

Amazon VPC is being billed as a way to let “you create your own logically isolated set of Amazon EC2 instances and connect it to your existing network.” No detail is provided about any “isolation” beyond the network configuration, i.e. its private IP addressing and VPN connectivity. We can only assume that VPC instances are co-located on public resources, shared with other EC2 or VPC users. Thus Amazon VPC is not “virtual (private cloud)” but rather “(virtually private) cloud.”

Don’t get me wrong, these are great new features for EC2, they advance the state of cloud computing as a whole, and they definitely do make EC2 easier to adopt by the enterprise. But this new offering doesn’t fundamentally address many of the security, control and compliance concerns held by the enterprise. By claiming “isolation” and naming the service VPC, the offering at best contributes to industry confusion around private clouds. At worst it may be outright misleading.

Thanks in part to an aggressive pro-cloud stance taken by the Obama administration, the federal government is positioning itself to be an aggressive adopter of cloud computing and cloud technologies, as evidenced by the recent GSA RFQ. As a result of the government’s push towards the clouds, there has been a strong desire on the part of agency IT leadership and practitioners to hear from real cloud users, providers, and technology suppliers. A number of civilian and defense CIOs and CTOs have taken “study tours” to visit firms like Google and Amazon in an effort to transfer back best practices and lessons learned back to their agencies.

In response to this need, and to help facilitate the sharing of cloud success stories with government, last week Appistry hosted a “Success in the Clouds” breakfast seminar in Washington, D.C.

I provided the “warm-up act” providing an overview of a variety of cloud computing topics and trends before inviting up Mike Rains from FedEx to speak. Mike shared FedEx’s real-world experiences as an Appistry customer and early adopter of private clouds for mission-critical applications, with much of his time spent addressing direct audience questions.

Here are my slides.

One of my take-aways from the event was a reminder of just how broad the cloud discussion is: At one of the breaks I was approached by an attendee asking whether I’ve seen customers using cloud computing as a way to reduce software licensing costs. I responded yes, as we’ve often seen a shift to cloud happen in parallel with a move to open source to accomplish just this. Her response startled me at first, “You mean to products like Open Office?”

To a server-side guy like myself, pushing desktop apps into the cloud is sometimes not quite as interesting as all the datacenter, application-tier and middleware magic that it takes to make that happen, but to end-user organizations the shift away from desktop applications and to web-delivered apps can provide tremendous economic value. It was a good reminder to be conscious of the many perspectives we bring to cloud.

John Foley of InformationWeek published a nice bit of investigative reporting in his recent post Amazon Web Services Secrets Revealed. The post tries to get at growth estimates of Amazon’s Web Services business from a number of angles and provides evidence of some impressive growth rates.

One data point cited in the post is some nice research done by Guy Rosen, posted to the InfiBase blog. Using his company’s tools he’s looked at the top 500,000 Web sites and determined that while only a small number of them are hosted on EC2, Amazon (a) hosts a disproportionate number of the large sites and (b) has experienced dramatic growth.

The actual estimate Guy provides for AWS hosted sites is very small on a percentage basis (0.28%) but it must be kept in mind that the hosting business is extremely fragmented and in fact a market share as low as 3.25% is all it takes to make it to the top of the “top hosts” list, according to data from the Netcraft Web site.

I find it interesting to see the AWS business measured this way because it supports a perspective I’ve held for some time: While some believe cloud computing results in the eventual dominance of just a handful of “big computers” (e.g. data centers, the list of which always includes Google, Amazon, Microsoft and whatever company employs the person telling this story), it’s always been my opinion that there will be many cloud providers and in fact, at some point in time, every hosting provider becomes a cloud provider in some sense of the word.

Cue up the arguments about “what really defines cloud computing” if you will, but this strikes me as an inevitable market shift.

At the highest level, what separates a hosting provider from a cloud provider are technology and business models. (And the former enables the latter.) The core technology required to launch an infrastructure-oriented cloud hosting business, e.g. one that provides access to virtual machine instances via an on-demand pricing models, is readily available today and the complex integration work that’s currently required to offer a complete solution will eventually give way to packaged tools.

Consider the fact that Plesk, one of the leading providers of “hosting automation systems” (i.e. control panel software) is owned by Parallels, a leading[1] provider of desktop and server qwwavirtualization software and emerging cloud computing player, and it’s easy to see the beginnings of the convergence of the hosting and cloud worlds.

Bonus Links:

• John’s article and the InfiBase data reminded my of an earlier post by Ruv, Amazon reserves the right to host your applications, which looked at the EC2 “reserved instances” pricing model and how it made the company’s offering price competitive with VPS hosting plans.
• In researching this article, I came across Krish’s recent CloudAve piece, Handful Of Monopoly Infrastructure Players – A Shortsighted Idea, which I’m not sure how I missed. Great article that compliments my own with a more “philosophical” and macro-economic perspective.

[1] A forum post found via a quick Google search suggests Plesk/Parallels holds some 30% market share relative to competitors like cPanel and H-Sphere; not hard data, but consistent with what I’ve seen out in the market.

Responding to an informative and well-researched–albeit a bit short sighted–study by McKinsey, a post earlier this week by Rajen Sheth on the Official Google Enterprise Blog helped to clarify for the world exactly what Google means when it talks about cloud computing. The post, a bit of a rail against the idea of private clouds, builds the argument for public clouds in part on the idea that Google and a few other companies have a lock on advancements in hardware infrastructure, software infrastructure, applications and innovation.

The fact is, as much as it might like to think so, if there is one area that Google does NOT have a monopoly in it is innovation, and I for one am irked by the implication that the enterprise will hand over the "keys to the kingdom" because it’s not able to innovate. Actually, enterprise IT organizations have access to many of the same innovative technologies that Google claims give advantage to public clouds.

That is the idea at the very heart of the private cloud "movement": that, through the aggressive adoption of innovative technologies like virtualization, commoditization, cloud platforms and eventually cloud-oriented software architectures, enterprises can achieve many of the advantages of public clouds without their disadvantages, which include concerns about security, visibility, control and lock-in.

So let’s look at Google’s arguments against private clouds in the order that they were presented in the original post, and discuss how enterprises can utilize the same or similar ideas in their private cloud initiatives.

Hardware Infrastructure

Google leads off its argument with a focus on the hardware it uses, arguing that its use of commodity-grade servers it uses are cheaper than the typical "enterprise-grade" server used in corporate data centers. No argument there. (Side note: I used this slide back in ‘05 to illustrate a similar point.)

What he seems to miss though is THE POINT of commoditization–that this is not cutting-edge proprietary technology only available to Google. The single-CPU server with SATA storage presented as an example of the lower-cost hardware platform that Google uses is available to any enterprise from a very large number of vendors at extremely low price points.

Rajen makes the great point that what enables Google to use very inexpensive, and inherently unreliable, servers is the fact that they’re "building reliability into the software" so that when the hardware does fail (inevitable with commodity boxes) the work continues on on another server.

Sounds very cool, but off-limits to the rank-and-file enterprise? Actually, no. This idea is one the key themes upon which Appistry was founded, and it lives on in our CloudIQ Engine product. Since this kind of software-based reliability (what we call "application-level fault tolerance") can be heady stuff, we built it into the platform itself, so that the developer doesn’t need to think about it.

Perhaps my colleague Guerry said it best in his very first blog post, "Cross-pollination, Beehives, the Game of Life, and You":

Did we actually write our software by observing bees? No, not directly; however, do stop and consider what a beehive teaches us. Individual bees, each important to the hive community, accomplish what needs to be done without central control, and with no single point of failure. Well, there is the queen, but let’s not quibble. If a bee dies, another picks up the work, and the hive continues by self-organizing, and self-healing.

The energy efficiency of Google’s servers and data centers is mentioned as a key advantage of public clouds, and this is an area that they have provided some leadership on. But the comparisons they like to make are to the "typical server" and the "typical data center" not to a data center that is pursuing a private cloud strategy. How do their servers compare to the best-in-class available from vendors like Rackable, who supplies high-density, energy efficient servers to the likes of Amazon, Yahoo, and also sells to, you guessed it, the enterprise?

The climax of the hardware-infrastructure-argument-against-private-clouds is the obligatory reference to Gartner analyst Darryl Plummer’s 2006 proclamation that enterprises spend some 80% of their budgets "keeping the lights on" versus driving change and growth in their businesses, followed by an assertion that:

The reality is that most businesses don’t gain a competitive advantage from maintaining their own data centers.

Perhaps, Google, but it’s about the applications, not the real estate. Many enterprises do in fact gain a competitive advantage by maintaining their own mission-critical applications which operate against proprietary data sources, are integrated to other back-end systems, and are tied to enterprise user repositories, all of which simply can’t and won’t be moved to the cloud immediately because of the issues mentioned above.

Software Infrastructure

The next area covered in the Google post is software infrastructure. While Google’s custom database, app server and web server is mentioned in this section, the bulk of the section focuses not so much on the quality and characteristics of the software infrastructure Google provides, but on the as-a-service model through which they provide it.

What’s obviously missing in this equation are the huge portfolios of existing applications that exist within today’s enterprises, none of which were written to use BigTable. This is the reality of cloud computing for the enterprise. It will evolve from the adoption and application of cloud-like technologies within the enterprise, first applied to today’s enterprise workloads. Over time, enterprises will adopt more cloud-friendly approaches to application architecture, but it won’t happen in a big bang and it certainly can happen behind the enterprise firewall.

That’s the guiding philosophy behind our product line, Appistry CloudIQ Platform. Our CloudIQ Manager is a tool used by infrastructure and operations teams to manage untouched applications (e.g. the existing apps) in a cloud environment — private, public or hybrid. And, CloudIQ Engine is a tool used by applications teams looking to create cloud-native applications with an inherent agility and ability to scale. In this way, CloudIQ Manager serves the role the of a private cloud “on-ramp” technology, and CloudIQ Engine is more of an advanced tool used to address specific needs.

Applications

The discussion around applications in the Google post focused on the virtues of application outsourcing, with email being used as the primary example. No knocks against Gmail–it rocks, but having just come from a panel of CIOs discussing this very issue, the challenges are not simply in justifying the cost of hosted email. For those CIOs the bigger issues were legal and compliance ones related to discovery and data retention. Corporate control over the data from a legal perspective is a huge issue as well: is Google willing or able to fight a 3rd party subpoena of my data to the same extent that I am?

But what if those organizations willing to adopt less feature-rich email and collaboration environments could take a page out of the Gmail playbook and deploy agile, private cloud-based email systems for 1/2 the cost of today’s systems, versus 1/3 the cost that Gmail can offer but without the accompanying legal and control issues. My guess is they would jump at it.

Innovation

As mentioned before, this is the one that got me going. Contrary to the opinion expressed in the post, today’s enterprise is better positioned to drive innovation through IT than ever before, thanks in part to technologies like those being developed to enable private clouds. Yet clearly the enterprise has more sophisticated requirements than the consumer user of SaaS services.

Conclusion

Rajen’s post ends by posing the question:

As companies weigh private data centers vs. scalable clouds, they should ask a simple question: can I find the same economics, ease of maintenance, and pace of innovation that is inherent in the cloud?

Unfortunately Rajesh, enterprises must also ask themselves the complicated questions: Can public clouds provide the security, visibility, control, compliance, support for existing applications, and legal protections required to even be considered for my applications?

Please don’t take any of this to mean that Sam Charrington or Appistry is a against the use of public cloud services by the enterprise or anyone else. Nothing can be further from the truth. WE ARE HUGE FANS!

That said, we strongly believe that enterprises must be empowered to choose the right cloud for the right job, and the model that we believe will emerge is one of a hybrid or federated cloud strategy centered of the private cloud but extending into the public clouds as the application use-case allows. Clearly we are not alone in holding this view. We continue to see strong end-user movement in this direction. Furthermore, both Forrester and Gartner have recently come out with proclamations of support for private clouds as the future of the enterprise data center.

So please, please, please keep up the good work, Google. The industry–no, the world(!)–needs more innovation around all flavors of cloud computing. But don’t discount the efforts of IT organizations to balance the constant drive for innovation with the unique requirements of the enterprise. And don’t discount the result of those efforts, the emergence of the private cloud.

This past Friday, I posted a Manifesto on “the” Manifesto (those of you who know me and read my posts or follow me on Twitter know which manifesto I’m talking about; if you don’t, you can read it here).

Yesterday the Open Cloud Manifesto was *officially* made public. We think it represents a great opportunity for the industry and we’re proud to support it.

The making of the manifesto (a made-for-DVD extra if there ever was one) generated a lot of discussion.  Good. As an industry we should be talking. Together, we’re at the very heart of where the IT industry is heading. We’re building the foundation for the future of IT (yes, that sounds a bit over the top, but I do believe that the cloud — in its many forms — will drive computing for many years to come).

Yes, we are going to compete…and compete aggressively. But history has shown that customers win when competition is conducted on a level playing field — a field based on openness and industry standards. We’ve seen it with the Internet.  We’re seeing it again with cloud computing.

I encourage you to read the manifesto and engage in the discussion to make it stronger. If you are a member of the cloud community, it is your responsibility to do what you can to improve it (the Manifesto and the community) for all. If you are a customer working with a cloud computing vendor, ask your them if they support the principles undergirding the manifesto, if not the Manifesto itself.