At CloudCamp St. Louis, Michael Groner, my colleague at Appistry, gave a talk on the MapReduce framework. The 5-minute Lightening Talk was a condensed version of the 45-minute presentation he delivered originally at the St. Louis Lambda Lounge (a local group focused on dynamic and functional programming). You can check out his slides below.

Michael received great feedback on his presentation and generated a lot of interest in MapReduce. As a follow-up, I thought it would be nice to feature a brief Q&A with him, which I present below:

Sam: What is MapReduce?

Michael: MapReduce is a software framework to support processing of massive data sets across distributed computers. MapReduce has its origins from functional programming where the  “map” and “reduce” operations assist developers in processing data sets easily. For applications requiring the translation of  gigabytes, terabytes, or even petabytes of data into usable data structures and then collecting to end results, MapReduce is a good candidate.

Sam: Where did it come from? Is it new?

Michael: MapReduce’s popularity surge started in 2004 with the publication of “MapReduce: Simplified Data Processing on Large Clusters” by Google. In this paper Google illustrated the framework they use to construct an index of the internet through a simple programming paradigm.

Sam: Why should people care? What does MapReduce have to do with cloud computing?

Michael: This framework contains many of the key architecture principles promoted for cloud computing:

• Scale – The framework is able to grow and expand in direct proportion to the number of machine applied to the system.
• Reliable – The framework is able to handle the loss of a node and restart the work somewhere else.
• Affordable – As reliability is accounted for in the framework, commodity hardware can be used. As scale is accounted for, a user can start small and add additional hardware as necessary.
• Simple – A user simply provides the implementation of a map and reduce function in order to process large quantities of data.

Due to these principles, the MapReduce framework has become one of the most popular platform tools for building cloud applications.

Sam: Can you give us some examples of who is using it?

Michael: MapReduce applications are used in by companies like Google, Yahoo, Visa, and the intelligence community, among others, to produce results from the large data sets they produce.

Sam: How should someone go about learning more?

Michael: To learn more I suggest reading the original Google paper and looking into a MapReduce framework such as the Apache Hadoop project.

Thanks, Michael!

Are you using MapReduce? We would love to hear your experiences with it!

Last week Thursday, the St. Louis cloud computing community came together for our first CloudCamp. If you caught any of my previous posts you know that excitement was really mounting in the run-up to the event, with the event selling out and sponsors signing on practically up to the very last minute. Well, the event itself didn’t fail to deliver on our expectations. Not by a long shot. CloudCamp St. Louis was a smashing success!

Congratulations St. Louis, on hosting your first CloudCamp, and thanks to everyone who sponsored, assisted with and attended the event!

Based on our estimates, over 130 people attended the event. The estimate isn’t too fuzzy, since we had 120 chairs in the main room and there were people lining the walls. This was 90%+ of the number of pre-registered attendees, which is much higher than typically seen in other cities. It’s a good thing we upped our catering number at the last minute. (We were fortunate to have had a very flexible and capable caterer.)

We anticipated a diverse crowd at the event, and this was confirmed by the show-of-hands poll I took at the opening of the event. Our audience identified themselves as approximately 40% developers, 40% IT infrastructure folks, 10% IT management and 10% business management. When asked how many of the audience had ever participated in an unconference, only a handful of people raised their hands. (At which point, I knew we were in for a fun night!)

Lightening Talks

There were four Lightening Talks featured at the event.

• Mike Bradley from Rackspace on The Rackspace Cloud
• Brian Blanchard on behalf of Microsoft, on Windows Azure
• Alex Miller, with a community on Distributed Caching in the Cloud
• Michael Groner, with a community talk on Map Reduce and Cloud Computing

I’ll be posting or pointing to these slide decks soon.

The first couple of lightening talks went a bit over the allotted 5-minutes and were each very focused on a specific vendor offering. The latter two in contrast were minimally promotional and very well received by attendees. I’m not trying to call anyone out here, but it’s worth mentioning that as an organizer, balancing the needs and desires of the sponsors against the needs and desires of the attendees is one of the most challenging aspects of running a CloudCamp.

Kyle Cordes, himself a sponsor through his company Oasis Digital suggested that each sponsor be given one slide and 10 seconds of pitch-time, max. I think this is a bit extreme, but I could go for 30 to 60 seconds. Anything beyond should be strictly educational and non-promotional and vetted beforehand as such. Thoughts on this?

Unpanel

After the lightening talks we held the Unpanel. The Unpanel — a bit of a CloudCamp tradition at this point — is a panel discussion in which audience members are selected as panelists and the audience also comes up with questions for the panel. The Unpanel is a great way to get folks’ juices flowing and into the unconference groove.

After a bit of prodding, we identified the panel and presented them with a pretty broad array of questions, including:

• Will we run out of bandwidth for cloud computing?
• How to address security in the cloud?
• How to migrate in/out of the cloud?
• How to address compliance in the cloud?
• How to address fraud in the cloud?

As you can imagine, none of these questions was fully addressed in the brief time each panelist was given to speak, but we accomplished the mission of spurring on discussion, at several points getting dangerously close to veering off into outright debate. (The relationship between virtualization and cloud computing was a particularly contentious issue.)

Breakout Sessions

We then channeled our creative and argumentative energy into the choosing and participating in breakout sessions. We had time for two consecutive rounds of breakouts, and access to six rooms of varying sizes, though we only ever needed four at once.

Before breaking up, we identified these sessions:

First round:

• The Enterprise Cloud Use-Case Panel
• Cloud Infrastructure
• Big Fail in the Cloud
• Fraud in the Cloud

Second round:

• Software Development Perspectives: Idioms, Architectures, Patterns for Cloud Computing
• Virtualization and Cloud
• Building Clouds
• Cloud Data and Storage

The majority of attendees attended the first two sessions on each list, which were held in the biggest rooms. In fact, the latter two sessions of the second round never really got any steam, and the folks interested in them either mingled in the food area or went to one of the two major sessions.

I spent most of my time in the Enterprise Use-Case Panel and the Software Development Perspectives sessions, both very well attended and quite interesting, but found myself popping in and out of each of the sessions each round, just to ensure all was going smoothly.

Looking Forward: Cloud Computing in St. Louis

After the breakouts, we pulled everyone back together for a brief wrap-up. We got a quick rundown from an attendee of each breakout. The energy was still high and we a pretty big crowd remained, considering it was 10pm.

Before leaving, I asked about interest in future CloudCamps and the possibility of starting up a cloud computing users group and there was strong interest in both. I’m excited to announce here that the St. Louis Cloud Computing User Group is now forming. Contact me if you’re interested in helping out or participating, and stay tuned for more information.

I’m just back to St. Louis from Las Vegas, where I’ve spent the last couple of days at the Gartner Application Architecture, Development and Integration show. I may get a post up about that later this evening, but honestly my mind is pretty squarely occupied with this evening’s CloudCamp, which I’m really looking forward to.

I’m excited to announce that we are down to single digits in terms of the number of “tickets” remaining, having long since exceeded our initial projections, which we thought were aggressive given the lead time, time of year, and all the other factors that we fretted over while deciding to do the event.

This is great news!

The other great news is that we’ve upped our catering order, and plan to make sure everyone is well fed and, er, “socially lubricated.”

We’re dependent on sponsors to make that possible, and we’re excited to announce that Rackspace and Microsoft have signed on over the past couple of days as Gold sponsors. Thank you both!

See you this evening!

Momentum continues to build towards CloudCamp St. Louis, which is just a few days away. Today we signed up a couple of new sponsors, Contegix and Datotel, who join Appistry, Savvis, Hexagrid and Oasis Digital in supporting the camp and the STL cloud computing community.

Over the weekend registration shot up by over 20%, and as of this writing we’re now one registrant shy of 120. (Will you be lucky number 120, 130 or 150? Register now and see. )

Finally, we’re announcing the details of an exciting session we will be conducting at the event.

One of the complaints most often heard from potential enterprise users of cloud computing is the dearth of case studies available about what other enterprises are doing. To this end, at CloudCamp St. Louis we will be hosting an “Enterprise Use-Case Roundtable” which will explore a series of case studies detailing the experiences of enterprise customers in adopting cloud technologies. The Roundtable will consist of a series of brief case-study presentations, followed by a moderated panel discussion through which attendees and panelists will engage in a broader discussion about enterprise cloud computing adoption.

Roundtable panelists include Bob Lozano, founder and chief strategist of Appistry, and author of the forthcoming Executive’s Guide to Cloud Computing (Wiley, April 2010); Benson Schleisser, technical vice president for Savvis, whose work has been instrumental in the launch of Savvis’ own cloud computing offering; and Suresh Mandava, founder and CEO of Hexagrid, who has 20 years of experience as an infrastructure architect of large data center environments at enterprises such as MasterCard and ExpressScripts.

(I’d love to moderate this panel, but I suspect I may have my hands full on Thursday. I’m still looking for someone to do it, so if you’ve got any ideas let me know!)

We’re excited about the Roundtable and the ability to highlight some real-life customer experiences at the event. We’re also very thankful for the support of CCSTL Platinum sponsors Appistry, Savvis and Hexagrid, without whom the Roundtable would not be possible, and whose support of the event as a whole has been outstanding.

Update: While writing this post, lucky number 120 checked in!

In many ways, CloudCamp St. Louis has been well over a year in the making. It’s been something I’ve been dying to do since Ruv, Dave, Jesse, Sara and I did the very first camp in San Francisco, back on June of ‘08. The first discussions about it between Benson and I were later that summer, but alas life (work, really) happens.

I suppose I’ve always felt a bit guilty that we hadn’t done one here in St. Louis yet, given all the work I’ve done with CloudCamp the organization and helping to put on CloudCamps in other folks’ cities. I’ve kind of considered it one of those “cobbler’s children” kinds of situations, i.e. they go barefooted. (My buddy Gary Woodcock over at Hexagrid calls it establishing “band cred” — making it big someplace else and then rocking out back home. Works for me.)

Needless to say, I was pumped when Benson, Ken and I finally decided to “just do it” a little less than a month ago and committed to pull together CloudCamp St. Louis under a very aggressive time table.

It’s been a blast so far, and we’re just under a week from the big day.

The community has really rallied behind the event. We’ve got great corporate sponsors, including Savvis, Hexagrid, Oasis Digital and of course Appistry, with more to come. And we’re also supported by a number of local groups, including the St. Louis Lambda Lounge, a group focused on functional programming languages, the St. Louis-based Strange Loop conference, and the St. Louis Java Users Group.

And we’re quickly approaching 100 pre-registered attendees. Woo hoo!

So, please join us on Thursday evening, December 10th, at 6pm for CloudCamp St. Louis. The more the merrier. The event promises to offer a fantastic evening of cloud computing education, discussion and networking (and of course food and drink!!) You’re guaranteed to have a great time, or your money back!!! (By the way, it’s totally FREE.

See you at CloudCamp!

I just posted the official announcement of “CloudCamp in the Cloud,” our upcoming virtual unconference. Please help spread the word!

CloudCamp Announces “CloudCamp in the Cloud” – First-Ever Virtual Unconference

October 5, 2009 – CloudCamp, organizer of the community-based cloud computing unconference series, today announced that it’s taking its popular event series virtual with the forthcoming “CloudCamp in the Cloud.” CloudCamp in the Cloud, to be held Thursday, October 22, 2009 from 12 noon to 3 pm Eastern Standard Time, builds upon the original live CloudCamp format providing a free and open place for the introduction and advancement of cloud computing. Using an online meeting format, attendees will exchange ideas, knowledge and information in a creative and supporting environment, advancing the current state of cloud computing and related technologies.

There are a number of opportunities to get involved with CloudCamp in the Cloud:

• ATTEND – Attending CloudCamp in the Cloud is free, fun and informative. Register now at http://bit.ly/UKbc1.
• PRESENT – CloudCamp in the Cloud encourages community presentations. If you have a cloud-related topic to discuss, visit the topics page to submit a proposal.
• SPONSOR – CloudCamp depends on corporate sponsors who provide financial assistance and other valuable donations. Current CloudCamp in the Cloud sponsors include Citrix, Enomaly and Appistry. If you would like to sponsor CloudCamp in the Cloud, please contact Reuven Cohen.
• ORGANIZE – CloudCamp is a non-profit, volunteer-driven organization. If you’d like to help facilitate CloudCamp in the Cloud, letting us know about your interest by emailing cloudcamp@googlegroups.com.
• SPREAD THE WORD – Help share the news about CloudCamp in the Cloud, by retweeting this announcement (hashtag: #cloudcamp), blogging about the event, and linking to the main information page.
  

Related Links
[1] http://bit.ly/UKbc1 [CloudCamp in the Cloud Registration]
[2] http://twitter.com/cloudcamp [CloudCamp on Twitter]
[3] http://www.facebook.com/group.php?gid=10128776220 [CloudCamp on Facebook]

Contacts
Program

• Dave Nielsen, (415) 531-6674, dave -at- platformd -dot- com
  

Sponsorships:

• Reuven Cohen, (212) 203 4734 x102, ruv -at- enomaly -dot- com
  

Media:

• Sam Charrington, (415) 727-1850, sam -at- appistry -dot- com
  

About CloudCamp
CloudCamp was formed in 2008 in order to provide a common ground for the introduction and advancement of cloud computing. Through a series of local CloudCamp events, attendees can exchange ideas, knowledge and information in a creative and supporting environment, advancing the current state of cloud computing and related technologies. CloudCamp has served over 5,000 CloudCampers in more than 50 events all over world, in cities like Amsterdam, Antwerp, Bangalore, Berlin, London, New York, San Francisco, Stockholm and Singapore.

via CloudCamp Announces CloudCamp in the Cloud – CloudCamp | Google Groups.

I’m in Las Vegas for the next couple days for the Enterprise Cloud Summit, and to help out with the CloudCamp we’re holding there Monday evening. If you’re around, ping me — I’d love to meet you.

Our partners over at Apptis were kind enough to post video footage from the recent CloudCamp Federal event. At the beginning of the event, each of the sponsors was given the opportunity to present a five-minute “Lightening Talk” on some aspect of cloud computing and where they fit in the big picture. My session is posted below. At the end of the video you’ll have the opportunity to review the other presentations from the event.

The slides are a bit difficult to make out, so I’ve included them below as well.

  With the U.S. presidential elections in the not-too-distant past, I thought it would be a timely opportunity to highlight a few of the results of a survey we conducted at the recent CloudCamp Silicon Valley. The survey is part of a series dubbed “Inside the Cloud” and is an effort to gauge the impact of cloud computing on the cloud community in general, and those attending CloudCamp particular, as well as uncover the issues and opportunities cloud presents.

We had 61 CloudCampers respond to the Silicon Valley survey, representing a mix of providers and consultants (and the ever elusive "other"). Through the survey, they provided feedback on cloud leadership, market conditions, innovation, challenges to cloud adoption, critical cloud attributes, and relevant cloud applications.

Some of the key findings include: 

• Amazon is perceived as leading the race for cloud king, besting Google by a 2-1 margin.
• Respondents expect an uptick in cloud interest as a result of the current state of the economy.
• Infrastructure is currently seen as the source of the most innovation in cloud computing right now.
• Cloud computing’s "Big Three" — security, reliability and scalability — continue to haunt developers’ dreams.
• There remains a considerable amount of uncertainty around what applications will win the cloud race.

The full survey report was released today complete with deeper analysis and statistics. You can read the press release here.

Have a question you’d like to submit for consideration in the next "Inside the Cloud" survey? Post a comment to let me know.

P.S. We’re tallying the results of a similar survey conducted at CloudCamp Federal in Washington, DC, which focused on the unique challenges and opportunities for cloud computing in the federal market (more to follow in a future post).

I can’t believe it’s taken me so long to post about the amazing CloudCamp Silicon Valley event we held a couple of weeks ago. But there’s no better time than the present, especially since it gives me a chance to plug next week’s CloudCamp Chicago.

CloudCamp Silicon Valley was held at Sun’s executive briefing center in Menlo Park on September 30th. The EBC is an amazing facility; thanks again to Sun for hosting us. We had more than 10 rooms filled to capacity with more than 150 people discussing various aspects of cloud computing. In fact, due to strong interest in the event a good number of walk-ins we found ourselves pushing the limits of the Menlo Park fire code. We may need to pony up for a door man/bouncer at the next event.

The event took on the traditional CloudCamp “un-conference” approach, with attendees driving the discussion by posting topics to the board. We had really strong participation and the board filled up quickly. There were a handful of slots open at the end, so I volunteered to facilitate two topics I thought would be of interest. The first was a session on “What is Cloud Computing?” targeted at those new to the field. The second was a more advanced topic on cloud application development: what it means, how it’s different, and what enterprise developers – Java and .Net – need to do to fully leverage cloud environments.

Defining the Cloud

About 25 CloudCampers joined me to explore what cloud computing really is. It was a highly interactive session (as most camp discussions are) where we talked about different approaches to cloud computing. In the end, the group consensus lined-up very well with the approach I’ve been advocating here on my blog (see especially the Blind Men and the Cloud post), that is, that trying to define cloud based on the technologies used to enable it doesn’t capture the full promise of cloud, and that the best way to define cloud computing is in terms of the set of characteristics each of us is trying to achieve with it. Some of the essential cloud characteristics we came up with in the session were:

A Heated Debate on Cloud Application Development

The second session I led was around cloud application development. There were about 20 or so very technical participants crammed into a relatively small room, many standing, and we ended up having a very lively discussion about software development in the age of the cloud. I took a show-of-hands survey at the start of the session and the group’s primary focus was on achieving scalability for their cloud applications, as opposed to simply trying to outsource the ownership and management of servers to a third party. (Not a surprising finding for a group of developers.)

There was a bit of a split in the group between whether cloud application development was, in fact, different than traditional application development at all, or if it was just more of the same. Many of those in the room felt that developing for the cloud required a different approach so that applications could take advantage of the resources the cloud made available. A couple of folks took the position that cloud development wasn’t really any different. (Chris Richardson did a great job of challenging the notion that a new approach was needed). I think the unifying idea between both positions is that cloud development necessitates the best practices that people have been talking about for a long time. Developing applications that can truly scale in the cloud depends on writing loosely coupled, highly modular code and being careful about the way state is managed. For more on this topic see the Guerry’s great series of posts on “Is your code cloud-ready and multi-core friendly?” (Part 1: Introduction, Part 2: Atomicity, Part 3: Statelessness, Part 4: Idempotence, with more to come)

Finally, the exceptional turnout also gave us the opportunity to more formally ask about the industry issues on the minds of CloudCamp participants. Appistry partnered with Cloud Camp to sponsor the first-ever Cloud Camp attendee survey. We’re in the process of tabulating the responses and will highlight some of them in upcoming posts. We’ll continue to sponsor these surveys to help the community enhance its understanding of cloud computing in the market.