Layerboom

Behold Sinners! The Apocalypse Aproacheth. No in all seriousness if you run a managed hosting company then your time is officially ‘up’. You won’t survive the coming hosting Apocalypse. Here’s why.

There are a few companies you may have heard of building large compute grids for consumption by the general public. They’re calling them their Cloud Computing products. IBM is building BlueCloud, Microsoft is building the Mesh, Amazon already has EC2, and Google has AppEngine. AppEngine is in a slightly different category than the others and the BlueCloud details are sparse, but they’re still worth mentioning. Of more immediate interest are Amazon and Microsoft’s solutions.

Microsoft is currently building their famous 300,000 server Data Center in Chicago. That’s roughly 3 times the number of servers that Google has. Microsoft has also announced several other Data center projects - each worth about $500 Million. It’s fair to say that that’s a lot of computing power, and it’s not all for MSN - Microsoft is planning on providing their platform in the cloud.

The real question is what Amazon will do when the Windows Cloud comes online. Microsoft has enough money in the bank to provide their 300,000 servers to customers for *years* without earning a single cent. That implies they can offer services at super low rates; Low enough to at least compete with Amazon’s EC2, which will support the Windows Server OS in fall 2008.

What happens with two huge cloud hosting companies get into a price war?

In the interest of self preservation they won’t make their services commodities - at least right away. But it won’t even matter. When you’re as big as Amazon, Microsoft, Google or IBM, you can afford to buy servers in such massive quantities that you could make money selling compute time for 10$ a month. The hosting space will change forever, because Amazon will eventually drop their prices by an *order of magnitude* and that has dire implications for the rest of the Mom’n'Pop hosting companies.

If thousands of companies can’t compete with Microsoft or Amazon on price, and they can’t compete in terms of convenience, then why would anyone use them? If you have to buy individual servers, or even servers by the rack, then you’re not going to get the price you need to be able to compete. You also don’t have access to the handful of specialized individuals and hardware required to make things work on such a grand scale.

The only answer is for all the smaller players to band together - to create a Federated Hosting environment, where together they can provide services that begin approaching levels of service and power that the Big 4 will offer.

Either way, we’re in an interesting period in the industry. Computing and the infrastructure of technology has become such a requirement for the economy that it will eventually become a general utility. The real question is who will be around.

Do you think it’s the end? We’re working on the answer, and your opinion is important.

Hi everyone!

I’m looking for some help answering some questions to help LayerBoom and some associated projects determine what companies and individuals really want in their own “Cloud Computing” environment.

I know the Term has been getting worn out, so to summarize, LayerBoom is providing software that helps companies build utility computing environments using existing hardware and network infrastructure. The idea is to provide software as a common platform, then provision resources across many distributed environments, and enable companies and academic environments to rent/share unused resources for extra revenue.

There’s a survey posted up at http://survey.layerboom.com

I’d appreciate your help and time!

-Trevor

(Also posted on http://trevoro.ca)

There are a lot of ways for companies to bootstrap, and even more companies and partners willing to lend a helping hand if you know who to ask, and where to look. One of the companies that’s willing to help is Sun Microsystems.

Through their Startup Essentials program, companies can get access to Sun gear and resources at heavily discounted rates. That’s great, because Sun gear is pretty much the best out there, and being competitive with other companies like Dell and HP on price will help get their technology into more Data Centers and up and coming businesses. But like all shiny objects, servers lose their luster. Even if they’re free.

In order to run a server you have to secure colocation space, pay for bandwidth, buy some switching gear and a router, and depending on your setup get UPS power. Sure you can get cheap switches to connect your high-end servers, but depending on your work load it would be like running a jet engine on bacon fat.

If you’re running more than a few servers you’ll probably need to get more than the standard ~3000W/Rack - especially if you have a SAN device. That means one full rack with extra power. Most older data centers don’t have the cooling capacity to handle todays dense and powerful systems, so your full rack will probably be half full (if you’re lucky), then you’ll have to get another rack if you want to expand. That involves more waiting. Average amount of time for a colo to provision a new rack? 4-6 weeks. You’ll also need to pay for installation, and every colo provides space on contract so you’re committed for a year. Time is money, and waiting 4-6 weeks to be able to expand your environment means you have less flexibility.

There are certain scenarios where getting colo still makes sense. If you’re running your own hosting environment *as a business* then obviously having the control over your environment is necessary. Companies with certain regulatory or security requirements will also need to stick with colo, but otherwise, why lock yourself into contracts and inflexible environments and hire extra staff to manage that operation, when you can just rent some servers en-masse and get super cheap bandwidth?

If you run the numbers you can get just as much if not more *power* for the same amount of money from companies like Joyent, Amazon, ServerBeach or Rackspace, and not have to worry about contracts. While it’s nice to have an awesome piece of kit, sometimes it isn’t worth the time or money.

I’ve been working on a project that lets you quickly move systems between your private Xen implementation and Amazon’s EC2 service. There are a lot of hurdles to get this to work, and most of them are surrounding how Amazon doesn’t let you download a Kernel or Ramdisk image out of S3 unless you’re the owner. You can download someone elses image if you’ve saved it as your own but you still can’t download the kernel and ramdisk. Also, EC2 has specific requirements for how the image is built. Here’s how you can get your image out of Amazon and run it locally on your own Xen hypervisor. I will assume you are already using Amazon Web Services and have created an account. If you haven’t then sign up.

Amazon calls their instance images Amazon Machine Images or AMI’s. If you want to be able to grab one of the many images from Amazon you can use download the Amazon AMI tools and AWS tools then do the following. You can download the tools here

Find and Download the AMI

$: ec2-describe-images
IMAGE   ami-cc6386a5    ubuntu-hardy-ruby/image.manifest.xml    848278689040    available       private         i386    machine
IMAGE   ami-386c8951    ubuntu-ruby-lapack/image.manifest.xml   848278689040    available       private         i386    machine
$:

Fields 3 and 4 contain important information. For this example I’m listing the images that I own. Optionally you can provide a switch that will list all Amazon images by including ‘-a’ to the end of the ec2-describe-images command.

Field 3 is the unique identifier for the AMI, and field 4 is the bucket and AMI “manifest” - or a file that describes the AMI. Because users can specify the name of the manifest, you should pay attention to this value when trying to run the next set of commands.

The AMI tools from Amazon include a utility called ‘ec2-download-bundle’. This will download the manifest file from the bucket, parse through to see what other files it needs to download, then it will reassemble the AMI image locally, and check its signature. The AMI’s are encrypted in small (usually 10 meg) chunks. The signatures for those chunks are also included in the manifest.

To download the first AMI listed above, run the following commands

mkdir 'image-to-download'
cd 'image-to-download'
ec2-download-bundle --bucket ubuntu-ruby-lapack -m image.manifest.xml --access-key $AWS_ACCESS_KEY --secret-key $AWS_SECRET --privatekey $EC2_PRIVATE_KEY

That will start downloading the bundle to your local system.

Rebuild the AMI

Now we have to unbundle the files

ec2-unbundle -m image.manifest.xml -k $EC2_PRIVATE_KEY

This will decrypt and reassemble the image from all the individual components in the list

Now you have an image named ‘image’ in your directory. You can take a look at this file by mounting it

mkdir /mnt/image
mount -t ext3 -o loop image /mnt/image
cd /mnt/image

If you’re lucky there will be copies of the kernel and perhaps the ramdisk in the /boot partition. Otherwise you’ve got to do something really tricky : You have to guess as to what kernel will work the best. Thankfully we have a good understanding of what’s required to boot one of these images.

If you’ve created an image for Xen already then chances are your kernel will work just fine, but your ramdisk might need some adjusting. A trick you can use is to chroot to the /mnt/image folder, specify which modules you want loaded and rebuild the ramdisk - then exit the chroot, copy the kernel and ramdisk out of /mnt/image and you’ll have all the components you’ll need.

I know what you’re thinking: That’s a lot of work / guessing

You’re in luck. While there are a couple sites for sharing pre-built Xen images, the community is nowhere near as large as the Parallels or VMware ‘appliance’ sites. Jailtime.org has a hanful of images but they don’t follow any sort of standard, and the disk layouts / configurations aren’t compatible with Amazon’s EC2.

LayerBoom has a Xen image that is completely compatible with Amazons AMI format, and it can run in your own environment. This means you can copy a system into Amazon from your test environment without any hassle. It also works with the Eucalyptus project, and can be booted in xVM server as well (Instructions are coming)

Download the Xen package

url: http://layerboom.com/files/xen/images/centos52-20080930.tar.gz
md5: d54a83fc22f1ec052db6ebe3c258ee45

u/l :root/password

Wiki.layerboom.com is up for anyone interested in reading or contributing to documenting the installation and administration of the Eucalyptus Cloud Computing platform. Rather than replace the UCSB documentation, the Wiki is an attempt to supplement that resource, as well as collect common issues in one place.

Check it out at wiki.layerboom.com

Everyone knew it was coming, but this morning Amazon announced it will be providing a CDN service.
The process is the same as uploading to S3, then you simply make a call that says “Put this in the CDN”. Customers will be automatically sent to the closest For a lot of companies that are using S3 as a poor man’s CDN already this should provide some extra performance, although there aren’t any details regarding how many edge locations Amazon will be building. That will be a major comparison point between an Amazon CDN and other major delivery networks that have edges in most major POP’s around the globe.

From the article

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections. Our goal is to create a content delivery service that:

Lets developers and businesses get started easily - there are no minimum fees and no commitments. You will only pay for what you actually use.
Is simple and easy to use - a single, simple API call is all that is needed to get started delivering your content.
Works seamlessly with Amazon S3 - this gives you durable storage for the original, definitive versions of your files while making the content delivery service easier to use.
Has a global presence - we use a global network of edge locations on three continents to deliver your content from the most appropriate location.

As Amazon continues to provide easy interfaces to rather abstract problems, it will be interesting to see how the other major players will respond. Companies like VMware and Citrix will no doubt have their own EC2′ish API, and other CDN providers will publish their own API’s, but the real question is how all of these things will integrate together.