The concept of Virtual Datacenter is that you have all the power of a datacenter at your command without having to worry about the details of actually running a datacenter. Chad Sakac from EMC wrote an excellent post in his personal blog about the transformation to a Virtual Datacenter.

One of the points Chad makes is the abstraction of the physical infrastructure. Quoting Chad:

“Every Layer of the physical infrastructure (CPU, Memory, Network, Storage) need to be transparent. Transparency means ‘invisible’. This implies a lot, and implies that the glue in the middle, like a general purpose OS, needs to provide the “API models” for those hardware elements to be transparent. ”

I latched on to this point because that is what we have been building at SoftLayer for the last few years. We realize that the abstraction of the physical infrastructure not only means that end-users don’t need to know how to manage the physical infrastructure, but that the abstraction can make more efficient use of resources (= money!).

Let’s talk about the advantages of virtualized infrastructure. Without virtualization, provisioning a web-facing server on the network would involve obtaining rack space, a server, licensing and loading an OS, finding a switch port, physically connecting a cable or three, setting up the switch port (I hope you know IOS), getting IP Addresses (hopefully you don’t have to go get more from ARIN), and adding a firewall and/or load balancer (more procurement, cabling, and configuration). Adding storage could be just as complex – also involving procurement, racking, cabling, and configuration. This doesn’t sound very efficient. In fact, it sounds a lot like creating a “circular device that is capable of rotating on its axis, facilitating movement or transportation whilst supporting a load”. It’s been done before and I’ll bet it’s been done better by people other than you.

Using virtualized infrastructure you should be able to perform the task with a few clicks of a mouse or a few API calls and have the functionality you need set up in a few minutes instead of days, weeks, or months. No worrying about procurement, physical constraints, or learning the specifics of network and storage devices from different vendors. All you should have to focus on is the running of your particular application. You shouldn’t have to worry about configuring servers, networking, and storage any more than you should have to worry about chillers, HVAC, generators, and UPS batteries.

Remember when the internet used to be about bulletin boards, e-mail and other random tasks like keeping up with CNN, ESPN or whatever news outlet you may fancy? It wasn’t that long ago, but after some time in the internet industry I have to tell you that I was amazed today by a real life representation of the evolution not just of the internet, but communications as we know it.

As I write this, it’s 4:00pm CST on January 20, 2009. The significance of this day will be marked in history by the inauguration of the 44th president, Barack Obama. Love him, hate him, whatever your position is, you cannot deny the sheer volume of intrigue as we enter into this presidency and its influence on the next 4 or 8 years, depending on how history plays itself out.

This volume of intrigue has officially impacted the internet in a manner yet to be seen prior to today, but in a manner that is likely to be seen more and more as technology continues to progress. In Softlayer HQ, we have a U shaped office the spans two sides of a corporate office building with the glass walls of the exterior creating the exterior barrier, while the interior barriers are your typical sheetrock, egg white colored walls. In between the Glass and the sheetrock lie some 60-100 cubicles. As I walked from conference room to conference room, I could easily see the video streaming of the inauguration on dozens of our employees computers. Some used the really cool CNN/Facebook stream, some used the MSNBC Stream, some used others, but you get the idea. The fact that live streaming video of monumental events occurs on a video screen; while the tasks at hand are being completed is something that old movies portrayed as beyond belief. It’s really impressive the technologies that are at our fingertips and the abilities that we have to utilize these in our daily lives.

Softlayer had the opportunity to experience a real life “so what does that mean for internet going forward” example today. Recently we were approached by a large scale content delivery firm with the expectation that they had been contracted to do live streaming of the inauguration. With a simple introduction we indicated that we were well prepped to provide you the turnkey infrastructure to accomplish their task. Without going into great detail, the infrastructure included 200+ servers, multiple load balancers, firewalls, and other ancillary devices. With the on-demand nature of our business we were able to enable the infrastructure to functional within a 4 hour period. Although stated to the customer, they had their reservations, but true to our stated deployment times, we met with flying colors, the expectations.

So the real test, Performance! Although still streaming through what is likely to be one of the biggest, most watched events on the internet, Softlayer increased sustained bandwidth north of an additional 30Gbps to our network IP over and above our usual sustained bandwidth levels. Utilizing the 200+ Gbps of capacity throughout our network, we were in a fortunate position to have the capacity and the infrastructure in place to support such a large event. I am sure that the cellular firms wish they had prepped for better capacity in terms of spikes in usage. With many hearts racing in the throughout the office, but especially in the network department due to the bandwidth graphs racing upwards, all of here at Softlayer are excited that we were part of the day’s events. The many many meetings that involved robust network discussions, capacity planning, future growth models, etc. were all validated today with this event. The ‘We’ll never use that much’ and ‘that’s overkill’ discussions have all been put to rest. By deploying 40Gbps to each rack and building upstream capabilities that have capacity not as an issue, but as a planning and growth tool, we are extremely excited about what the future holds in terms of online, internet communications. We are looking forward to the next generation of internet technology as it becomes more and more robust. Our mantra remains firm as the leader in next generation virtualized data center services and we look forward to realizing the things that movies portray as beyond belief.

• There are more than 111 countries covered by our customers
• At least one customer lists a residence somewhere in the Amazon Rainforest
• We have at least 1 customer for every state in the USA
• The southernmost city is Christchurch, Canterbury, New Zealand
• The Northernmost city is Lakselv, Norway
• We don’t have a customer who lists a residence in Antarctica yet.

Consider this one a warm-up. My next map will show this same data, but broken out by our datacenter locations. (Dallas, Seattle and Washington D.C.)

Think about how fast things changed from Y2K until now. So much technology; the way we listen to music, HDTV, BluRay, Hydrogen, Electric, and Hybrid automobiles, the green movement. Change is everywhere. If you look back to 1900 or even the 1950’s changes were much slower and they weren’t nearly as life-changing. A change to someone’s daily routine back then wasn’t a life changing stressful event like it can be now.

So how do we cope with all these fast changes today? Some form of outsourcing. Think about it, even going out to eat is outsourcing. Some of my single friends have never even used the kitchen equipment in their homes. They outsource their kitchens! Most use the fridge to keep the adult beverages cold but that is to drink with the Papa John’s pizza they have delivered.

Ok, so now for the real reason this blog fell out of my head. I was talking to my mom yesterday and she was talking about quilting. She is a quilting nut. My whole house is buried in quilts. My wife loves them. I am pretty sure my brother’s house has quite a few as well. We have a wedding quilt with pics of the wedding and signatures and quotes from people that were at the wedding. We have a “charm” quilt with 1020 or so different fabrics in it including fabric from my favorite shirt (at that time). We have too many quilts to name and I don’t know what about half of them are called really. They all have “quilty” names.

So for those of you who don’t know a whole lot about quilting (and I don’t either) there are machine quilted quilts and there are hand quilted quilts. Machine quilted quilts are faster to finish but aren’t worth as much money and can stand the test of time better. Hand quilted quilts take much longer to finish and can’t stand up to as much punishment but are worth more money in the long run.

So, I have always wondered how my mom can bust out all these quilts so fast. The ones that are machine quilted always made sense. She can make them faster and they are tougher. But she gives us several that are handmade and I could never figure out where she got all this free time. I mean she is retired and sleeps till like 11am! Anyway, to make a good quilt you have to go search for all the right fabric, sew the top all together, then the bottom, and then quilt it all up nice and pretty – like toilet paper really. Batting on the inside makes it fluffy.

It takes time to do all those things and she just keeps on making them at blazing speed. So I finally figured it out yesterday when on the phone with her she mentioned she dropped 3 quilts off to be quilted. It didn’t even faze me at first and then it hit me, she is cheating! I said, “You did what with the quilts?” So I learned yesterday that some of my quilts I thought my mom hand quilted, were finished by people I don’t even know! I guess outsourcing is even to the point where grannies can do it to trick their grandkids! Where will it all stop!

In this fast moving age, it probably won’t so don’t miss the outsourcing train; it can save you some time – time you can spend with your family making quilts if you choose.

PS. I wonder if I set the record for using the letter “Q” the most times in a single blog?

You have to step back and look at it from a different angle. If you aren’t ready to outsource the whole farm just yet, then you can go about it in a couple of different ways. One, you can outsource your sandbox, development, and/or test environment. We all know that with SAS 70 and SOX you have to have all of these (or most of them anyway). And outsourcing might be a good way of getting them in place. The cool thing about outsourcing any or all of those are you have a pristine environment and if it does get polluted somehow you can just reload the OS quickly and painlessly and try to tear it up again. Outsourced servers are great for this type of scenario. You can even get a few servers and carve them up virtually and have even more toys to play with. Now, you can just go buy new servers and have this in house but when they break or they are obsolete then you get to buy more. With an outsource model you can buy 1 or 100 and have them for 1 month or 2 years, it’s up to you, your needs, and your budget. You can add hardware, memory, change the OS daily, and only buy the License for a month instead of having to buy it outright when you buy your own servers. I personally believe this is a really good way to get acclimated to outsourcing and test the waters both with yourself and your boss. You always have to make sure they are ok with the way you are doing things. Well, sometimes anyway.

Another option with outsourcing is outsourcing production. Some bosses out in the world aren’t ready for this yet, but they will be. They like keeping their data close by and having multiple copies and instances and USB keys with copies on it, etc. That’s just the nature of data. Now we all know that you can have the same if not more redundancy in the outsourced model too, it is just hard to explain to them sometimes. I have to give them credit. Think about all the data in the world and how much of it we need to use every day. If folks like them didn’t demand that we techies keep it safe the world might have a bad day, I know I would. I use tons of data everyday (might be a fun blog).

If you decide to outsource dev/test or production you have the ability to scale quickly and accordingly when dealing with technology. Not having to be bogged down by worrying about hardware lead times, dealing with accounts payable, the receiving dock, and all the other worries you have when buying hardware is a liberating feeling. I know what you are thinking; I have been over this side of it a few times so I will just leave it at that but the numbers and today’s technology make it all come together and make good business sense.

Outsource IT!

These are the famous noises that come from the NOC every so often. I swore and swore that I wouldn’t get sick. To be honest, there was something going around about four months ago, and I was just about the only one that didn’t get sick, and I was King-of-the-NOC!

Not this time.

Emails were sent out — “Clean your workstations — wash your hands — don’t throw your used Kleenex tissues at other NOC personnel — and for the love of God, don’t get sick”. Oops. So, one by one, each NOC technician started getting sick. One down, two down, three down…

Then it hit me.

You know how it starts, don’t act dumb. It all starts with that sore throat, that isn’t that sore, but makes you wonder if you’re getting sick, and everything ends up becoming a psychological battle of “do’s and don’ts” to get better, before you get any worse. It never works. You start feeling that sore throat, which gets worse as every hour goes by, as you start overdosing on Vitamin C drops/pills. Then you think, “I don’t just need Vitamin C, right!?” So, then you did around the infamous Softlayer NOC Pharmacy, and start overdosing on off-brand multivitamin, Centrum wanna-be’s**.
Things get worse.
So, by the end of the shift, your throat feels like it’s on fire. You have to make a Wal-Mart run at 12am in the morning (depending on your shift), and you buy every little piece of medicine you think you might need to make life better while you are…Sick.

So, for a few days, you end up chugging cough syrup, feeding on Centrum wanna-be’s, Vitamin C pills/drops, Halls Mentho-lyptus “Mountain Menthol” cough drops, Airborne Formula (more on this later)…and VITAMIN GUMBALLS!!
That’s right, folks. We have vitamin gumballs, and they are GROSS! The pink one is okay, probably the best out of them all, but it still taste like rubber. Ugh. Now, as for the Airborne Formula, I just don’t trust it. I mean, people say it’s GREAT, however, I need proof. I mean, come on, it was created by a second grade teacher. Was this teacher a doctor before he/she decided to actually teach kids multiplication? Think people. Think.

Overall, most of us in the NOC got sick, including me. I’m just now getting over this, while I still fight off a tickle cough, but I’m sure this will never be the end. So folks, keep taking that Vitamin C and that Centrum wanna-be, and don’t get sick. I’m sure next time I’m NOC gonna get sick!

** Centrum Wanna-be is what I call Off-Brand (Equate) Multivitamin Tablets.

Dear Mr. Starbuck’s (aka Howard Schultz),
As a frequent patron of your fine establishment for many years, covering many locations in multiple states and multiple countries, I need to ask a personal favor. PLEASE stop messing with me, with the idea of me being anyone other than ‘big business’. See I live in Dallas, Texas and this past weekend I had the urge to actually forego my normal $5 cup o’ java at any of the 50 Starbucks within the 4 mile radius of my house and actually go to a local establishment that has some great beans. I wasn’t going for a cup, but I was going for an actual bag of beans that I might be able to take back to the house and brew a random cup of sissy coffee (the flavored coffees that actually make hair fall off your chest, rather than put it on there like the SENOMA blend from the aforementioned Starbucks). BTW, for you Dallas’er’s my preference for my random sissy coffee fix is a Cafe Brazil.

Location aside, I drive past the 50 Starbucks in route to the closest Cafe Brazil which is about 5 miles away. My coffee of choice is the ‘Snickerdoodle’, so I was thinking I will big bag it and get a pound, maybe two, to ensure my fix is completely covered. I walk in to the aroma o’ joy that comes along with a coffee house. To an addict of caffeine, it’s kind of like Vick’s to a cold! All employees eye me and my girlfriend as we smile our way to the counter. As we are walking up, something just isn’t right and we can tell immediately something is amiss. Where are the bean’s that all of the other locations have? Where are the grinders? Being sure they are in the back or on the other side of the restaurant, I say with confidence, I want the biggest bag of beans I can get my hands on. The response, without a bat of an eye, was “not here sir, Starbucks forbids it!” WHAT THE $!%$? Again, the CSR at the counter say Starbucks told the landlord that they forbid anyone else in the shopping square to sell Coffee Beans to- go and went as far to tell me that he could not even pour me a cup of the coffee in a to-go cup, as per this was also forbid by Starbucks. This disappointment was seen in both of our faces and the CSR could tell that tears may be near, so the obligatory ’sorry’ was thrown out with a ‘can we do anything to make this right’ comment?

This is unacceptable to me. As the loyal readers of theinnerlayer and all of the employees of Softlayer can attest, Caffeine in any form is like the blood through the veins of this company. Pound for pound, employee for employee, I would challenge the caffeine intake of Softlayer against any other company in the world. Pot after pot of, yes Starbuck’s, is brewed hourly, if not minutely. Literally, cases of Monster are brought in weekly to support the efforts here. With the new JAVA Monster, the numbers may just fly right off the charts. Hence, the frustration

Mr. Starbucks, as one of the founders here at Softlayer I can tell you that all of us think about dominating our segment of the world, planned for it and expect it. Surrounding myself with the smartest people I have ever been around gives me a comforting feeling that all of these goals will be achieved. With the support of these smart people I refer to, we all have a standing order internally that to get to our stated goals; the idea of alienating customers by self serving goals has to nipped in the bud. We are a services company to the masses which means we believe that natural competition is healthy and that continuing to strive to build the bigger, better solution, customers will always be the winner in the equation. If we believed that exclusionary practices and pure heavy weight domination was the proper way to win, we would have thrown our money that way, but the open market allows us to stay at the top of our game, remain cutting edge and push for innovation and automation that will allow us to grow our customer base because we have a better solution for the customer, not because we don’t allow the customer to have any other option. I/we may be a small fish in the pond so to speak, but I think you might be able to learn something from my statements. I’m off to get my fill of caffeine, but not sure Starbucks will be my first choice for the next short while.

Sincerely,
One un-caffeinated, unhappy customer
(Sean Charnock)

1. The People – one of the best teams that I have ever met, and now, have the pleasure of working with
2. Vision – as someone recently said, how often do you get a chance for a “do over” without being Bill Murray in Groundhog Day?
3. Industry – how cool is it to be able to play with new technology and help shape the way people will do business
4. Acronyms and Buzzwords – as anyone that has worked in the telecom or technology industries can attest, this is the best place to be if you want to assemble new words based simply on the first letter of each.

Even before the Internet, (did that time really exist?), telecom hardware vendors, service providers and others in this little universe loved to create acronyms to make technology sound complicated. And of course, it created employment for thousands of people, which according to many of those techies, offered no real value, to help translate all of this into something that the consumer and investment community would buy… Welcome to the world of marketing in the Internet Age!
One thing that all of these people that came to be known as “Gurus” or an even buzzier buzzword – Evangelists – learned was to standardize on at least part of the acronym. Out of this concept was born:

• _AN (G, L, M, S, W)
• _EO (G, L, VL, S)
• _DSL (H, A, S, V)
• _SP (A, M, S)
• _2_ (B2B, B2C, P2P, M2M)

And each of these led to spin-off acronyms, like DSLAM, FRAD, ATM (the network not the cash machine), and my favorite – BE – which said a lot about what we were all doing back then. Acronyms became the patents of the original dot.com era (which we did not actually call Web 1.0, but more on that later). Of course, we also learned different naming systems and adjusted the English language to suit this purpose. Capitalization rules were thrown out along with the baby and bathwater. Capitals now appeared in the middle of all company names (yes, we even did it here…) and products thanks to another phenomenon of the era – the mandatory use of compound words.
Best of all, the digital age gave birth to an industry designed to make all of us look like techies – the acronym dictionary. Many of the generation that graduated into the telecom and Internet revolution of the early 1990s (including yours truly) built careers on the ability to string acronyms together to define the future of networking and ultimately, life. The common toolkit for all of us: Newton’s Telecom Dictionary and a fluffy cloud graphic from the MS Office clip art files – that was generally used by “advanced” gurus and not something to be thrown around by anyone with less than 12 months of experience in the dot.com trenches.
The web generation also taught the masses about versions and a fundamental rule in software: avoid buying version 1.0 and never launch a product called version 0. Even the web generation of telecom -mobile carriers – figured this one out. 3G has long been touted as the utopia of communications, but we had to get through 2.5G first. Never mind that v1 and v2 seemed to actually work. Thus, versioning took its rightful place at the left-hand of acronyms.
Jumping ahead a decade or so, we are now firmly in the grips of Web 2.0 and a new set of buzzwords and acronyms. If you don’t have an avatar living on a virtual street in a virtual world that spends virtual time stuck in virtual traffic driving to a virtual job, you probably are like me – part of the 1.0 generation. I too used an Apple IIe in school, but for me it was grade 10, not grade 1! And then there was the Newton which did not exactly fit into a palm, but that story will have to wait…
Like all things, buzzwords and acronyms have evolved as well. We are now using words like “grid”, “utility” and “always on” to reflect the way we are connected and work. And of course we all need to be Irish for more than 1 day per year. Don’t get me wrong. SoftLayer agrees that green is good. We have turned our facilities green in search of the “green”. Basically, we believe the best way to help our customers with access to technology on a real-time basis is by being as operationally efficient as possible. So, we have forced ourselves to be “green” in everything that we do. But, I digress…
My favorite current acronym root is “_aaS”. We can thank the failure of ASP (part of Web 1.5) and its lesser known cousin – AIP – to catch on for this latest iteration. The good news is that almost everything that we can think of can be called an “X as a Service”. Our internal top 10 list includes:

• Architecture as a Service
• Communication as a Service
• Data Center as a Service
• Hardware as a Service
• Network as a Service
• Platform as a Service
• Software as a Service
• Storage as a Service
• Virtualization as a Service
• Widgets as a Service (the favorite of our Starbucks fan)

But, don’t take my word for it. Whatever as a Service will be hard to avoid. Now that I think of it, MaaS (Monster as a Service) might be a big money maker around here.
I know we would all like to see Service as a Service, but that may need to wait until Web 4.0 when we have all been virtualized. Personally, I can’t wait to see what comes next. Given our place in the universe, I think that SoftLayer will have some influence and that’s why I’m here…

Ok, I see that you are back, are you still employed? We are hiring if you need a new job resumes@softlayer.com

Ok, really, how do you think that conversation would go? I have had that same conversation with ex-bosses and owners of small and medium sized businesses in the past and most of the time they don’t go very well. Granted they were a few years ago so hopefully times are changing.

I have been told a few times, “no, I don’t want to pay $300 per month for a server we don’t own and put my data on it! That is ridiculous, just go buy me a new $3,500 server and we will put it in our local Datacenter, Server Room, Broom Closet, Bathroom, Office Managers office…” well you get my drift.
“But Sir, with this outsourced server we could easily have off-site backups, more processing power, some cool redundancy and it will not annoy everyone in the office with the loud fans and heat generation. And when we have a power outage in the office and everyone goes home for the day, they will be able to work from home because the server will still be online. Oh yeah, and our company website and email will still be functioning as well.”
“Are you insane? Those challenges are so easy to overcome. We will simply add a small air conditioner to the broom closet and buy a big UPS system that will keep the server alive in the event of another power outage, and we can hire a service to come by every morning to pick up tapes and deliver them to an offsite bunker. Instead of a single connection to the internet we can buy two and have redundant connections also.”
“Sir, I am no accountant, but by the time you pay someone to keep up with the depreciation of a new server, buy and install a small A/C unit and UPS unit, pay for a 2nd internet connection that will sit idle and pay a service to DRIVE here daily I really think the outsourced server would be cheaper. Not to mention in the event of data loss we could get the data restored to the server much quicker than waiting on a service to physically bring it to us.” An interesting note here is, I don’t care what kind of offsite data bunker you have, the Monster in Cloverfield IS going to destroy it so think multiple copies of data in multiple cities!
“Well I have made my decision; we will not be outsourcing my very valuable data – Hackers might get it, it is more secure here, so leave my office. Before you go could you please try to get my printer working again, and I am getting this annoying pop-up about spyware and it seems that my ITunes files have lost their license and I used to have a folder called Docs on my desktop with everyone’s salary in it that is missing and my PDA will not sync…(zzzzzzz) — OUTSOURCE IT!

Hard disk drive manufacturers recognized both the reality and the advantages of being able to predict normal hard disk failures associated with drive degradation sometime around 2003. This led a number of leading hard disk makers to collaborate on a standard which eventually became known as SMART. This acronym stands for Self-Monitoring, Analysis and Reporting Technology and when used properly is a formidable weapon in any system administrator’s arsenal.

The basic concept is that firmware on the hard disk itself will record and report key “attributes” of that drive which when monitored and analyzed over time can be used to predict and avoid catastrophic hard disk failures. Anyone who has been around computers for more than a day knows the terrible feeling that manifests in the pit of your stomach when it becomes apparent that your server or workstation will not boot because the hard disk has cratered. Luckily, we ALL of course back up our hard drives daily! Right?

All kidding aside even with a recent back up just the task of restoring and getting your system back in working order is a serious hassle and it’s not something you get the luxury of scheduling if the machine is critical to operations and failed in the middle of your work day or worse yet, the middle of your beauty sleep. That is where SMART comes in. When properly used SMART data can give “clues” that a drive is reaching a failure point–prior to it failing. This in turns means you can schedule a drive cloning and replacement within your next regular maintenance window. Really aside from a hard disk that lasts forever what more could an administrator ask for?

SMART drive data has been described as a jigsaw puzzle. That’s because it takes monitoring a myriad of data points consistently over time to be able to put together a picture of your hard disk health. The idea is that an administrator regularly records and analyzes characteristics about the installed spinning media and looks for early warning signs that something is going wrong. While different drives have different data points, some of the key and most common attributes are:

• head flying height
• data throughput performance
• spin-up time
• re-allocated sector count
• seek error rate
• seek time performance
• spin try recount
• drive calibration retry count

These items are considered typical drive health indicators and should be base-lined at drive installation and then monitored for significant degradation. While the experts still disagree on the exact value of SMART data analysis, I have seen sources that claim at least 30% of drive failures can be detected some 60 days prior to the actual failure through the monitoring of SMART data.

Of course not all drive failures can be predicted. Plus some failures are caused by factors other than drive degradation. Consider drives damaged by power surges or drives that are dropped in shipping as good examples of drive failures that cannot normally be detected through SMART monitoring. However in my humble opinion even one hard disk failure prevented over the course of my career is something to celebrate–unless you happen to own stock in McNeil Consumer Healthcare, a.k.a. the distributors of Tylenol!

So what does this have to do with SoftLayer? Well I am certainly not claiming that SoftLayer is going to predict all your hard drive disasters so there is no reason for you to back up your data. In fact, I recommend not just backing it up but backing it up in geographically disparate locations (did I mention we have data centers in Dallas and Seattle?). What I do mean to share is that technologies like SMART data are just one of the many ways SoftLayer is currently investigating to improve what is already the best hosting company in the business.

I should know. I was tasked with writing the low-level software to extract this data. That’s right. SoftLayer has engineers working at the application layer, down at the device driver layer, and everywhere in between. If that doesn’t give you a warm fuzzy about your hosting company, I don’t know what will.