Well, we’ve done it again! Last week we had a record SLales day and a monumental achievement for Team Softlayer. We sold and fully delivered 208 servers last Monday breaking our previous record of 117 in a single day. There will be no official press releases about this or anything like that other than this blog, this is normal day to day operations here at Softlayer, or as I like to call it – Monday.
We are indeed making quite a dent in the hosting industry, who else out there can claim that they can accomplish this colossal achievement? We are growing at a massive rate here and all the while keeping the same high level of service that we have worked extraordinarily hard to earn and maintain.
We’ve come a long way since the opening of our doors 2 years ago. However we still have a long way to go. We have our Washington DC datacenter coming online in May to offer better service to our East Coast and European clients and we have integrated our new API 3.0 for our clients to have even more control over there servers and our services never before seen in the industry today. 2008 is going to be an astounding year for our clients as well as Softlayer.
Here is the much anticipated Song of the Year. So to keep up with new DC openings and doing a blog about it, I felt I had to attempt to out do Sleepless in Seattle. I believe with a little help from Dave we have done just that! Enjoy!
When I was 14 I got my first tech job as a tech support guy for a local “mom and pop” internet service provider, from there on out I have been in many data centers in the North West working with multiple companies of all caliber. From National Dial-up Internet Service Providers to small webhosting companies that have had their stuff collocated in many of the area’s datacenters.
When I was about 20 I decided I was burnt out on the internet and wanted to try Central Office build outs for a national telecommunications company installing their fiber and DSL network in Washington and Oregon. The one thing that I learned in the Telco industry is to do nice and neat work. Work that you could trace a single cable in a bundle and follow it from point A to point B.
After a few years of doing the same thing over and over, I figured it was time for me to get back into the Internet as it was way more challenging for my ever-thinking mind.
So I took my nice and neat skills and worked on a contract for Microsoft building out a data center in a top secret location in the Puget Sound. This was by far one of the nicest and cleanest datacenters I had ever seen. After that I went to work for some other area datacenters doing systems administration work. I helped them do a migration of two datacenters into one. I helped build out a datacenter, and I helped by trying to make the datacenter as nice as Microsoft’s along with as neat as the Telephone companies COs.
During this time I really noticed SoftLayer Technologies was Ahead of the Rest when it came to the internet utility hosting Industry. I quickly wanted to learn everything about this company, and being the nerd that I am, figured I should buy a server from this company. I Bought one and went to lunch thinking I might have a call or e-mail saying that my server will be done here within the day. Wow! 45 minutes later? “These guys are on top of it”, I thought.
Then one day I was browsing Webhostingtalk.com (this is my equivalent to your teenager’s myspace.com addiction) and noticed that SoftLayer just released a P.R. about signing a deal with InterNAP for a 10,000 server datacenter in Tukwila so I figured this company’s features are so freaking amazing and cool. “I just need to try to get a job at this location with this really cool company”, I said to myself. I sent off a Resume and a little info about myself. I did not hear back from them for a while. I figured my quick-witted humor may have rubbed the HR department the wrong way, or maybe I wasn’t qualified, or too qualified.
SoftLayer finally called me back. I was as happy as a 10 year old getting a dirt bike for his birthday — they wanted an interview.
So I go in and tour the facility and do my interview with the interviewing committee, I have to say it was one of the most intense interviews I have ever had with the technical questions that was asked along with just a hard interview process, though I left that day knowing I would be getting a call from SoftLayer as I felt I sold myself to them on my skillset.
I have to say it is really relaxing and challenging working for a world-class company in a world-class datacenter. There is a great deal of stress that comes with our job in this industry, and when the datacenter and management have everything in order from the get go and it hasn’t been patched together it makes your job as a Systems Administrator a little less stressful. I do my daily walks of the datacenter in Seattle looking at thousands and thousands of racked servers that are set to standards which is weird when I’ve worked for places that use motorcycle tie-downs and zip ties to secure your rack to make them ‘Earthquake’ ready.
I now sleep at night knowing if there is an earthquake we will be prepared and your data and machines will be safe in SoftLayer’s Seattle N+1 datacenter. We have a wonderful team of build engineers and systems administrators that work around the clock to keep your virtual datacenter up and running. I wouldn’t want to be at any other place for 40+ hours a week!
By Romeo Rodriguez on Thursday, February 21st, 2008
As you may have guessed SoftLayer isn’t just sales team members, Datacenter Managers, and Development team members. There is also a pretty important group of people who hideaway in their cubicles and can be seen running around our state of the art server rooms from time to time. I am of course talking about us DC Techs; you might know us from our ticket signature “SoftLayer CSA.”
I had a question brought up for the first time while on a phone call with a customer, his question was,
“What is it like to be a Datacenter Technician?”
I could only laugh just a little bit as I looked around the office and saw several of my co-workers engaging in the organized chaos we call Datacenter Operations. You see, with datacenter operations there is no “daily routine” to follow, there isn’t a “what to expect” sheet posted somewhere to prepare us for the day. We have to rely on experience and each other to keep our beloved customers happy. So would you like to know my answer to this customer?
“It depends on the ticket I’m working!”
I say that because this particular customer was calling about a networking issue. In this instance I was his “network engineer”, helping him resolve an issue with secondary IP addresses. As I said before, not every issue is the same from one minute to the next so it keeps us on top of our game. One second I am a networking engineer, the next a hardware technician, the next a Systems Administrator. On some occasions we DC techs can be all three at once! It’s because of this fact that I enjoy coming to work each and every day. I never know what problem will arise or what I will learn in the coming hours.
I decided to write this after a very long shift, because I think a lot of our customers and people who read this blog would like to know what exactly it’s like. Of course there are good days and bad days, sometimes we make mistakes or take a little longer to reply to a ticket than we should. But for the vast majority of the time, our phone calls are ending with “Thanks so much!”, and our tickets are ending with “Great Job, You guys are awesome!”, and our customers are going to sleep knowing their server is in good hands.
Now what question do all of us DC Techs have? That’s simple:
What is it like being a SoftLayer customer?
Judging by everything I have seen recently, with our company expanding to Seattle, building new datacenters, and shattering several of our own sales records, I think we’re doing a pretty good job of putting everything you want from a dedicated hosting provider at your fingertips. There is always work to be done, and I speak for everyone here in the office when I say the most important thing to a DC Tech and the company as a whole are our bosses, and we currently have around 4,500 of you around the world and growing!
In Steve’s last post he talked about the logic of outsourcing. The rationale included the cost of redundant internet connections, the cost of the server, UPS, small AC, etc. He covers a lot of good reasons to get the server out of the broom closet and into a real datacenter. However, I would like to add one more often over looked component to that argument: the Spares Kit.
Let’s say that you do purchase your own server and you set it up in the broom closet (or a real datacenter for that matter) and you get the necessary power, cooling and internet connectivity for it. What about spare parts?
If you lose a hard drive on that server, do you have a spare one available for replacement? Maybe so – that’s a common part with mechanical features that is liable to fail – so you might have that covered. Not only do you have a spare drive, the server is configured with some level of RAID so you’re probably well covered there.
What if that RAID card fails? It happens – and it happens with all different brands of cards.
What about RAM? Do you keep a spare RAM DIMM handy or if you see failures on one stick, do you just plan to remove it and run with less RAM until you can get more on site? The application might run slower because it’s memory starved or because now your memory is not interleaved – but that might be a risk you are willing to take.
How about a power supply? Do you keep an extra one of those handy? Maybe you keep a spare. Or, you have dual power supplies. Are those power supplies plugged into separate power strips on separate circuits backed up by separate UPSs?
What if the NIC on the motherboard gets flaky or goes out completely? Do you keep a spare motherboard handy?
If you rely on out of band management of your server via an IPMI, Lights Out or DRAC card – what happens if that card goes bad while you’re on vacation?
Even if you have all necessary spare parts for your server or you have multiple servers in a load balanced configuration inside the broom closet; what happens if you lose your switch or your load balancer or your router or your… What happens if that little AC you purchased shuts down on Friday night and the broom closet heats up all weekend until the server overheats? Do you have temperature sensors in the closet that are configured to send you an alert – so that now you have to drive back to the office to empty the water pail of the spot cooler?
You might think that some of these scenarios are a bit far fetched but I can certainly assure you that they’re not. At SoftLayer, we have spares of everything. We maintain hundreds of servers in inventory at all times, we maintain a completely stocked inventory room full of critical components, and we staff it all 24/7 and back it all up with a 4 hour SLA.
Some people do have all of their bases covered. Some people are willing to take a chance, and even if you convince your employer that it’s ok to take those chances, how do you think the boss will respond when something actually happens and critical services are offline?
If you haven’t been to a Whataburger, I’m sorry. It’s an amazing fast food chain that sells not only the freshest made-to-order burgers, but they’re also open 24-hours a day, and their breakfast is second to none (Chris Menard has a clinical addiction to their taquitos). The problem with this is that they only exist in the South. I’m in the North. In Seattle, Washington to be precise—accompanied by our go-live team to manage our newest datacenter and make sure the launch goes smoothly.
On the bright side (no pun intended, it hasn’t stopped raining since we landed), it has. We have assembled an amazing team, the datacenter is absolutely spectacular, and the locals have been very friendly. Efficiencies we have built into our normal daily operations over the last two years have basically allowed us to “drag and drop” our datacenters as needed, where they are needed without having to reinvent the wheel every time we launch. Since the deployment is simple, we can focus on service upgrades—like the latest 40-Gigabit rack-level connections—while we roll out a new facility. Connectivity you could use…say…to look for a Whataburger near you http://www.whataburger.com/one_near_you.php (I look every day). We’ve already flown through our first historic Seattle Truck Day, and had a second one to boot. We’re provisioning droves of machines for new and current customers who are taking advantage of our network architecture, tools, and StorageLayer to create their own custom solutions. In a nutshell, we have brought a new DC online and maintained the ability to provide our customers with the same cutting edge hardware and innovative utilities that they have come to expect in Dallas.
On the darker side, with everything is going so well, it leaves a lot of time to sit and think about a tasty Whataburger. With jalapenos. And bacon. Ugh.
