Safehaven Run Book Automation – A small change with a big impact

In an earlier post, we discussed SafeHaven for CenturyLink Cloud Disaster Recovery- as-a- Service (DRaaS) solution and the benefits it offers IT Administrators.  As we noted, failing over a multi-tiered application when executing a disaster recovery plan is critical. However, it’s not always as easy as it seems.  In order for a multi-tiered application to recover correctly, the VMs upon which it depends start up according to a prescribed “recovery plan.”  For instance, it is usually necessary for the database to be running before application servers boot and, similarly, necessary that application services be running before webservers boot. Safehaven for CenturyLink Cloud’s latest feature enhancement, Run Book Automation, allows end users to configure custom shut-down and bring-up plans for each group of IT systems that received disaster protection.  For example, delivering web services often involves a set of interdependent workloads that need to start in a specific order and taking into account time intervals between applications.

For each group of IT systems within the CenturyLink Cloud, users can pre-configure and test recovery plans in the Safehaven Console that identifies bring-up and shut-down order, actions, delays, as well as any custom script to be executed as part of the recovery operation.  Once the plan has been loaded, executing becomes as simple as a point-and click operation.

For each group of IT systems within the CenturyLink Cloud, users can pre-configure and test recovery plans in the Safehaven Console that identifies bring-up and shut-down order, actions, delays, as well as any custom script to be executed as part of the recovery operation.  Once the plan has been loaded, executing becomes as simple as a point-and click operation.

Conversely, when the original data center recovers after a disaster event, the challenge becomes how to efficiently restore protected IT systems so that they resume normal operations in the production environment.  With this “failback” challenge, SafeHaven also provides full Run Book Automation. Users use the same interface shown above to specify custom recovery plans for failback to the original production data center.  So why is this seemingly minor feature so important?  Well, because in business time is money.  And when your applications are down, you are losing money.  Aberdeen recently released a report that stated mid-size businesses experience on average 3.5 downtime events per year, lasting 3.4 hours and costing over $880,000.¹ This equates to about $1,233 for every minute your business applications are not running when they should.  Let’s say, for example, that through the nature of Safehaven’s Run Book Automation allowing multiple applications to be brought up at the same time and nearly eliminates the risk of human error during the process, you save 15 minutes of downtime a per hour.  Using the assumptions above, this means that implementing a disaster recovery (DR) plan with Safehaven for CenturyLink Cloud could save your business $18,500 per hour in downtime costs.  And, when you take into account the average number of events and downtime, your estimated savings in this example become $220,150.  That’s no small chunk of change.

Make sure your DR solution is intuitive enough to protect your business both technically and financially.  Learn more about Safehaven for CenturyLink Cloud here.


Inside Look at CenturyLink Cloud Development Center

Cloud. It’s a disruptive force. Here in Seattle some may think of it only in terms of weather, but at CenturyLink we know it’s something much greater and its impact on our organization is being felt company wide. It’s been less than a year since CenturyLink acquired Bellevue-based Tier 3 and branded it CenturyLink Cloud. At the time we announced our plans of opening the Cloud Development Center in the Seattle area, saying “Tier 3’s products, roadmap and vision are now the foundation of CenturyLink’s cloud strategy and anchor the new Seattle-based CenturyLink Cloud Development Center.”

Why Seattle? It turns out Seattle is the center of the cloud universe. Forbes ranked it the best city for tech jobs. We all know that Amazon has an enormous campus in South Lake Union, Microsoft is headquartered in Redmond, but did you know Google’s development center for cloud is based in Seattle and Kirkland? And, since the entire country believes it rains here 24/7, can you think of a more natural place for the Cloud Development Center to be placed?

Officially opening on October 14th, the CenturyLink Cloud Development Center encompasses almost 30,000 square feet in Bellevue, WA. Beyond its partnership with CenturyLink Field – go Seahawks! - CenturyLink has truly invested in our presence in Seattle. Today our more than 1,500 local employees serve our consumer and business customers with services ranging from 1 GIG broadband services to our three local data centers offering a wide-range of colocation-to-cloud. At the Cloud Development Center the engineering staff, which has doubled in size since November, will grow to more than 250 professionals. Anchored by a Cloud Executive Briefing Center, the Cloud Development Center will host CenturyLink clients from around the globe for discussions surrounding the evolving role of cloud in Hybrid IT strategies. The Cloud Development Center will also be a hub for community events such as startup up and developer meet ups – one example is the upcoming Cloud Foundry Meet Up we’ll host on October 16th.

Form Meets Function:  Designing for Today’s Workforce The Cloud Development Center features a flowing, open design, which is reflective of our employees’ personality.

CenturyLink Cloud Office Space

One cultural artifact:  During the job interview process, all potential employees are asked to name a favorite movie and why. Yes – this is all about understanding how the potential employee thinks. This movie question has become so ingrained into the new center that meeting rooms are named after movie places: Death Star, Jack Rabbit Slims, Thunderdome and the Bat Cave. And, the break room wall (pictured) has a few favorite lines on it – all suggested by employees. Can you name the movie they came from? (See the end of the article for the answers) There is also a little local flavor; the tables pictured were fabricated by the manufacturer who made the reclaimed distressed wood tables at CenturyLink Field.


Watch for more over the next month about the CenturyLink Cloud Development Center as we get ready to celebrate its Grand Opening on October 14th.

Movie Wall Quiz Answers: We have a hulk – The Avengers Do or do not, there is no try – Star Wars V – The Empire Strikes Back Fear is the mind killer – Dune Come get some – Army of Darkness

[UPDATED] “Shellshock” Vulnerability & What You Need to Know

A new vulnerability was recently identified in the “bash” shell that a default component of most Linux operating systems deployed globally today. This vulnerability – dubbed “Shellshock” - is being compared to what was experienced earlier this year with the Heartbleed bug because of the widespread use of the impacted Linux operating systems.

Shellshock has been assigned the highest risk rating of “10” according to the Common Vulnerability Scoring System (CVSS). Why? The vulnerability can be exploited across the network, it does not require any authentication to exploit, and exploiting this vulnerability is simple.

Unmanaged Customers - Patch Your Systems in the CenturyLink Cloud Immediately

If you have instances running a Linux operating system in CenturyLink Cloud data centers, you are likely affected.  Our unmanaged customers are responsible for day-to-day configuration and deployment of these systems, so it is the customer’s responsibility to remediate any affected systems.

We recommend you apply the updates for this vulnerability as quickly as possible. This is especially important for those servers running Apache web servers as there are published exploits already circulating for Apache websites.

Managed Customers – Request Patching via Ticket with Managed Services Help Desk

Customers running managed environments (including Apache) on CenturyLink Cloud will have their systems patched upon request. To initiate a request, open a ticket with the CenturyLink Cloud Managed Services team. CenturyLink hosting engineers and operations are currently working with multiple software vendors to enable the necessary critical patches for quick resolution.

Actions Taken by the CenturyLink Cloud Team

CenturyLink Cloud has assessed our infrastructure and we will be updating all OpenVPN servers with the patches that fix this bug. You will receive additional communication from us when those updates are scheduled. Any additional updates will be posted to this blog article so please check back regularly.

Information on Patches for Each Linux Distribution:

[2014-09-25 9:30AM PT] Original Post

[2014-10-06 11:29AM PST – All externally facing systems including customer OpenVPN servers managed by CenturyLink Cloud have been updated]

Private Cloud: Tradeoffs and Transformation

A few weeks ago, we announced CenturyLink Private Cloud – a new approach to the private cloud segment that offers breakthrough simplicity for large enterprises.  CenturyLink Private Cloud is designed for those looking to deploy a transformational private cloud, instead of eking out incremental gains.

Entering a new market segment is a significant undertaking for any product organization.  Every solution requires a series of trade-offs —just ask any product manager - and development of the CenturyLink Private Cloud is no exception.  So what were the trade-offs that we made, and how do they compare to other private cloud alternatives?  For a little insight into how we evaluated the private cloud market landscape and our decision making process, read on.

Recognizing the Challenges of Hybrid Cloud – Part IV

Hybrid cloud is becoming a standard operating model for many organizations. But how can you realize the expected agility when there are so many challenges ahead of you? In this series of articles, we’ve dissected each challenge and proposed some corresponding solutions. Whether you’re facing security and network concerns, or integration and system management issues, it’s critical to have a proactive plan in place. This final article rounds out the discussion by looking at ways to address the issues around portability, compatibility, and your existing toolset.

Solutions to Hybrid Cloud Challenges

In many cases, a hybrid cloud is the combination of complimentary – but not identical – computing environments. This means that processes, techniques, and tools that work in one place may not work in another.

Compatibility. Gluing together two distinct environments does not come without challenges. Now, it’s possible that you have the same technology stack in both the public and private cloud environment, but the users, technology, and processes may be dissimilar!

  • Move above the hypervisor. Even if your public cloud provider supports the import and export of virtual machines in a standard format, no legitimate public cloud exposes hypervisor configurations to the user. If you want to have a consistent experience in your hybrid cloud, avoid any hypervisor-level settings that won’t work in BOTH environments. Tune applications and services, and start to wean yourself off of specific hypervisors.
  • Consider bimodal IT needs. If you subscribe to the idea of bimodal IT, then embrace these differences and don’t try to force a harmonization where none exists. Some traditional IT processes may not work in a public cloud. If the more agile groups at your organization are most open to using the public cloud and setting up a hybrid cloud, then cater more to their needs.
  • Be open to streamline, and compromise. The self-service, pay-as-you-go, elastic model of public cloud is often in direct conflict with the way enterprise IT departments manage infrastructure. Your organization may have to loosen the reigns a bit and give up some centralized control in order to establish a successful hybrid cloud. Look over existing processes and tools, and see which will not work in a hybrid environment, and incubate ways to introduce new efficiencies.

Portability. One perceived value of a hybrid cloud is the ability to move workloads between environments as the need arises. However, that’s easier said than done.

  • Review prerequisites for VM migration. A virtual machine in your own data center may not work as-is in the public cloud. Public cloud providers may have a variety of constraints around choice of Operating System, virtual machine storage size, open ports, and number of NICs.
  • Embrace standards between environments. Even if virtual machines are portable, the environmental configurations typically aren’t. Network configurations, security settings, monitoring policies, and more are often tied to a specific cloud. Look to multi-cloud management tools that expose compatibility layers, or create scripting that re-creates an application in a standard way.

Tooling and Skills. Even if you have plans for all of the items above, it will be hard to achieve success without robust tooling and talented people to design and operate your hybrid cloud.

  • Invest in training. Your team needs new skills to properly work in a hybrid cloud. What skills are most helpful? Your architects and developers should be well-versed in distributed web application design and know what it means to build scalable, resilient, asynchronous applications. Operations staff should get familiar with configuration management tools and the best practices for repeatedly building secure cloud environments.
  • Get hands on experience. Even if you’re using a private cloud hosted by someone else, don’t outsource the setup! Participate in the hybrid cloud buildout and find some initial projects to vet the environment and learn some do’s and don’ts..
  • Modernize your toolset. The tools that you used to develop and manage applications 5-10 years ago aren’t the ones that will work best in the (hybrid) cloud today, let alone 5-10 years from now. Explore NoSQL databases that excel in distributed environments, use lightweight messaging systems to pass data around the hybrid cloud, try out configuration management platforms, and spend time with continuous deployment tools that standardize releases.

Taking the Next Steps

Hybrid cloud can be a high risk, high reward proposition. If you do it wrong, you end up with a partially useful but frustratingly mediocre environment that doesn’t stop the growth of shadow IT in the organization. However, if you build a thoughtfully integrated hybrid cloud, developers will embrace it, and your organization can realize new efficiencies and value from IT services. How can CenturyLink help? We offer an expansive public cloud, a powerful private cloud, and a team of engineers who can help you design and manage your solutions.