devopsy.kinsta.cloud

Author: Robert Reeves

  • Lessons Learned From the Salesforce Outage

    Lessons Learned From the Salesforce Outage

    Everyone has bad days. Every company has been through some kind of outage due to a buggy database deployment. Even the best of the best, with highly trained staff, world-class best practices and well-thought-out processes make mistakes. On May 17, Salesforce.com had a bad day.

    What Happened?

    The company deployed a faulty database change script that broke permission settings in production and gave users read and write access to restricted data. This opened the door for an unauthorized employee to steal or tamper with the company’s data. As a result, Salesforce needed to take large parts of its infrastructure down to find and properly fix the issue. The outage lasted 15 hours, 8 minutes. According to Gartner’s cost-of-downtime formula ($5,600/minute), this outage cost approximately $5 million. Plus, since so many companies rely on Salesforce, it was a very visible and embarrassing outage. (Just take a look at #SalesforceDown and #permissiongeddon on social media.)

    Salesforce had to shut everything down because of the way databases work. It’s not as easy as pulling a single application. Who knows how many Salesforce employees worked like mad to take the whole database down, find the offending database script, and restore everything—all because of one change script. That’s not a fun way to spend a weekend.

    What Does This Tell Us?

    Historically, Salesforce customers have experienced very little disruption in service. On the day of the outage, many loyal customers were tweeting about how rock solid the service has been, and that’s impressive.

    That being said, this outage should be a wake-up call for users to realize their dependency on their platform, which has become a more integral part of how we conduct business. I’ve heard anecdotes of entire offices being unable to complete work that Friday.

    The customer reactions show that they clearly have their stuff together over at Salesforce. What this outage shows is less about any shortcomings of this company specifically; but rather that everyone has blind spots, no matter how robust the testing process is.

    Lessons for IT professionals:

    • Don’t forget the database. Unfortunately, this problem is all too common. There are countless cases of a database change script being executed in production and causing unexpected issues. This indicates that the database is often the forgotten part of the software test and release cycle. Unless companies start to seriously consider the database when it comes to change management, things like this will continue to happen.
    • Automate everything. Manual efforts fail. Humans make mistakes. Companies such as Salesforce that handle sensitive customer data should not put it on the internet unless they have automated every aspect of the system, including the database. Companies need a robust DataOps process, one that includes production-like data and automates the validation database changes.

    Lessons for end users:

    • Establish a continuity plan. Organizations that depend on systems such as Salesforce need to have a business continuity plan to continue operations if they go down. This includes syncing Salesforce tasks and calendar with Office365, Exchange or Google Apps. However, since Salesforce has become the system of record for most companies that use the technology, having a backup of the data will be very difficult—after all, the point of a company using the platform in the first place was to avoid hosting its own data.
    • Demand accountability. Additionally, users can and should demand more accountability from software vendors. This can take the form of demanding a refund, but also customers need to speak to their account manager to understand what steps Salesforce is doing to make sure this does not happen in the future. It’s 2019 and companies update software all the time without taking down their production systems at the end of a quarter. Moreover, companies also can demand that companies don’t change production systems, outside of necessary security issues, during business hours.

    The bottom line is that, while the mistakes that led to the Salesforce outage are very costly and highly visible to customers, they are also entirely preventable.

    — Robert Reeves

  • Back to DevOps School: What Tools are in Your Backpack?

    Back to DevOps School: What Tools are in Your Backpack?

    The 2018-19 school year is in full swing, and just like the kids are going to the bus stop with their new backpacks and shoes, it’s time to refresh your DevOps tool chain and look to the future. Remember, just like education, DevOps never ends. It’s a lifetime pursuit that constantly improves your life.

    The DevOps tool chain of the future must include:

    Trunk Releases

    Time to get your branch and merge skills down pat. Git is the best source code control to offer this to development teams. Companies should be actively moving to git to allow their development teams isolated feature development with pull merges and pushes to the trunk for releases.

    If you are not using git, it’s time to start justifying why you are not. Given the speed at which businesses are moving, it is time eliminate blockers for development to release features. Git is built for speed and large numbers of dispersed developers. Remember: it was built by Linus Torvalds to support Linux kernel development.

    Database Development Tools

    When planning for your DevOps tools, seriously consider database development tools as well. But, if you want comprehensive results when developing your database, you need tools with a DevOps flare.

    If we didn’t have the efficiency and speed that we got from Agile software development, we wouldn’t have created DevOps. So, bringing Agile to database development was a must for database DevOps to solve problems. One tool in particular that we use supports some incredible features for Agile development. A database is different than an application; because of this, a database will have state. So, having a database instance we can compare to with our team is critical. The tool we use has this functionality, which has allowed us to thrive. It also lets users work with existing source code control along with a live database to help database developers. Because of this, users can execute this functionality against database objects such as stored procedures and tables.

    Application Release Automation

    Application software can be difficult to manage. Often, they have too many components to manage easily, which is where automation can help. Application release automation helps your team move from learned and taught processes to basic scripts that are easily scalable and can help deploy applications rapidly. If you’re working with a team that cares about data and DevOps, Application release automation is a critical tool.

    Test Data Management

    No matter how rapidly you are able to deploy applications, you’ll need to test. The faster you deploy, the more tests you will need to run. But a massive problem with testing can be the lack of quality. Many applications pivot functionality depending on the data stored in a datbase. So, providing data that is or, at least, matches the production data is critical to guarantee high test quality. Without the right data, you could be running massive amounts of tests that are also providing bad/false results. And, with DevOps, you’re committed to improving your software and working to enable more testing for that software.

    Don’t Forget the Database

    A key finding of the “2018 State of DevOps Report” is that integrating database change management into your DevOps toolchain early in the process drives higher performance.

    This is obvious to anyone that has released software before. However, for large siloed enterprises, this is non-obvious. For too long, we have been told to leave the database to the DBAs. Well, it turns out that approach (though sensible in the ’90s, when we started it) is no longer working for us.

    Your application will not run without database schema and logic changes about 80 percent of the time. So, there is no reason to have a separate manual process. Here’s someone saying that exact same thing in 2017: https://www.youtube.com/watch?v=DDZIiGT3jzs

    Now What?

    Having all the right tools is a necessary first step, but it doesn’t stop there. To fully succeed with these tools, a few key strategies must be put in place as well:

    One Ring to Rule Them All

    DevOps has taught us to break down silos. Yet, we see silos now being created based on tool choices. Companies should standardize DevOps tools across the entire value chain. If some parts—say Dev and Test—are using Artifactory, for example, the operations team should also use it and not rely on the release teams to put the release on a shared network drive. Ops should use Artifactory, too.

    This will require some horse-trading. Some teams will dig their heels in on a tool because it provides value specifically to them. Well, so be it. As one of my favorite Venture Bros. character, Dr. Henry Kissinger, says, “Compromise, my friend, is the essence of diplomacy, and diplomacy is the cornerstone of love.”

    Cloud First

    When I hear “cloud” from IT leaders, I immediately think of Inigo Montoya from “The Princess Bride” when he said, “You keep using that word. I do not think it means what you think it means.”

    The National Institute of Standards and Technology has defined cloud computing to have five essential characteristics:

    1. On-demand self-service
    2. Broad network access
    3. Resource pooling
    4. Rapid elasticity or expansion
    5. Measured service

    Your DevOps tools should also have these characteristics. Specifically, if you are not allowing external teams to utilize your tools and require someone to make a request via ticket first, you are doing it wrong.

    Time to learn how to offer DevOps as on-demand self-service.

    — Robert Reeves

  • DevOps: The (Absolutely Critical) Cloud Enabler

    DevOps: The (Absolutely Critical) Cloud Enabler

    A couple weeks ago, I gave a talk titled, “DevOps: The Cloud Enabler” at a conference. As I was putting together some requisite materials previewing my session, I made a few mistakes. (Yes, I’ll readily admit it.) I used the words “should” and “can” and “consider.” Instead, the words “must” or “not optional” or “operationally imperative” make better sense.

    Companies are moving to the cloud (or at least endeavoring to); that’s a fact of IT life. If you’re not doing it, your competitors are lapping you. That may not be what you want to hear, but hey, reality bites. The cloud is simply the most scalable, flexible, efficient way to run a business these days. Cloud was just a buzzword for the longest time, but companies in all industries are actually putting it to work now. If your CIO isn’t at least in the midst of forming an application migration plan to the cloud, it might be time to light a fire under him or her; otherwise, start reading the writing on the wall and look for a new gig.

    Easier said than done, though, right? I’ve heard many a CIO say, “We’re going to have all our apps in the cloud by 2018!” No, you’re not. You don’t know how. You don’t have the tools and road map to get there. Let me enlighten you with some of those stronger, more urgent words I mentioned earlier. In your move to the cloud, DevOps MUST be the conduit. Period. Stop. End of argument. Game, set, match. Not optional; it’s imperative.

    Here’s why: One of the most fundamental problems that’s part and parcel of a move to reliance on the cloud is that IT orgs want every scrum team to have its own environment, complemented by an individual database instance. Eventually, that leads to creating a distinct database instance for every single developer. You probably can see where this is headed.

    I’ve used this comparison time and time again, but cloud and database instances become like the wire hangers in your closet you accumulate every time you pick up clothes from the dry cleaner. They multiply over time and, all of a sudden, you seemingly have a million on your hands, with no idea where they came from and no good way to get rid of them. To compound things, once the proliferation begins, it’s hard to stop. Shut down a database instance (as with an application instance), and you lose the underlying data. No can do.

    That reality spells disaster for the database administrators (DBAs) being flooded with thousands of development SQL scripts and database changes across multiple app release cycles across multiple environments. For DBAs to attempt to keep pace with this rise in change requests, organizations all but have to go through DevOps’ economies of scale. DBAs need reprieve, and DevOps can be just that. David Linthicum put the solution thusly: “Stop separating the development and operations skills; they are tightly coupled and need to be peers. You’ll find this is the hardest part of the process of moving to devops and the cloud.”

    Spot on. To be more prescriptive:

    • Developers need to understand what DBAs are dealing with as part of the cloud migration and know how their work rolls downhill.
    • Automation needs to reign supreme, removing manual work.
    • Developers need to have the ability to be involved directly in cloud ops and enact 95 percent of changes themselves.
    • DBAs need to be freed to oversee the 5 percent of changes that truly need their attention and management skills.
    • The flow from cloud application updates to cloud DB changes needs to be seamless.

    In bringing development and DBAs into sync, DevOps then becomes the vital remedy, the critical cloud enabler.

    TL;DR: If you have DevOps alignment, processes, responsibilities and tooling properly implemented in advance, then when you decide to move to the cloud, you can simply flip the switch from on-premise to the cloud. One click and you won’t miss a beat. Seriously, it’s that easy. Sound appealing? I know.

    If you truly want to get to the cloud, you must have a sound DevOps strategy that encompasses application development, web and the database. Without that in place, swim at your own risk. This is the year that we’ll see much more of the true impact of DevOps. And that’s coming just in the nick of time, because without DevOps, CIOs and their IT orgs would be in for some nasty surprises in their all-important moves to the cloud.