devopsy.kinsta.cloud

Tag: policy

  • There are Few Enough Silver Bullets

    There are Few Enough Silver Bullets

    I was working through this week’s blog this morning, and it was laser-focused on a narrow topic. I had examples of why too much of a good thing is bad, how absolutism about methodology is hurting the majority of organizations out there, and how to get past this issue to keep improving what IT does for the organization.

    And I was nearly done when I decided that there are far too many areas of IT that suffer from absolutism, so I’d offer general symptoms and steps that work to get out of any of them.

    What do I mean by ‘absolutism’? Well, it used to be—and, in a couple of areas, still is—true that people would say, “We are an X shop.” And mean it. If you were a Cisco shop in the 1990s or 2000s, that is all you bought. Competitors were not allowed; nor was consideration of competitors. This applies to methodologies as well as product categories. In fact, today it applies to various methodologies more than products.

    And it’s bad for the organization, no matter where it comes from. While with vendors, it streamlines purchasing and offers the (in)famous “single throat to choke”, it also causes inefficiencies in costs (from your perspective, they are a monopoly, and some large vendors gained infamy by trying to take advantage of that fact), and causes people to do things in the least efficient way because that’s what the chosen methodology/solution requires.

    Sometimes, we manage to do the dance and avoid “One True Way”—take the move to cloud and continuation to multi-cloud, for example. While cloud vendors do not offer customers much leeway in pricing or priority (we customers are all cattle, not kittens), multi-cloud still offers us the opportunity to use the cheapest or best-suited cloud vendor without much drama.

    Many times, we are not so effective at avoiding the idea of a silver bullet. Take Agile. If there was a One True Ring of this decade in IT, I’d want it to be containers, but it would actually be Agile. Most IT departments zealously guard that Agile is the One True Way. Early-stage new product development is astoundingly well suited to Agile, and we needed it—if just for that part. Late-stage product maintenance, where changes are not feature-laden monuments to systems complexity, are also really well-suited to Agile and we should be happy to have it. In between? Large updates that can span data centers—or even continents—with a geographically dispersed workforce and hundreds (or more) of hours of development to get one sub-feature completed? Not really a great fit for Agile, and while many of us have said that for years, here is hoping things are changing. As I’ve said before, as maligned as waterfall was, it did the job for decades and handles large complex projects well. Maybe we need a replacement more in line with the times; one that incorporates what we’ve learned from Agile. I’d love that, actually. But what we don’t need is “We’re an Agile shop.” It is less useful than when it was said about products (at least if you only chose a single product, you got single sourcing and streamlined purchasing. With methodologies, you don’t even get that).

    Note: Agile is an example, not the point of the post. Avoid “One True Way” across your organization. Root it out and examine it. Sometimes, a single solution with a single specialized skillset is the perfect answer—at least for now. Far too often, “We are an X shop” is someone’s preference codified into corporate policy.

    To get past One True Way, you have to be there showing its weaknesses to management. There is some skill overlap when allowing multiple approaches or products, but if management starts prattling about that, simply point out the number of development and/or CI/CD environments the org is currently supporting. To show the overhead, track hours spent. Staff shortages have been endemic to IT for years. Make certain you know how much time is spent in overhead for the product or process, and try to map that to what could have been achieved with that time. It’s not perfect, but it will be illustrative. And don’t stop. Do your work, but point out every time that One True Way makes the job harder or more expensive.

    You are nailing it. Don’t let things like Agile keep you from churning out great apps for the company. I’ve been on teams where the time spent in process overhead was dragging down the ability to deliver quality. Fight that. Keep kicking it, and use the best tools/processes for the job.

  • OpenAPI Specification: Perception vs. Reality

    OpenAPI Specification: Perception vs. Reality

    The OpenAPI Specification (OAS) (formerly known as the Swagger specification) provides a way to describe and document REST APIs and their components. It includes details on endpoints, their operations, parameters needed for the operations, expected responses for every operation, authentication methods and even annotations. OAS is an easy format to learn and read, and can be understood by both humans and machines. As you might imagine, this kind of “accessible” documentation is useful and beneficial in a number of different use cases for both development and security teams.

    Developer Relationships With OAS

    For developers, OAS can be used to generate documentation for APIs. Proper documentation can help other developers figure out how best to integrate and use these APIs with their own application.

    Even so, many developers have a love/hate relationship with OAS. Let’s face it, documenting is a far cry from developing. Although it is easy to learn, the actual act of documenting is tedious, manual and time consuming. Frankly, it’s probably one of the least glamorous parts of a developer’s job, and, therefore, many developers only do the absolute minimum. As a result, OAS can be incomplete or, in some cases, missing entirely. Not to mention, when the API is updated, how often is the OAS updated, too?

    In the case of incomplete or nonexistent documentation, developers will muddle through with trial and error, even though this can slow down the development process.

    If out-of-date OAS proves an annoyance for dev teams, what’s the impact on security?

    Security’s Relationship With OAS

    OAS can be consumed by both humans and machines, and both are critical for security’s use of OAS.

    OAS helps security teams (humans) become aware that the API exists, since its documentation is often part of the CI/CD process.

    By examining an OAS file, a security team can understand what the API is supposed to do and, from that, determine if it conforms to or violates the organization’s security policy. As an example, security teams might use the OAS to make sure that authentication and authorization are being implemented correctly, and that sensitive data, like PII, is not being exposed.

    Security teams might also deploy tools (machines) that use the OAS to enforce policy based on what’s in the documentation. As an example, the OAS may specify that, for a particular parameter, a string should be expected; it should be no more than four characters and it should contain only letters and numbers with no special characters. By validating input based on what is defined in the OAS, security tools can check that the API does what developers intend it to do.

    This plan sounds great on paper, but …

    Perception versus Reality

    Imagine a perfect world, in which every API is well-documented as part of the development process. As that API makes its way through the development pipeline, it does so hand-in-hand with an associated OAS. At the appropriate time, before production, security is made aware of the API and the OAS, the team evaluates the details, validates that the API matches policy, gives the thumbs up and the API goes into production. Security and development teams live in harmony and everyone is happy.

    We don’t live in a perfect world. Instead, we must deal with questions such as:

    • Does the OAS exist? In the most extreme cases, the OAS doesn’t even exist. It was never created as part of the development process, so it can’t be used for any sort of awareness. These misses happen more often than you think, giving rise to the problem known as shadow APIs. These APIs slip through the cracks, and make it into production with no documentation at all. While you might be saying to yourself, “Our company would never let this happen!” you should also consider that numerous APIs are stood up by development teams as a test, as part of a beta or as a quick-and-dirty way to get something done. There are more of them than you think, and these shadow APIs create unrealized risk.
    • Is the OAS complete? In slightly less-scary territory, let’s consider an incomplete OAS. You might be saying to yourself, “We have a top-notch development team, well-defined development processes and strict policies around documentation. No API makes it into production without an OAS!” Perhaps, but the devil is in the details, and those details are often missed. I’ve worked with countless customers who have happily and confidently produced an OAS in which critical details don’t match the reality of the API. Sometimes they’re not complete. Sometimes they’re missing all together. The industry calls these missing details shadow endpoints. Yes, someone did produce an OAS, and security was aware of the API, but a lot of critical detail is missing.
    • Is it being kept up-to-date? Development cycles seem to be moving faster and faster, and as documentation is one of the least glamorous parts of a developer’s job, updates can be missed, dismissed or, at the very least, lag behind. We commonly see multiple versions of an API, and the OAS does not reflect the new capabilities of the latest version.

    While a non-existent or outdated OAS presents a modest concern for API visibility, it creates a substantial concern when trying to serve as the foundation for API security.

    You take a huge risk if you depend solely on OAS for security. More often than not, an OAS is not accurate, lacks critical details and is only as useful as the data you feed into it. Gaps, including shadow APIs, leave out critical details, making enforcement impractical.

    The inaccuracy of OAS leaves customers with two choices. You can either enforce security policies based on strict adherence, and risk blocking legitimate traffic, or you can enforce based on loose adherence and risk letting attacks through. In either scenario, your organization is unduly vulnerable, and significantly less secure than your dev teams intended or your security teams would like.

    OAS is Only a Starting Point

    OAS provides a great starting point for documenting your APIs. Don’t make the mistake, however, of basing your API security strategy on what’s in your OAS. It’s simply not accurate enough. Taking that approach will leave your company’s vital data and services at the mercy of API attackers who will quickly know more about your APIs and what they expose than your OAS does.

  • Challenges in Leading a DevOps Team at a Fortune 100 Company

    Challenges in Leading a DevOps Team at a Fortune 100 Company

    A manager, an evangelist and a godfather all walk into a bar.

    Okay, this isn’t a bad joke, this is an article about how managers can implement DevOps in large organizations, especially where culture and organizational change hamper efforts. But, I promise, we will talk about managers, evangelists and godfathers.

    Uldis Karlovs-Karlovskis (@UldisKK) works at Accenture Latvia. Accenture is a global consulting giant, employing 400,000 with teams and clients across the globe. Uldis helped bring DevOps into the organization and shared his journey—the good and the bad—at theAllDayDevOps conference in his talk titled, “Lessons in Leading a Fortune 100 Team to a DevOps Philosophy.”

    Uldis joined Accenture in 2012 and faced three top challenges:

    1. How to maintain quality at such growth?
    2. How to enable continuous learning?
    3. How to stay fast and flexible at their size?

    His team was small, they held daily stand-ups that were productive, and they were flexible. Life seemed good, and then his team grew into a “spider.” They still held daily stand-ups, but they were useless because it was just people reporting and no one understood what is going on.

    What did Uldis do? Well, he didn’t freak out, if that is what you are wondering. Actually, he did. One morning he, “woke up, realized I had 25 people under me, freaked out and skipped the job for the day.” He walked around, thinking about it and wondering what to do. He thought about his three to four types of DevOps projects and all of the non-project activities that were monopolizing his time. He realized he focused on the urgent over the important tasks, working 10 to 12 hours per day.

    What is a manager to do? Have people help him, of course. Uldis decided to create an unofficial role that sounds interesting: the subject matter expert (SME). They would work on non-project activities, increase team engagement, improve satisfaction and assess DevOps maturity in projects. Initially, people were motivated to be SMEs, but they were always busy with other things. So the effort failed.

    Uldis had a new idea: Instead of him leading the SME team, he let them decide who wanted to do what, which SME team they wanted to join and who would be the lead SME for each team, and let them set their own goals and how they wanted to work, etc. This helped, but still didn’t solve the problems. Uldis was still burdened with the tyranny of the urgent.

    Back to the drawing board, and a new idea was born: evangelists. Uldis broke down 50 SMEs into seven subgroups and added a little hierarchy because the flat structure wasn’t working. Each evangelist led a guild—one of seven. Team members were to talk to their evangelist before coming to Uldis with questions.

    But, alas, Uldis was still just handling escalations and talks.

    So, what do you do to bring order? Bring in Godfathers.

    Now, team members could always go to the Godfather for help. Uldis decided that, generally, each person would have four topics they may want to talk about. For each topic, they have a person to talk to:

    • SME addresses how to contribute to the company in general
    • Godfather addresses escalations, such as not happy with the team lead
    • Evangelist addresses questions about the person’s skills
    • Team Lead addresses project tasks

    Uldis now realized the benefit of this approach: In a way, now, everyone is focused at some level on sales. As we all know, this is important yet very difficult in any organization—but especially when everyone is an engineer.

    While his approach has helped him and his team, he also acknowledged, “If I did the same thing today, I would fail, but maybe for different issues.”

    Todat, Ulids is looking at improving his culture. As we talk about at AllDayDevOps time and time again, good culture is key to successful DevOps. Uldis, looking at Accenture, noted it doesn’t fit into a typical organizational structure. Reading literature, industry buzz, etc., he realized something was missing from these models, but he wasn’t sure what. He knew they needed to be better.

    He discusses three dominant types or organizational cultures:

    • Pathological – Eight hours of work and your job is done
    • Bureaucratic – Two to 12 hours of work if the process requires it
    • Generative – Two to 12 hours of work and I’ll deliver it

    Uldis is trying to be at the third approach, and the key to achieving it is knowing that if you are having a bad day, you should leave your work for tomorrow when you will be more productive. Tomorrow it may take more than eight hours, but you will put your best into your work.

    His journey is ongoing, but he finished his talk covering a few things they do on his team:

    • Onboard new joiners through a continuous delivery pipeline
    • Meet weekly at lunch
    • Half-yearly performance assessments

    He also provided four takeaways:

    1. Everyone by default is good
    2. Intrinsic over extrinsic
    3. Engagement is your work responsibility
    4. Let them lead

    Uldis told an honest and open story about his journey with the hope it will help others. To hear it firsthand, watch it for free here. You can watch any of the 2017 AllDayDevOps sessions free of charge here.

    — Derek E. Weeks

  • SecDevOps: Security Automation By Example – The Firewall Change

    SecDevOps: Security Automation By Example – The Firewall Change

    Security Automation By Example

    The Firewall Change

    Just when you thought DevOps was the new black, along comes SecDevOps.  Yes folks, like most things in life, the new cool is already here.  Before I move on to trying to explain SecDevOps, please accept my mea culpa because for many people DevOps is yet to be clearly defined.  I can imagine your frustration that I’m going to try and explain a new term based on a loosely defined term.

    I’m not going to try and provide a complete definition of SecDevOps, but instead I’d like to provide what I believe to be important attributes of the SecDevOps movement.  In this post, lets discuss security automation and use a firewall change process as an example.

    One important attribute of SecDevOps is heavy use of security automation.  In particular when I think security automation, I tell people to use IFTTT thinking.  Stolen from the service whose tagline is “put the Internet to work for you”, I explain security automation simply as “if this, then that”.  Its a highly simplistic, but poignant way to imagine ways in which one would automate workflow.

    A simple act of changing a firewall has huge potential for security automation.  The SecDevOps firewall change process goes beyond changing a few lines in a config, but also provides an independent and automated verification process that most organizations don’t perform today.

    Consider what happens when a firewall policy is changed.  First the policy change request is reviewed and approved.  Then the policy change request is assigned and acted upon.  Many times, but not always, the change is verified by another person through configuration review.

    Security automation takes this process one step further.  Imagine when a firewall is changed that a security audit event is automatically logged.  Next, an automation component notices the firewall change and automatically spins up a remote scanning instance in a cloud provider.  Next, that scanning instance is contacted using APIs to perform a remote network scan.  The scan completes, results are exported and compared with prior scan results automatically.  Finally, the automation component automatically updates the firewall change request in a ticketing system showing the scan results.  We have now completed an automatic, independent and autonomous firewall change verification.   That simplified yet powerful workflow is security automation and completes what many people refer to as a closed loop.

    The new SecDevOps method for a firewall change not only provides a closed loop process, but does so by automating the verification steps.  Joining traditional security processes with new API driven cloud services is a true force multiplier that many organizations will soon wish they had.

    In upcoming posts, I’ll also discuss some of the other important components of what I believe make up the new black of SecDevOps.