Category: DevOps Practice

  • Best of 2019: Breaking Down the OWASP API Security Top 10, Part 1

    Best of 2019: Breaking Down the OWASP API Security Top 10, Part 1

    As we close out 2019, we at DevOps.com wanted to highlight the five most popular articles of the year. Following is the fifth in our weeklong series of the Best of 2019.

    As a result of a broadening threat landscape and the ever-increasing usage of APIs, the OWASP API Security Top 10 Project was launched. From the start, the project was designed to help organizations, developers and application security teams become more aware of the risks associated with APIs. This past September, the OWASP API Security Top 10 release candidate (RC) was finalized and published on OWASP.

    In my previous article, I provided a high-level view of the interaction between API endpoints, modern apps and backend servers, in addition to how they’re different from their traditional browser-based counterparts. I also discussed why this project was so important to the contributors and the industry overall.

    In this article, I aim to clarify the first five risks by highlighting some of the possible attack scenarios to help organizations and end-users understand the dangers associated with deficient API implementations. The following discussion follows the same order as found in the OWASP API Security Top 10.

    API1:2019 – Broken Object Level Authorization

    Attackers can exploit API endpoints vulnerable to broken object level authorization by manipulating the ID of an object sent within the client request. What this means is the client can request information from an API endpoint that they are not supposed to have access to. This attack normally leads to unauthorized information disclosure, modification or destruction of data.

    Example attack scenario:

    Say for instance there is an e-commerce platform that provides financial and hosted services to a group of different online stores (shops). The platform provides an API used to gain access to revenue charts for each of their hosted stores, and each store should only have access to their own revenue charts. However, while inspecting the client request from a single store who wants to gain access to their own revenue charts, an attacker can identify (find) the API endpoint for those revenue charts and identify the URL in use, for example /shops/{shop name}/revenue_data.json. Using the names of other stores being hosted on the e-commerce platform, an attacker can create a simple script to modify the {shop name} ID object in subsequent requests, and gain access to the revenue charts of every other store.

    API2:2019 – Broken Authentication

    Being different than authorization discussed above, authentication on the other hand is a complex and confusing mechanism concerning APIs. Since authentication endpoints are exposed to anyone by design, the endpoints responsible for user-authentication must be treated differently from regular API endpoints and implement extra layers of protection for credential stuffing attempts, in addition to brute force password and token guessing attacks.

    Example attack scenario:

    Suppose an attacker obtained a list of leaked username/password combinations as the result of a data breach from another organizations. If the API endpoint handling authentication does not implement brute force or credential stuffing protections such as CAPTCHA, rate-limiting, account lockout, etc., an attacker can repeatedly attempt to gain access using the list of username/password combinations to determine what combination(s) work.

    API3:2019 – Excessive Data Exposure

    By design, API endpoints often expose sensitive data since they frequently rely on the client app to perform data filtering. Attackers exploit this issue by sniffing the traffic to analyze the responses, looking for sensitive data that should not be exposed. This data is supposed to be filtered on the client app, before being presented to the user.

    Example attack scenario:

    Image an IoT-based camera surveillance system allows administrators to add a newly-hired security guard as a system user, and the administrator wants to ensure the new user should only have access to certain cameras. These cameras are accessible via a mobile app the security guard uses while at work. The newly hired security guard’s mobile app makes an API request to an endpoint in order to receive data about the cameras and relies on the mobile app to filter which cameras the guard has access to. Although the mobile app only shows the cameras the guard can access, the actual API response contains a full list of all the cameras. Using the sniffed traffic, an attacker can manipulate the API request to show all cameras, bypassing the filtering on the mobile app.

    API4:2019 – Lack of Resources and Rate Limiting

    It is common to find API endpoints that do not implement any sort of rate limiting on the number of API requests, or they do not limit the type of requests that can consume considerable network, CPU, memory and storage resources. The amount of resources required to satisfy a request greatly depends on the user input and endpoint business logic. Attackers exploit these issues causing denial-of-service attacks and associated endpoint outages.

    Example attack scenario:

    Let’s say an attacker wants to cause a denial-of-service outage to a certain API that contains a very large list of users. The users’ list can be queried, but the application limits the number of users that can be displayed to 100 users. A normal request to the application would look like this: /api/users?page=1&size=100. In this case, the request would return with the first page and the first 100 users. If the attacker changed the size parameter from 100 to 200,000, it could cause a performance issue on the backend database, since the size parameter in use is so large. As a result, the API becomes unresponsive and is unable to handle further requests from this or any other client.

    API5:2019 – Broken Function Level Authorization

    Although different than API1 above, exploitation of this issue requires the attacker to send API requests to endpoints that they should not have access to, yet are exposed to anonymous users or regular, non-privileged users. These types of flaws are often easy to find and can allow attackers to access unauthorized functionality. For example, administrative functions are prime targets for this type of attack.

    Example attack scenario:

    To illustrate this further, imagine during the registration process to a certain application that only allows invited users to join, the mobile app triggers an API request to GET /api/invites/{invite_guid}. GET is a standard HTTP method used to request information from a particular resource. In this case, the response to the GET contains details about the invite, including the user’s role and email address.

    Now, say an attacker duplicated the request and manipulated the HTTP method by changing GET to POST. POST is an HTTP method used to send information to create or update a resource. The URL would look like this: POST /api/invites/new/{“email”:”hugo@malicious.com”,”role”:”admin”}. In this case, the attacker easily exploits this issue and sends himself an email invite to create an admin account.

    In the context of the five risks above, one could easily image many similar attack scenarios. Those provided were just examples of the nearly unlimited possibilities when attacking vulnerable API implementations. As you can see, the risks above are primarily caused by errors or oversights. I believe these risks could easily be managed or nearly eliminated when organizations improve their secure coding practices, especially when it comes to the way they’re utilizing APIs.

    — Erez Yalon

  • Best of 2019: Implementing Message Queue in Kubernetes

    Best of 2019: Implementing Message Queue in Kubernetes

    As we close out 2019, we at DevOps.com wanted to highlight the five most popular articles of the year. Following is the fourth in our weeklong series of the Best of 2019.

    The adoption of Kubernetes as enterprises’ tool of choice is increasing very rapidly. Consequently, the architectural consideration and operational requirements to support production environments have gained increasing attention. A major challenge to consider when planning a scalable microservices architecture (MSA) that is orchestrated by Kubernetes is how to manage the heavy traffic of messages in the system.

    After migrating from the monolith architecture to a microservices architecture or setting up a new MSA, each microservice holds its data model and is disengaged from the rest of the system. When scaling, it is observed that a project can grow to hundreds or more microservices, and the messaging traffic can become enormous in systems that generate millions of messages a day. Hence, there is a need for a new robust communication between the microservices to be established.

    Using Point-To-Point connectivity such as REST to solve the communication gap in Kubernetes can end up creating a complicated structure between the services. A consequence of this is the need for comprehensive maintenance by the development team each time service requirements are changed. This is where a messaging queue system comes in handy to take care of the messaging challenge in MSA and Kubernetes. Each service communicates with one focal point (message broker queue) in its own language and the message queue system is in charge to deliver the messages to the services which are waiting for it.

    To build a well-managed messaging solution for Kubernetes, the message queue system should be native to Kubernetes, support the relevant messaging patterns, robust and secured. Being native to Kubernetes is a critical parameter to achieving a healthy system in Kubernetes. It includes a simple and fast deployment to Kubernetes (and not outside of the Kubernetes cluster), low DevOps maintenance and well connected to Kubernetes ecosystem for logging, monitoring and tracking.

    Hybrid Cloud

    A hybrid cloud solution enables enterprises to make use of both on-premises and public cloud services. An advantage of a hybrid cloud is the flexibility it offers–allowing workloads to alternate between the two environments when capacity and costs change, for example. Sensitive workloads and data can be hosted in the private cloud or on-prem while less critical workloads are hosted in a public cloud. Regulatory requirements for data handling and storage can be provided in the private cloud. A commercial advantage of public cloud services is cost, both in terms of CapEx and OpEx, as the scalability it provides allows an organization to pay only for the resources it uses. Deploying message queue in Kubernetes provides the technology that allows the hybrid clouds to smoothly and transparently connect and interact.

    Use Cases

    There are many use cases for message queues. The message queue needs to support diversified messaging patterns, enabling flexibility in creating different use cases.

    Here are some of the common use cases a messaging queue system should support in Kubernetes.

    Multi-stage Pipeline

    The synchronous pattern is used when we need to process the messages in a coordinated way in a workflow. Pipeline use case allows processing messages in a sequence between services. Each service can be considered as a stage, and the message passed between all the stages in a sequence. In some cases where the message is lost or cannot be processed (for any reason), the dead-letter queue mechanism takes the message and process it in a pre-defined way.

    Job/Task Distributed Queue

    The A-synchronic pattern is used by some producers to many consumers when we need to distribute tasks/jobs between workers. In this use case, there is no need to coordinate between tasks. Each service is pulling messages from the queue to process it. The sequence of processing the messages is not important as each worker takes the task from the queue and process it in his own time.

    Figure 1: Tasks/Jobs Distributed Queue.

    Stream

    The A-synchronic pattern is used when we need to stream data from many external sources (big data, IoT) and process it in a dedicated service such as database, pipeline, machine learning, storage, etc. This case typically aggregates many producers to a small number of consumers. The order of messages is not important, but the delivery guarantee is important.

    Pub/Sub Real-Time

    The A-synchronic pattern is suitable for cases of a small number of producers sending real-time data to many consumers. A service acting as a publisher sends a message to a channel where a set of services subscribe to that channel and get the messages.

    Application Decoupling

    Connectivity between services such as edge, API, databases and storage act like a router, and message order is not important; timeout and sync between producers and consumers are important.

    Ease of Use

    One key feature of the microservices architecture is its ease of use. Deploying it saves your organization time and money by unifying development and operations workflows. The message queue system needs to be easy to use as well. Its ease of use and DevOps’ friendliness should minimize the need for dedicated experts and, at the same time, accelerate development and production cycles. While considering this, it’s important to ensure support in high volume messaging with low latency, efficient memory usage and fundamental messaging patterns (real-time pub/sub, request/reply and queue) are not compromised.

    Gradual Migration to Kubernetes

    When IT professionals evaluate the significant advantages of using the Kubernetes environment, they cannot ignore the operational requirements necessary to maintain the revenue-generating monolith infrastructure. Legacy systems also contain the processes and data related to the on-going business and are crucial for the service contingency.

    To support a gradual migration that keeps the business operation, the Kubernetes message queue must enable connectivity between the old and new environments. A bridge is required to connect the legacy side to IBM MQ, TIBCO, MSMQ or Kafka (just to mention a few popular enterprise solutions), to the Kubernetes message queue that allows a seamless bi-directional transposing messages between the services in the legacy monolith environment and the microservices-based deployment in the Kubernetes environment.

    The bridge installed in the legacy environment is “listening” to the MQ on behalf of the relevant services in the Kubernetes environment, ensuring the designated messages are transferred from the monolith to Kubernetes and vice versa. This bridging capability enables gradual migration by implementing a step-by-step replacement of components from the old environment or the creation of new services that can still connect with the legacy resources.

     

    Figure 2: An example of gradual migration from MSMQ on the on-premises environment to AKS.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Lior Nabat

  • Best of 2019: The Time Has Come for an Engineering Approach to DevOps

    Best of 2019: The Time Has Come for an Engineering Approach to DevOps

    As we close out 2019, we at DevOps.com wanted to highlight the five most popular articles of the year. Following is the third in our weeklong series of the Best of 2019.

    Do you want to know how to do DevOps? Are you doing DevOps but not satisfied with the results you are getting? DevOps is complex. It does not come with a prescription, installation guide, user’s manual or maintenance manual. It doesn’t even have a standard definition. How can anyone expect to get good results with DevOps without clear and definitive guidance? There are plenty of books that describe different aspects of DevOps and customer user stories, but up until now there has not been a book that frames DevOps as an engineering problem with a step-by-step engineering solution and a clear list of recommended engineering practices to guide implementers.

    My book, Engineering DevOps: From Chaos to Continuous Improvement … and Beyond, provides a step-by-step engineering prescription that can be followed by leaders and practitioners to understand, assess, define, implement, operationalize and evolve DevOps for their organization. It provides a unique collection of engineering practices and solutions for DevOps. By confining the scope of the content of the book to the level of engineering practices, the content is applicable to the widest possible range of implementations.

    This book was born out of my desire to help others to do DevOps, combined with a burning personal frustration. The frustration comes from hearing leaders and practitioners say, “We think we are doing DevOps, but we are not getting the business results we had expected.”

    There are many documented references, courses and other sources of information that explain a multitude of different aspects of DevOps, but there is not a comprehensive step-by-step prescriptive guide that details how to engineer a DevOps solution for any organization given any starting point.

    Engineering DevOps takes a strategic approach, applies engineering implementation discipline and focuses operational expertise to define and accomplish specific goals for each leg of your DevOps journey. This book guides the reader through a journey from defining an engineering strategy for DevOps to implementing The Three Ways of DevOps maturity using engineering practices: The First Way (called “Continuous Flow”) to The Second Way (called “Continuous Feedback”) and finally The Third Way (called “Continuous Improvement”). This book is deliberately intended to be a guide that will continue to be relevant over time as your specific DevOps, and DevOps more generally, evolves.

    Structure of this book

    This book is organized as an engineering reference guide presented in five parts as follows:

    Part I: What Is Engineering DevOps and Why Is It Important? is organized into three chapters. Chapter one, “What is Engineering DevOps?,” explains engineering concepts and terms used in this book that have proven to work with many clients I have encountered during my consulting experiences. My depiction of a DevOps Engineering Blueprint is presented for as a useful, practical “big-picture” reference for discussing engineering the major parts of DevOps. Chapter two, “Nine Pillars of DevOps,” describes a classification of DevOps practices that I have found to be useful characterization of DevOps engineering practices readily applied when engineering DevOps implementations. Chapter three, “Why Is Engineering DevOps Important?,” explains the benefits of taking an engineering approach to engineering DevOps.

    Part II: Engineering People, Processes and Technologies for DevOps provides a comprehensive explanation of recommended engineering practices for the higher levels of the DevOps Engineering Blueprint. DevOps transformations are not normally understood or applied using strict engineering principles. Engineering practices are presented in nine chapters. “How Should DevOps Be Engineered?” includes a discussion of DevOps Engineering Maturity Levels for each of the Three Dimensions (People, Process, and Technology), Twenty-Seven DevOps Engineering Critical Success Factors and Lean DevOps Value-Stream Pipeline Engineering.

    I describe recommended engineering practices for the top layers of the DevOps Engineering Blueprint in the next chapters, which include “Value-Stream Management (VSM),” “Application Release Automation (ARA),” “Version Management,” “Continuous Security (a.k.a. DevSecOps),” “Service Catalog,” “Governance,” “Site Reliability Engineering (SRE),” “Disaster Mitigation” and “Recovery.”

    Part III: Engineering Applications, Pipelines, and Infrastructures Engineered for DevOps provides a comprehensive explanation of recommended engineering practices for the lower levels of the DevOps Engineering Blueprint, presented in six chapters as follows: “DevOps Applications Engineering,” “CI/CD Pipelines Engineering,” “Elastic Infrastructure Engineering,” “Continuous Test Engineering,” “Continuous Monitoring Engineering” and “Continuous Delivery and Deployment Engineering.”

    Part IV: DevOps Seven-Step Transformation Engineering Blueprint provides a description and tools for my approach to realize and evolve DevOps. The seven steps are Visioning, Alignment, Assessment, Solution Engineering, Realization, Operationalize and Expansion. By using these tools, DevOps leaders and practitioners can create, implement, operate and expand their DevOps across the organization. The chapter goes further to explain how to evolve DevOps from a successful First Way DevOps (Continuous Flow) toward realizing more advanced Second Way (Continuous Feedback) and Third Way (Continuous Improvement) DevOps implementations. This part includes a discussion of “Beyond Continuous Improvement”—a look at emerging technologies shaping DevOps in the future, and how you can prepare your DevOps and yourself for the future. This part includes a discussion of how to set up an effective DevOps engineering training program that supports continuous learning of DevOps engineering skills needed to maintain and enhance DevOps.

    Part V: Appendices, Continuous Learning, and References includes materials and sources I have found most useful for engineering DevOps.

    To get a copy of my book “Engineering DevOps: From Chaos to Continuous Improvement … and Beyond,” or start reading for free, click here.

    — Marc Hornbeek

  • Best of 2019: Critical Skills Developers Need to Avoid Getting Left Behind

    Best of 2019: Critical Skills Developers Need to Avoid Getting Left Behind

    As we close out 2019, we at DevOps.com wanted to highlight the five most popular articles of the year. Following is the second in our weeklong series of the Best of 2019.

    Burnout is a recurring theme at DevOpsDays events around the world. The modern developer faces constantly and rapidly-evolving challenges: the agile movement reducing the development lifespan, the DevOps movement shifting production operational responsibilities into development processes and the complexity of cloud offering new architectures from SaaS through to FaaS, as well as microservices.

    As these shifts occur, often ignored are the operational skills needed to support an application running in production. Developers expect to support production and be on call 24/7, but are IT leaders giving them the necessary tools and training? More and more, we need developers to be prepared and able to seamlessly oscillate between focused specialist and broad generalist. This article highlights a few key areas where software engineers need to bolster their skill sets to remain agile and effective in building applications.

    DevOps and SRE Assume Broader Skills

    Non-developers may interpret agile computing as developers creating a world where they get to code rather than document, follow process or deliver to plan. In reality, agile took the lengthy cycle of waterfall releases and replaced it with the frenetic continuous delivery of valuable software. DevOps was born from the need to deliver application updates rapidly and regularly.

    The monitor phase of the DevOps infinite loop originally focused on ensuring application problem detection and diagnosis was possible. Moving from detection to prevention, DevOps practitioners had to shift their testing and quality processes. Developers were asked to code the application, while also creating the testing systems to validate their code and building the production executables. Progress here led to additional responsibilities. As development moved to agile and DevOps, infrastructure transformed into software defined infrastructure (SDI).

    The separation of the operating system from the server hardware through virtual machines started the journey to SDI. Application containers removed the OS requirement and added cross-server mobility. Kubernetes and Docker Swarm allowed developers to encode application requirements in the recursively titled YAML. After containers, microservices is a further refinement that isolates an action into a separate piece of code—which is well suited for workload-based pricing of cloud environments. SDI’s capabilities bring us to where infrastructure definitions require a software development lifecycle (SDLC) of their own.

    The outcome of all this is a massive increase in complexity that developers must consider as they code components of an application. It requires additional considerations of the infrastructure and business environment the application will run in. Developers can define the best YAML definition of needs for their new application, but does not mean the production infrastructure will be able to provide a matching environment.

    Skills Developers Must Enhance

    Here are some key areas where developers would benefit from honing their skills in order to stay relevant amid growing complexities mentioned above.

    Networking

    The single purpose of computer networking is to move data from the memory of one computer into the memory of a different computer. With TCP/IP protocols as the default network stack, the flexibility needed to survive a nuclear attack comes with a lot of variability in configuration. At the basic level, network components must be configured to identify how many bits of an address to use when searching for another network entity via Classless Inter Domain Routing (CIDR).

    Software engineers need networking skills to understand how their software definitions may conflict with the eventual production network infrastructure. For instance, network definitions in the YAML descriptors for Kubernetes/Docker are generally set with a /nn configuration (e.g. /16). Conflicts arise when you overlap the pod addresses with physical server addresses or have a mismatch between development pods and production pods. If your pods-per-node count differs between pre-prod and prod environments, it is possible the CIDR settings will differ as well—as a colleague regrettably learned when using /8 rather than /16, leading to the largest production outage in their career.

    In addition to these basics of network addressing, there are concepts such as micro-segmentation needing definitions of performance and connection requirements that span across containers, virtual machines and bare metal servers.

    Security and Segmentation

    Historically, applications have relied on production infrastructures with their firewalls and DMZs for protection from the outside world. That castle moat approach is insufficient for today’s threat landscape where applications live everywhere. Not only do enterprises need to protect against outside connections and attacks, they also need to prevent accidental connections from production applications to development, QA and testing environments. Crossing the streams can cause exposure of production data in test environments and, even worse, corruption of production data.

    Ensuring that application components only communicate to specific services in specific environments is addressed with micro-segmentation and its ability to manage and monitor secure code-to-code communication. Control with this detail comes with its own challenges as incorrect or insufficient definitions could leave a production application unable to communicate with needed API’s or microservices. Making diagnosis in micro-segmented environments harder is the possibility that traffic for other transactions or applications may be successfully crossing the same environment. This is when you may hear the classic “it is working fine in test” response.

    Micro-segmentation offers significant security benefits yet requires developers to clearly identify the required connections and components of any communication. The best time to note this is while coding, making this an important skill for the modern developer.

    Dev and Ops Synchronicity

    The synchronous arrival of DevOps and SDI provides a mutual opportunity for developer and operations teams. While developers need to learn production on-call skills and operations teams need to learn SDLC skills, both teams also need to remain current in security concepts. Some organizations tackle DevOps by creating merged teams, though members may continue to stay in their lane. Whether merging teams, or maintaining separation, it is beneficial to hold brown bag sessions to share information and help educate each other. Ask your operations peers to talk about their three most common production issues and the three most popular tools they use.

    An important aspect of the application on-call generalist is having the right tool supporting them to effectively diagnose production outages and slowdowns. Whether developed as an in-house solution (with incumbent technical debt) or selected as a product, you will need something that can see from end-to-end and allow the code-centric software engineer to drill into databases, storage, networks and code components from an application point of view without having to learn siloed management and monitoring tools for each aspect of infrastructure.

    While individuals come into a DevOps transformation with I-shaped specialists, they must leave as T-shaped engineers, thus enhancing their value to current employers and helping their career.

    — Marco Coulter

  • Best of 2019: Five Microservices Worst Practices

    Best of 2019: Five Microservices Worst Practices

    As we close out 2019, we at DevOps.com wanted to highlight the five most popular articles of the year. Following is the first in our weeklong series of the Best of 2019.

    Microservices are moving to the mainstream. The IDC FutureScape: Worldwide IT Industry 2019 Predictions states that by 2022, 90% of all new apps will feature microservices architecture that “improve the ability to design, debug, update and leverage third-party code.”

    With so many organizations adopting or about to adopt microservices, they must be doing something right. In fact, just considering the model is a positive step toward delivering quality software, faster. However, many companies are leaving microservices benefits on the table because there are many pitfalls on the road to success. It’s about so much more than just a technology choice and starting to code microservices.

    Indeed, breaking your apps into microservices isn’t enough—you have to manage them, orchestrate them and deal with the data they create and modify. Before you even touch the keyboard it’s a good idea to think about a few points of strategy to ensure success.

    Here are five microservices worst practices and how to turn them around.

    Implementing Microservices Without the Prerequisites

    You can’t take Calculus II without taking Calculus I. Similarly, you can’t roll out microservices without the proper prerequisites. To fully leverage microservices, companies must consider modularity and domain boundaries, and the fundamental distributed systems theory should be considered and established. In addition, while each service is independent, there are operational requirements that must be met. These include DevOps; PaaS; containers or immutable VMs; service replication, registry and discovery; and proactive monitoring and alerts.

    Implementing Microservices Without Making Changes to the Development Culture

    The biggest mistake organizations make when moving to a microservices architecture model is underestimating how it’s changing the way they need to think about applications and especially the way they think about the teams developing those applications. Traditionally, there are many people involved in the development of an application, with each coming at it from his or her own perspective—the UI specialist, the middleware specialist, DBAs, sysadmins and so on. What worked for one might not work for the other when it gets thrown over the wall, and it often takes a lot of time and frustration to make everything work for everyone. Trying to do microservices without changing that model and mindset will almost certainly set your team up for failure. Instead, think in terms of cohesive, cross-functional teams, each fully responsible for its own unique services and its own set of customers for those services. Each team’s working with other teams as if it’s a business-to-business (B2B) relationship. It’s not an easy cultural transition, but it’s one that is essential for microservices to work.

    Going Too Big and Broad Out of the Gate

    An organization that goes all-in on microservices all at once is almost certain to run into problems that could completely derail the initiative altogether. Pick a small project, and staff it with people who see the potential of microservices and are excited to work with the technology. Once that project has been successfully completed, use it as a proof point to promote the further use of microservices across the organization and to get others on board.

    Misplaced Expectations

    Organizations are right to question the performance impact that microservices will have when migrating an existing monolithic application to a microservices model. But their concerns—and expectations—are often misplaced. Yes, there can be a performance hit when decomposing existing monolithic applications in favor of microservices. But what organizations need to understand is the goal of microservices is deployment speed over API invocation speed.

    Isolating the development of a service within a team (as mentioned earlier) means the service can be updated as needed—whether that’s weekly, daily or even hourly—in response to business, competitive and security demands. In its Worldwide IT Industry 2019 Predictions report, IDC notes it is “already seeing enterprises that have shifted fully to these new approaches (and supporting tools and technologies) that dramatically accelerate their ability to push out digital innovation through their organization at 50-100 times (or more) the frequency of their traditional approaches.”

    Picking the Wrong Applications to Microservice-ize

    This worst practice is linked closely to the previous one. Organizations that pick the wrong applications to split into microservices or decompose an entire monolith without thinking about which parts might be best as microservices (and which would fail miserably as microservices) will have the odds stacked against them from the get-go. A financial application or a high-volume application that makes multiple calls may not be good candidates for microservices. Or, it may be that parts of these applications would work well in the microservices model even if the monolith as a whole doesn’t. The idea is to examine each legacy application carefully and to build new applications with microservices in mind from the beginning.

    If you’re thinking that it takes a lot to establish a microservices foundation, you’re right. It’s a big investment, and it may take some time before there is a return on that investment. But organizations that move toward microservices in a thoughtful, purposeful way—exploiting the best practices of those that have gone before them—will likely find themselves more agile, more secure and more competitive.

    — Eric Schabell

  • Predictions 2020: Infrastructure and Ops Trends to Watch in 2020

    Predictions 2020: Infrastructure and Ops Trends to Watch in 2020

    Today, modernizing systems is more than simply moving technology to a new location. It requires an IT stack of a new generation of technologies and tools to work.

    As we head into the new year, let’s take a look at the top eight trends that will shape how we will develop and deliver software. 

    Kubernetes

    Yes, it’s big and getting bigger. The pace of enterprise adoption of this leading container orchestration solution will only gain momentum in 2020. Look no further than the 2,300% attendance increase from KubeCon 2015 (500) to KubeCon 2019 (12,000). As more companies begin to see the direct financial and operational benefits to the flexibility, scalability, automation, high availability and portability of Kubernetes, the demand for developers and engineering experts in this complex software will grow accordingly. Case in point: A 2019 national job search on LinkedIn turned up 16,744 open positions for Kubernetes-related roles. Given the already large and still developing Kubernetes community, we should also see an increase in the speed and volume of feature releases. 

    Microservices

    It’s predicted that by 2022, 90% of all apps will feature microservices architectures. The growing business imperative for software engineering teams to produce and release software in ever shorter cycles—continuous delivery—has paved the way for a huge surge in microservices, the ideal architecture for CD. As enterprises continue their accelerated move to the cloud in 2020 and beyond, they increasingly prize the flexibility to try out the new technology stacks that microservices provide. All businesses want to optimize resources, minimize downtime, reduce infrastructure costs and avoid technology or vendor lock-in. In 2020, microservices architectures will continue to offer up more and better of all these benefits as the migration from monoliths picks up speed. 

    Service meshes

    The rise of microservices has led to a parallel boom in service meshes. This will only accelerate, given how integral the current class of service meshes—Istio, Linkerd, Consul, Envoy—is to the successful functioning of microservices. As a software designed to enable the thousands of microservices that make up an application to talk to each other, it’s quickly become an essential component with a vendor ecosystem poised for substantial growth. Feature-rich Istio, backed by IBM, Google and Lyft, is the leading Kubernetes service mesh, and that shows no signs of changing in 2020, especially given the relative ease of setting up and running it. If your enterprise has embarked on the journey from monolith to microservices, it will soon be crossing paths with service meshes. 

    Observability

    In the recent past, the sheer complexity of modern distributed systems has outstripped the tools designed to allow teams to see into them. Enter observability—which isn’t monitoring. Like good helicopter parents, software engineers, application developers, SREs and DevOps teams want to know and understand what their systems are doing at all times. LightStep, Datadog and Honeycomb, to name a few, are vendors poised to offer logging, tracing and metrics solutions to the rapidly growing number of enterprise teams building cloud native systems. Keep a close eye out for OpenTelemetry, an open source observability framework currently a member of the CNCF Sandbox. 

    SRE

    One of the great challenges ahead for this still-emerging field is for enterprises of varying sizes and work cultures to find a form of SRE that works. What succeeds at Google may not be the recipe for SRE implementation at smaller tech enterprises or startups. The role of SRE and the expectations that go with it remain fluid. The SRE’s constant imperative to automate will only add to the profile of critical SRE and DevOps tools such as Terraform, Ansible and Jenkins. Ponder Amazon’s Prime Day 2018 losses of up to $99 million for a clear understanding of the business-critical importance of SREs in the years to come. The job market reflects this new reality. 

    Security

    The shift-left movement in the software development lifecycle is on. The days of bolted-on post-production security are rapidly fading. DevSecOps, though still a buzz term, is here to stay, with more practice and use cases added to the theory. There are huge opportunities here for automated application security tools that support the software development workflow. 

    Then there’s privacy. The GDPR took effect last May and the California Consumer Privacy Act takes effect January 1, 2020. On top of these, there’s HIPAA and the regulatory compliance pressures on businesses in the area of security, which are already enormous—plus the immense challenges of cloud security. A 2019 Symantec survey on cloud security threats reports that 65% of businesses in the cloud still aren’t using multi-factor authentication. The same survey also notes that an astounding 85% of Symantec’s cloud customers aren’t employing security best practices. Look for businesses to tend to security earlier in the process. 

    Serverless

    It’s a boom time in serverless, with Google’s Functions, Microsoft’s Azure Functions and IBM’s OpenWhisk all offering their serverless solutions following the 2014 rollout of AWS Lambda. Look for more businesses to adopt a serverless-first approach as they upgrade legacy systems and see the substantial cost saving and application development benefits. Startups, too, will see the cost-cutting upside and the agility that serverless brings. The changes in application development and architecture in the cloud as a result of serverless are hard to overstate. With the vendor solutions now on offer, developers are freed up to do what they most like to do: run code and add functionality to their application, faster, without worrying about scaling and availability. Like microservices, serverless will see a huge upsurge in usage as the age of the monolith fades. 

    DevOps

    With the demands of cloud computing and far greater enterprise investment in dedicated teams, DevOps in 2020 is poised to make the transition from policies and principles to actual practice. The addition of baked-in security to the DevOps realm—DevSecOps—adds urgency to the automation of building, provisioning and deploying secure software. ML- and AI-driven apps have arrived, and DevOps practices will play a large role in the development workflow. The goal of zero-touch automation looms large, and this adds urgency to the need for clean data, organizational buy-in and more integrated systems. AIOps, though getting plenty of attention, is still young and its benefits to IT operations are still more theoretical than actual–but stay tuned.

    As we head into 2020, it will be exciting to watch these trends unfold. DevOps teams can expect the way we develop and deploy applications–in any architecture–to continue to evolve. By keeping a pulse on new developments in Kubernetes, microservices and other new technologies, they’ll be well prepared to bring their businesses into the new decade.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Kelsie Pallanck

  • Don’t Expect Miracles

    Don’t Expect Miracles

    Let’s run some fun scenarios to start this one off, shall we?

    • “We’ll decide what language to use at a meeting next month. Until then, get to writing code!”
    • “We might deploy internally, though Azure is a strong option, and we’ve discussed GCP. Now get operations up and running, we’ll tell you when we’ve decided.”
    • “The code is changing constantly, get moving and write the documentation.”

    All three of these scenarios are equivalent. The difference is how we, as an industry, view them. The first two, we (collectively) would never tolerate. The third? We do that to the documentation team every day. In DevOps and agile, we do it every day in a never-ending iteration. And it has to stop.

    Two Different Documentations

    For those not aware, there is documentation for internal use and documentation for external use. While we’re certain you have internal covered with self-documenting code, this blog will focus on external use, because if you expose and API, documentation is mandatory. If you’re not an end-user product, any UI requires documentation, too. So to be clear, for this blog we are talking generally about external documentation for B2B systems, and specifically about APIs, though UIs can have the same issues, depending upon how complex your market is.

    We Generate Docs

    Ever tried to use auto-generated documentation? I’ve yet to try one generated set of API documentation that I liked (they say Google Places has it, but my understanding is devs actually write that documentation and it’s reviewed by tech writers, so maybe we count that as one I liked, but its process is not yours).

    Generated docs are only as good as the developers working on the code–very much like self-documenting code. If someone puts “This parameter is a placeholder for the spline development iteration” into a generatable comment, users are going to be totally clueless what that means. Heck, I just dreamed that statement up, and while we can all see it turning up in generated API docs, even I don’t know what it might mean.

    Documentation Slows Up Dev

    Duh? If you are loathe to commit to completing a project until you know the parameters, why would you expect a documentation team to be any different? Of course they need the final product for review, just as the business wants to see the final app before release. The product your docs team produces is detailed documentation about what is. If you change what is underneath them, they look bad for having inaccurate docs. When really it should be IT that looks bad for not considering the entire value chain when delivering. If you deliver an API that is poorly documented and people don’t use it, you have delivered nothing of value to the organization.

    What to Do?

    Put documentation on your DevOps teams. I mean an actual person whose responsibility is documenting. If you’ve got technical writers, use one of them. If you don’t, find someone who likes helping people use your toys, and give them time to do it. For some organizations, technical evangelists can fulfill this role if you don’t have them too busy helping active users. Have them sit in on meetings, treat them as just another piece of both agile and DevOps. They need to know when functionality, URI or whatever else changes, so they can update the docs. The best place to get that info is at stand-ups or design meetings.

    Just like always, they’ll want to be there for design, but they’ll also want to be there for implementation decisions/changes. Because current documentation is the only worthwhile documentation. Give them sign off on release–not release of code (that’s a done deal) but publication of APIs. Before you let users at the access points, make sure they have solid instructions in how to use them.

    Risks

    Everything is a trade-off, but this one has some pretty easy choices to make–fewer people use your API because your documentation isn’t great (outright sucks in many cases), or you run some risks of slower iterations and you dedicate resources to documentation. Given that if no one uses your API, it is all wasted time, a bit of slow-down and man-hour investment is an easy choice to make.

    Tech writers are generally cautious people who want perfection–because this is their product, and they want it right. This can cause unnecessary slow-downs in an agile/DevOps world. The easy answer here is to look for the same traits in documentation DevOps team members that you look for in other DevOps team members–ability to adapt, understanding that everyone is just trying to make it better and willingness to work through changes quickly to deliver.

    Just like you don’t want the DBA that says “no” to every request and you have to argue with just to move the project forward–I worked with this guy, genius, but really frustrating to have on an agile team–you don’t want the documentation person that adds weeks to timelines so they can get it right. This risk is easy to manage. You know how to choose people for a fast-moving environment, just follow your gut.

    Profit

    Well, okay, maybe not profit, but deliver a solid product quickly with sufficient documentation that users not steeped in your corporate shared knowledge will understand and be able to use. That’s the goal of public-facing APIs, to be used. And it’s the goal of docs people to make certain those calling the APIs understand what they’re getting into and how to make the most of it.

    Keep on cranking out cool solutions and crank out better docs. Your customers are counting on it. I’ve had my team go find alternative solutions based on shoddy API documentation. If I can’t use it, why would I keep trying if there are alternatives?

    — Don Macvittie

  • Predictions 2020: 7 DevOps Predictions Shaping 2020

    Predictions 2020: 7 DevOps Predictions Shaping 2020

    2019 was an exciting year for DevOps practitioners, with DevOps adoption continuing to grow at a fast pace. Most organizations were either executing or evaluating their DevOps strategies. So, what is 2020 going to bring in DevOps, infrastructure and application automation? Here are seven DevOps predictions for emerging trends in the new year.

    Customer Experience

    CX will become the guiding light of the DevOps journey, and not necessarily just technology. This is a big mindset shift from the early days of pure-technologist enthusiasm. As DevOps matures, we have to remind ourselves that it’s all about making business faster, not just the cool tech.

    Take advantage now:

    • Tie DevOps achievements to business needs and results to get executive buy-in and budgets.
    • Successful QA teams will start looking at their role as guardians of customer experience and not just testers.

    Trust

    Everyone will be talking about trust—not only in the context of zero trust networks but also in the context of making it possible for Dev and Ops to trust one another as the foundation for DevOps.

    Take advantage now:

    • Proactively invest in creating a culture of trust. Identify causes for lack of trust in your value stream and make trust a 2020 goal.
    • Encourage product-level conversation and define methodology and ownership when it comes to value stream management, application and infrastructure automation. It is the foundation that everyone is working on.
    • Invest in staging environments—Staging environments are needed to ensure things work before you put them in production. For many of us, jumping directly to automatically deploying to production is a big step. The risk can be mitigated by investing in staging environment automation, which will be the first step toward deployment to production.

    Value Stream

    The DevOps toolchain is going away. In 2020 we will think value stream. The industry is maturing beyond the notion of chaining together dozens of open source tools and trying to make it work together. In previous years many felt comfortable saying that if they have a CI system in place it means they “do DevOps.” This is no longer the case. DevOps is not just CI, and DevOps cannot be established in silos. Mature DevOps needs to take into consideration the entire value stream—from planning a new capability and the business requirement, all the way to the customer getting more value and a wonderful experience.

    In 2019, Gartner announced it’s the last year of the ARO MQ, as it’s not the right perspective. In 2020 we finally gain a system-level view on the value stream. More players are announcing end-to-end value stream management including Atlassian’s new Forge platform, GitLab and CloudBees. Companies are maturing and adopting system-level solutions. 

    Take advantage now: 

    • Plan for platform operations. System-level value stream perspective requires roles that take ownership of orchestration and automation throughout the value stream and not just in silos.

    The SRE Role Will Continue to Mature

    The SRE role that started with Google bringing software development practices into Operations will continue to evolve. Increasing volume, velocity and variety of data will make AIOps solutions thrive. Also, increasing levels of automation will require smart ways to handle dynamic infrastructure and applications without losing control, and being able to track changes all the way back to the coder. 

    Take advantage now:

    • If you haven’t implemented the SRE role yet, consider how SREs can be part of your teams in 2020. Define SRE areas of responsibility and KPIs, and how the SREs communicate with the teams. Is it a centralized position or are the SREs part of the team?
    • Make sure you provide SREs with the tools they need to succeed including:
      • What are the observability solutions you use? Who uses them today? How do you analyze it and how is it communicated?
      • What are the production environment orchestration, deployment and configuration tools that you use today?  How do you connect them to application and infrastructure automation that happens before production?

    Seamless Infrastructure

    We will have to make infrastructure seamless for our teams. Internal customers are also customers, and their experiences will be just as important. Self-service and empowering teams will be the main themes in the bridges that need to be built between Dev and ITOps. Public cloud, Kubernetes and serverless technologies will grow. And while it makes automation more feasible than ever, the emphasis will shift to self-service and to the end users. With more companies adopting similar approaches, the need to develop internal expertise will decrease, and we will see more off-the-shelf solutions.

    Take advantage now:

    • Map how infrastructure is used throughout your value stream, and work on unifying the approach in product-level.
    • Make sure you build self-service solutions with strong focus on the end-user in order to enable and balance both speed and control. Make sure developers and ITOps are equally represented when determining an approach to infrastructure automation.

    Infrastructure Automation

    Infrastructure automation is going to get smarter, with a lot of AI and ML. Making sure automation is properly governed will also increasingly be important. Take into consideration security and compliance, as well as cost.

    Take advantage now:

    • Assess infrastructure automation practices throughout your value stream. Who builds the automation? What teams and systems consume it? How do you enforce security best practices (e.g. no passwords in public repositories, using proper data sets for testing, compliance of automation blueprints)? How do you enforce cost control (e.g. tagging, reviewing cloud spend reports)?
    • Invest in control mechanisms before you lose control, especially in public cloud automation. If you don’t have governance in place, look at baking governance into automation, rather than adopting traditional cloud management approaches. Invest in automatic de-commissioning of infrastructure, automatic tagging, analysis of infrastructure consumption data and connecting it back to the business requirement, secrets and key management and access control.

    SaaS

    Open source is nice. SaaS is great. While big open source projects set the pace (K8S, Github) and are strongly backed up and funded by the industry’s giants, smaller projects are increasingly difficult to scale and glue together, especially when trying to assure enterprise readiness. The need to “do it yourself” is gradually fading, as more vendors offer products and services for supporting DevOps with clear business and support models and, as important, with OOB integrations with the ecosystem. Just Rest API is no longer enough.

    Take advantage now:

    • If your enterprise-readiness requirements make it difficult for you to scale DevOps initiatives, consider evaluating new SaaS offerings that eliminate much of the heavy-lifting and gluing that were typical to the skills-heavy early-days implementations. If you were concerned with security, cost and skill requirement, today you can find tools that are purpose-built to support DevOps for enterprise at scale. The good news is that most of the vendors are well-aware of the criticality of integrating to your ecosystem, making it easier to achieve your end-to-end value stream vision.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Maya Ber Lerner

  • Untapping the Potential of DevOps, Part 4: Tips for Launching a DevOps Team

    Untapping the Potential of DevOps, Part 4: Tips for Launching a DevOps Team

    DevOps has become a hot strategy for IT teams across just about every industry—and for good reason. DevOps is built around increasing the velocity of the development pipeline, and in businesses where time to market is a key to success, it’s easy to imagine why DevOps has gained traction. While the tools and processes put in place are certainly essential, building the right team—from top to bottom—is maybe the most important factor for getting the most out of a DevOps strategy.

    To help get the most out of DevOps initiatives—and to set teams up for long-term, sustained success—here are some basic tips for creating the team that will take development pipelines to new heights.

    Change Is Coming

    When kicking off a DevOps strategy, it’s clear that changes will be made. But, the extent of those changes may be underestimated. To get the most out of DevOps, it’s not just about changing up the tool stack or switching up meeting styles. The mindset around DevOps is at least as important as the processes. To squeeze the most juice out of a DevOps initiative, make sure the company and leadership understand what all DevOps entails and create the understanding that it requires a lot of internal changes to the existing culture. Getting leadership on board with the cultural changes that come with DevOps will go a long way toward driving success.

    Talent Talks

    This may go without saying, but it is imperative to build a team with the right skills for the task at hand. It can be easy to try to shoehorn people into roles, especially when trying to move quickly. But make sure to set individual talent up for success—both employees and the company will benefit. This means having DevOps engineers and leadership with the skills—whether they are technical, interpersonal or something else—to perform the work required.

    Get Your Budget Right

    DevOps helps create efficiencies. Leveraging automation, maximizing value delivery and getting products to market faster are all key tenets of a good DevOps strategy. But that doesn’t mean you should skimp on the important items. Like any project worth doing, it’s a lot easier (and more successful) when it is budgeted properly.

    Make sure to have an adequate budget to create a team, perform the work and support the appropriate environments and infrastructure to perform the DevOps tasks and initiatives that need to be done. Doing this in advance will save a lot of headaches down the line and help develop a clear understanding of what is really needed to get from point A to point B—and help keep spending in line while the DevOps train is moving.

    It’s Not Just About IT

    Remember there are two parts to DevOps—dev and ops. It is often easier to get the developer side to buy into the strategy—after all, many of the tenets of DevOps speak to the technical teams. But overall buy-in is necessary for cooperation and involvement of all teams needed to achieve goals. That means development and operations—plus other departments within the organization, such as sales, marketing, the executive team, etc. Getting everyone on board with the process and rowing in the same direction will fundamentally change the mindset around how to design and deliver products. When the whole team is moving in lockstep, that’s when success can really be visualized.

    By keeping the thoughts above in mind when planning a DevOps journey, organizations can help ensure teams are organized and set up for long-term success. But remember, every organization is different and has different requirements. What may work for one organization may not for the next.

    When considering the potential of DevOps for your organization, it can be tempting to dive right in and skip some important considerations for setting up the initiative. But the key points listed above are tried-and-true and carry weight, no matter the size or scope of business. Having a leader who acts as a champion for DevOps efforts is essential, so IT professionals tasked with leading the organization into the DevOps transformation should arm themselves with these tips to be better positioned for success.

    — Jonathan Fries

  • Predictions 2020: The Year of DevOps Empowerment

    Predictions 2020: The Year of DevOps Empowerment

    Every year since mid-decade, it seems, one publication or another has declared it to be “the year of DevOps.” Allow me to make the case that it really will happen in 2020.

    The reason? IT leaders are finally empowering their engineers with full-service ownership.

    In a world in which speed, quality and precision will continue to determine the difference between continued success and unmerciful failure–particularly with regard to delivering best-in-class, end-to-end experiences for all stakeholders–full-service ownership is a matter of reality and necessity. Not to mention common sense.

    Developing and delivering products and services that maintain pace and meet the needs of businesses is dependent on achieving faster release cycles without sacrificing quality. But running reliable systems at ever-increasing speeds presents huge challenges that software teams can meet by changing their service ownership policies. The good news is DevOps practices have been gaining considerable traction because they help teams deliver with increased velocity coupled with reduced risk. This enablement, similar to any change introduced, takes time and practice to make work for your needs.

    Code It, Ship It, Own It

    Full-service ownership grants engineers greater responsibility for the code they write and the services they develop during production. The importance of owning and operating your own code and taking responsibility for all deliverables should not be underestimated, because it is the engineers, after all, who should be–and have earned the right to be–accountable. They are the ones designing architectures and systems, coding software and writing runbooks. They are the ones on whose shoulders the overall quality of service sits. They are the ones who are on call when issues arise (invariably, as Murphy’s Law implies, in the middle of the night).

    If they are the ones who code it, then they are the ones who should ship it and own it. The days of engineers “throwing code over the wall” to operations or depending on site reliability engineering (SRE) teams to guarantee the performance of services in the wild are no longer relevant, much less workable.

    That’s particularly important going forward, considering the DevOps market is predicted to grow 18% to nearly $13 billion by 2025. As DevOps tools and solutions become more ingrained in IT operations, the most logical and proactive way to manage this growth, and the inevitable chaos it will bring, is to empower IT with full-service ownership.

    The Forces Behind Full-Service Ownership

    This is happening for three reasons:

    Accountability

    It’s a simple truth; consumers implicitly expect the services they use to work seamlessly and with minimal latency. Considering that consumers will simply move to another site when a page is slow to load or returns a 404 error, the payoff in terms of increasing business’ competitive advantage is immense.

    Full-service ownership promotes higher-quality work because it gives engineers clear visibility into how the code and services they’ve created are performing and impacting their customers’ daily activities. 

    Reliability

    It’s not rocket science to ask the scientist who built the rocket to fix it when it breaks. Likewise, businesses can reduce the amount of downtime and customer impact when a serious incident occurs by immediately bringing in the engineers who wrote the code or created the service. 

    Full-service ownership offers another important advantage: overall institutional knowledge among IT teams. Individual engineers benefit greatly from code reviews, on-call hand-offs, daily standups, casual information sharing and other exercises that build redundancy in knowledge and expertise.

    Continuous Improvement

    A common misconception in the engineering world is that “on-call” is synonymous with “always-on.” That in itself is counter-productive because it can lead directly to alert fatigue. In an ideal world, teams will build time in their development work to go “off-call,” a practice that alleviates alert fatigue and that full-service ownership encourages.

    It also fits perfectly with an integral element of the culture of engineers: a commitment to continually refine and improve their code, product, services and alerting. The most effective process goes beyond just fixing problems, it also includes getting ahead of them, by automating the work needed to prevent them, freeing engineers up to focus on the work that matters most. 

    Five Benefits of Full-Service Ownership

    1. The quality of the code will increase.
    2. Engineers will know when they are off-call, and alerts will be covered by another developer with equal expertise.
    3. Team knowledge and skills will increase through a deeper understanding of the codebase.
    4. Services will become more reliable.
    5. Operations will become more agile.

    When engineers are empowered to code it, ship it and own it, businesses put themselves in a better position to enhance the customer experience and reduce the impact of disruptions, which enables them to focus more on innovation. By defining and refining roles and responsibilities, full-service ownership also reduces the chaos associated with incidents, eliminates silos and unnecessary layers and, ultimately, promotes a culture of empowerment and accountability.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Tim Armandpour