devopsy.kinsta.cloud

Tag: infrastructure

  • Dependability is Crucial for Real-Time Experiences

    Dependability is Crucial for Real-Time Experiences

    Millions of people around the world experienced a major disruption after an October 2021 outage saw Facebook, WhatsApp and Instagram go offline. A few days prior, the popular workplace collaboration platform, Slack, also went down for some users. And in the last few weeks of 2021, Amazon’s AWS caused considerable global chaos after significant networking issues in its busiest data center, US East 1, disrupting streaming services such as Disney+ and Netflix.

    Downtime can be costly, causing organizations to lose $5,600 per minute and up to $300,000 per hour, on average, according to Gartner. But it’s not only financial and reputational loss that businesses need to worry about. Outages also seriously downgrade—and in the case of real-time experiences, completely disrupt—the customer experience.

    At a time when consumers are demanding more live and interactive digital experiences, organizations can ill afford disruption or downtime, as customers quickly move to other services that ‘just work’.

    It’s not just the capacity within a real-time system that matters, but also globally distributed failover capabilities when something does go wrong. While organizations could try to build their own failover mechanisms, it’s extremely difficult to failover so services continue seamlessly, regardless of the status of its underlying infrastructure. Simple failover systems not only take longer to resume but typically suffer data loss during the time that a region is disrupted. Businesses need to know that even if, say, AWS suffers an outage, they won’t be affected.

    This is why it’s vital that organizations get it right when looking to develop their own real-time infrastructure. Managing tens of thousands if not hundreds of thousands of concurrent connections while synchronizing data in real-time means that if anything does go wrong it’s immediately obvious to users. If tech behemoths like Amazon and Facebook can’t get it right, what chance do organizations with far fewer resources have when building and managing their own real-time infrastructure?

    The Steps to Success With Real-Time Infrastructure

    Regardless of whether it’s for synchronized multi-user collaboration, audience engagement or instant updates, developers have a hard enough time developing even the most basic of real-time infrastructure, let alone making sure it’s dependable. There are four key elements to achieve dependability: Predictable performance, data integrity, reliability of service and high scalability.

    • Predictable performance: When we think of performance we usually imagine more computing power or speed. But when it comes to real-time, it’s not just about minimizing latency and bandwidth, it’s about having a predictable system. If developers have global latencies and bandwidth within certain operating boundaries, it provides certainty in uncertain operating conditions. This makes it easier to design, build and scale features around this certainty.
    • Data integrity: As we all know, the internet can be an unreliable network. Users abruptly disconnect and reconnect all the time, resulting in data that gets lost, is delivered in the wrong sequence or that is sent multiple times. As a result, organizations must ensure their infrastructure can guarantee high integrity without sacrificing anything else.
    • Reliability of service: The live nature of delivering updates in real-time requires total reliability. Outages can cause distrust and annoyance from users. Organizations could find their reputation taking a hit or customers quickly jumping ship if reliability proves to be a recurring issue. To prevent this, organizations should develop infrastructure that’s fault-tolerant and redundant at both a global and regional level so they can deliver a great experience even if there are multiple failures.
    • High scalability: More often than not, delivering an experience in real-time can result in unpredictable traffic; for example, a huge number of people wanting sports updates for a World Cup game. If thousands of devices connect over a short period of time but capacity can’t scale up quickly enough, then an outage will occur. To avoid this, developers must use infrastructure that can be elastic and meet changing levels of demand. Otherwise, the user experience is seriously affected and real-time capabilities crash when the network is swamped with connections it can’t cope with.

    While organizations often build their own infrastructure with the best of intentions, it’s easy for them to run into trouble. Most lack the internal expertise, resources and time; plus, going it alone often results in a huge technical debt. It’s also a massive undertaking, due to the amount of ongoing complexity and cost involved.

    Third-Party Partnerships Enable Dependability

    Unless organizations have the resources, finances and expertise of the likes of Amazon or Google, then building your own real-time infrastructure is just setting yourself up for failure. Instead, they should save on development cycles and focus on generating revenue more quickly by offloading their real-time infrastructure needs to a third party. This will mean organizations can design without limitations and focus on solving the challenges that really matter to them while putting in place real-time infrastructure that is as dependable and future-proof as possible.

  • Calculating Infrastructure Automation ROI

    Calculating Infrastructure Automation ROI

     

    DOWNLOAD the EBook >>

     

     

  • How to Build an Options-Based Observability Strategy

    How to Build an Options-Based Observability Strategy

    The global pandemic forced companies to accelerate their digital transformations, pushing them toward new technologies and architectures as they struggled to adapt to a changing world. Companies wanted fast development and deployment and were happy to trade off efficiency, reliability and sometimes, security, to pursue their goals. Now that the pandemic is receding in many parts of the world, the priorities are rebalancing as companies grapple with choices they made under fire.

    Observability has emerged as a way for enterprises to better understand sprawling infrastructure and application environments. According to Gartner, “Observability enables organizations to reduce the time it takes to identify the root cause of performance-impacting problems. In particular, and in contrast to traditional monitoring, operators can freely interrogate data post-hoc without the need to pre-program dashboards.”

    This sounds promising and the market responded quickly, but predictably. Large and monolithic monitoring platforms embraced the term and added new features, making them even larger and more monolithic. These monolithic platforms represent a failed choice-based strategy. They assume that all observability data is collected, centralized, integrated and only analyzed by a single tool. Enterprises following a choice-based observability strategy find themselves locked-in to multi-year investments without the flexibility to react to evolving technologies.

    While this has been good for vendors in the incumbent monitoring space, end users haven’t benefited from these enormous platform deployments because the choice-based strategy is built on the faulty assumption that a single pane of glass serves all of your monitoring and observability audiences, from SREs to DevOps and SecOps.

    A choice-based strategy also forces customers into limited options in the future as technology continues to evolve and requirements change. Defaulting to choice-based strategies also leaves little leverage for contract negotiations, data integrations and data access.

    Infrastructure and operations teams need to take a different approach to their observability strategy. Instead of focusing on buying products and making choices, teams need to shift their thinking to creating options. The choices you make today limit your options tomorrow. Tomorrow is unpredictable and uncertain. Taking an options-based approach allows you to be flexible and adaptable today and into the future.

    An options-based observability strategy is built on three pillars:

    • Establishing your observability strategy principles
    • Building composable environments
    • Staging your commitments

    Let’s look at each of these.

    Establish the Principles of Your Observability Strategy

    When asked about their observability strategy, many often leap into detailed plans covering every imaginable use case. They talk about the current state of their infrastructure and how things will be at the end of three or five years, including the skills and functions they’ll need in that time. If you’ve read the above, you’ll recognize this as a choice-based strategy: All the choices they need to make over the next several years are already laid out. This leaves little room for adaptation and options as the world evolves around you.

    Instead, think of your observability strategy as a single principle, or a collection of principles, guiding your observability strategy. Let’s say your core principle is that observability is a key enabler of operational efficiency across SREs and DevOps. We can build on that operational efficiency principle to define what key elements of our strategy has to include:

    • Data sources have utility-like accessibility; always on and available across tools.
    • Metrics are primarily driven by total cost of ownership and optimization.
    • Governance is centralized to the team distributing data to downstream systems.

    These elements are different if your core principle is something like customer intimacy or risk management.

    With principles and strategy in hand, we can focus on the next phase: Composing the observability environment.

    Compose your observability environment

    In an options-based strategy, the key success factor is avoiding large ecosystems that limit flexibility. Many vendors take a “give us everything” approach to observability. These approaches appear to start and end at the destination platform, but they begin even earlier, at data origination. It’s common to see agents only sending data to one destination, limiting the usefulness of the data by locking it into a silo.

    Staying modular means selecting tooling that abstracts the data sources from destinations and puts the power back into your hands. By using an abstraction like an observability pipeline, you can send data from one source to multiple destinations, while managing that data in any way you see fit. For example, this is where your centralized governance can occur.

    When new tools are introduced, upgraded or deprecated, an observability pipeline insulates the rest of your environment and users from those changes. A modular approach also gives you more power in contract negotiations with your other vendors because you’re less reliant on one vendor for all of your observability needs.

    Stage Your Commitments

    A choice-based observability strategy is a large commitment, often taking years. Once you start, stopping becomes impossible. That’s simply unacceptable in today’s environment of uncertainty. Choice-based strategies limit your ability to test, experiment, and learn. At the same time, our infrastructure is critical and it must be reliable.

    An options-preserving approach may involve experimenting with freemium products; only paying once the product’s value has been proven. Cloud-based options are another consideration since they are commonly subscription-based and offer easy try-before-you-buy purchasing styles. Even better, cloud-based options don’t require using your own infrastructure to test new options.

    Every IT team feels the need to react and respond to demands from the business. That urgency to make choices, to solve the immediate problem, can have long-term implications for future flexibility and agility. By structuring your observability strategy around a long-term vision based on preserving options, your team can remain adaptable as technologies and business requirements evolve.

  • IT Ops Takes Center Stage

    IT Ops Takes Center Stage

    When people talk about the last 18 months, most will remember it as a disruptive time when nothing felt certain. Yet, for many IT Ops and network operation center (NOC) leaders, it was the year they stepped out of the darkness and into the spotlight. Suddenly, it mattered deeply who was maintaining those services we relied on to shop, work, eat, live – and it became clear that IT Ops teams – who were previously thought of only as the number you called for a password reset – had been keeping our businesses running all along.

    While there may have been some recognition before the pandemic, that awareness certainly gained momentum during 2020 and, now, shows no signs of stopping.

    As Ben Narramore, senior manager of operations with Sony Playstation Network said at a recent customer conference, “Today, I am able to talk to leaders across the company that I would never have talked to a year ago. They didn’t care about operations, they didn’t care about the NOC, didn’t care about incident management, and now they’re like, ‘Wow, this is great stuff,’” he said. “I’m excited right now. We’re at a place in operations where my team has evolved, doing more than just problem-solving.”

    I am excited to hear sentiments like that, because for the last 10 years or so, application development has been the golden child of IT. The faster you could crank out code, the faster the business could move. For years, every dollar and every resource went to building out amazing applications and services, leaving the IT Ops people to keep it all available and performing. Sure, some companies like Hulu and Expedia had the dynamics worked out and built a modern, optimized organization that could run as fast as it built, but most companies paid a high price in time, focus and resources to keep their services up and running.

    For most companies, it felt like a war of speed versus availability. Engineering wins when they can deliver what the business needs, quickly. They do not need to look beyond their group for much and – if it works as intended – they are good to go. IT Ops, on the other hand, needs a broad view of the entire technology landscape. They need to think about the availability of all services, which requires them to have visibility across the entire enterprise stack, often dealing with the downstream implications of outages and incidents that cross networks, hardware, software and clouds.

    Now, it’s 2021, and the war is over. And it turns out, the rise of operations wasn’t about a revolution. It was about recognition. IT Ops teams needed a seat at the table. They finally got the chance to explain that they want to move fast, too – to streamline operations, consolidate awareness, automate wherever possible and help support and enable the velocity the business demands, side by side with their partners in engineering.

    Jeremy Bragg is an infrastructure automation architect at Abbott Labs. I thought his role description was really compelling; here’s how he explained it: “Probably the most interesting aspect about being in automation is that we break down silos. We cannot just be a server team or a network team or a mobility team. We work across the board with all the teams to try to further their causes. We ask ourselves: where are we doing manual work today that we could automate?”

    Over the next five years, I believe more teams will start to take advantage of AI and machine learning to set the pace for the modern IT organization; more of the mundane work will be automated so they are free to lead innovation, not just react to it.

    Ben Narramore also has a lot of passion for that very topic. “If I look at my teams in the last ten years, we were firefighters; we’re good at it out of necessity. Now, we’re at a point with cloud computing [and] with AI that gives us a chance to reassess our team. I’m excited about how our team has evolved from day-to-day problem solving to looking at our business with the future in mind. Being able to automate manual operations and workflows gave us the ability to do new, more exciting work, as well as helping out with incident management and problem management.”

    My own take on IT is that the shift is happening, and I am excited to have a front-row seat as IT Ops teams step out of the shadows and into the light, taking their well-deserved place as strategic partners at the table, and helping drive the business forward.

  • Empowering Developers Through Lean Code

    Empowering Developers Through Lean Code

    Digital transformation is widely understood as a critical component for increasing user adoption, fueling workplace productivity and innovation, and driving business growth. Software has become increasingly important across countless industries, teams, and functionalities, and as a result, companies are beginning to understand that being an industry leader and gaining competitive advantages starts with software development.

    Easier said than done. Modern software apps are a complex orchestration of infrastructure, deployment pipelines, notifications, user experience, data filtering and processing, and error handling. A good engineering team will be able to account for each of these items and integrate them into a larger, comprehensive infrastructure.

    So what happens when you don’t have a solid, in-house team guiding your software maintenance and development? Consider a lean code development platform; a developer service that pares down the explicit requirements of standard functions, interactions, errors and software infrastructure into implicit scripts, templates, configurations and API requests. Using a lean code solution will give your team the power to build higher-quality features more quickly than they otherwise would on their own.

    There are several ways that lean code platforms can simplify processes for developers. Primarily, these platforms turn explicit requirements into implicit interactions, eliminating the need for developers to pull elaborate infrastructure to solve complex problems. Lean code also gives developers the freedom to customize everything to fit where it is needed, while limiting the amount of code necessary to do so. Lastly, lean code platforms provide tools that assist developers at all levels of the software stack, significantly increasing productivity while cutting down on time and workload.

    How Does it Differ from Low-Code?

    While many people might be unfamiliar with the term “low-code,” they’d probably be surprised to learn that they use it all the time. Low-code describes platforms that require minimal code in order to build applications and processes.

    Essentially, these programs enable users with little to no coding or software development experience to create and use applications. The idea of low-code is to introduce audiences to intermediate level software development concepts such as object-oriented programming, interface design, conditional logic and code syntax.

    However, no matter how helpful these low-code approaches might be, many users still run into the problem of wanting more customization than the low-code system is designed to give. In fact, many users become frustrated with the term “customization” because it’s viewed as the enemy, indicating a need for more advanced software development services than the consumer is able to provide.

    Lean code platforms, on the other hand, are designed to specifically meet developers’ needs at all stages of the software development life cycle. In essence, lean-code prioritizes simplicity, efficiency, flexibility and accessibility with the goal of making software development accessible on a larger scale.

    Instead of encouraging you to reduce your development team, lean code platforms allow users to expand development through exploring new avenues and abstracting away the explicit requirements of traditional software development.

    What Does Lean Code Have to do With Productivity Infrastructure?

    Many forward-thinking companies have gone beyond simply building technical solutions to specific development problems. Instead, they’ve taken the advancement of lean code platforms to a new level by providing fixes that meet developers at the central stages of the integration process.

    Because lean code offers such integrated solutions to developer challenges, the approach is increasingly becoming the norm across the bulk of the productivity infrastructure category — an emerging market category that combines elements of cloud communications services, API platforms, low-code development tools, business process automation and AI software development kits.

    If you’re looking to implement a lean-code platform, consider the following features:

    • Intelligence: By making it easy to pull information from unstructured data sources and flip it into more meaningful material, machine learning (ML) replaces the manual effort to build, implement and maintain models.
    • Automation: Workflows, triggers, templates and notifications can be abstracted away for common use cases.
    • Security: When handling user data, security is a top priority. From protecting health records to pet names, data security is crucial.
    • Connectivity: The ability to connect to a relevant data source can make or break an infrastructure.
    • User experience: User interfaces should follow expected conventions for whichever task the user is trying to accomplish. Maintaining a successful user experience requires constant evaluation.

    How will Lean Code Become Market-Leading Software?

    Lean code development platforms have emerged as market-leading software development solutions in three major areas:

    • Productivity Infrastructure: More companies have pivoted to providing full-stack development services that assist developers in building out native communications, search and indexing, scheduling and financial experiences.
    • Software development and deployment workflows: Continuous integration/ continuous deployment (CI/CD) pair down complex build and deployment pipelines to basic configuration files.
    • Infrastructure deployment: Infrastructure as a service (IaaS) products assist in deploying scalable web services via API requests and simple configurations.

    Where are lean code development platforms headed?

    It’s safe to assume that, given their ability to provide efficient and effective infrastructure to development teams, lean code has a bright future ahead. The number of developers in the space is only growing. A Slash Data survey from 2019 estimated there to be just under 19 million active developers across the globe and expected the number to grow to 45 million by 2030.

    As the market for software development rapidly expands, it would make sense to assume that the need for a platform that solves problems across the entire software ecosystem would, too. Lean code development platforms are emerging as a top choice for companies that want to build out a leading software platform into the future.

  • Protecting Data from Insider Threats

    Protecting Data from Insider Threats

    “Insider threat” can be defined as the threat an employee or a contractor will use their authorized access to, wittingly or unwittingly, do harm to an organization’s security. Certainly, as organizations become data-driven, embrace data democratization and release new applications at breakneck speed, the probability of an insider threat is rising sharply.

    A common example of an insider threat is granting permissions to on-call SRE engineers who may need to access system tables across multiple databases, inspect contents of S3 buckets and make changes to the production environment to fix availability issues. Every two weeks, the composition of the team may change, and the composition of services and databases may vary dramatically.

    While it is impossible to completely safeguard against insider threats, organizations can better protect their data by investing in best practices that (a) reduce the likelihood of such a threat playing out, (b) limit the blast radius from an insider attack and (c) help detect the attack as quickly as possible. The third point is especially important – the Ponemon Institute’s recent Cost of Insider Threats 2020 Global Report claims that it takes, on average, 77 days to detect and contain an insider attack.

    A DevOps Framework for Protecting Data From Insider Threats

    For security measures to be effective, they must be non-invasive, especially in agile organizations where DevOps teams are expected to push releases several times a day.

    Implementation of Access Policies-as-Code

    Implementing access policies-as-code automates workflows for user and service access authorization. Using this approach, manual policy practices are replaced with “code” (policy files readable by your team and by the systems that enforce the policies) so that the right people on your team can request, approve and publish policy changes instantly.

    As an example, here’s how the above scenario may be implemented as code:

    Engineers can add themselves to on-call status with a code push in GitHub to a Terraform script, with only the team lead authorized to approve this action. (See example 1, below.)

    insider threats
    Example 1

    Using frameworks like Sentinel, changes to production environments can be limited to only on-call engineers. (See example 2, below.)

    insider threats

    Example 2

    Similar policies can limit only those engineers who are able to read unencrypted content of S3 buckets by fetching keys from Vault. (See example 3, below.)

    insider threats

    Example 3

    The above examples demonstrate guardrails that allow engineers to perform their duties, while drastically reducing the administrative burden on leads, resulting in fewer errors. An additional benefit of this approach is that it forces a public review of all changes to policies, which makes the security model more robust.

    Zero Standing Permissions

    Zero Standing Permissions (ZSP) is a DevOps-friendly security tenet that requires any administrative privilege to be available to users only when needed, and only for as long as needed. It enables teams to configure access, via administrative privileges, to highly critical components of their infrastructure for a limited duration, with restrictions on the scope of those privileges.

    Here are a few examples of how teams can implement ZSP for their databases:

    • Create an SRE role on the database that can update and delete database tables and a private Slack channel of db-admins that can allow on-call engineers to assume this role.
    • Implement a service that can be used to request and grant access via this SRE role for a default of 60 minutes. After that time, the access is automatically revoked and needs to be granted again.
    • Further implement policies in BrewOPA that restrict an SRE from dropping tables and databases, which avoids any disasters.

    Audit Trail for Privileged Activity

    SIEMs are increasingly the nerve center of every security organization. While effective threat-hunting and incident response requires several years of expertise, DevOps teams can significantly bolster the security of their cloud platforms by investing in log collection and basic analytics for all privileged activity in their environment.

    These tools include services such as the following:

    • Session recording for all Linux boxes using Cmd
    • Logging activity across databases with Data Cloud services
    • Centralizing and collecting all environment logs using services like CloudTrail.

    These logs can be visualized in tools like Kibana, where teams can look at activity pattern anomalies. These could include:

    • Someone opening up an FTP connection to outside the organization
    • Creating a manual copy of tables outside of the database
    • An increased number of authentication failures across resources

    While the concept of “least privilege” as a safeguard against insider threats to data has existed for some time, the advent of cloud-native and DevOps methodology has exacerbated the challenge of implementation. In recent years, security-as-code and infrastructure-as-code have emerged as good models for implementing security solutions, and are rooted in the above framework. The future encourages collaboration across various stakeholders, documents security policies for easy review and builds on existing tools and workflows. It will help limit insider threats, limit privilege sprawl and prevent “user fatigue.”

  • 4 Ways to Scale DevOps in 2021

    4 Ways to Scale DevOps in 2021

    January is a month often dedicated to planning, setting intentions and preparing ourselves professionally and personally for a fresh year. In my own capacity as a DevOps and SRE leader, many of the clients I work with will reinvigorate internal efforts to streamline how they work in Q1.

    I often advise clients to make very concrete goals and milestones for their technology organizations to become more agile, streamlined and modernized. Setting these goals is helpful to articulate to the wider business where you want to go and how you plan to get there. Here are a few DevOps-related initiatives your team may want to tackle in the first half of this year.

    #1: Tighten Security through DevOps

    Securing your organization’s infrastructure is an evergreen priority for technology leaders. When done properly, security should be baked into all DevOps processes (sometimes called DevSecOps) from the design phase all the way to application maintenance. To be effective and not slow down development, security processes need to move at the speed of today’s fast-moving environments, which is near impossible, or, at least very challenging, unless security is well automated.

    Automated security measures can be applied to the full life cycle of product development and should make development time faster and smoother in the long run if you can catch vulnerabilities early. Identify automation tools that work for your team and experiment with what works best.

    Similarly, in fast-moving environments, it’s tempting to play fast and loose with privileged access to unblock individuals and teams. Lax policies for user rights exposes the organization to too much risk from potential hackers. Implement a solution or protocol to keep access controlled. Policy and governance are critical for well-rounded DevSecOps. Communicate policies regularly with the entire organization. Ensure that all engineers know how to comply and enforce them.

    #2: Make DevOps an Organizational Culture

    Much like security, DevOps needs to evolve from an individual role to a cross-functional, holistic, cultural approach to developing software. Organizations should foster a DevOps culture rather than identifying specific individuals in DevOps roles.

    So how do you know when this culture shift has happened? A good pulse check is to see how quickly your company can adapt to change. How mired is your team in legacy code or infrastructure? Can individual teams operate autonomously? Teams and individuals that are independent and accountable are able to continuously experiment and learn at a much faster rate.

    DevOps cultural transformation won’t happen overnight – but it can be expedited if embraced by the entire company. I typically advise companies to focus on making incremental, consistent progress that builds autonomy, trust and transparency. These are the building blocks of a thriving DevOps culture.

    #3: Simplify Your Infrastructure

    Complexity, over time, creates duplication, unnecessary systems and technical debt. All of these have a cost, and create security vulnerabilities. Create organizational goals to clean up and reduce complexity. I often suggest teams create a roadmap to move toward self-service and building platforms. As you focus on building a DevOps organizational culture, you will likely notice that teams will identify opportunities to simplify and standardize, which will also remove complexity.

    #4 Review system SLAs and SLOs

    Establishing and monitoring key metrics are a fundamental part of any SRE role. Service level agreements (SLAs) and service level objectives (SLOs) need to be defined and monitored to ensure your team is fulfilling them. System availability (uptime, latency, etc) is core to any cloud SRE team. Define what reliability means for your organization. January is a good time of year to start fresh and articulate these metrics, how they relate to your core business and what investments or changes need to be made to achieve them.

  • Up Your DevOps Game: It’s Time for NoOps

    Up Your DevOps Game: It’s Time for NoOps

    NoOps is here and it’s more than just a buzzword or an ideal. The next generation of platform-as-a-service (PaaS) platforms are evolving to minimize the developers’ operational burden and reduce dependency on DevOps for deployment. Dev teams that don’t want to deal with operations or don’t have in-house operations resources should consider deploying applications on a NoOps platform.

    To understand the concept of NoOps, we need to start with an understanding of how deployment management has evolved over the decades.

    From Hardware and Software and Back Again

    It’s been a long trend of back and forth between developers and “deployers.”

    It was once mandatory for software developers to be fluent in the hardware for which they were creating software. They wrote the code and then worked with the operations team to deploy it on in-house servers. Everything was tightly coupled. The first mainframes—IBMs and SiliconGraphics machines—ran on custom-built software, and the developers were responsible for the deployment and management. The developers had the machine first, and then had to create the code to make them run—assembly or low-level language or even hardware code; rope memory, in some cases.

    Then came the x86 revolution, when hardware became more standardized and was decoupled from the complexities of deployment. Developers started focusing on the software and organizations had separate teams that would manage the deployments.

    With the advent of virtualization and infrastructure as a service (IaaS), the deployment of software now depended on APIs for infrastructure provisioning, and we returned to an integrated development and deployment function—in other words, DevOps.

    The Problems DevOps Created

    The problem then became one of choice: As APIs multiplied, developers had to decide which APIs would be most relevant, efficient and useful to deploy their software—and on what infrastructure: bare metal, virtual machines such as EC2 and VMWare, Kubernetes, Swan, OpenShift, serverless, Lambda, Knative, etc.

    Not only did the teams have to decide on what IaaS to deploy the actual software, but they needed to also choose how and where the layers and logs would be monitored—and even where those logs would be stored. They need to consider things such as traceability and horizontal or vertical scaling issues, to name a few.

    This has resulted in the creation of bespoke permutations and combinations that work great … until vendors’ upgrades require them to keep up with tens or hundreds of configuration changes. Most teams are keeping it together, but it definitely can be done more efficiently.

    Now It’s time to Up the Game to NoOps

    It’s time for the next approach: Limit the number of choices to create standard best-in-class operations that deliver economies of scale and easily evolve with minimal hassle.

    NoOps simplifies cloud operations—everyone can do things the same way. NoOps aims to “completely automate the deployment, monitoring and management of applications and the infrastructure on which they run,” according to Forrester, which coined the term.

    NoOps is about standardizing the approach to deployments and reducing the number of variables, bringing simplicity. At its core, NoOps is focused on automating deployments and executions that are predictable and repeatable.

    NoOps & Standardization

    The development and increasing adoption of containers are critical to the entire NoOps philosophy. Containers provide the ability to independently deploy services and applications, automating and standardizing the process to deploy anything, anywhere. Using containers delivers the tremendous portability that hasn’t been seen since the development of generic hardware.

    With encapsulation within the container, whatever is running inside will behave the same no matter where it is deployed.

    The NoOps-containers movement will transform the entire DevOps industry.

    Let’s go back even further. In the 1800s, few homes had electricity. They hooked up whatever appliance they had to whatever generator they were running and kept their fingers crossed. Once governments took over the role of bringing electricity to every home with standardized voltages and outlets, the appliance industry took off.

    A true NoOps platform will provide a standard “socket” that works for everything. The developer will simply build the app, and it will “plug in” anywhere. With NoOps, developers don’t have to worry about how the application will work, deploy or scale, nor do they have to consider reliability or elasticity. The platform will deliver it automatically.

    The “Gift” of Containers

    Containers free developers from managing infrastructure and reduce the operational burden associated with building and deploying applications. They also minimize application conflicts among different environments and improve portability and team collaboration.

    Containers make application deployment easier than ever before. Continuous integration doesn’t change as part of this process. Adopting containers as part of NoOps simply automates continuous delivery—developers write the code, commit the changes and push it to the container’s remote repository. Based on the Docker file, an image will be built and the containers that run your application will be deployed. Since all deployments are rolling, no downtime occurs.

    Leveraging containers provides speed, agility and flexibility to any infrastructure, thus bringing cost savings and dynamism. Containers can flexibly host microservices, which can be scaled individually, as necessary. Using CPU, memory or even custom business rules, developers can set auto-scaling policies so applications are ready to serve an increased number of requests whenever needed. As the platform monitors application performance, it can use these insights to intelligently architect a right-sized infrastructure.

    Putting NoOps Into Play

    If you’re considering adopting NoOps, look for a PaaS provider with the expertise to make the right platform choices for you based on your application development language of choice. For example, if you are looking to host a Ruby application, ensure that the provider has the requisite Ruby expertise.

    A true NoOps platform will streamline deployment, automatically scale based on your business requirements and provide outstanding performance. The automation dramatically boosts productivity and reduces costs associated with deploying and managing applications.

    NoOps platforms monitor operations and automatically resolve issues using built-in observability mechanisms and intelligent alerts. Centralized logging, metrics and traceability make it easy to identify issues and initiate fixes without expensive downtime. Moreover, the platform should ensure strong security boundaries and adhere to applicable compliance regulations.

    When “going NoOps,” also make sure that the solution provides comprehensive documentation and 24×7 support for emergencies that would normally be handled by an in-house DevOps team.

    Wrapping It Up

    NoOps can eliminate much of the work associated with deploying and monitoring applications by simplifying and streamlining operations and reducing the need for extensive in-house DevOps support. While it’s not a one-size-fits-all solution, it delivers great value helping companies of all sizes from start-ups to enterprise teams save time and money and better redirect their resources toward high-value activities, creating the best code to deliver the highest quality products.

  • 7 Important Truths About Chaos Engineering

    7 Important Truths About Chaos Engineering

    As a relatively new practice, chaos engineering has plenty of myths surrounding it, from randomly shutting down production systems to requiring huge investments of time and money. There’s a lot of confusion over the purpose, the value and the practice of chaos engineering. This presents a problem for DevOps teams, especially since more than half of the teams surveyed in the 2019 Gartner DevOps Survey listed improving system reliability and release quality as one of their top five DevOps objectives.

    In this article, we’ll clear up some of this confusion by presenting seven important truths to help you make an informed decision about chaos engineering and how it can help your engineering organization.

    Truth 1: Chaos Engineering is not Chaotic

    What comes to mind when you think of chaos engineering as a practice? If the answer is causing random production outages, you’re not alone. Tools such as Chaos Monkey popularized this idea, but for most teams practicing chaos engineering, it is a well-planned, controlled process that aims to mitigate chaos rather than cause it.

    The goal of chaos engineering isn’t to add chaos, but to mitigate chaos.

    It’s true that it involves creating potentially harmful conditions on otherwise healthy systems. For example, we might test our application’s ability to handle load by increasing CPU usage on our servers. With enterprise solutions we have full control over which systems are affected by this test (known as the blast radius), how much CPU we consume (known as the magnitude) and how long it runs for. We can also immediately stop the test and roll back its impact in case of unexpected consequences.

    We also know exactly what conditions we’re introducing into our systems and can revert the changes at any time. Yes, we’re still causing harm, but we’re doing so in a way that helps us learn about our systems and reduces the risk of failures, both real-world and induced.

     Truth 2: Developers Care About Reliability

    Developers aren’t only interested in building new features. While development teams often prioritize feature development, this is likely due to business-driven initiatives resulting in rapid release schedules. Developers—especially those who have responded to incidents or worked through bug reports—understand the value of resilient software. They just don’t have time to build it.

    The problem is that this creates a reliability gap in our applications. Rapidly building features without adequately testing their resilience creates failure modes, leading to problems that developers need to go back and fix at the expense of new projects. The longer we go before finding these failure modes, the more likely we are to experience unexpected behaviors and outages in our applications, and the more expensive a solution will be.

    Increasing development velocity without testing for reliability increases the likelihood of failures, causing our expected availability to drop.

    To prevent this gap from forming, we need to prioritize and provide the appropriate tooling so developers can test their code and improve the resilience of our applications. This leads to our next truth.

    Truth 3: You Have Enough Time for Chaos Engineering

    Chaos engineering saves you the time you would spend responding to, troubleshooting and fixing production incidents. As you build resilience through chaos engineering, you’ll reduce the engineering time, effort and costs spent on incidents and can focus more on your core competencies.

    That said, adopting chaos engineering does require some time investment. Engineers will need to learn the principles and practices, adopt new tools, run experiments and implement fixes. However, this is relatively minor compared to the cost of production outages. Without chaos engineering, your engineers will need to pause their regular work and rush to restore service. This is time they could be spending on feature development and other value-creating tasks.

    A few minutes of proactive testing can uncover defects that would’ve taken hours to fix in production.

    Truth 4: Your Systems Support Chaos Engineering

    You don’t need to use specific hardware, software, or cloud platforms to benefit from chaos engineering. While some tools have specific requirements, most only require access to the host operating system. Some can even target specific resources, such as containers.

    Truth 5: You Don’t Need Comprehensive Observability (But It Helps)

    You don’t need a comprehensive monitoring or observability practice in place to benefit from chaos engineering. Basic metrics provided for free by your cloud provider can still provide important insights into how your systems are responding to experiments, and some chaos engineering tools provide basic monitoring for you.

    That said, chaos engineering provides significantly more value when used with monitoring and observability. We want to know how our systems are responding to our experiments, and the way we do that is by collecting metrics and traces. Even if you’re currently implementing a monitoring solution, chaos engineering can help streamline this process by helping you determine which metrics to track, fine-tune your alerts and avoid alarm fatigue.

    Truth 6: Chaos Engineering Pays for Itself

    Like any new tool or practice, chaos engineering requires an upfront investment of time and money. Solutions need to be evaluated and purchased, and engineers need time to warm up to the practice. However, the value it provides far exceeds these costs. Using chaos engineering can help you:

    • Avoid losses in revenue and productivity due to incidents and engineering time spent resolving incidents.
    • Improve your ability to generate sales by helping you build reliable systems and prepare for periods of increased demand.
    • Save costs by helping you optimize resource allocation and test for non-essential infrastructure.

    Truth 7: You Can Get Started Today

    Getting started with chaos engineering can be as easy as deploying an agent to your systems and choosing a resource to target for experimentation. You don’t need to have a stable, mature SRE practice or enterprise-grade infrastructure. In fact, you can use chaos engineering to help establish your SRE practice or accelerate migration to a new platform.

    The most important truth is that no system is perfect, and there are failure modes hidden all throughout our applications and infrastructure. Chaos engineering lets us uncover these failure modes and transform reliability from myth to reality.

  • How to Make a Self-Healing IT Infrastructure a Reality

    How to Make a Self-Healing IT Infrastructure a Reality

    In 2019, enterprises worldwide reported that every hour of infrastructure downtime cost them an average of $301,000- $400,000. If a system is down for five hours, that’s $1.5 million lost—and that’s on the low end. Five hours of downtime can impact companies for months down the road. It’s in the hands of DevOps practitioners to ensure systems are up and running and budgets are kept in line. But if they don’t have the proper tools on hand that allow them to juggle their responsibilities while adding value to the business, it makes their jobs that much more difficult—or, dare I say, impossible.

    Teams simply can’t afford to go offline to fix incidents, but neither can they afford to manually sift through data to find the root cause. By automating the incident management process, developers have more time to focus on building new products and capabilities that drive revenue. With a self-healing IT infrastructure, teams can tackle these issues before they become larger issues costing millions of dollars.

    Imagine you’re running to catch a bus and your heart rate increases, but when you sit down, it doesn’t come back down to a normal rate. This could quickly become a much larger, even fatal issue. In this instance, the body should “self-heal” to get back to a normal rate. On a similar note, a self-healing IT infrastructure allows teams to quickly get back on track by fixing issues before they become a $1.5 million problem.

    Let’s look at how DevOps practitioners can turn a self-healing IT infrastructure into a reality.

    Self-Healing Infrastructure Needs Observability and AI

    Observability is the practice of collecting deep data from applications and services to provide insights through three core components: logs, traces and metrics. While these three components are essential to a self-healing infrastructure, they are most powerful when used together. Similar to our senses of sight, hearing and touch, they each tell us something different but are equally important. When combining the power of all three, development teams can determine where, when and how an incident occurred and take action as needed.

    We all know the pain of traditional monitoring tools. They don’t surface immediate issues or identify root causes. They rely too heavily on developers to interpret and analyze immense amounts of data then sift through it in a tedious and time-consuming process requiring extra budget for more human power, and extra time to identify and remediate incidents. Using today’s cloud-native self-service approaches to combine observability and artificial intelligence, teams can set up tools themselves that automate the process of gathering and correlating metrics, logs and traces at machine speed to provide a complete, intelligent and actionable picture of what’s happening and why. And that’s where a self-healing IT system starts.

    The Future Is Within Reach

    By ingesting observability data, applying AI to analyze that data and create insights around root causes, then leveraging automation in a closed-loop, dev teams can not only see what’s happening but also take action on it for remediation. When this closed-loop process is done intelligently at machine speed, that’s when self-healing becomes a reality.

    The idea of a self-healing IT infrastructure doesn’t have to be a distant vision. In fact, the democratization of cloud computing and advanced data science have put the required observability technology within reach of teams of any size with any budget. When artificial intelligence and observability come together, DevOps practitioners can operate less and innovate more.