Category: DevOps in the Cloud

  • Cloud Migrations Demand Risk and Compliance Maturity

    Cloud Migrations Demand Risk and Compliance Maturity

    The COVID-19 pandemic brought undeniable disruptions for organizations and their employees whether business, personal or otherwise. Across the globe, businesses and governments alike were forced to try and manage these disruptions. For nearly all organizations, digitization initiatives were accelerated in a short period of time, from implementing work-from-home policies and launching new applications to support a distributed workforce to adopting artificial intelligence (AI) to adapt supply chain processes and more.

    According to Gartner, by 2022, 30% of all security teams will have increased the number of employees working remotely on a permanent basis and by 2023, 40% of all enterprise workloads will be deployed in cloud infrastructure and platform services. The distributed cloud enables organizations to provide products and services when they’re needed in this era of work-from-anywhere, whether to their employees or customers.

    Even in sectors such as energy and utilities, which are historically heavily reliant on standard on-premises installations and had often avoided cloud adoption, business leaders are realizing the value of migration, especially in light of the global pandemic. These businesses are able to deliver digital products with speed, experiment with tools such as artificial intelligence (AI) and robotic process automation (RPA) to increase productivity and to lower the total cost of ownership (TCO) across assets, among other benefits. Still, cloud migration risks abound, and new challenges may arise. A risk that cannot be ignored is the cybersecurity risk. Meeting regulatory IT compliance and managing risks involved with cloud computing are top challenges facing those migrating their workloads to the cloud.

    Cloud Migrations and Security: Clear Risk and Compliance Gaps

    Many cloud providers such as Microsoft Azure, AWS, Google Cloud and others have a global network of service models that include compliance teams and consulting organizations that help with risk and compliance for their cloud instances, whether public, private or hybrid cloud. Many of them have even built tools for customers to use to implement basic risk and compliance management in-house, leveraging relevant data and applications. However, monitoring and meeting security and compliance controls that span people, processes and technology for cloud environments, and in the broader context of the enterprise, is complex. This is one cause of cloud migration risk, and is a challenge for many other reasons; lack of measurement, visibility and accuracy are three of the greatest risks when migrating to the cloud. The point solutions that currently support most cloud instances don’t elevate the posture of cloud environments to that of the enterprise risk posture, and the majority of assessments still remain point-in-time and qualitative. Metrics are fractured and far from holistic, and very few, if any, solutions can provide insight beyond compliance and into real-time risk management.

    IT and security regulations and standards are filled with requirements that were created before the cloud became a commodity. In the energy sector, for example, cloud security isn’t taken into account because regulators and industry leaders couldn’t fathom those platforms becoming as pervasive as they have, because on-premises installations were standard to the industry. On-premises installations are still a mainstay in energy, power and utilities, and for those who have become more comfortable with cloud migration processes, there is a clear and pressing need to leverage their human capital, processes and technologies to implement robust risk management practices.

    Beyond regulatory compliance lags, many distributed organizations opt to have multiple providers in place, requiring a multi-cloud approach to compliance requirements and risk assessment. As more organizations consider cloud migration risks and begin shaping their cloud migration strategies, there are some innovations that address risk management and compliance in the cloud, but not many. Measuring, managing and reporting on compliance frameworks, making the shared responsibility model actionable, and getting a view into risk are all serious challenges. Cloud providers will continue to mature and bring new innovations to their services, but, to date, there hasn’t been a lot of anticipatory work done in this area. The focus has largely been on creating reactive solutions. In heavily regulated countries, the challenges only become greater.

    Leverage AI Automation for Compliance and Risk Management

    There is a shift occurring in cybersecurity and IT risk management, calling for the dramatic disruption of the legacy IT governance, risk and compliance (GRC) space and demanding a reevaluation of how we manage compliance and risk in the digital age. For years, data has been aggregated manually and analyses performed on out-of-date information. With the increasing availability of automation, the five functions of the NIST Cybersecurity Framework – identify, protect, detect, respond and recover – are becoming more continuous in nature and shifting into real-time management, from assessment to reporting and more.

    Leveraging this technology in the cloud is no exception, but those who look to reinvent their approach must look for solutions that go beyond the siloed capabilities of cloud security posture management solutions and similar markets.

    Ultimately, the true test of this next-generation approach comes when organizations are able to roll all of this data up to risk. With risk metrics that are supported by drill-downs, trend reports and risk profiles, executives can get the visibility they need into their posture with the most up-to-date data, informing their key business decisions. Using this next-generation approach to risk will inform global expansion, allow executives to evaluate risk across lines of business, and increase cyber maturity in any cloud-based organization.

  • Too Many Organizations Get Public Cloud Wrong

    Too Many Organizations Get Public Cloud Wrong

    The cloud has been getting a lot of press in the past few years—and with good reason. The elasticity and flexibility of resources are transformational for businesses. However, there are also drawbacks. For most enterprises, it’s a significant undertaking to migrate on-premises applications into a public cloud environment.

    Modern applications today are rarely self-contained entities. Application dependencies occur when software components, applications and servers rely on one another to deliver a service. The typical enterprise manages hundreds, or even thousands, of interconnected components and services, forming a complex web of dependencies.

    If an application is migrated to the public cloud without understanding the dependent services, severe performance degradation can occur with applications that are still on-premises. For example, moving an e-commerce shopping application to the public cloud but leaving a key database on-premises will result in consumers experiencing a slower response.

    The Repatriation Problem

    IDC research shows that fewer than 20% of enterprise applications have been moved from onsite data centers to the public cloud, not because companies do not see value in the cloud, but because the process of getting those applications to the cloud is complicated.

    A survey of 350 cloud decision-makers in the United States and UK commissioned by Virtana and fielded by Arlington Research in November 2020 confirmed this. More than two-thirds (68%) of respondents have already migrated 25% or more of their applications to public clouds, supporting the IDC finding.

    Clearly, businesses recognize the value of the cloud. But another surprising result confirms that the migration process may be problematic. A whopping 72% of respondents said they have moved one or more applications from the public cloud back on-premises. Considering the time and effort cloud migrations require, this is a striking result. While there may be cases where repatriation is undertaken strategically to support evolving business needs, the reasons cited by respondents show most were rollbacks required because of unanticipated problems.

    There Is No Single Reason Why

    Almost half (41%) of respondents said they repatriated applications that should have stayed on-premises. The reality is that not all workloads are a good fit for the public cloud environment. You need a solid understanding of the various attributes of different workloads, such as data, back end, privacy and security requirements, to determine their inherent suitability for a public cloud environment.

    Technical provisioning issues challenged 36% of respondents. The sheer number of configuration options available—combined with the variety of computing dimensions that you have to take into account—means there are a lot of puzzle pieces that have to fit together, or you could run into problems.

    Almost one-third (29%) of respondents reported application performance degradation in the public cloud. The public cloud is a fundamentally different environment than an on-premises data center, and creating a hybrid environment further alters the nature of the infrastructure, so you cannot simply expect workloads to attain the same performance levels as you migrate them. And if performance slows to an unacceptable level, any benefits you gain by moving applications to the cloud are overshadowed.

    Around one-fifth (21%) admitted that they selected the wrong public cloud provider. Cloud providers are not all the same, and picking the wrong one, for any reason, can be an expensive and disruptive mistake.

    Finally, one-fifth (20%) of respondents repatriated applications because of unexpected costs in the public cloud. The public cloud’s pay-for-what-you-use model is appealing compared to the on-premises capex model, where you pay for capacity regardless of whether you use it. But there is a dark side; you have to pay for what you use in the cloud even when that usage is unexpected. Whether that is due to poor planning, an unanticipated surge or workload drifts over time, getting a nasty surprise in your end-of-month bill can put a big dent in your cloud budget.

    Eliminate Repatriation With Better Planning and Observability

    Any one of these reasons alone reflects a challenging setback to overcome. Unfortunately, many organizations run into more than one of these problems. In fact, one-third of respondents cited two or more of these reasons for repatriation, and 12% experienced three or more of these issues. This seems to indicate that organizations’ migration planning efforts are inadequate. It is not for lack of investment—78% of survey respondents said that even during the current challenging economic climate, they are continuing their migration to the public cloud at the same or an accelerated pace, signaling they believe public cloud deployments to be a strategic investment.

    The real problem is a lack of precision in observability—of workload characteristics in the data center pre-migration and of workload utilization and performance in the public cloud post-migration. The better you understand your applications, in minute detail, within the context of the environments they are moving from and to—a ‘know before you go’ approach—the less likely you will have to hit the “undo” button on your public cloud migration.

  • How to Respond to Rising Cloud Costs

    How to Respond to Rising Cloud Costs

    As organizations ascend to the cloud, their cloud expenses are floating to the same lofty heights. But are all cloud costs necessary? The answer is a resounding no — there’s a lot of waste.

    In 2020, organizations wasted $17.6 billion on idle cloud resources and cloud overspending, according to ParkMyCloud. Data egress, upgrades and subscriptions to SaaS tools also crowded the cloud invoice inbox. With digital innovation a pressing concern, cloud spend is forecasted to rise significantly in the coming years. Without the right fiscal discipline in place, companies face escalating fees.

    I recently met with Eugene Khvostov, VP of product management at Apptio, to discover how organizations can rein in cloud and SaaS spending. According to Khvostov, IT departments require increased visibility, optimization and responsibility if they are to curb the cloud checkbook. Below, we’ll see what he means and discover useful tips for reducing unnecessary expenditures. Following these general guidelines could help right-size your cloud offerings across AWS, GCP, Azure and other cloud-based tools.

    Where We Are With Cloud Costs

    The pandemic has ushered in significant growth in the cloud computing and SaaS space. We’re also seeing a multi-cloud fracturing effect, as cloud service providers like GCP and Azure reach parity with AWS offerings.

    Cloud tooling grants extraordinary flexibility, elasticity and scalability. Being able to spin up or wind down an instance on a whim has permanently altered how software architecture is built. Yet, with great cloud power, comes great responsibility. “How do I make sure I’m not wasting money in the cloud?” Khvostov asked.

    According to Khvostov, as more organizations move to hybrid and decentralized multi-cloud models, more and more questions arise: What is the total cost per service? Who is spending what? Is cloud spend actually driving growth, and, if so, how much? How are new tools being procured?

    Three Cloud Financial Management Strategies

    To address these concerns and plan for cloud spending increases, Khvostov recommended three general strategies. These three phases can be thought of as research, remediation and prevention.

    1. Access and Visibility

    “You’ve got to have access and visibility,” said Khvostov.

    First, companies require access to their spending data. In this case, large, general invoices are not granular enough. You need breakdowns by service, said Khvostov, to access more fine-grained spending habits.

    Second, you need to equate those spending habits to team usage. This requires improved visibility into your organizational structure and service usage tracking at a team-by-team level.

    2. Optimize

    Next comes optimization. Ensure cloud footprints are appropriately sized for each application, and ensure you don’t have any idle resources, Khvostov said.

    Where do you have the opportunity to optimize? Well, it could be as simple as shuttering an orphaned volume, such as an Amazon Elastic Block Store (EBS), which could persist even after an EC2 instance is terminated.

    “BS volumes tend to persist unless you select the Delete on Termination,” AWS explained. So, check your configurations. Amazon provides a way to detect this using OpsCenter.

    3. Responsibility

    Once things are optimized, you need to maintain standards. “You must ensure teams drive responsibility for spend management and optimization,” said Khvostov.

    But how do you instill an optimization mindset and empower engineers to take action? Khvostov said it’s all about demonstrating a measurable return on investment (ROI) and continuing discussions between stakeholders.

    A recent report from A Cloud Guru found that 90% of organizations plan to expand their cloud services over the next one to three years, but 44% lack a plan to upskill their workforce. Most groups have their work cut out for them if they are to assign accountability for cloud costs ownership.

    Specific Cloud Costs That Bite

    So, what are some other particular areas we should be watching? Well, arguably, the most common cost is hidden data transfer fees. Surprise ingress and egress fees can be costly — take the story of Chris Short, whose personal site was charged $2700 for an unexpected 30.6 TB data transfer.

    Watch out for other cloud costs around:

    • Services automatically launching or autoscaling.
    • Idle resources due to “orphan” instances or overprovisioning.
    • Price lock on granular commitments. You could be stuck in a savings plan on an Amazon EC2 R5 instance, which would be lost upon upgrade to R6. (Khvostov acknowledged that AWS has remedied this situation, to an extent).
    • Legacy, less-efficient code can rack up cloud costs, The Information reports.

    The Cloud Economist Role Is Trending

    To unearth these costs, some believe a brand new role is required. A cloud economist is a cloud financial manager that helps spot fees, directs further cloud optimizations and calculates overall ROI from cloud investments.

    The rise of cloud financial offers is an accelerating trend, Khvostov said, one that requires the right tools and practices within an organization to become a trusted ally. “It’s definitely a legitimate role in the sense that it’s critical to organizations,” said Khvostov. “It’s specialized enough that it requires technical and financial skills.”

    Hiring a cloud economist, or building a cloud financial center of excellence, could provide a lot of value. Yet, gauging the ROI of this role isn’t immediately obvious. Some optimizations could take time to visualize, Khvostov said. A certain change could instantly invoke 20% cost savings, but you may not register that impact for 30 days or more.

    “It’s not just about cloud infrastructure — it’s about your SaaS spend, as well,” Khvostov added. He believes companies desperately need a window into what infrastructure-as-a-service (IaaS) and automation, like Chef or Jenkins, cost, along with the time savings and dollar ROI their usage equates to — because, he said, “You can only right-size what you know.”

    Trimming Cloud Spend and Cloud Costs

    Some IT folks simply fail to understand the implications of their cloud resources. Surprisingly, a recent survey by SoftwareONE revealed that 30% don’t realize that everything in the cloud is metered. To rectify this knowledge gap, organizations could use better awareness into their cloud holdings.

    The cloud is the future, and we must prepare responsibly. Yet, having fiscal discipline in a world where every software function is commodified “as-a-service” is tricky — teams require financial visibility into their cloud holdings. To rein in costs, Khvostov recommended treating it as a continuous process: access and visualize spending data, optimize your existing cloud footprint and disseminate responsibility among teams.

    Once you have a better frame of reference, you can begin to right-size cloud, right-size SaaS and appropriately right-size the team. Eventually, with the right multi-cloud and SaaS financial management strategies, perhaps that cloud bill will start to feel a little lighter.

  • Why Every DevOps Team Needs a Spot Instance Strategy

    Why Every DevOps Team Needs a Spot Instance Strategy

    Most DevOps teams use the public cloud extensively and focus a lot of energy on reducing cloud costs. According to one estimate, U.S. businesses spent $14.1 billion in 2019 just on wasted, unused cloud resources. Spot instances are one of the most important ways to reduce the cost of public cloud services.

    Spot instances are not a new development. Amazon was the first to announce this pricing option in 2009, turning cloud computing into a market influenced by the dynamics of supply and demand. Microsoft Azure took more than a decade to follow suit, announcing its spot virtual machines offering in May 2020.

    The spot pricing model is simple on the surface but can be complex to implement. Spot instances are how cloud providers sell their unused capacity. When compute instances are not ordered by anyone via the regular, on-demand pricing model, the cloud provider puts them up for auction at very favorable prices, usually around 10% to 20% less than the on-demand cost.

    However, it’s not so simple to get this 80% to 90% discount. When another cloud customer requests the spot instance, the cloud provider sends a notification, and you need to immediately move your workloads before the instance is terminated. This makes it difficult to use spot instances for stateful, database-driven applications, or those that require high availability.

    This is not the only challenge. The market price fluctuates and is much more volatile than on-demand pricing. This means the level of discount that can be achieved over time is relatively unpredictable.

    However, there are at least three ways savvy DevOps teams can make great use of spot instances:

    1. Automation—DevOps teams are proficient with a range of automation tools, including configuration management, infrastructure as code (IaC) and cloud provider auto-scaling tools. These can be used to manage workloads in clusters and automatically fail over when a spot instance terminates.
    2. Dev/test environments—DevOps teams are responsible for setting up development and testing environments, which are extremely suitable for spot instances because they can usually tolerate brief interruptions, and, in many cases, are not stateful.
    3. CI/CD jobs—running jobs on Jenkins, GitLab and similar tools can be easy to scale on spot instances. These jobs are stateless, and if an instance drops, it’s easy to rerun the job on another.

    AWS Spot Instances

    AWS Spot Instances let you buy unused Amazon EC2 computing power at a significantly discounted price. You can specify a price, and when a spot instance is offered at that price, it is launched with the Amazon Machine Image (AMI) of your choice.

    How Spot Instances Work on AWS

    Spot instances are priced at a variable spot price, which is adjusted according to supply and demand conditions. To see current rates, use the AWS Spot Instance Advisor.

    You create a spot instance request, specifying what instance types you are interested in, and the availability zones (AZ) in which they should run. If capacity is available, and their current price is less than your maximum bid, instances are launched.

    Spot instances continue running until:

    • Capacity is no longer available (because the instances were requested by on-demand customers).
    • The price has risen over your maximum bid.
    • You request to terminate the instance, or it is automatically terminated by auto-scaling.

    You can also order spot instances with a predefined duration—you then pay a static hourly rate for that entire duration (even if market price changes in the interim).

    Automation Options

    You can automatically scale instances on Amazon EC2, including both on-demand instances and spot instances in a single auto-scaling group. If spot instances are not available when scaling up, the group can use regular on-demand instances.

    Amazon also supports mixing in reserved instances and savings plans (additional ways to save on on-demand instances by committing to a certain period of time or total capacity). So you can combine multiple saving methods in the same auto-scaling group.

    You can improve availability by deploying applications across multiple instance types running in multiple AZs. By allowing multiple instance types, you tap into multiple pools and increase the chances of obtaining a spot instance when you need it.

    Azure Spot Instances (Spot VMs)

    Azure offers Spot VMs that give you access to unused compute capacity. You can request a single spot VM, or launch multiple spot VMs using an Azure VM Scale Set (VMSS). Spot VMs replaced the previous Low Priority VMs feature, which let you purchase VMs that were in low demand on Azure for a reduced price.

    The spot price of VMs on Azure depends on the total capacity available for that specific instance size and SKU (instance type) in the Azure region. Azure commits to changing pricing slowly—avoiding sudden spikes—to maintain pricing stability and make it easier to manage budgets.

    Like on Amazon, discounts fluctuate significantly, and spot VMs can be up to 90% cheaper than the base price of the same VM.

    How Spot VMs Work on Azure

    The Azure Portal provides access to Azure spot VMs. When you create a spot VM, you can see the current price for the selected region, image and VM size. For consistency, prices are always in U.S. dollars, even if you use a different base currency for billing.

    There are two options for eviction of spot VMs—you can choose the condition on which spot VMs will be evicted:

    • Maximum price eviction—You set a maximum bidding price, and when the spot VM rises over that price, it is evicted.
    • Capacity eviction—You always pay the current price of the VM (without setting a maximum price), and when Azure does not have sufficient capacity of the requested VM type, your VM is evicted.

    When a VM is evicted, Azure applies an eviction policy called Stop / Deallocate. This means the instance is paused, but attached disks remain, and you are still charged for them. When the price goes down or capacity becomes available, the instance is restarted and continues working on the same disk data.

    Automation Options

    Azure provides virtual machine scale sets (VMSS), which can automatically increase or decrease the number of VMs running your application. You can create a scale set that includes spot VMs, and as your application scales, more spot VMs will be added as they become available. Spot scale sets operate in a single fault domain and do not guarantee high availability. Unlike AWS, Azure currently does not allow you to mix on-demand VMs and spot VMs.

    Both Amazon and Azure provide robust capabilities for cost savings using spot instances. Azure’s offering is newer and provides less-advanced bidding and auto-scaling capabilities, but these are expected to be added as the service matures.

    Whether DevOps teams choose to run in AWS, Azure, or both, they literally cannot afford to ignore spot instances, especially for low-criticality workloads like dev/test and CI/CD job execution.

  • Engineering Practices Can Overcome DevOps Challenges

    Engineering Practices Can Overcome DevOps Challenges

    When I think about the challenges of DevOps transformations, I love the viewpoint expressed by Ryan Harmon, strategic accounts manager at Trace3: “DevOps is a complex maze that has many leaders frustrated. In many large enterprises, ‘DevOps’ is simply a buzzword they are striving to achieve but struggle with what the end state looks like or even where to start. There are many layers across the organization that are instrumental to truly executing on DevOps. DevOps is not something you go get a quote for and simply buy. It’s an evolving journey.”

    In my previous article, The Time Has Come for an Engineering Approach to DevOps, I agree with Ryan.

    Too many organizations launch into a DevOps transformation initiative without considering the full scope of what it takes to engineer a successful DevOps solution. DevOps is more than “just” a CI/CD pipeline or toolchain, as complex and challenging as this alone may be to accomplish, for even one application. DevOps is more than simply a culture change, as difficult as that always is.

    There are 24 DevOps practice categories that need to be mastered to realize and evolve well-engineered, high-performance enterprise DevOps solutions. Not all 24 need to be mastered to begin the DevOps transformation, but, ultimately, all of them do need to be mastered before the solution will achieve the high performance levels to which many enterprises aspire. The good news is, successful enterprises have demonstrated that the investment in people and technologies required for the journey is worthwhile, and a strategic approach will get you there systematically if you follow a disciplined engineering implementation roadmap.

    As explained in my book, Engineering DevOps, the core of every successful DevOps practice includes nine core pillars. These are the pillars to focus on as you start on your DevOps journey. The following paragraphs illustrate requirements for each of the nine pillars.

    Leadership – Leaders set an inspiring DevOps directional vision for the organization, and proactively stimulate and sponsor team activities that help progress toward goals.
    Collaborative culture – The organization encourages cross-functional collaboration, shared responsibilities and works to reduce restrictive boundaries between organizational functions.
    Design for DevOps – Products are architected to support service-oriented, modular, independent packaging, testing and releases, in accordance with the principles of twelve-factor apps.
    Continuous integration – Changes to software code and artifacts needed for releases are frequently merged, built, tested and packaged into executable artifacts for application releases.
    Continuous testing – Software tests that are most important to verify the results of pipeline stages including pre-flight, integration, regression, performance and release acceptance tests are automated as much as possible.
    Elastic infrastructure – Infrastructure-as-a-service (IaaS) resources (i.e., software, computing machines, storage and networks) are integrated into a toolchain for automating builds and testing environments to efficiently support variable workload demands.
    Continuous monitoring – Applications, pipeline tools and infrastructure components are instrumented with metrics, and the collected data is analyzed to manage health and performance.
    Continuous security (DevSecOps) – Security practices, processes and technologies for applications, databases, pipelines and infrastructures are integrated into the culture and pipelines to assure security.
    Continuous delivery – Release artifacts are validated, prepared for deployment to production and may be automatically deployed to production once acceptable deployment measures are achieved.

    A Comprehensive Engineering Blueprint

    While the above nine practice categories are the core of every successful DevOps solution, 14 additional practice categories are essential parts of a comprehensive engineering blueprint for accomplishing high-performance DevOps.

    DevOps infrastructure practices are used continuously by the nine core practice categories as follows:

    Version management systems – Every stage in the end-to-end DevOps value stream depends on keeping track of versions of software source code, configuration data and executable images using source code management systems, artifact repositories and configuration management systems.

    Infrastructure-as-a-Service – Virtualized cloud and dedicated resources must be served up ephemerally to meet variable workload and storage demands, which requires mastering engineering practices for orchestrating dedicated and cloud, hybrid cloud and multicloud services, and deployment of applications to the infrastructure.

    DevOps pipeline practices, which constitute a layer just above the nine core practices, include two additional categories that make use of the nine core DevOps practices.

    Application release automation – Directing and monitoring software changes through the CI/CD pipeline requires process orchestration and data analysis working at a level above the CI/CD pipeline itself.

    Value stream management – A deployed service, composed of multiple application microservices and data services, requires orchestration and analysis of a federated set of services, end-to-end, across the value stream from planning through to operations.

    Management Practices

    DevOps management practices include four categories that make use of the DevOps Pipeline practices, as follows:

    Service catalogs – Organized collections of software stacks, tools and pre-configured pipelines, made available through portals, enable DevOps-as-a-Service operations and simplify access, configuration, control and evolution of DevOps resources needed by enterprise application teams.

    Disaster prevention and recovery – Proactive disaster monitoring coupled with high-availability configurations and disaster mitigation practices, such as automated processes, work together to minimize the blast radius of disastrous events, and orchestrate rapid restoration of essential DevOps services, when needed.

    Site reliability engineering (SRE) – A combination of practices which include service level objectives (SLOs), error budgets, monitoring, work-sharing policies, toil reduction, proactive anti-fragility practices and blameless post-mortems help to “shift left” production wisdom from operations into development, thus improving the efficiency, readiness, security and resiliency of applications and infrastructures in production.

    Governance – The scale and scope of enterprise DevOps solutions are controlled using governance policies-as-code implemented through service catalogs, management dashboards and error budget policies.

    Project Practices

    DevOps project practices include four categories that guide enterprise adoption and evolution of DevOps, as follows:

    Adoption – Enterprise-wide DevOps adoption strategies that engineer and build DevOps capabilities in logical phases. This begins with a small set of representative applications that serve as a model for other applications; this method has been shown to be more successful than “boil the ocean” strategies that attempt to transform all applications at the same time.

    Model applications – Applications – that can serve as a model for greater enterprise adoption – are selected and monitored according to criterion that spotlight the value of DevOps for other applications.

    Return on investment (ROI) – DevOps investments must compete with other IT projects for resources. An ROI model that demonstrates measurable cost savings is an approach that has proved to achieve greater numbers of DevOps project approvals compared to other methods.
    Continuous improvement – Competitive forces and industry trends require that DevOps solutions continue to evolve to keep up with enterprise requirements.

    The Human Element

    Human development practices include two categories, as follows:

    DevOps engineering – The development of DevOps engineer roles, skills and the structure of DevOps engineering within an organization is critical to the success of designing, building, maintaining and evolving DevOps solutions.

    Continuous learning – A DevOps continuous learning strategy is critical for the development and transfer of DevOps knowledge needed to develop and continuously improve DevOps for the enterprise.

    In summary, the importance of practice categories – and specific practices within each of the categories – will evolve as each application does, alongside the changing needs of the enterprise. DevOps is complex, but a comprehensive engineering strategy that considers the 24 DevOps practice categories described in this article can help enterprises achieve the high-performance DevOps solutions they need to justify the investment and to meet business goals.

  • 3 Common App Modernization Pitfalls to Avoid

    3 Common App Modernization Pitfalls to Avoid

    Application modernization has been a consistent theme for organizations year after year. Businesses today still rely on their legacy applications for the majority of their revenue generation, from mainframes to Java EE. New digital transformation initiatives and mandates have pushed application modernization back to the forefront, as a new generation of automation platforms and best practices are helping these organizations avoid the pitfalls of the past and accelerate their journey to cloud-native architectures.

    Here are the three most common application modernization pitfalls and how to avoid them.

    Lack of a Clear Organizational Destination

    Application modernization journeys need a north star – a destination point that development, infrastructure and DevOps teams can rally around. In order to establish a clear, unified direction, everyone must share a precise understanding of that destination. That journey starts by defining a set of business goals you are trying to accomplish (such as cost savings, business agility, resiliency and/or competitive advantage) that can guide and prioritize the technical decisions.

    Next, you need to decide which application modernization technologies, platforms, and infrastructure will help you deliver on those business goals (e.g., Red Hat OpenShift, Kubernetes, serverless, hybrid, private or public cloud, etc.). Setting common architectural standards will help avoid disparate future stacks and ensure a common shared platform for microservices architectures. For example, if an application starts with Java WebSphere, should the goal be to move to a modern container framework like Spring Boot, or a specific microservices Java framework like Quarkus? To avoid analysis paralysis and internal drag, set tangible objectives and targets that prescribe how teams are expected to prioritize, select and modernize these applications and how their progress should be measured, communicated and shared.

    Taking Shortcuts – Don’t Fall For the Old ‘Lift and Shift’

    The more one modernizes, the more value one gets from the cloud – but not all modernization efforts are created equal. For example, ‘lift and shift,’ putting a monolith into a container and application refactoring into microservices are all called “modernization.” But only refactoring allows enterprises to realize the true benefits of the cloud, including elasticity, regained engineering velocity and cost savings. Don’t fall into the quick-and-easy modernization temptation with lift and shift or brute force containerization. You will still have a monolith, in the end; you’ll simply end up paying more and trading one set of monolithic problems for another.

    Not Bringing Everyone Along – No Developer Left Behind

    Modernization is never just about technology. Modernization requires that your team’s processes (for example, agile practices), structure (vertical domain teams as opposed to horizontal teams – UI team, for instance) and skills must all adapt. The organizational culture needs to grow to adopt DevOps and cloud-native best practices to take advantage of the deep development, build, deployment and operational benefits that are available. Take the time to train and educate so everyone is brought along. Modernizing legacy applications can introduce a whole new set of technologists, responsible for those applications and the supporting infrastructure, into the cloud-native world. This is a win-win for everyone.

    In other words, the organization needs to have crystal clear answers to the where (destination), why (path) and how (team) to ensure success and avoid the three most common pitfalls of application modernization.

  • Security Policy Management and Hybrid Cloud with Tufin

    Security Policy Management and Hybrid Cloud with Tufin

    In this TechStrong TV / DevOps Chats interview, Colby Dyess, director of cloud product management at Tufin, joins Mitch Ashley to discuss security policy management across hybrid cloud, multi-cloud and cloud-native security. Recorded in late 2020, Colby shares some experience from working with customers during the rapid migration to the cloud in 2020.

    The video and podcast audio of the conversation is below, followed by the transcript. Enjoy!

    TechStrong TV Video

    DevOps Chats Podcast Audio

    Transcript

    Mitch Ashley: I’m joined today by Colby Dyess who is director of cloud product management at Tufin. Welcome, Colby. Welcome, it’s good to be talking with you. Would you introduce yourself, tell us a little bit about you and of course, tell us about Tufin.

    Dyess: Yeah, well, you’ve got my name and my title, so I’m dialing in here from beautiful Southern New Hampshire where it’s fantastic weather. So, I’ve worked for Tufin Technologies, Tufin is really well known in the industry for providing security policy management, mostly in the organizations like the Global 2000 complex environments where you need to make changes to the network environment and you need to be able to do it quickly, safely, ideally you do it automatically and it’s secure. The Tufin product suite helps organizations do that and we’ve done that in the traditional space. Of course, our customers have adopted cloud, cloud native platforms, Kubernetes, and we’ve been taking these same capabilities for securing the environments and bringing them into the cloud, so no matter where you put your workloads, we can help you, and that’s mostly what we’ve been focused on.

    Ashley: Good stuff. Boy, the center for the universe for a lot of folks. Kinda speaking of that and just thinking about the timing of where we all are, you know, there’s a lot of statistics. I run an analyst firm as part of MediaOps organization called Accelerated Strategies Group. Anyway, we have a lot of data showing some of the acceleration that’s happened, both in digital transformation and also move to the cloud, just in the last six to nine months. I don’t think that’s news to anybody, right? I don’t know if you need data to say that, because it’s happening probably in most people’s organizations.

    Dyess: Right.

    Ashley: If I put on my security hat for a moment, we always get plenty of notice when we’re making changes and doing new projects and accelerating stuff—sarcasm—so, I’m sure there’s a lot of security folks who are scrambling still just to not only keep pace, but make sure we’re all doing a good job of securing environments that our developers are taking this into, making sure we’re understanding the services we might be using now that we weren’t before, cloud services that we’ve been in. I’m sure you’ve seen a ton of that happening with your customers.

    Dyess: Yeah. It’s no surprise. Organizations have been moving to public cloud environments for a long, long time, at least, technically, for the past 15 years, they’ve been doing it, but it hasn’t always been sanctioned, right? Let’s say over the past five or eight years, organizations have really begun to intentionally adopt the cloud. And now this year, of course, everybody’s moved to the cloud about as quickly as they could.

    What we’re seeing, at least what I’ve seen a lot is, security is now scrambling to try and get ahold of some visibility—what’s really going on inside these cloud environments—and trying to put some sort of guardrails around the services that the Dev teams are using. That’s a real challenge because they’ve been focused so much on how to secure most of the on prem environment or use firewalls to protect the cloud space, but now they realize they’ve got to go a bit deeper, because the threats are pretty real out there and they’re really dynamic. And if you’re not sure how to secure the cloud environment, you’re going to leave yourself open to threats.

    So, security has really been spending a lot of time trying to learn more about cloud and cloud native security controls. We spend a lot of time helping customers figure out the difference between cloud native security controls and on prem stuff and then how to combine them so they can be assured it’s all secured. We see that a lot, yeah.

    Ashley: Well, you know, I mean, we’ve all been doing firewalls and intrusion prevention and access controls and the list goes on.

    Dyess: Yeah.

    Ashley: So, the topics aren’t new, but the environments are different and the services. It’s not like you can go into the cloud—no, let’s use all the things I’m normally using in my on prem environment. Maybe you can get some software versions of that. Sometimes it’s just better to use because it’s better integrated, you spend less time making the environment work by using what’s provided by the cloud provider.

    Dyess: Yeah. That pattern is the same everywhere. I’ve seen that you’re gonna go to a new environment, you wanna take some of the things that have worked for you in the past and carry them forward. And a lot of times, that makes sense as you try to move quickly into the cloud. If you’re not familiar with cloud native security controls, you probably think, “I’ll just use firewalls and the process for managing firewalls in the cloud.” And there are a lot of options to do that, a lot of the traditional vendors in that space have tried to make a pivot to offer their same services, same capabilities in the cloud environment. So, it’s no surprise that we see a ton of that.

    At the same time, those processes around managing all that stuff still takes too long. If you moved to the cloud, you went there for agility, right? You want to rapidly build applications and get them out into the market as quickly as possible. But if you’ve got to fill out a ticket to get a firewall changed and it takes three days for that stuff to be processed, you’re no longer as agile as you used to be when you could twiddle a little configuration in a Terraform template, redeploy, and—poof, you’re up and running all over again. That’s something that security teams—

    Ashley: If you take another two days to answer that ticket, you’re probably gonna have a fourth after your third _____ while you’re add it to the mix, right? Some developers are gonna be like, “I’m outta here, let me go set up the next environment.”

    Dyess: Yeah, it’s too difficult. You can’t wait. You don’t want to see shadow IT any longer, because IT was taking too long to respond, but you’ve got to respect that IT’s there, the security site of IT is just really trying to help protect the business.

    But you know, you also saw Amazon and AWS, they both announced their own firewalls, right? Azure did a little bit back and AWS just did within the past month. And I think that’s really just paying homage to the fact that those are really good technologies. Firewall technologies really are good stuff and it’s more than security groups or NSGs, it goes beyond that, and the businesses really need these. These do address customer problems. So, we’re seeing a lot of folks ask about how do they adopt those new to the cloud provider, new technology, and how do they integrate them into the very agile processes? And I expect we’ll see a lot more of that.

    In fact, what we are finding that we do often is try to help organizations see how they can get the most value out of the cloud native security controls, because these cloud environments are really, they’re pretty solid. And like you pointed out, the security controls are integrated into the services. So, it makes a lot of sense to use the services, they’re there to protect your assets and they’re so integrated in the services, you really should be trying to use them, but it’s a learning curve for security, so it means a lot to try and help them get the visibility and then start to affect control over that, so that they don’t have to slow down the rest of the business, swapping out, let’s say, a cloud native control with a legacy control [Cross talk].

    Ashley: Mm-hmm. It is that tradeoff, and I think that’s one of those where it’s sort of, I think it puts security teams in a position where, are we gonna stick to our guns of this is the environment we’re gonna have consistent across all of the environments we operate in, or for sake of speed but not sacrificing security, maybe we do look at some other options and can those integrate into the monitoring and management systems as opposed to using the same firewall in every environment that we take with us there, right?

    Dyess: Yeah, yeah.

    Ashley: Easier said than done. I appreciate it’s not an easy job.

    Dyess: Right. You know, whether you decide to use the cloud native security controls or use the traditional kinda controls, the virtualized versions of those or a combination of them, one thing I think we’re sharing is, we see a lot of conversations around how—which technologies to adopt and then how to actually manage those technologies.

    And from my point of view, it seems like we’re almost too focused on getting down into the weeds about the specific security control and, as a security person, I think it makes a lot of sense to step back a little bit and try and focus on what’s the overall security policy. Instead of thinking what firewall should I use, should I use one from the cloud or should I use from my traditional vendor, maybe a better question to help organizations remain agile is for security to be able to focus on what the security policies are, what the guardrails should be, and then begin to get the practice of checking cloud environments against those guardrails, against these policies as part of your CI/CD pipeline, the whole buzzword of shift left.

    That’s a real thing, right? That’s really important. That security can focus on defining security policies where they have the strength, but the actual implementation of enforcing the security policy is probably handled by the cloud operations or whoever actually twiddles the bits on the security controls. Like, if you can marry those two things together and you can do it in the CI/CD pipeline, then that’s a huge win.

    We’ve seen organizations move a lot faster because security focuses on what they do and they get it codified in a way that can play in a pipeline, and now the app teams, they can make changes to an application or a cloud Ops can change the configuration of an environment and get an alert saying it doesn’t comply with the security policy. That’s great. To step away from choosing which sort of security device you want to use and focus on do things comply with the security policy? That’s a major shift, and one that seems to be really hard for organizations to do. I think to kinda get away from security being hands on with the control and now focused on the policy—it’s trust but verify, I think.

    Ashley: [Laughter] Back to that old saying, right?

    Dyess: Yeah.

    Ashley: You know, in a way, it also comes down—you know, we’re used to, let’s first take an inventory and make sure we know what’s all out there that we’re trying to secure.

    Dyess: Sure, of course.

    Ashley: But it’s equally just as important to have, whether it’s change management practices or software that can help you with notifying when changes occur or understanding, so when a change occurs, how does that map back to policies that we have in place and are we still in alignment, are there gaps that we have to take care of? Because the environment isn’t—it isn’t an environment we control everything in any more. It changes by itself, because other people are changing it along with us. So, it seems like it’s a more dynamic environment than maybe what we had 10 years ago, if you reflect back a bit.

    Dyess: Intentionally more dynamic, right? Businesses have to move faster, we also have to react more quickly to customer demand or competitive threats or whatever. So, the fact that we’ve got these dynamic environments is just a reflection of what we actually needed to have built, anyway. And now, we need to find a way to incorporate that back into our regular practices.

    So, you made the comment about being able to track what’s changed and when and all that, but if you adopt more of an infrastructure as code model, you already have that. You don’t need a CMDB—maybe you do, but you don’t have to have a CMDB in place when you could go back to your Git logs and see, oh, here’s the pull request for the change that we made that allowed for this security group to be open or for this Azure firewall to have this particular rule. You can actually see that information, but these are, on the security side, these are new tools. As a security person, when was the last time you looked into a Git repo to see what the configurations were? [Laughter]

    Ashley: The repo—right, GitOps and security, right?

    Dyess: Right. Right, but we’ve got to get somewhere around there, as a group collectively, whether we’re writing the code or we’re managing the operations or we’re managing the security collectively, we’ve got to be able to collaborate and use these tools that I can’t say are merging, they’ve been around a long time. Terraform and Helm, Ansible, Chef, Puppet, the whole bit—these things have been around for a while, but to incorporate them now into our security practices is a pretty big leap for folks.

    Ashley: Mm-hmm.

    Dyess: So, we really try to encourage that the security team gets the visibility that they need, that’s the first thing, as you pointed out, right? You absolutely need to understand what’s deployed, what’s talking to what and whether or not things are properly segmented.

    You’ve got to first get that visibility, but I think before you start jumping into let’s throw a bunch of other control points in there, consider what resources are there and how can you, on the security side, participate in automation, participate in these flows that your CloudOps, DevOps, SRE, whomever had set up. Because there’s a really good opportunity now on security to be part of this agile process and to allow the business to be secure, but to do so in a way that won’t break the developer productivity.

    Ashley: Mm-hmm.

    Dyess: And that’s, it’s uncomfortable, because it’s a big change, but it’s a change that I know it can be done successfully. We’ve worked with companies that have done that, you know, make the conscious decision.

    Ashley: I almost wish we had stepped back and redefined shift left, because if, you know, I were king of the world for a nanosecond, I would think about it as, it’s not just about getting security people to work with the software teams earlier in the life cycle of software—yes, it’s that. But think about it this way of, the old world was, set the policies and then how do we implement the enforcement mechanisms, right, to make sure that those are being followed.

    Dyess: Mm-hmm.

    Ashley: I think in the world we’re in now, it’s, security people are, if you were gonna set up a security practice in an organization today, what I would do is, let’s start with how software is created, let’s work with the team about how the entire tool chain is set up, how that flow happens, both for application software, but also for the infrastructure and how that gets created where we’re talking about test environments and production environments and maybe you’re stepping into infrastructure as code if it really, truly is kinda dynamic Terraform or whatever kind of environment.

    But start there, because if you understand how that’s done, now you know how to secure it. You know how to work with teams to help them secure it, then the next layer would be getting into the software architecture itself around microservices and things that are more porous than we’re used to, kinda the hard outer shell of monolith applications of the past.

    That, to me, is the mental shift, the big mental shift for a security team to not necessarily be an expert at all that stuff, but understanding that’s what you’re really securing now.

    Dyess: So, two things that stand out, like responses I can imagine people having right now and that’s, I don’t have time to go and learn all that stuff. [Laughter] I’ve already got 20 other tickets for all high priority things. I’ve gotta get on that right now, so I can’t go and learn all that stuff. Valid, that’s definitely the case in a lot of organizations, yeah.

    And another problem related to this is security teams who maybe do want to know, maybe have some time to do that, but struggle that the application teams themselves don’t even know what makes up the application and what connections do they need. So, even the folks who you think would know because they built it, you’d think they would know the answers to some of these questions, but it turns out that they don’t—in a shockingly large number of scenarios, that’s the case.

    Ashley: Mm-hmm.

    Dyess: And so, if you’re gonna shift left, you’ve got to address not just the problem of the security team getting visibility in this—or maybe, actually, it’s the same thing. The application teams have to be able to come to the table to say, “This is what the application looks like. Here’s how it’s put together. Here are the resources it needs and here are the services that will rely upon it.”

    Ashley: Mm-hmm.

    Dyess: And so, if application groups can come to the table with that, then a security team can look at it usually very quickly, look at a map and say, oh, I can see where it needs to be segmented this way. I can see where he’s a resource that has too close of an access to PII, PCI, right? And so, that conversation could start a lot sooner if it’s part of your normal bill practices, you were generating these views, you had some way to produce an item that a network team or security team could look at, right? They don’t need to look at the source code, they need to look at your network topology, they need to understand what you’re connecting to.

    So, if you can bridge that gap, that goes a long way. So, to your point, it’s not just like, “I just need to check if it doesn’t comply with the policy.” Sometimes, it’s even just trying to define the policy at the outset.

    Ashley: Yeah, on what you’re trying to secure.

    Dyess: Yeah, but also, while security never seems to get the heads up about the things that are coming down, the joke that you made at the beginning, it’s also the case that when the application team is building something that they don’t really have all the rules, they’re not sure what all the restrictions are, and there are certainly plenty of guardrails, guidelines that they could have if you give them access that they could pull from an API to do their own checks.

    So, you know, there’s a model. We’ve done this before in the industry, back when I wrote code, we had a separate QA team, so we’d write code, we’d do a little bit of testing, but just enough, frankly, to get us through the build, right? And then we throw it over the wall and it went over to QA. And then thankfully, our development practices matured to a point where we realize we actually need to integrate testing as part of what we do. So, as developers, we had to own some of that testing, some more of the testing, but also importantly, the test team got involved earlier in the application life cycle, the model you were talking about. So, the QA teams could come in and learn a bit more and begin to set up some of the additional tests and participate in the build process, so what they did showed up in NMake.

    So, I think security is on a path like that as well. I think security, instead of app teams just throwing it over the wall and asking security to figure out how to lock it all down, I think we’re going to, and in some places, some customers for sure are moving to a place where security is part of the process earlier, and if they can say, “Here’s some of the tests,” so now the guardrails, they can contribute some of that and they can go into this build process, then now you’ve really begun to get that shift left.

    But I know this can happen. We’ve seen it happen before. Now QA is just integral to what we do. You can’t even imagine putting a product out without some part of QA being part of the build process and tests happening really early on, test planning happening early on. I imagine security just going in that same sort of loop.

    Ashley: Yeah, of course Tufin’s been in this space for a while, has built up, I’m assuming, a body of knowledge about this. If you were gonna advise maybe a new customer or an existing customer on—okay, we’re always playing catch up, we’re really playing catch up now with how much more cloud services that organizations are using in such a short period of time, what are some of the sort of best practices or recommendations of, “Here’s to sort of get the first thing established, here’s the next thing to work on, here’s the next thing to work on.”

    Dyess: Yeah.

    Ashley: Any suggestions about that?

    Dyess: Sure. We have this whole crawl, walk, run methodology, and it’s designed to help organizations take these small steps towards getting to really secured environments. And the first step right away is just to get visibility, which is kind of a “no, duh” statement, but it’s actually one of the big problems that orgs have. You’ve got to be able to see what’s deployed inside your cloud, what’s deployed inside your cluster, and which resources are communicating with each other, where are the dependencies.

    So, understanding that is a really major first step. It’s true for the application teams, but for the security, it’s to know are things properly segmented, where are the biggest risks? So, getting that awareness, that’s really the crawl phase. And in the walk phase, that’s where you take what you’ve learned about what’s deployed, what talks to what, and whether or not things are segmented properly and then you begin to isolate. Here are specific accounts that we need to tackle or here are clusters that we need to tackle—like, hone in on the higher priority items and begin to define a security policy for those cloud native environments.

    And as you develop the cloud native policies, which you then share with the application teams or the cloud operators, whoever’s responsible for configuring it. As you develop that sort of, that skill, you’ve gotta get to the run phase, and the run phase is when you begin to automate this. You automate generating security policies, you automate validating security policies, but at the very beginning, where most organizations are at right now is, first get the visibility and then the second step of defining a security policy. Not defining what is my security group definition, not defining what is my AWS firewall definition, but what is my security policy? What are the rules that should govern access to applications, assets, resources of certain categories? Those are two big steps to take right away.

    Ashley: I think that’s some great advice. I’m curious, any thoughts as we—I think we all want to turn the page on 2020 and kinda get to 2021, and hopefully, we’re looking at a different kind of a year. But one of the things about 2020 is, we’re not gonna go back, I don’t think. We’re not gonna go back to, let’s rewind to January and start working in that environment again, right? Everything is changed with remote work or how much we’re using the cloud.

    What are some things that you’re anticipating for 2021 as you look forward in the next 6 to 12 months. Do you kinda think it will be topics maybe not as talked about yet but will be more topical in 2021? I’m just kinda asking you—

    Dyess: Are you asking for predictions of 2021?

    Ashley: You know, I don’t see it, but I’m pretty sure, there might be a crystal ball right behind your chair, just look back a little bit there and kinda check back. [Laughter]

    Dyess: [Laughter] Well, I think, given the maturity of the market adoption of public cloud environments, I think we’re going to see more security teams getting their arms around how to help in securing these cloud environments without necessarily always relying on the traditional devices. We’re already starting to see that even now. So, I think we’ll see more of the cloud native security control adoption. I know organizations have seen the benefits of doing that to lower cost of operations, just total lower cost of ownership, plus the benefit of being able to say when there’s a problem, I can call my cloud vendor, I don’t have to worry about vendors poking at each other. So, I think we’ll see more of that.

    I think it’s the Kubernetes space that’s probably the most interesting, because it’s the least well known for most folks on the security side.

    Ashley: Mm-hmm.

    Dyess: So, I’d say maybe over the past year, there’s been a lot of emphasis, just like with cloud, slap a firewall on it and we’ll solve the problems of micro-segmentation later, but later is gonna happen real quick, and I’d expect in 2021, you’ll see securities start to get more involved in getting visibility into the clusters and then looking to define the security policies.

    And not—again, not to try and define the policy of an individual cluster, but just the general rules, because there’s no way we can keep up with what will really be thousands of services running inside of a cluster.

    Ashley: Yeah.

    Dyess: So, we’ll see that. I think we’ll see more organizations at the security side have to deal with the scale and the rate of change and begin to participate in the automation processes that the Dev teams are using to manage it.

    Ashley: Maybe that’s something we’ll hear more frequently. I wouldn’t be surprised if the—let’s think of it as cloud native security, not just as security, sort of think about it as one of you architectural approaches to how you do security.

    Dyess: Cloud native. The native controls of the cloud provider—yeah, that’s a big shift for folks, I think.

    Ashley: It is. We’ll see if something like that happens. And I think the things you said made a lot of sense. Well, it’s been great talking with you, it’s the first chance you and I have had to talk before, and—

    Dyess: Yeah, it was good to talk to you, Mitch.

    Ashley: – a lot of fun. Tell us a little bit about where folks can learn more about Tufin.

    Dyess: Yeah, of course, you can go to Tufin.com, that’s T-U-F-I-N dot com. On the cloud side, like what’s really relevant to what we’re doing here, we’ve got a lot of resources at Tufin.com/try-securecloud. It’s cloud native security controls. It’s really designed to help you get visibility and control the cloud native security components. Great stuff there, free to use product. There’s a great trial for people to give a go. Tufin.com would be the place.

    Ashley: Great. Perfect. Thanks. Well, we wish you the best and a even better 2021, we’ll look forward to that.

    Dyess: Yeah. From your lips, man.

    Ashley: [Laughter]

    Dyess: Good luck in 2021, everybody.

    Ashley: There you go. Hang onto that, hang onto that. Alright, well, take care. It was very nice talking with you, Colby—Colby Dyess from Tufin.

  • Survey: The State of Database DevOps

    Survey: The State of Database DevOps

    Enterprises are undergoing rapid changes. Throughout the past year, companies have been pressed to learn a diversified set of technologies and methods to remain agile in a changing economy. In this digital-first world, data plays a heightened role; in tandem, so does the delivery of data and the persistence of the database layer itself.

    Applying DevOps principles and automation throughout the software life cycle is critical to maintaining a high level of efficiency. As with application development, databases are not immune to the DevOps takeover. Database DevOps is a discipline that considers the processes used to automate database changes and deployments — a field growing in mindshare as databases diversify.

    Seventy percent of companies now use more than one database in their stack. Many organizations now use a blend of on-premises and cloud databases. Styles are diverging, too, as non-relational databases like NoSQL gain favor, and new open-source databases emerge to fit niche circumstances. All of this means database operators may require an increased emphasis on database DevOps to handle additional complexity.

    The 2021 State of Database DevOps report, conducted by Redgate, surveyed 3,200 database professionals on the general state of their DevOps adoption. The survey also inquired about database change automation and deployment and underlying database technologies. Now, five years in, the study continues to reveal shifts in practices across industries.

    Below, we’ll review the report’s main takeaways to see how companies implement database DevOps in practice.

    DevOps Adoption Increases Across Projects

    To meet pressing digital innovation needs, DevOps has surged in importance across all industries — 74% of organizations have adopted some form of DevOps, and 26% of respondents reported applying DevOps practices across all projects — a significant uptick from previous years. The report acknowledges a slight increase in DevOps adoption among larger company sizes.

    So, which DevOps practices are companies employing the most? Well, in terms of DevOps for application development, the majority of organizations (50% or above) have adopted version control, automated builds, monitoring, automated unit tests, static code analysis and semi-automated deployments in production. Only 41% of organizations surveyed have fully automated deployments to production. These methods can be viewed as a basic benchmark for modern DevOps.

    The study demonstrated that in most organizations, DevOps is a communal undertaking. More respondents reported adopting a community of practice, rather than relying on individual decision makers, to disseminate DevOps strategies and direct planning. According to the report, a community of practice is a group of people that “makes decisions, provides recommendations, or is consulted.”

    Lack of Skills Tops DevOps Obstacles

    The pressure is on engineers to produce to meet digital transformation demands. However, organizations experiencing a developer drought may not possess the talent to introduce DevOps practices. It’s no surprise that the common obstacle hindering DevOps adoption was a “lack of appropriate skills in the team.”

    Other cultural barriers to DevOps adoption include disruption of existing workflows and a lack of alignment between teams. Interestingly, a lack of executive support was cited as the lowest hurdle to DevOps adoption, indicating universal consensus on DevOps goals. If we consider what technical barriers prevent improved software delivery and operations, 41% noted insufficient automation in release processes and 39% expressed hindrances due to legacy code.

    Continuous Delivery For Databases Increases

    Database DevOps appears to be growing in production use too. For the purpose of this study, Database DevOps is defined as “automating the delivery of database changes.” According to the report, 51% of respondents are already automating some parts of their database deployment process.

    Though growing year after year, database DevOps is nowhere near ubiquitous. Compared to DevOps in application development, database DevOps has lower overall use. For example, only 36% are implementing automated builds. And, only 22% adopt continuous delivery (CD) for all databases. Nevertheless, this could quickly change, though, as 80% of respondents are working to implement CD for database changes or have already implemented it.

    Setting up a CD process for database changes could increase overall software delivery speed. The top two drivers for adopting database automation are increasing the speed of database changes and freeing up time for other work. The third top reason, minimizing application downtime, rose in rank this year — which makes sense, as the desire to create more reliable systems is an ever-present concern.

    However, to reap these rewards, organizations must overcome some significant challenges, like synchronizing application and database changes and maintaining high speeds for database automation, among other concerns.

    Organizations Adopt Hybridized Database Approaches

    Companies increasingly use a mix of different database technologies and styles. We also see an increase in cloud and hybrid database estates.

    In terms of database styles, relational databases still make up the majority of market share — 98% of respondents have at least one relational database. However, non-relational databases, like NoSQL, are gaining popularity. Forty-seven percent of database professionals have a NoSQL database in their enterprise. Interestingly, only 2% use non-relational databases exclusively, indicating it’s a niche use case.

    Enterprises are now using more databases than ever before. Seventy percent of database professionals reported having more than one database in their stack. This figure rose nearly 10% since last year. Database types are multiplying too — 45% have more than one type of database. In terms of database management systems, Microsoft SQL Server holds the most use, at an 82% usage rate. Others like MySQL, Oracle, PostgreSQL, and MongoDB trail behind.

    Enterprises are also increasing their use of cloud and hybrid database deployment models. Eighty percent of respondents host a cloud database, and 65% report a mix of cloud and on-premises estate.

    One possible explanation for the diversified database effect is an increase in public cloud provider usage, combined with the rise of multicloud strategies. Database options are increasing as well — many free, open-source databases can cater to unique project needs.

    This “polygot persistence” could also simply be a natural occurrence within large enterprises supporting many application nuances. As Martin Fowler describes, “… any decent sized enterprise will have a variety of different data storage technologies for different kinds of data.”

    Quality Database Ops = Quality Software Performance

    In summary, non-relational databases are gaining steam, cloud and hybrid database models are becoming more common, and large organizations are increasingly using multiple database types. The landscape is forecasted to diversify further, meaning database DevOps is stepping up to manage this complexity.

    The report indicates a correlation between quality database Ops and overall software quality, noting that organizations that release more frequently typically deploy database changes in much less time. To achieve these goals, “… the next area for database DevOps is in the data engineering and analytics space,” writes Pramod Sadalage, director, data & DevOps, ThoughtWorks.

    While it’s important to take a vendor-endorsed study with a grain of salt, it’s safe to say we will likely see more interest in database DevOps, especially as DevOps themes continue to permeate through the entire spectrum of software development and delivery.

    For more information, you can download The State of Database DevOps report behind an email gate here.

  • Cloud Misconfigurations Threaten Cloud Migration

    Cloud Misconfigurations Threaten Cloud Migration

    If we’ve learned one thing during the COVID-19 pandemic, it’s that digital-native business models are essential to survival. That means cloud adoption is more important than ever before. Research shows that the public cloud market is expected to grow to $308.5 billion in 2021, an increase of 16% from 2020.

    But while the cloud holds incredible potential, we have an outstanding security issue to address. That issue is cloud misconfigurations. Through 2025, 99% of cloud security failures will be the responsibility of the customer, quantifying the need for security professionals to turn their attention to security hygiene issues like cloud misconfigurations.

    Digging Into the Cloud Misconfiguration Problem

    To get to the root of the problem, we must fully understand the risk of cloud misconfigurations and how they differ from more traditional vulnerabilities. While traditional vulnerabilities can be patched, cloud misconfigurations can create vulnerabilities in otherwise secure applications and infrastructure. Imagine cloud infrastructure like a highway system and cloud misconfigurations as road hazards. They can be incredibly dangerous and lead to accidents.

    As a specific example of cloud misconfigurations, let’s look at identity and access management (IAM). Poorly configured IAM, such as misconfigured roles or a lack of multi-factor authentication (MFA), can lead to compromised administrative accounts. If a threat actor hijacks a legitimate administrator account, they potentially can take full control of an entire cloud environment. Because IAM sits above the cloud infrastructure layer and all workloads and data within, once it is exploited a threat actor will often be able to circumvent your other security barriers, such as network segmentation, leaving you defenseless.

    Cloud Misconfigurations Can Easily Scale

    Where misconfiguration risk dramatically increases is through the adoption of cloud-native applications and practices like infrastructure as code (IaC) templates used by DevOps teams. These templates offer teams greater speed and scale for building and managing applications, but the downside is that misconfigurations can be unknowingly replicated from development environments to production environments (where sensitive data is stored) at greater velocity.

    In short, the biggest risk to organizations right now is scaling these misconfigurations through the cloud. As the cloud grows in adoption and scale, we’re witnessing these misconfigurations scale right alongside it. Where before, a misconfiguration might have been limited to a siloed application or environment, today, that same misconfiguration can impact the entire organization without checks in place to catch it. Worse, many cloud storage buckets have logging disabled, meaning once threat actors are able to identify a misconfiguration and access an internal cloud bucket, an organization won’t even be able to see what data was accessed.

    Understanding the Shared Responsibility Model

    This all points to a higher-level, hygiene-related issue to keep in mind when securing cloud environments: responsibility. Particularly, organizations struggle to understand the shared responsibility model and how it applies within their own organizations. While some may fail to delineate between cloud provider responsibilities and their own, the major issue at stake is the shared responsibility between various internal teams that often goes undiscussed and undocumented until there is a security event.

    When thinking about software-as-a-service (SaaS), to a much lesser degree the responsibility falls on consumers. However, when considering infrastructure- and platform-as-a-service (IaaS/PaaS) and all the moving parts such as network, user credentials, resource configurations, workloads, identity configurations and more, cloud consumers become responsible for much more. A key consideration to note is that one can never outsource accountability, no matter which cloud model is used. To put this more simply, if one puts data in any cloud provider, they are still accountable for that data.

    Enabling a Holistic Cloud Security Strategy

    When advising security and business leaders on how they can better secure their cloud environments and applications, I highlight what my team calls “The Big Cloud 5,” a set of recommendations developed to help organizations adopt a holistic cloud security model that accounts for proper security hygiene and shared responsibility.

    • Gain awareness and deep cloud visibility
      The very first step to ensuring cloud security is understanding how teams are using cloud technologies, leveraging shadow IT and cloud provider APIs. This allows you to get situational awareness and make informed decisions today as well as in the future. This is not a one-time event, but something you’ll need to do continuously.
    • Set guardrails to automatically prevent the most serious cloud misconfigurations
      Drawing lines in the sand around the most offensive (and potentially destructive) misconfigurations that should never exist in an environment is key to automating protection in the cloud. This will help keep templates and practices controlled, so that poor hygiene doesn’t inadvertently take root and spread. Think of this as your “dirty dozen.” What configurations should never exist in your cloud environments?
    • Standards are the precursor to automation
      One can’t automate what hasn’t been standardized, and while there aren’t widely accepted security standards yet, key stakeholders in an organization must be in agreement about how to secure cloud infrastructures.
    • Train and hire security engineers who code
      To fully leverage APIs, security teams must have engineers who know how to code and automate security processes. An assessment of skills that exist across your security team (e.g., knowledge of coding in the likes of Python or Ruby) can point to areas in training and hiring that need further investment.
    • Embed security in the development pipeline
      Map out who, what, when, where and how your organization pushes code into the cloud. Once mapped, identify the least disruptive insertion points for security processes and tools, so that they can exist in as much of the development pipeline as possible.

    If you feel your organization is a step behind on proper cloud security hygiene, know that you’re not alone. Earlier in 2021, the Cybersecurity and Infrastructure Security Agency (CISA) released a report highlighting the importance of strengthening security configurations based on the uptick in successful cloud attacks, often attributed to poor hygiene and the mixed use of computing devices in the remote work environment. While this is a growing problem, it is also a reversible trend. Best practices, cyber hygiene and a shared responsibility model can help companies safely and securely migrate to the cloud.

  • Survey: DevOps Growing Pains in Salesforce Environments

    Survey: DevOps Growing Pains in Salesforce Environments

    IT organizations that have adopted software-as-a-service (SaaS) application platforms from Salesforce in part to jumpstart their digital business transformation initiatives are experiencing some DevOps growing pains.

    A survey published this week of 230 executives, managers, and members of Salesforce delivery teams conducted by Beagle Research on behalf of Copado, a DevOps platform for IT teams building custom applications, shows that while 82% have increased or maintained the size of their Salesforce development teams, the change fail rate jumped from 23% last year to 33% in 2020 when compared to a similar survey conducted a year ago. The survey also notes time to recover increased from 110 hours in 2019 to nearly 196 hours in 2020.

    Andrew Leigh, chief marketing officer for Copado, said those results generally reflect the nature of the more complex application initiatives being launched on top of Salesforce applications. The survey also noted 55% of respondents don’t consistently automate their Salesforce deployments, with less than a quarter of respondents (23%) having a way of automatically deploy a configuration from development through testing to production.

    In fact, in teams with fewer than 10 members, only 28% of survey respondents automated Salesforce deployments compared to 56% of teams with more than 100 developers that have automated their deployments, the survey found.

    The survey also noted that more than a third (34%) of respondents said features that development teams create may not be adopted. That issue suggests there is more work to done in terms of better aligning development teams with the goals of the business, Leigh said.

    Organizations building applications on top of Salesforce don’t typically embrace DevOps best practices until development teams have 25 or more members, Leigh noted. Salesforce environments generally require their own DevOps platform, because applications are constructed using a set of objects and tables unique to the platform. Those objects and tables are most often invoked using a set of low-code application development tools. Nearly three-quarters of respondents (73%) employed at least one commercial Salesforce development tool.

    Overall, the survey finds 82% of organizations have increased their budgets for or effort toward digital transformation, with 53% reporting they have exceeded their performance goals for the year. The survey also identifies a set of elite DevOps performers that are reporting 30x shorter lead times; 46x more frequent deployments; 5x lower change fail rate; and 37x shorter time to restore. Elite performing teams are 65% more likely to have adopted continuous integration than low performing teams, the survey notes.

    More than half of respondents (53%) said that they found it easy to find people who have DevOps experience in a Salesforce environment, with salaries typically in the $90,000-$140,000 range.

    The survey also finds the majority of organizations (82%) expect the systems integrators they contract to build applications to help define their DevOps processes.

    As organizations continue to adjust to new business realities in the age of COVID-19, there’s no doubt more custom applications will be built and deployed on top of SaaS application platforms such as Salesforce. The challenge is finding the best way to accelerate the development of those applications in a way that consistently produces the desired result.