devopsy.kinsta.cloud

Category: DevOps Practice

  • The DevOps Sweet Spot: Inserting Security at Pull Requests (Part 2)

    The DevOps Sweet Spot: Inserting Security at Pull Requests (Part 2)

    In part one of this two-part series, I explored how organizations can more effectively automate security quality decisions and discard doing automation for automation’s sake. I shared why security scans need to be faster, more reliable and comprehensive. Only then can security be meaningfully automated into developer workflows without slowing them down.

    But, the other part of this is understanding where these security scans should take place throughout the SDLC. Why? Because inserting security too early or too late in development and pre-production can have drastic impacts on remediation efficiency, developer productivity and more. 

    Shifting Security as Far Left as Pull Requests

    Identifying vulnerabilities once they’re in production is the riskiest, as well as the most time-consuming and expensive remediation. Working in pre-production test environments or builds reduces the risk, but not much of the time or efficiency in remediation. Conversely, inserting code analysis into developer IDE’s is too early. While in theory the IDE could be a good place for security feedback, in practice, it slows down the developer’s machine and comes at a huge cost of productivity.

    Instead, commits and pull requests are the ideal place to insert security because it delivers the right vulnerability information to the right developer at the right time. Combine this with integrations into ticketing and other systems and the full developer workflow can be meaningfully automated.

    Simply by getting the right developer, the right vulnerabilities information breaks down one of the biggest challenges in AppSec. Developers typically outnumber AppSec team members by 50-1 or more—hence a security scan that generates 100 vulnerabilities creates a tremendous amount of work for a single AppSec professional. However, assigned to the right developers, at an average of two per person, the challenge immediately becomes more manageable. Still accuracy matters because, while the burden may be spread out, developers triaging false positives is an expensive resource focusing on a low value task.

    Getting developers security feedback at the right time is also vitally important. When developers get immediate security feedback, vulnerabilities become easier to fix because the code is still fresh in their mind. Furthermore, fixing vulnerabilities early has compounding benefits. Since insecure code never goes into production, not only is the attack surface reduced, but additional code that is dependent on the vulnerability never gets written. This minimizes the most expensive and time-consuming security fixes, which require not only fixing the underlying vulnerability, but also rewriting dependent code. Best of all, with immediate, regular, comprehensive and accurate security feedback, developers will ultimately learn and adopt more secure coding practices overtime.

    Lastly, by inserting security at the commit or pull request, security and developers can establish a shared responsibility model on more realistic terms. When security can commit to accuracy and timely KPIs, developers can commit to triaging and fixing vulnerabilities faster. This reduces MTTR, puts a downward trend on defect density and reduces the application’s overall attack surface. It also reflects the realities that AppSec professionals have a limited ability to triage vulnerabilities, because they don’t know the code intimately and that only developers can fix the underlying vulnerabilities.

    By solving the fundamental challenges of speed, accuracy and comprehensiveness to deliver the right vulnerability to the right developer at the right time, security earns the ability to ask more from developers, and CISOs and CTOs can jointly accept responsibility for protecting applications and data.

    The rate at which software is being created and shipped isn’t slowing down, but many organizations have struggled to find where security most effectively fits into this process. By inserting security at the pull request, not only can developers identify and remediate vulnerabilities faster and more accurately, they can better automate security quality decisions. By taking this approach, organizations are fundamentally changing how engineering and security collaborate within enterprises.

    — Chetan Conikee

  • Moving from NOC to the SRE Model

    Moving from NOC to the SRE Model

    Is Site Reliability Engineer (SRE) another case of “Do what Google does”? When viewed with a discerning lens, the practices and tooling high-performing engineering teams deliver to the market can be very beneficial. The SRE model is one of them and stands to benefit DevOps teams substantially.

    If I could change the name from Site Reliability to Service Reliability I would. Applications are increasingly more service oriented, architectures are based on services and the SRE model is generally more customer centric than previous application support approaches. SRE has become the production support companion for modern application development.

    While the Network Operations Center (NOC) and SRE primary objectives are to focus on application uptime, the responsibility of supporting applications is much greater. Those who support modern applications are under pressure from the business and development to respond fast and accurately. This has been done previously by a NOC only.

    In its most traditional form, the NOC was a 24x7x365 staffed room where all telemetry from various aspects of infrastructure and the application were presented on screens, usually with one screen dedicated to sports and video games. If you staffed the NOC, your job was to keep an eye on the application, spot any alarms and if there is an alarm, to find the right person—usually based on a spreadsheet—to fix it.

    The NOC has also classically been more tied to the infrastructure than the application. They would get alarms for the application, but they’d be unable to dig in and figure out what specifically in the application was causing the issue. The NOC model services waterfall development and monolithic architectures just fine. But now, with rapid release velocity and new application architectures, a new approach is needed. This is where SRE comes in.

    Customer Obsession

    The first big difference between the SRE model and the traditional NOC model is that the SRE role does not start when something breaks, it starts before. The most high-performing SRE teams I’ve seen always work backwards from the customer. They use the customer to decide what metrics to measure and collect, and how they will interpret the metrics.

    Google talks about RED (Request, Errors, Duration) metrics but these are in no way best practice for everyone, nor the only approach. They are the best practice for Google, perhaps, and how their users interact with the application. Things such as saturation, latency, etc., can be even more important. Also, it is not the individual metrics that matter, it is their relationship to each other and, ultimately, to the customer’s experience. The customer obsession bleeds into everything SREs do—from setting up automation to educating the rest of the technical teams.

    To be customer obsessed, SRE teams need to have good visibility into what is going on.

    Observability

    I’m sure you have heard the new popular term “observability.” You might have thought to yourself, “Isn’t that just monitoring?” Observability’s foundation is monitoring, but it’s purpose is to go from the static point-in-time view of applications that traditional monitoring has been used for. It also addresses new challenges that show up as application architectures become increasingly more fragmented with services, and more complex to monitor holistically.

    Observability in some ways is about removing the human observer. In the NOC, you have staff watching aspects of the application and infrastructure to decide when there is an issue. Platforms that are designed to be observable have the detectors to tell you when there is a problem. It also does not create an assumption that those on-call have existing underlying knowledge of how all layers of the application work in advance because with modern applications, this assumption is not possible.

    Observability is both a tooling approach, but also a strategy for building applications. Applications should be built to be observable, which means the application provides sufficient telemetry to support its observability. Monitoring is basically part of your application now.

    Additionally, application degradation (such as latency) is the new downtime, so the ability to measure depth and breadth of application activity is critical. Observability helps identify reduced performance of the application as well, or impending issues in services that could lead to catastrophic events.

    Automation

    There is one big assumption that comes with the SRE model: automation and the interest to automate. The SRE model enables greater context and visibility across the organization, but it has to do that without increasing human effort. Without automation that is not possible.

    Automation includes:

    • Going from insights from observability to incident response.
    • Gathering context from systems to report, to observability tools and incident response.
    • Mechanisms to support those who are on-call with self-service remediation.
    • Automation to understand similar incidents.
    • Automation to alert/page/call people based on on-call schedules and policies.
    • Automation to recommend responders based on their incident response history.

    Anyone Can Be On-Call

    One of the huge benefits of the SRE model is anyone can be on call. The practices and automation of the SRE model allow anyone to be added to a rotation and on-call schedule. Most companies have this as a dedicated team, and in those who are transitioning from the traditional NOC to modern, that is exactly the first change that takes place. They leave the dedicated physical room and are on-call virtually instead.

    Not just people who support the application are on-call, the SRE model allows for developers and quality engineering to be on-call as well. Arguably, you can’t embrace shift-left without embracing SRE.

    Context

    Automation without data does not create a truly observable and actionable environment. Leveraging data across your delivery chain that can support action is critical. For example, this means the ability to leverage tracing to pinpoint production issues down to a service, or embed more context as part of the payload of alerts to bring more clarity to those on-call.

    Data collection is one thing, but not knowing how you are going to leverage the data makes it useless. The SRE model is obsessed with leveraging data and doing controlled experiments to ensure the data collected supports the uptime and experience for the user.

    Stewardship and Strategy

    Finally, SREs spend more time on stewardship and strategy than the traditional NOC. Some SRE groups run DoJos for the organization to help spread the idea of quality and application support across all technical teams. They will collaborate with developers so they better understand aspects of the application the SRE processes can leverage to support it better. SREs also spend a lot of time creating the strategy of application production support for the organization. Choosing key metrics, deciding how automation will be setup and consumed and creating policies on how on-call should function.

    Before you get upset, SRE does not mean the NOC is dead. The modern NOC can be driven by an SRE model, and I’ve seen this in-place in several companies. The NOC is also necessary to support many on-prem applications.

    What happens is, instead of a staffed room, the NOC becomes a group of engineers who are building automation, setting policies and practices, and stewarding quality across all teams. Similar to DevOps, SRE has elements of culture, principles and practice. Organizations who have a team structure where the NOC is responsible for application support, oftentimes do not change this for SRE, they become the SRE.

    The NOC is changing, and has to change as modern applications become more complex, release velocity increases and users become more demanding. The SRE model is an approach to transform the NOC into a modern version of itself or a whole new organization and strategy. The goal is not only to support applications when something breaks, but to make sure customer’s expectations are met, and systems support current and future development efforts, without increasing the human workload.

    — Chris Riley

  • Catch Emerging Security Risks Earlier by Leveraging Kubernetes Audit Logs

    Catch Emerging Security Risks Earlier by Leveraging Kubernetes Audit Logs

    When a Kubernetes deployment suffers a security breach, it can be difficult for security teams to diagnose and identify the source of the intrusion and its impact. Each Kubernetes cluster may host multiple applications, and each application may be composed of dozens or even hundreds of interacting microservices deployed as pods.

    The complex and dynamic communication between the applications’ components, as well as the automated Kubernetes orchestration activities, makes it difficult to understand the structure of each application and differentiate between normal behavior and malicious behavior from the observed network activity. The ephemeral nature of pods, which may exist only for moments to perform an activity before they are disposed of, makes this task even more difficult. A pod that was vulnerable may be gone by the time security teams are investigating a breach, or resume normal operation as the attacker pauses its abuse or pivots or escalates the attack to another Kubernetes resource.

    K8s is by nature dynamic, flexible and scalable, and it is this same advantage for cloud native application developers that becomes a challenge for security professionals. How can one understand the source of a breach, let alone how it propagated through the system, without deep understanding of the architecture of the application?

    One solution to the problem of tracing breaches is to examine the Kubernetes audit logs since they record all cluster-administration activity, even for pods that may have existed for seconds and are now gone. But K8s audit logs are complex and often convoluted, requiring deep expertise to understand. Therefore, security teams need to not only increase their proficiency with K8s audit logs, but also to apply automation and software tools to the K8s logs to detect suspicious behavior after the fact and even during their occurrence.

    In fact, there are many potential threats that can be discovered when observing audit logs for suspicious behavior. They include:

    • Stolen or misused credentials, enabling hackers to gain access to K8s-based clusters or pods.
    • Misconfigured Rules Based Access Control (RBAC), enabling lateral attack propagation, privilege escalation and unauthorized data access or manipulation.
    • Exploitation of vulnerabilities in the Kubernetes API Server, enabling bypassing of authentication, authorization, admission control or validation of cluster administration requests. This lets users gain access to privileged and sensitive resources.
    • Violations of security policies diverge from compliance requirements and best practices.

    Detect Breaches in Near Real Time by Watching K8s Audit Logs

    There may be a vast time span between an initial breach and the detection of the compromised application or the unauthorized access of secure data and private information. Security teams must seek to close the gap between a breach attempt and its detection as much as possible, while still being able to diagnose the causes of a breach after it has been detected and assess its scope and impact.

    Since K8s deployments can be vast and intricate, machine learning and artificial intelligence are preferred tools to monitor audit logs and flag suspicious behavior. An automated tool could continuously scan logs to learn what normal behavior looks like in the company’s clusters, and then flag network behavior that is unusual or suspicious for security teams to qualify, prioritize and investigate. This methodology would enable security teams to investigate possible security violations in near real time to protect and secure the applications.

    In addition to detecting security risks, security tools that leverage K8s audit logs should provide reporting that is easy to understand and consume by security teams. By providing the correct user experience, companies can make their K8s logs accessible to more than the hard-core experts, effectively opening a new vista of security possibilities for more security professionals. The difficult learning curve of Kubernetes will no longer be a wall between K8s pros and newcomers with less experience, relieving pressure from understaffed security teams.

    Step Up to Security and Compliance Needs

    As Kubernetes moves toward the mainstream, with roughly 71% of Fortune 100 companies using Kubernetes as their main container orchestration tool, companies must also look toward compliance with standards and government regulations designed to protect financial transactions and private or personal information such as PCI, GDPR and HIPPA. The penalties for violating these standards and regulations can be severe, so companies should proactively adopt K8s audit log monitoring as a way to identify breaches earlier, to limit their impact on customers and data, and to document the behavior of the application and would-be hackers for possible legal inquiries. An effective audit log strategy will be vital in determining what happened in the case of a breach.

    Leverage Audit Logs for Real-Time Security Observability in K8s

    Many tools exist to secure the perimeter of the data center and to identify possible security risks during application development. The benefits of analyzing and leveraging Kubernetes audit logs at runtime have been relatively overlooked. K8s audit logs are a prime way to comprehend the behavior of any cloud-native application orchestrated by Kubernetes, as well as the best way to comprehend how the Kubernetes infrastructure itself can be abused by an attacker to compromise the applications it is hosting. As such, security teams have a responsibility to deploy tools and gain a proactive, real-time view into K8s audit logs in order to minimize the time between breach and remediation and to validate compliance conformance at runtime. Such a thoughtful approach to Kubernetes security will go a long way to protect your deployments from harm.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or Aug. 2020.

    — Nitzan Niv

  • Atlassian Adds No-Code Automation Tool to Jira Cloud

    Atlassian Adds No-Code Automation Tool to Jira Cloud

    Atlassian announced today it has added a no-code tool to enable DevOps teams to make it easier to customize Jira project management software without having to learn how to code.

    Sean Regan, head of growth for software teams at Atlassian, said Automation for Jira will now become a core part of Jira Cloud that will be made available to all 65,000 customers using Jira software. Originally developed by Code Barrell, the tool was acquired by Atlassian last year. Previously, it had been available as an add-on adopted by about 6,500 customers, he said.

    The goal is to enable administrators that track projects to reduce the amount of time they spend on manually entering data. On average, organizations are spending 51 hours a month on administrative tasks involving the management of projects, Regan noted. Organizations should take inventory of those tasks and use Automation for Jira to start automating the tasks that are the most repetitive, he said.

    Automation for Jira, Regan said, is part of a larger ambitious effort to make Jira software accessible to more than 100 million active end users. By employing a visual tool to automate tasks it should become easier for a broader number of organizations to customize processes by defining rules that trigger events. Organizations will not be forced to depend on developers to write code to create those workflows, he noted.

    Automation for Jira

    Automation for Jira is also part of a concerted effort to modernize a venerable Jira project management application that has been widely employed by software development teams for decades. The ultimate goal is to enable software development teams to spend more time building applications versus trying to ascertain the status of any given development project, said Regan.

    The no-code Automation for Jira tool is also designed to integrate with collaboration tools Slack and Microsoft Teams within the context of a set of best DevOps practices. In fact, Regan noted any time there is an issue on a software development project it almost always comes down to a lack of collaboration between all the stakeholders involved.

    There is, of course, a lot more appreciation for project management software among DevOps teams that typically are trying to organize around multiple projects at the same time. A delay on one project might present an opportunity to reallocate resources to accelerate another. That can’t happen, however, if no one is aware of what bottlenecks exist in the first place.

    Not everyone appreciates all the nuances of project management. The less obtrusive project management becomes the more effective it tends to be. Developers are especially notorious for not always updating project management applications, which usually results in project managers having to nag them to share information that other teams are counting on before they can move forward. Hopefully, there may come a day when all project management applications are updated automatically. In the meantime, the less time spent on manual tasks the happier all the members of the software development team are likely to be.

    — Mike Vizard

  • The DevOps Sweet Spot: Inserting Security at Pull Requests (Part 1)

    The DevOps Sweet Spot: Inserting Security at Pull Requests (Part 1)

    Today’s organizations are increasingly benefiting from the modernization of the software development lifecycle (SDLC), including the adoption of cloud, DevOps, Agile methodologies, containers and more. Thanks to this modernization, organizations are innovating and releasing updates faster than ever before. But these benefits aren’t without challenges.

    Application security arguably tops the list of challenges. Security has inherently manual bottlenecks, and is unable to keep up with the pace of automated DevOps pipelines. The result is that security is often left behind, overlooked or simply ignored in the development process. Developers aren’t willing to slow innovation down today for the potential of security breaches tomorrow.

    As a result, more and more applications are being shipped into production with vulnerabilities and security flaws. Today, most of the vulnerabilities are found in post-production either via red teams, bug bounty programs or siloed and irregular code analysis scans. At best, these vulnerabilities are fixed in the next version that’s shipped. At worst, your organization is making headlines for millions of personal records being breached.

    However, security in the modern SDLC is becoming more symbiotic thanks to two key advancements—speed and accuracy. Overcoming these two challenges enables security to be inserted into developer workflows earlier than ever before. By automating security with every pull request, development organizations can finally embrace security without slowing down innovation.  

    Automation — Security at the Speed of DevOps

    These days, many vendors claim to insert security into DevOps pipelines, often during the build. While it’s true that security scans may be automatically started, they still finish in siloed processes that don’t keep up with the pace of releases. We call this “automation for automation’s sake.” It does not measurably improve defect density, slow the accrual of technical debt or reduce mean time to repair (MTTR). The only value that automating this way provides is checking a box on a compliance form.  

    So, how can organizations more effectively automate security quality decisions? 

    First, security tests and scans need to be dramatically faster. Legacy code analysis takes such a long time that many vendors recommend sacrificing comprehensiveness by scanning only differentials. Even then, they still aren’t fast enough to keep up with CI/CD. It goes without saying that organizations cannot meaningfully automate security into processes such as builds or pull requests, that take just a fraction of the time.

    Second, the results need to be more reliable. The goal of security automation should be to fail insecure code. However, historically, code analysis tools are plagued by so many false positives that no code would ever pass any test. Only with reliable results can insecure code be confidently failed. 

    Third, security scans must be comprehensive as false negatives lead to vulnerabilities going undetected into production. This is, in part, because developers are not asking the right security questions when scanning their code. Traditional code analysis tools leverage pattern matching techniques—they look for symptoms of a vulnerability that are common across all code bases. For example, lack of input validation from sources is a classic cause of injection vulnerabilities. However, there are many vulnerabilities that are unique to each code base, such as business logic flaws, data leakage, hard-coded literals and insider threats.

    For example, all applications are purpose-built to meet specific business needs, meaning vulnerabilities may go beyond coding errors and present themselves as business logic flaws in the design of an application. Take the First American Financial breach. Thanks to an insecure direct object reference (IDOR), an attacker has the potential to manipulate an internal implementation object. In the case of First American Financial, this was a URL or form parameter. Given the asymmetrical advantage that attackers have, defenders must protect from all possible vulnerabilities, but attackers need to only find one vulnerable route. As such, partial security can be as problematic as no security.

    Only when security is completely comprehensive—reliable enough to fail with confidence and fast enough to keep up—can it be meaningfully automated into developer workflows without slowing them down.

    Yet, comprehensive security scanning is only part of the equation. In the second article of this series, I’ll dig into where these security scans should be taking place in the development cycle to make them most effective.

    — Chetan Conikee

  • Better Apps and Better Security When You Shift Left

    Better Apps and Better Security When You Shift Left

    Tens of thousands of people and hundreds of cybersecurity vendors descended on San Francisco at the end of April. While the RSA Conference was the primary draw, there are a number of peripheral events that go on throughout the week as well, such as the Cloud Security Alliance Summit, the CIO/CISO Interchange, BSides San Francisco and the Qualys Security Conference.

    I was able to attend some of the Qualys Security Conference and I had the opportunity to see Alex Mandernack, security solution architect for product management at Qualys, presenting on the need to shift left when it comes to cybersecurity.

    Mandernack outlined some of the issues organizations face today when it comes to security. Migrating to the cloud and embracing DevOps culture means more ephemeral, elastic environments and compressed development time frames. He pointed out that these trends make security more challenging, but also described the opportunity they present to change our approach to streamline and improve security.

    The Traditional Pipeline is Broken

    The traditional model of application development is much more linear. Each app team builds their own image—often in a silo or vacuum disconnected from other teams. Penetration testing is conducted on a sporadic or infrequent basis, and eventually the app is deployed. Vulnerability scans are run on the app in production, and issues or deficiencies are reported back to the developers to address or incorporate into the next build.

    The process is slow and cumbersome. It also has the potential to introduce redundancies that waste resources and make the organization less efficient. As companies embrace DevOps and move to the cloud, the traditional development pipeline simply doesn’t work—especially not for security.

    DevOps, and Cloud, and Containers — Oh My!

    The cloud and DevOps fundamentally change the development lifecycle. Development time frames are compressed, and the process is a self-feeding, iterative circle rather than a repeating linear approach. The traditional approach of identifying and resolving security issues after an application is deployed won’t work in a DevOps environment because by the time issues are identified and fixed, the app itself may have already moved on to the next version.

    Cybersecurity and the development pipeline have enough challenges with the cloud and DevOps, but there is also a newer technology trend that exacerbates the problem—containers. Containers are designed to get code quickly and to operate fast. They are also built for scalability—spawning and spinning down hundreds or thousands of containers constantly to keep pace with demand.

    There is a diverse array of tools and platforms associated with cloud, DevOps and containers. Organizations often employ a complex collection comprised of some combination of AWS, Azure, Docker, Kubernetes, OpenShift and others. The result can quickly become a black hole when it comes to security—which is why it’s important to push security left into the CI/CD pipeline.

    Shift Left to Help Security Teams Do Better

    In a DevOps culture, developers and security teams need to think about security sooner. It’s important to get the security tools into the process earlier. Thankfully, with the DevOps lifecycle and the tools commonly used within a DevOps environment, it can be done.

    For starters, companies should build a golden image. Developers shouldn’t have to patch and update the base operating system before they even start developing. The golden image takes a baseline operating system such as Ubuntu or CentOS, and completely patches and configures it so everyone has a standard, consistent image to develop on.

    With the golden image in place, the next step in shifting security left is to implement testing and scanning in the CI/CD pipeline. Put vulnerability information at the fingertips of developers as they’re developing and implement vulnerability gates in the pipeline to ensure that applications that don’t comply with policies and meet minimum requirements are not allowed to be deployed.

    New Role of the Security Team

    Part of shifting security left includes providing tools that make security self-service for the DevOps team. Automating actions with scripting, APIs and CI plugins ensures that security is simplified and streamlined to provide value for developers.

    In this shift left scenario, the security team has to redefine its role in the process. The security team is still vital, but rather than being the police or a roadblock in the development process, the role of security focuses on verifying and auditing the process. The security framework should have dashboards and live data that enable the security team to monitor activity and identify trends that may need further attention.

    Better Security, Faster

    There is often conflict between developers and security teams, but there doesn’t need to be. Mandernack stressed that developers don’t want to create insecure apps either.

    Shifting security to the left into the CI/CD pipeline streamlines the whole process. Ultimately, the result is higher quality, more secure applications that are developed faster.

    — Tony Bradley

  • DevOps Brings Pressure to Bear on Database Management

    DevOps Brings Pressure to Bear on Database Management

    The rise of DevOps is starting to have a significant impact on the rate at which databases now need to be deployed and updated. A global survey of more than 2,000 developers, database specialists and IT leaders conducted by Redgate Software, a provider of tools for automating the management of databases, finds nearly half (49%) are now deploying database changes in production at least once a week or more.

    Kendra Little, DevOps advocate for Redgate Software, said most of the changes are being driven by the adoption of best DevOps practices that are now having a cascading impact on other functions in the IT environment.

    However, the survey shows the adoption of DevOps remains uneven. Only 18% of respondents said they have adopted DevOps across all projects, while 36% said they have adopted DevOps across some projects. Another 14% said they are at the proof-of-concept stage. Only 13% said their organizations have no plans to adopt DevOps at all. Survey respondents in the financial services and insurance industry sectors have the highest levels of adoption, with more than 77% of respondents saying their organizations are either in proof of concepts (16%), have adopted across some projects (41%) or have adopted across all projects (20%).

    Progress appears to be rapid among those that have adopted DevOps. Well over a third (37%) report that only 1% or less of their deployments introduce code defects that require hotfixes. That compares to 30% for all other groups.

    The survey also revealed that more than half of respondents (55%) perform most or all of their business-critical database deployments while their systems are still online, with 75% of that group reporting that 10% or less of those deployments caused defects that require hotfixes. Nearly half (46%) are also employing some of automation to deploy databases, according to the survey.

    Nevertheless, the survey makes it clear challenges remain. Disruptions to existing workflows and business was cited as the top barrier to adoption (23%), followed by lack of skills (22%) and lack of alignment between developers and operations (16%).

    Little said the survey makes it apparent that database management needs to become more aligned with the software development life cycle (SDLC) with organizations transitioning to DevOps. Over a quarter of respondents (26%) said it is still somewhat or very difficult to get code reviews for database changes. A further 16% said that no code reviews are done for database changes early in the development cycle.

    Less clear is who precisely is managing the database. Developers in the Age of DevOps are exercising more influence over what databases are employed. However, Little noted that 43% of organizations still have a database administrator managing their databases. That compares to 34% that rely on developers and 21% that have a combination of both. Over time, Little said most organizations will continue to shift toward a combination of both. That, of course, will require a significant amount of cultural change within most IT organizations, where all too often developers have tended to view DBAs as a bottleneck to be eliminated whenever and wherever possible.

    — Mike Vizard

  • Provide Continuous Value with Product-Centric Delivery

    Provide Continuous Value with Product-Centric Delivery

    Software engineering and management has borrowed many practices over the years from traditional industries such as construction, automotive and manufacturing. One such practice is the notion of managing software development and delivery based on projects.

    According to the Project Management Institute (PMI), a project is defined as “a temporary endeavor undertaken to create a unique product, service or result. A project is temporary, in that it has a defined beginning and end in time, and, therefore, defined scope and resources.”

    Many enterprises today manage software delivery via a project-centric software delivery approach. New software projects are identified based on probable business cases. Business opportunities are identified and prioritized based on potential business benefits and estimated cost. This is done with the assumption that the exact market needs, the estimated cost, start and end dates, and potential benefits are certain and known up front. Once the business case is approved by senior management, projects are initiated. In an environment where the scope, schedule and cost of projects can be defined upfront, this provides a predictable approach for software delivery management and governance. 

    But, do we live in a world of total certainty? In the volatile and uncertain world in which we live—with the very fact that software is not a physical good—accurate predictions of market needs, cost, timing and end-user benefits are far from reality. In addition, companies are competing in the marketplace by exploring innovative ways to create and deliver new products and services.

    Why is the project-centric delivery approach used by so many IT organizations if it is a challenge? Projects can no longer be the vehicle for software development and delivery if enterprises want to stay ahead of the competition. Then, what is the alternative approach?

    Introducing the Product-Centric Delivery Model

    A product-centric delivery model is a new way of organizing people, and continuously delivering valuable software to customers.

    Let’s take a look at the steps an enterprise needs to follow in order to transition from project-centric to product-centric software delivery. 

    Identify Products Instead of Projects

    What are software products? Organizations across the world provide value to their customers in the form of goods and services. These goods and services are called products and are often driven by software and IT systems. For example, to a bank that provides digital banking services to its customers, its online banking solution is a product, not a project. Similarly, for a retail company, its catalog and product search becomes a product. These kinds of products are different from the actual physical goods that these companies sell.

    Organize and Fund Stable Teams Around a Product, Not a Project

    Once your products are identified, the next step is to organize and fund your software teams around these products. These cross-functional teams include members from business development, IT development, support and operations. This team must be empowered to build and deliver the product on their own and will collaborate, decide what to build, how to develop it and how to manage the product with frequent updates. The team must be outcome-oriented, long-living and cross-functional. They can be organized as a team-of-teams or squads, resulting in better collaboration, focus and ownership from any corner of the world. This will also help in scaling the overall capacity of the product. A common shared services team, supporting multiple products might be an added advantage. 

    Define Product Vision, Define Objectives, Deliver and Measure Value

    After the products and the teams are in place, you need to define the product vision. The product vision defines the reason for its existence and the long-term goal that it is trying to achieve. This also aligns business objectives with IT delivery and provides guidance for all decisions made regarding the product development and delivery.

    The business stakeholders define the product vision and derive the near-term objectives that will guide the teams to further decide the set of initiatives they need to work on.

    The teams then identify relevant product features, plan and deliver those in short increments or time boxes, thereby delivering continuous value to end users.

    At the end of each time-box, stakeholders will measure the product outcome with the stated objectives and changes in the market needs. The findings will help the business decide on whether to continue investing in the product or not.

    Conclusion

    Product-centric delivery paves a new way of organizing people and continuously delivering software value with more emphasis on outcome rather than the output. It provides a continuous flow of work to the product teams and continuous value to the end users. It aligns the business strategy to software development and execution, while empowering the teams to decide what is best for their product.

    — Prabhu Arumugham

  • A Deep Dive Into AI for IT Operations

    A Deep Dive Into AI for IT Operations

    Traditional IT administration and management techniques are reforming. By employing artificial intelligence (AI), IT operations are taking an interesting turn in the field of advancements. The term “AIOps” stands for Artificial Intelligence for the IT Operations. AIOps uses AI techniques and algorithms to monitor the data as well as reduce the blackout times. The dominance of digital businesses is introducing technological changes in their operations to streamline them and eliminate the overhead of traditional operational management. 

    Digital companies are generating abundant and information-rich data whose analysis and monitoring do not demand delays. From data collection to storage and data processing to analysis, all operations need to be frictionless to deal with them in an efficient manner. In the last couple of years, the AIOps market has exploded which is ultimately achieving the goal of reducing delays in problem-solving and identification.

    Paving the Road to AIOps

    The story starts with the need of AIOps and digital transformations. Cloud adoption and the shift toward other digital technologies are revamping the IT industry. It also includes the adoption of innovative solutions in the IT operations to make them frictionless. The technologies that are widely used in IT operations vary from AI to machine learning, big data to Internet of Things (IoT) and Application Programming Interfaces (APIs) for various use-cases to edge computing. All these innovations are penetrating into IT operations making them more smooth and sleek.

     AIOps platforms enhance and automate IT operations, using the following different approaches:

    • Using machine learning algorithms and neural networks for better analysis of gathered data. The data is gathered using IoT devices and tools connected with the source and sending data to the cloud for storage. 
    • The system automatically interprets if something is wrong in the operations. The errors are detected and solved immediately without introducing delays in the operations that are dependent on each other to complete.
    • Reducing human power by automating the processes that include the effort of individuals. 

    Components of AIOps

    The two basic components of AIOps that accelerate innovation in IT operations are machine learning and big data. Instead of spending human effort and time into cleaning, processing and analysis of the raw data collected from different platforms, IT operations are able to do all the tasks by employing machine learning algorithms and techniques to judge the nature of data and deduce necessary information from it.

    Factors Boosting the Emergence of AIOps

    AI is a promising technology that holds the confidence of building applications that can help the IT community. AI provides options for reliability, scalability and generic tools that can be deployed to promote the IT operations and boost them without utilizing human power.

    Factors boosting digital transformation in the IT industry:

    • Complex IT Infrastructure: Its operations face many challenges regarding data management and infrastructure. Manual IT operations are becoming harder to handle such as data collection, processing, cloud management, database efficiency, third-party management tools and SaaS integrations. Complexity management is a task that needs proper attention and the environment to deal in a better way. 
    • Handling Abundant Data: IoT technology is penetrating rapidly at an industrial level. IoT devices produce an abundant amount of data. Handling this much data is not the job of individuals. Therefore, by using AI-based algorithms the data is interpreted and analyzed. The performance is monitored using efficient algorithms. 
    • Increased Computing Power: The networking infrastructure has become robust. Therefore the IT industry is in dire need to enhance the processing speed also. By deploying IT solutions that harness AI and machine learning technologies, the computational power of the system can be increased. 

    Enterprises Using AIOps

    The IT companies that are complex and large to which human power is not enough for overall operations are using AIOps to automate the tasks.

    The adoption of technology has a huge impact on the success of IT companies:

    • Development Teams: Companies that have a DevOps model and struggle to manage the roles between the organization are using AI and other innovative technologies to smoothen their processes. With this, developers increase the responsiveness, agility and clarity in the IT operations.
    • Cloud Computing: Cloud computing has a big future. The companies store their huge data in the cloud and all the operations related to cloud data management need to streamline, as it has its own challenges. Therefore, to make the processes flow without errors and monitor the access, there needs to be automation.
    • Digital Transformation: Companies around the world are all active in adopting technology solutions. Whether it is a fintech, travel, healthcare or e-commerce company, all businesses are automating their processing by first digitizing them and providing the user’s facility of utilizing applications based on newly emerged technologies. Information handling can be done by making operations handled by robotics and blockchain technology. AIOps removes friction from the IT operations.

    — James Efron

  • Don’t Let ‘Hindsight 2020’ Be Your DevOps Strategy

    Don’t Let ‘Hindsight 2020’ Be Your DevOps Strategy

    The idea of DevOps originated a little over a decade ago with a desire to ensure efficiency in the overall software development and delivery process. Constant operational hurdles that have been hampering flawless and frequent delivery of applications have led organizations to think beyond and to come up with continuous improvements in their processes and tooling. Thus, DevOps has evolved into a solid practice to aid enterprises with an end-to-end feedback enabled framework that spans across various functions and stakeholders to design, develop and deliver quality software.

    The IT landscape over the last few years has changed significantly. Companies are building secure applications and large volumes of workloads are being transformed to the public cloud. Nowadays, firms are actively applying AI technologies to transform their businesses more than they were just a few years ago. Some companies are still investing heavily on building their own complex AI platforms involving expensive infrastructure, but most companies have realized the value in leveraging AI as a service offered by leading cloud providers.

    Digital initiatives and IoT have further increased the use of cloud-based services and the need for automated deployment and delivery of applications and services. Seamless integration of the CI/CD pipeline for on-premise or deployment to the cloud is possible only with a solid DevOps practice. For organizations that are ambitious and striving to get ahead of the game in order to accelerate the growth of their businesses, DevOps is indispensable.

    To successfully implement some of the current major trends in IT, organizations are required to have a mature DevOps practice in place.  In October 2019, Gartner published its predictions on technology trends for 2020. We’ll pick a few from those to discuss the crucial role that DevOps is expected to play in them, and the critical success factors for companies to be advanced forward.  

    Strategic Trends in IT and the Role of DevOps

    Hyperautomation

    Gartner’s analysis on strategic technology trends for 2020 has hyperautomation as the number one trend. Hyperautomation, which enables organizations to function as autonomous digital enterprises, will be leveraging latest technologies such as 5G networks, intelligent edge devices, advanced data analytics, expanding RPA with AI, etc. Firms aspiring to implement AI and ML on a grand scale will want to take advantage of the diverse range of complex machine learning algorithms implemented on scalable cloud infrastructure and offered as services.

    Instead of recreating everything from scratch, organizations will simply upload their data to the cloud and train algorithms based on their business processes and needs. For hyperautomation initiatives to be successful, organizations need to implement end-to-end fully automated DevOps processes with the relevant tooling to specifically address the varied needs from several perspectives. Speed, agility and ensuring an error-free repeatable process will be basic requirements for hyperautomation.

    On-demand deployment of transient infrastructure on the cloud, driven by infrastructure as code and configuration as code will be a primary requirement for rapid development and testing of artifacts that form components of hyperautomation. Fully automated pipelines for the creation and deployment of containers will become a core requirement not only for hyperautomation but in general for most applications. The involvement of DevOps to manage data virtualization, obfuscation and uploading data to the cloud will become more significant to increase the overall efficiency of the automated CI/CD pipelines for hyperautomation projects. 

    Democratization of IT

    Democratization of technology is about making technology more accessible to common people. Gartner has predicted this to be another major trend for 2020. Smartphones and the internet have already played a major role on this front and have enabled people with the technology, tools and data to help make better decisions. Democratization of technology in the IT world is about enabling the IT community and it gets more and more interesting day by day.

    With container technologies such as Docker Desktop that can run on a user’s laptop, a developer now has access to pre-configured container images of various operating systems and other platforms that can be pulled from container registries within seconds. Cloud services such as IaaS, SaaS and PaaS add to this by providing more independence to the development teams apart from making the entire process agile and faster. Democratization at these levels also leads to setting a different level of expectations on the overall velocity of development and delivery of applications. All these services in isolation, i.e. when not connected together through a strong DevOps process and toolset, will not produce the expected level of agility and speed.

    An end-to-end DevOps process works all the way: from triggering the build of a container image as the code is built, deploying an environment on the cloud, integrating with other components, testing artifacts with automated testing and finally to producing a production-worthy artifact. When IT teams get more democratic, they will run their domain the way they perceive beneficial and produce the best possible outcome with quality and speed to make their teams and the organization successful. DevOps will be the means to achieving that goal.

    Internet of Things

    Edge devices are getting smarter by the day. Mass manufacturing of intelligent controller boards, such as the ESP 8266 and ESP 32, is making it extremely cheap to build edge devices with data storage and processing capabilities locally within the device itself instead of communicating with a cloud based central system for command and control. However, such intelligent devices need to be kept up-to-date on security patches, bug-fixes, firmware updates, etc. A solid DevOps framework that integrates testing of patches/fixes, and deploys tested artifacts to remote edge devices is an absolutely essential component of an organization’s IoT ecosystem.

    Some controller boards run light-weight versions of Linux and have the capability to additionally run a Docker engine. This helps ensure consistency through deployment of updated containers via a DevOps pipeline to the edge devices. Convergence of cloud and digital transformation initiatives, especially in IoT over the last few years, has widened the scope of DevOps. Major cloud providers are already offering several services to integrate edge devices with a full-fledged CI/CD pipeline leveraging DevOps processes and tools. 

    Blockchain

    Application developers and IT managers spend a great deal of time either fixing bugs or recovering systems from breakdowns caused by bugs. If best practices, processes and tools are put to proper use, this wasted effort can be greatly reduced and the entire development and delivery process could be made more efficient. Blockchain demands building a strong foundation in IT and raises the need for higher quality in software development, delivery and IT operations on the whole.

    Organizations have to take a closer look at their current capabilities and process maturity before embarking on a journey involving blockchain. Firms that have faced failures with blockchain can certainly vouch for the fact that the failure wasn’t caused by the concept of blockchain or the algorithms that drive it. Rather they had resulted because the rest of the IT systems weren’t reliable enough to keep up with the strict demands of the blockchain systems. DevOps and automation will certainly help improve the security, quality, reliability and repeatable deployment of applications. When the systems that feed the blockchain application go through such increments of improvement, the net effect is that the organization is getting closer to implementing a solid blockchain solution.

    Success Factors

    Firms are moving fast toward the cloud for IaaS and PaaS services. As DevOps, CloudOps and SecOps are converging, CIOs are becoming conscious about building organizations that encapsulate these functions under one leadership in order to set shared goals and ensure synergies across these functions. Organizations with strict compliance requirements and governance would need to involve their operations teams to lay the rules regarding using cloud services.

    Application development teams and users are still dependent on several operations teams to design and implement secure cloud connectivity and to establish the rules around using cloud services. While a strong governance is absolutely essential, there is a high possibility for firms to lose their agility and get caught once again in a mess if these operational processes are not smoothen and streamlined. For the strategic initiatives on implementing the new trends to succeed, it is vital for DevOps to be able to operate without any hurdles.

    Below are a few critical success factors worth considering.

    Dev and Ops in a Box

    Organizational structure plays a key role in determining the success of DevOps. Keeping applications development, DevOps and infrastructure operations very closely aligned with overlapping goals, KPIs and roadmaps is a key to success. Companies that operate these functions independently without proper alignment fail to establish an organization-wide DevOps culture and rarely reap the full benefits of DevOps.

    Full-Stack Engineers for DevOps

    It is important to develop a DevOps culture across the company. At the same time, it is equally important to bring in experts who can evangelize DevOps and take a completely fresh look at how things are done. Experienced DevOps professionals can pin-point areas for improvement, and additionally help accelerate the adoption rate of DevOps. Settling for high-quality resources that have full-stack experience is absolutely essential if you are in for a long run with DevOps. Global companies should seriously consider placing such experienced resources close to the development and operations teams. While it may be cost-effective to recruit resources offshore, that may not be such a good idea if your active development and infrastructure teams are onshore. 

    Trained Internal Resources

    Training internal resources are extremely important in order to carry out the recommendations from experts and to make DevOps a habit. Trained resources have to be deployed immediately to work on DevOps initiatives alongside the experts. To establish synergy and increase collaboration, it is a good idea to send development and operations resources to some common training courses and sessions.

    Wrapping Things Up

    Research organizations are closely observing the direction in which most organizations are moving toward in order to improve their speed, efficiency, agility and their overall ability to provide services that are secure and reliable. Business strategy and technology feed each other and as a result, some key trends in technology emerge from time to time.

    We looked at some of the trends that are going to be prominent in 2020 and the role that DevOps would have in implementing these technologies. Regardless of any specific technology, it is important to automate the integration and delivery of applications. DevOps processes and tooling will help a great deal in accelerating the pace of implementing new technologies and leveraging them in business applications.

    The organization of IT teams and a focus on recruiting the right talents will determine the overall success of DevOps. Training internal staff would help sustain the culture of DevOps in companies and fuel further growth in this space. DevOps will not be an optional function any more. It will be an essential component of the culture, and firms that adapt themselves sooner to this culture will be the ones most successful in implementing the current and future trends in technology.

    — Sridhar Asvathanarayanan