Category: Enterprise DevOps

  • The State of DevOps Report 2019 Is Out

    The State of DevOps Report 2019 Is Out

    The Accelerate State of DevOps Report 2019 is out like every other year. Now, let’s talk about the key findings of this year’s survey:

    • The Industry Continues to Improve, Particularly Among the Elite Performers: Proportion of high performers have tripled, now comprising 20% of all teams. This shows the improvement is real for those who really are doing the right things.
    • Delivering Software Quickly, Reliably and Safely Is at the Heart of Technology Transformation and Organizational Performance: It is evident that speed, stability and availability leads to organizational performance in terms of profitability, productivity and customer satisfaction.
    • The Best Strategies for Scaling DevOps in Organizations Focus on Structural Solutions that Build Community: High performers favor strategies that create community structures at both high and low levels of organization. Having established community of practices, having proof of concepts, helps them become more resilient and successful. Dojos are becoming more and more important.
    • Cloud Continues to Be a Differentiator for Elite Performers, and Drives High Performance: The use of cloud as defined by NIST Special Publication 800–145, is predictive of software delivery performance and availability. High performers are 24 times more likely to execute all five capabilities of cloud computing:
      1. On demand self-service
      2. Broad network access
      3. Resource pooling
      4. Rapid elasticity
      5. Measured service
    • Productivity Can Drive Improvements in Work/Life Balance and Reduction in Burnout, and Organization Can Make Smart Investment to Support It: To support productivity, organizations can foster a culture of psychological safety and make smart investments in tooling, information search and reducing technical debts through flexible, extensible and viewable systems.
    • There Is a Right Way to Handle the Change Approval Process, and It Leads to Improvements in Speed and Stability and Reductions in Burnouts: Heavy weight change approval processes such as CAB, negatively impact speed and stability. In contrast, having a clearly understood process for changes drives speed and stability, as well as reduction in burnouts. It is nice to see ITIL 4 has adopted this best practice, and the CAB there is no longer mandatory.

    The study was done on varied departments and industries.

    Software Delivery and Operational (SDO) Performance

    Over the last 6 years, a set of key metrics has been developed to measure SDO performance which provides a high-level system view of software delivery and performance and predict an organization’s ability to achieve goals. Last year, one operational metrics were added and found this measure helps delivering superior outcomes.

    This puts the focus and proves operational performance adds to the overall performance of the organization and helps in achieving the outcomes. Availability becomes an important component to recognize the outcome and co-create value. In ITIL 4 we also find this being brought out in the Service Value System. Importance of operations in adopting DevOps is very important.

    The SDO for Elite Performers as per the Accelerate State of DevOps Report 2019

    This year the report has shown two research models to help drive improvement in both improvement in performance and productivity.

    Use of cloud has been one important factor. How it is used leads to the benefit generated out of it.

    The Five Essential Characteristics of Cloud-Related Result:

    Only 29% of the respondents who said they use cloud infrastructure agreed or strongly agreed they met all the five essential characteristics of cloud computing as defined by NIST.

    The report talks about technical practices.

    Executing for maximum effect:

    • Concurrent efforts at team and organization levels
    • Continuous delivery using automation
    • Team-level technical capabilities
    • Organization-level technical capabilities

    Factors within are:

    Importance of architecture still remains important at organization level. Relook at the change process, it’s an important point highlighted in this year’s report. Use of cloud services is giving more improvement. DR testing is as critical as any other testing.

    Change management needs to be tuned to facilitate continuous delivery. It should not be heavyweight and on the other hand do not compromise risk. Points to be looked at in change management are:

    1. Implementing segregation of duty
    • Use Version Control
    • Use Infrastructure as Code
    1. Role of CAB becomes more strategic
    • Helping teams to implement capabilities that improve performance by providing guidance and resource
    • Weigh important business decisions that require a trade-off or sign-off at higher levels of the business
    • Shifting detailed code review to practitioners and automation and providing more time for strategic decisions to leadership
    • Transition from gatekeeper to process architect and information beacon
    1. Clear change process with proper delegation of authority

    Psychological safety helps in improvement of performance.

    This report talks about improving productivity. How do we improve productivity? The following factors can help:

    • Useful and easy-to-use tools
    • Technical professionals and tools
    • Internal and external search
    • Technical debts

    Finally, the approaches to improve–heatmap of DevOps transformation strategies by performance profile.

    Learn more about the Accelerate State of DevOps Report 2019 and attend the DevOps India Summit 2019 to hear the global experts.

    — Niladri Choudhuri

  • From DevOps to DevSecOps: Owning Cloud Security

    From DevOps to DevSecOps: Owning Cloud Security

    The cloud is a complex environment, with different groups managing different cloud services and environments. The most basic division is between the cloud provider, who provides the core infrastructure, and the customer, who leverages services or builds their applications using resources leased from the provider. Within larger organizations, different cloud resources, data and applications may be owned by various lines of business, each with their own budget and development resources.

    From a security perspective, this sort of divided ownership of the cloud can create severe challenges in terms of end-to-end visibility, control and compliance. Of course, the cloud is a shared infrastructure, and when it comes to security events, there will likely always be some shared responsibility. But the lack of a centralized security strategy and inconsistent security policies and standards leading to potential grey areas of responsibility can create serious security gaps, and put critical data and other digital resources at risk.

    Cloud Provider vs. Customer Responsibility

    When respondents to a recent survey by IHS Markit were asked, “who is truly responsible when there is a vulnerability or security event in their cloud deployment?” the answer reflected a lack of understanding of the basic division of responsibility. Even in the best-case scenario, where it should be clear as to who is responsible for a security event (such as a vulnerability in the cloud platform), only about half of respondents were able to identify the root cause and responsibility.

    Complicating things further, there is yet another set of tools as to which responsibility should be clear. The list of security vendors available as add-ons through a cloud provider’s marketplace is exhaustive. The cloud providers themselves also offer a range of security solutions (built into their platforms as services). And many of the cloud management platforms, hypervisors, orchestration tools and other bits of cloud infrastructure have various levels of security available.

    However, the truth is adhering to the shared responsibility model, whenever the customer applies any configuration to these tools, they become his responsibility. Nonetheless, divisions of responsibilities are not always clear for different events.

    The best course of action is to identify and apply best practices to any resource and service consumed in the cloud. A root cause analysis (RCA) playbook should identify possible threats and plot actions where possible, and plot strategies for determining the root cause of an event. This includes identifying all key stakeholders, determining which investigation tools are available for threat analysis, establishing processes for orchestrating responses and resources between all parties and digitize and map those details to actual cloud technologies. Further, any response should be automated as much as possible.

    Security Responsibilities Within the Customer Domain

    Once the primary divisions of responsibility for a dealing with a security event is worked out between providers, vendors and customers–such as determining if a security breach is the result of a corrupted service or a misconfigured system–most cloud customers still have to address the divisions of responsibility within their own organization.

    The challenge is, when it comes to cloud security, too many companies don’t have a good handle on who is responsible for what. Part of the challenge is many organizations have taken an ad hoc approach to cloud development. As a result, today’s organizations, with an average of five different cloud environments in place and two or more in development, may be hamstrung by the challenge that each cloud instance might be owned by different lines of business.

    For many reasons, these groups may be reluctant to turn over the security of their individual cloud environment to the central security team. For example, speed is a critical component of DevOps efforts, so any security implementation that interferes with their primary objective of delivering applications is going to be a problem.

    Part of the reason is traditional IT security teams rarely know much about cloud environments, and the security tools they recommend often bottleneck development. However, handing over to the DevOps teams building out cloud applications and environments is problematic as they have little to no expertise when it comes to security.

    For example, one of the biggest challenges of many cloud-based security tools is the time and skill required to properly configure and manage them, let alone adjusting those configurations to eliminate false positives, or to ensure they don’t miss critical events or drop legitimate traffic. All that requires expertise most DevOps teams don’t have.

    Shifting from DevOps to DevSecOps

    Resolving this challenge can be as simple as adding a cybersecurity specialist to each of the various DevOps teams. Once integrated, DevSecOps can help navigate the fine line of the shared responsibility model to ensure both development and security requirements are met. They can negotiate between the customer/vendor/provider responsibility model, provide strategies for ensuring consistent security across and between cloud instances, and balance protection with performance.

    Part of that is achieved through the selection, deployment and management of tools designed to meet the twin challenges of speed and security. For example, many SaaS-based security solutions, such as web application firewalls, are self-scalable, allowing web applications to grow as needed without compromising protection. Choosing the right tool can also ensure it can be easily stitched into an application transaction with minimal effort. Some of them even have the advantage of having already included deployment, maintenance, scaling and fine-tuning functions.

    The proper application of automation plays an equally critical role in this process. Checking configurations and scanning objects for malware are time-consuming activities that can get pushed to the side when other issues arise. Automated cloud security solution selected and designed by the DevSecOps team can provide comprehensive configuration assessments, dynamically secure stored data, automatically scan for public cloud vulnerabilities, identify misconfigurations that put data at risk, scan files stored in the cloud and prevent the unauthorized downloading of sensitive information.

    Building a DevSecOps Team Requires Executive Support

    DevOps has become part of C-suite and board-level discussions, attesting to the growing critical value of web applications and digital transformation as part of the broader business strategy. However, if the frequency of breaches and the growing concerns of CISOs are any indication, executives aggressively pushing for cloud solutions often have a mistaken understanding of the nature of the security risks that cloud adoption and careless DevOps programs can introduce into their organization.

    Therefore, it is critical executive leadership is educated on the risks associated with expanding an organization’s digital footprint, and as a result, its new potential attack surface. Expanding DevOps to DevSecOps enables the integration of security into new cloud environments on day one. It can also ensure RCA cloud security playbooks are developed and followed, and help introduce and manage security solutions designed to protect critical digital resources without introducing unreasonable risks. Finally, it can provide a much-needed buffer against fines and penalties associated with regulatory requirement violations should their cloud deployment become a conduit for a severe system breach.

    — Lior Cohen

  • GitHub Enterprise Server Update Extends Enterprise Push

    GitHub Enterprise Server Update Extends Enterprise Push

    GitHub has expanded its push into the enterprise with the 2.18 release of GitHub Enterprise Server, which includes a bevy of updates intended to make administrators’ lives easier.

    Mario Rodriguez, head of product for GitHub Enterprise, said the enterprise edition of GitHub is seeing triple-digit growth as large companies more aggressively embrace DevOps tools and processes that already have proven themselves to be effective when building and deploying large-scale web applications.

    To make it easier for administrators of those environments accessing either an on-premises edition or software-as-a-service (SaaS) version of GitHub Enterprise Server, the latest edition enables administrators to employ an issue-transfer capability to delete an issue in one repository and move it along with its associated metadata to a new repository in a single step.

    Other additions to GitHub Enterprise Server include an assignable commenters capability that allows maintainers with write access to assign an issue or pull request to anyone who has commented on it, regardless of their organization membership or repository permissions.

    If an issue or pull request belongs to a milestone, the name of the milestone now also will display on the project card and in the project card details sidebar. An issue or pull request from milestones can be added or removed using the details sidebar, and project cards can be organized by milestones using the search bar.

    GitHub is now also providing administrators with repository templates that allow them to own or have write access to a template repository. That capability allows anyone with access to the repository to generate a new repository with the same directory structure and files. If an administrator creates a user-owned repository, they also now will be able to watch it for updates automatically. However, forks no longer will be automatically watched, even for those granted push access.

    Rodriguez said GitHub is also starting to extend the scope of its platform to address best DevSecOps processes. Expanded dependency graph support adds security alerts to projects that rely on “yarn.lock” files. Those alerts help developers stay on top of vulnerabilities that impact their dependencies, which Rodriguez said is critical because security issues these days can wipe out all the productivity gains previously made in a DevOps process.

    An expanded audit logs capability allows administrators to see changes to settings for organizations and repositories, in addition to seeing when PAT and SSH are being used to access information and when there’s been a change in organization and repository membership.

    In general, Rodriguez said GitHub Enterprise Server, which gets updated once a quarter, provides 95% of the GitHub repository capabilities accessed by more than 40 million developers to share code and download open source software. Enterprise IT organizations clearly want to replicate the GitHub developer experience behind a firewall to ensure no one is inadvertently accessing proprietary software that provides them with a competitive advantage. The challenge, of course, is not so much setting up the repository as much as it is defining all the DevOps processes that need to revolve around it.

    — Mike Vizard

  • Cloud Complexities Hinder Effective Enterprise Data Management

    Cloud Complexities Hinder Effective Enterprise Data Management

    Enterprises everywhere are rushing to embrace automation and AI to improve existing business processes and innovate. But they’re learning that if APIs are the glue they use to build their systems, data is the fuel that drives these efforts forward.

    According to data management provider Actian, however, enterprises are experiencing significant challenges when it comes to obtaining fresh data, effectively wrestling with the complexity of data and getting to the point of driving innovative initiatives.

    The Actian Datacast 2019: Hybrid Data Trends Snapshot surveyed 300 IT decision-makers and professionals in businesses with 259 or more employees. Ninety-four percent of respondents said it’s important for them to have a system in place that makes certain users keep getting data, yet 58% of respondents admitted it is only somewhat likely, or not likely, they are actually getting fresh data to their users. So much for smart, real-time decision making.

    That could explain why just 34% of respondents who said they are using data for decision-making are using it to push innovation forward, instead, the majority are using it for traditional business operations and reporting.

    “Business as usual operations keep enterprises running from day-to-day, but limiting data to operational reporting tasks means missing a key piece of the data puzzle–new insights that lead to awareness of products, markets, consumer trends, strategy and more,” said Actian in a statement.

    The costs of data management are also taking their toll. The survey found 84% of enterprises would deploy more data if they could do so more cost-effectively.

    Those costs could be driven, in part, by the complexity created with data being stored within application siloes and across multiple cloud environments.

    “Over 50% of these businesses say data complexity issues, due to siloed applications, are a top barrier to entry for accessing data and gaining effective real-time insights,” said Actian.

    Actian concludes, unsurprisingly, that enterprises must get to the point that they can exploit their data in real-time and better manage data across cloud platforms.

    “Slower decision-making is only one consequence of having to wait for data to become available for analysis,” said Jack Mardack, vice president at Actian. “There are exciting new analytics use cases, like customer-360 and hyper-personalized real-time offers, that simply don’t work with stale data”

    “Enterprises are struggling to keep up with the growing demand for business-as-usual reporting that it leaves no viable bandwidth for generating new insights. This requires that enterprises provide not only the necessary data management infrastructure to make data from all their sources available to analytics discovery, but also that they invest in and empower data teams that can pursue new insights from new data spaces,” stated Actian.

    — George V. Hulme

  • Accelerating Software Delivery Doesn’t Mean Compromising on Quality

    Accelerating Software Delivery Doesn’t Mean Compromising on Quality

    Whenever we’re pushed for time and forced to rush, we know we’re entering an unspoken deal with the devil. Whether it’s cooking the family a last-minute dinner, ironing a shirt before a wedding or writing against a strict deadline, we’re acutely aware it’s highly unlikely the output will be a fair reflection of our best efforts.

    Seldom does “hurry” equal “success”; mistakes will inevitably happen–that’s all part of the age-old mission to maintain the equilibrium between speed and quality. But what happens if the work, such as enterprise software delivery, is perpetual by its very nature and the only goal is to deliver a continuously improving experience to discerning end-users? How can such a compromise exist in such a fast-paced environment?

    It’s a challenge, that’s for sure–especially for software delivery teams at traditional large organizations. Unlike tech giants and other digital natives that were conceived around the very idea of a business model driven by incredible digital experiences, mature enterprises–your banks, insurance companies, retailers and so on–are learning on the fly, and serve and support vast product portfolios from across the business.

    That doesn’t mean they can’t compete–far from it. They just need to rethink how product-related work flows across their value streams to better empower their talented test and quality assurance teams who, with the right information available in real-time, are able to ensure customers are regularly receiving high-quality products in a timely manner.

    Trust in Your Test and QA Teams

    In most instances, enterprises can dramatically enhance the quality and speed of their software delivery without heavily investing even more resources into their test/QA organizations. Their testing and QA teams can create order out of chaos, and are in a unique position to understand and assess the health of the product’s development and delivery to answer questions such as:

    • Are all the requirements fulfilled?
    • Are their implementations defect free?
    • Are the known defects tolerable if they’re deployed into production?

     In many organizations, it’s up to the testing team to declare whether a product is ready, or if they must stop-ship. To effectively handle this responsibility, the testing team must have access to the right information at all times. It was once hoped that test management tools could be the single source of truth (at least when it came to managing defects).

    Such a dream turned out to be dead in the water because all other specialists across the software delivery value stream use their own best-of-breed tools to create and manage their work (represented as artifacts such as requirements, features, epics, stories and so on).

    And because these specialized tools do not naturally integrate and automatically flow information, testers (and QA teams) end up on time-consuming scavenger hunts to find the most current information—which dramatically undermines their ability to quickly and accurately determine the health of any product. It’s a vicious cycle.

    There’s More to Product Development than Defects

    Each specialist in the software delivery process has their own point of view on the shared artifacts. Testers need access to requirements to create tests. But the business analysts, product managers and product owners that write the requirements do not use the test management system, preferring their own purpose-built tools for developing requirements and user stories.

    Similarly, project managers use PPM tools, while IT support uses ITSM tools. And so on for each discipline across the key stages of the value stream from ideate and create to release and operation. While each of these disciplines are linked together by their desire to create and deploy the highest quality product possible, their ability to work together is hampered by the fact that their tools do not work together. And when these teams don’t work well together, it’s end users and the business that suffer.

    Knowledge-Sharing Across the Value Stream

    Getting these tools to work together can be accomplished in several ways:

    1. A Manual Process: such as logging into another team’s tool, emailing a colleague for the information or consulting a spreadsheet.
    2. Batch Data Transfer: exporting defects and requirements from one system and bulk importing them into another. Of course, artifacts loaded this way have only static information; they are not continually updated with the most current status.
    3. Automated Integration Across Toolchain: If these systems are integrated and the information in them is continuously synchronized, everyone has access to the most up-to-date information in real-time. Practitioners can even collaborate with one another via this information, all while remaining in their tool of choice.

    The third way is the only way to obtain the end-to-end visibility and traceability that is required for the vital one source of truth into the health of your software products. Only then can you access a single collection of consolidated data to analyze and optimize the quality of the software delivery process.

    By connecting these tools and synchronizing all product-critical data across the value stream, the QA process is strengthened and streamlined. For instance, when a tester creates a defect in a QA tool, it’s automatically updated in the PPM tool and the Agile Planning tool. The same is true for user stories, requirements and test plans—a free-flowing, always up-to-date stream of information that makes or breaks the speed and quality of a product’s development.

    A Single Well-Oiled Machine for Quality Control 

    True value stream integration doesn’t just enhance alignment between testers, developers and business analysts, it empowers all specialists involved in planning, building and delivering a software product. Business analysts, product managers, product owners, developers, architects, testers, system administrators, project managers, scrum masters and help desk professionals are operating within a single unified system by sharing and collaborating on the same information.

    With the test management system synchronized to other systems, the testing team gains a centralized view on a product’s health.

    Testers

    • Have access to product information in their test management system.
    • Have access to the most up-to-date information, eliminating costly rework due to inaccurate information.
    • Can collaborate on artifacts by updating and augmenting them in their tool, knowing their updates will be automatically reflected in all other systems syncing that artifact.
    • Can better leverage the features in their tools when those features rely on artifacts created by other teams—such as automatically creating tests from requirements.

    Testing Teams and Managers

    •  Gain better visibility into their products by receiving reports that pull data from multiple systems, rather than having a siloed view of their product data.
    • Get more collaborative, efficient teams. Practitioners no longer use ineffective email threads to communicate, they collaborate on common artifacts in their tool of choice.
    • Achieve a strong return on investment—each practitioner can save 20 minutes each day by automating manual processes, which can add up to millions a year in productivity savings and avoiding wasting precious human intellect.

    Moreover, with a fully integrated value stream, the QA and testing teams can institute standardized processes and tools, without being undermined by the tools and processes of other organizations. The centralized test management system can become the single pane of glass into a product’s development, providing test managers and executives with the insight into the chaos, allowing them more control and better decision-making. And, crucially, ensuring products are not only delivered on time but that they’re also continuously delivering delightful end-user experiences. No more deals with the devil, just the benefits of knowing the devil in the details.

    — Jeff McCollum

  • Challenges of Big Infrastructure Monitoring

    Challenges of Big Infrastructure Monitoring

    The growing complexity of today’s IT reality is indisputable: Software is becoming more complex, and the infrastructure it runs on is equally so. This complexity comes, in part, from adding layers upon layers to the infrastructure we manage—from bare metal servers to VMs to containers to function-based computing—and from how quickly we cycle in new technologies and how slowly old technologies are cycled out.

    Ephemeral and multi-generational infrastructure (often side by side) are the new normal. Old and new technologies co-mingle, and while new technologies may be shiny and exciting, they quickly become ordinary as the next new thing comes along.

    Here’s a sobering stat: In Q4 2017, IBM’s mainframe business grew by 32%. So even while cloud, AI and security initiatives are on the rise at IBM, mainframes sustain a bulk of their business. Old tech perseveres, and we have to find new solutions to bridge the gap between old and new.

    The hodgepodge of technologies—and the challenges they bring—are compounded further when we’re talking about big infrastructure.

    What Qualifies as Big Infrastructure?

    For the purpose of this discussion, big infra is an organization that has been around for at least 10 years. That means they’ve had success, they’ve grown and their infrastructure has grown along with them, and the technologies at play have expanded as well.

    Your organization’s exact age, size and revenue aren’t particularly material; what matters is you’ve had enough time to accumulate resources, people, practices and technologies to create what I think of as a trail mix problem—basically enough legacy infrastructure to start creating problems.

    The Challenges of Big Infrastructure

    The challenges of big infra stem from the technology itself and from the way people use that technology. As an organization grows, its teams often become larger and more siloed, and departmental goals become disconnected. Even when working toward the same goal, departments diverge in their methods for reaching that goal.

    Now, add tools and technology to the mix: Over time, teams choose tools they prefer, or build their own, which works out okay for a while until they have to collaborate with another department that uses different tools and a completely different set of practices.

    Now, add in different work styles and personality types and company politics, and everything gets tangled up fairly quickly. Trying to keep track of who’s using what, how they’re using it and what’s working (or not) is no easy task.

    One customer, for example, had one team using Chef and was all in, and were trying to get buy in from other teams as well, but there was another team that was early days Docker, hand rolling Docker hosts, and didn’t want to adopt Chef. Then there was a different team who was early days Kubernetes and didn’t want to use Docker, and yet another team using Puppet. Once they had to work with each other, things broke down—someone has to win out. You almost have to maintain a logical map of what team uses what tools as a precursor to prepare for a meeting with these business units. Not only is there no consistency in how applications are deployed, there’s also no consistency in how these different technologies are monitored.

    For example, in the past, Netflix stated monitoring accounts for 30% of their infrastructure cost, and recently mentioned they’re planning to spend $1.9 billion on infrastructure in 2019. Think about the scale and complexity involved—they’re storing and analyzing the logs, behavior and metrics for systems supporting some 125 million subscribers.

    Connecting Disparate Data Across Big Infra

    So, how can you streamline and connect all of the data spread across fragmented tools and systems?

    One solution is a data lake, a centralized repository that lets you funnel data into one place, so you have a holistic view of your infrastructure. In reality, big infra ends up spawning multiple data lakes that become siloed from one another.

    You also likely have a square peg/round hole situation—data in one format needs to be transformed to another format before it can get into one of the data lakes.

    This is where a monitoring tool becomes essential. Look for a monitoring tool designed to collect data in disparate data types and transform it into any number of formats, so you can set up an automated workflow to get data flowing from legacy systems (e.g., mainframes) alongside modern infrastructure like containers.

    An example of this idea is with Sensu, and our support of the Nagios plugin spec and Prometheus exporter metrics (amongst many other metric formats). By using a monitoring tool that works with various metric formats, companies can plug the service checks they’re already using plus data from multi-cloud infrastructure into their monitoring setup and get valuable information about their systems.

    A visualization tool such as Grafana is the icing on the cake. With Grafana, you can hook up data wherever it’s stored (e.g., Graphite, InfluxDB, Prometheus, etc.), visualize the data and display it on shareable graphs and dashboards. Using a monitoring tool as a data source for Grafana means companies can display monitoring incidents and inventory—and even mix that data with different data sources on the same Grafana dashboard. Combining a holistic monitoring solution with Grafana’s visualization capabilities gives a data-driven view of your infrastructure that you need—even if it remains a multi-generational, multi-siloed, multi-data-laked, 2,000-different-format-typed trail mix of big infra.

    — Sean Porter

  • Agile QA at Scale: Software Testing Strategies for Enterprise Teams

    Agile QA at Scale: Software Testing Strategies for Enterprise Teams

    Change is hard; it’s a universal truth not many people would disagree with. When you’re forcing (or strongly encouraging) change in a group of people, it’s especially hard. Even if that group is small, there’s bound to be miscommunication, disagreement and other friction points that make the change hard to implement.

    Now, multiply those difficulties by a factor of five. Now 10, 20 and so forth. This is the challenge of growing and expanding an organization. While you need to bring more people on board to fuel your growth, build a better product or improve the leadership, you also need to manage the complexity introduced by a skyrocketing headcount.

    More and more teams are accepting the challenge of switching to agile QA because it saves time, improves team collaboration and results in a superior product in a consistent, repeatable way. But most of the how-to articles and strategy guides out there only focus on small teams. What about the enterprise team ready to make the switch?

    Let’s look at a few software testing strategies for enterprise teams who have—or are in the process of—adopting agile QA in their organization.

    Outline a QA Strategy that Is Scalable

    Scalability starts early. You don’t want to set off running with a strategy that doesn’t fit your current team or is unable to expand with your team as it grows in the future. It’s key to adopt an approach that is flexible in all areas, from the technical aspects (tools and infrastructure) to team hierarchy and how resources are managed. If you don’t have a QA expert in-house, you may consider hiring a QA vendor to mentor you on testing approach, scalability best practices and team structuring.

    Bring on the Right Skill Sets

    Know what your business priorities are, what types of skills and technical expertise those priorities require and hire the right team for the job. With a solid team in place, you can scale up or down as needed. It helps to bring in both generalists (people who can handle a bunch of different domains) and specialists (people with a niche focus or specialty area). As you grow, your rock-star engineers will, too—they’ll move into management roles, continuing to motivate, lead and challenge their direct reports.

    Choose Metrics that Make Sense

    If you don’t measure progress and team output, you won’t know when it’s time to scale—or in which area that scaling is required. Be sure to invest time and care into the process of defining your success metrics, as they’ll have big implications for how your team is structured and how it grows in the future. Also be aware many success metrics focus too narrowly on code coverage and the number of tests executed. These are important numbers, but not as important as the overall quality of the product, which should be your “North Star” metric.

    Automate What You Can

    One of the easiest ways to scale without immediately adding headcount is to invest in automation testing. By automating regression testing and other intensively manual tests, you can free up engineers to work on higher priority projects, save more of the team’s time and cut resourcing costs. Plus, automation can help improve the quality of the product by reducing the number of human errors.

    No experience with automation testing? There are QA vendors out there offering expert automation teams who can learn your product, culture and processes in no time flat.

    Monitor and Maintain

    If you can spare it, devoting headcount specifically to monitoring the errors and alerts that arise during testing can help expedite the rest of the team’s work. When engineers don’t have to go back and work on every little fix or patch, they can focus on driving through their higher priority workload. Nitpicking the small stuff just isn’t feasible at the enterprise level, especially if you’re employing gifted and talented engineers.

    Here’s What You Need to Know

    1. How to Scale Your QA Team Effectively
      • Involve QA as early as possible
      • Design effective test strategies
      • Focus less on narrow metrics; more on product quality
      • Switch to smaller, more frequent releases
      • Test based on real-world user experiences
    2. Attributes of Agile QA at Scale
      • Automation is a part of the process
      • Engineers take ownership of their modules
      • Quality is top priority (“North Star”) of the whole team
    3. Strategies for Scaling QA
      • Understanding business priorities
      • Building a skilled team that reflects those priorities
      • Creating a roadmap for troubleshooting quality issues
      • Implementing technical and managerial strategies that scale

    Wrapping Up

    Everyone knows change is hard, especially when it requires people to work differently, work harder or work together in ways that are new, inefficient (only at first) and potentially uncomfortable.

    Successfully scaling agile QA is one of the best ways to ensure you’re consistently delivering a quality product to market. No wonder why teams small, large and enterprise-level are taking on the challenge—and with these strategies, you can too.

    — Anand Ramakrishnan

  • Weighing the Cost of Improper DevSecOps

    Weighing the Cost of Improper DevSecOps

    Simply put, data breaches are terrible news for companies. And, the costs associated with such attacks continue to escalate. A recent IBM-sponsored report found an average price of 3.92 million per breach.

    Not adapting security automation and vulnerability scanning into development pipelines could have a drastic effect not only on cost but workload efficiency and team morale. With these sorts of headaches, it’s vital to consider the repercussions for not adopting secure (and lean) armaments and auditing procedures.

    Can organizations adopt DevSecOps beyond the buzzword it has become? In this article, we’ll uncover some startling evidence on the state of Kubernetes vulnerabilities and discuss what companies can do to rectify.

    Trends in Kubernetes and Istio Vulnerabilities

    It’s astounding how much in DevOps is left exploitable. Secrets are routinely unknowingly left in the open due to human error. These are the most common gateways for attackers. Plugging these gaps is the first, albeit obvious, step.

    There’s proof of an epidemic, too. A recent study conducted by Alcide ran a live production analysis of over 500 cloud-native environments across the globe. According to their findings, “89% of deployment scans show companies are not using Kubernetes’ secrets resources, with secrets wired in the open.”

    This means the majority of Kubernetes environments share susceptible to data breaches in some form. Among the 5,000 scans run, the Alcide report found secrets are commonly left open in a few areas, including:

    • Workloads that had to connect to cloud resources that exist outside the cluster
    • AWS access keys to connect to those resources
    • Tokens for workloads that require API access to external databases
    • Credentials for S3 buckets

    Gaps in Best Practices for Kubernetes and Istio

    Kubernetes brings many nuances to the table. It’s a sophisticated system, with a wide range of use cases and cloud-native experiences. These require extensive auditing and compliance checks to safeguard deployment.

    According to Gadi Naor, CTO and co-founder of Alcide, Kubernetes requires a good level of expertise. However, “there is a knowledge gap that DevOps doesn’t have,“ said Naor.

    Much of this gap lies in Kubernetes secrets handling and network policies. On top of Kubernetes, service meshes are a next evolutionary leap. Istio, an architecture layer for creating network policies is ten-fold more complex, according to Naor.

    On network policy creation, “75% of the scanned deployments use workloads, which mount high vulnerability host file systems; while none of the surveyed environments show segmentation implementation using Kubernetes’ network policies,” stated the Alcide study.

    These are compelling platforms, but best practices must meet the growing number of potential attack vectors.

    Why You Should Approach CI and CD Security Separately

    Many pundits now link continuous integration (CI) and continuous deployment (CD) together as if they’re synonymous. However, they are separate concepts. Securing them, in the same manner, could decrease overall efficiency.

    Scanning all resources created in the CI phase against security auditing could waste time and resources. For example, if you’re standing images for CVEs and only deploy 20% of those images, that is a lot of wasted exercises.

    Rather than scanning all environments in the CI phase, Naor believes you should focus on scanning 100% of images for CD. Therefore, security tooling is leaner, and anything deployed will be assuredly safe.

    Takeaways on the Cost of Poor DevSecOps

    Infrastructure components such as Kubernetes and Istio are powerful allies to the DevOps minded teams. These environments do come with their nuanced security concerns, which must be addressed. Otherwise, the data breach price tag is high.

    To intercept system frailties before the black hats do, you must consider the following:

    • Scan for the Obvious: Utilize automation to discover secrets, keys, tokens, left in public repositories.
    • Scan in a Lean Way: Maintain a lean security environment that retains your engineer’s attention.
    • CI vs CD: Recognize the security nuances between integration versus deployment.
    • Add Auditing Tooling Strategically: Adopt compliance and auditing tooling that doesn’t create friction with existing developer workflows.

    As Naor said, “Don’t ignore the obvious stuff.”

    — Bill Doerrfeld

  • Radical Transparency in DevOps and ITSM Helps Achieve Excellent Customer Experience

    Radical Transparency in DevOps and ITSM Helps Achieve Excellent Customer Experience

    Here’s what I’ve noticed after spending several years in service management: DevOps is coming up more in the conversations I’m having with service desk leaders. Usually, these conversations start with one of the similar questions: DevOps is going to be the death of ITSM, correct? Alternatively, some consider DevOps as a challenge, and ITSM doesn’t seem so relevant because you don’t have in-house development going on.

    I always answer these questions in the same way: ITSM is relevant, and DevOps is not a threat to it. There’s more to it, however. ITSM professionals can still achieve their most relevant goal–excellent customer service.

    If you allow me a moment, I’d like to offer some background as to why excellent customer service is essential for every ITSM professional, then walk through some of the ideologies or frameworks used and how they interact. I’ll close this piece by discussing what radical transparency means in practice and how it can help you achieve an excellent customer experience.

    Some Background

    When discussing DevOps, we usually see three parts: Development, operations and the actual business. The business demands different things from both development and operations. Development teams are asked to push out code as fast as they can, and operations teams must keep the existing services available and make sure everything is stable. Development is trying to fulfill as many business needs as fast as they can, while the operations team is accountable if things go wrong.

    These might be extremes, but these are the traditional silos that have their expertise, priorities and goals, but there is no holistic overview. These internal organizations also use their way of working, based on a framework or ideology: DevOps tends to use agile, the business uses lean and ops focuses on ITIL. These ideologies can clash and compete at times.

    Because so much is happening with DevOps, we see significant developments. The speed of development and deployment is increasing, leading to additional continuous delivery advancements and technology implementations. The collaboration and breaking through walls between the development team and the operations team for deployment has come a long way within many organizations.

    Meanwhile, ITSM professionals face more and more challenges. Apart from the existing incident handling, they encounter incidents about newer, faster releases. Also, how are they supposed to deal with knowledge management, an already complicated process in itself? The increased speed of development makes it look as if they are always running behind. Getting out of that reactive approach is not getting easier.

    Customer experience and satisfaction is the number one priority in ITSM, creating quite the challenge. In ITSM, all input comes in: If things do not work at all or break down; if things do not work as expected; and if users need help to get things working. Good feedback also usually comes in via the service desk. When examining customer experience, there is a thin line in how fast you can go: If functionality is delivered too quickly, customers will not be able to use it. When delivered too slowly, customers will be unhappy about it as well.

    You can argue that a new silo comes into the picture–namely support. The support team keeps things up and running. Likewise, there are communication links between the customers and the other silos where they can filter bugs to development, info on any outages to operations and feedback on how the customer perceives the value of the functionalities to business.

    What if you don’t have in-house development? Well then, DevOps and the link with ITSM can be even more challenging, meaning you only have part of your DevOps and some of your support outsourced. For one application, perhaps you are relying on one or multiple vendors, but it might be the case that your vendor is dependent on another vendor.

    In other words, the process looks more like a plate of spaghetti than a smoothly laid smooth IT process. There are significant benefits to outsourcing, which is why we keep using outside vendors for some of our operations. In these cases, consider the shift of responsibility, the extra expertise, stability and continuity you gain. However, there are also downsides, such as loss of transparency.

    A Shift in Responsibility

    Shifting responsibility does not mean the same as a shift of accountability. Your ITSM team will still have to take ownership because even if you are not controlling everything. You have to be in control. In the end, you are still responsible for your customer experience and happiness. Customers do not want to wait two years for new features because the update of the vendor cannot take place for whatever reason. Also, it would be best if you had insight into what is going to happen for your users, and you need extra insight regarding how everything is working.

    Silos make this a challenge without outsourcing and are even more so the case when outsourcing these responsibilities. Before we move forward with solutions, let’s first take a step back and look at the frameworks and ideologies the different silos are using. As discussed, first, we have ITIL, typically used in operations.

    “ITIL is the most widely accepted approach to IT service management in the world. ITIL can help individuals and organizations use IT to realize business change, transformation, and growth,” stated Axelos on their website. It’s about aligning services to the needs of the business and supporting its processes.

    Next, we have lean, which is about maximizing customer value and minimizing waste. Lean is where value stream mapping comes in. You identify what works then look at how you can improve flow. Finally, agile, the cultural aspect of DevOps.

    At first sight, each of these processes seems contradictory, but actually, this is not the case. The key is using each of these principles correctly. For example, consider CAB, which comes out of ITIL. In many cases, the A in the acronym stands for “authorization” instead of “advisory.” Knowing that might make things much better and can open up the possibilities of automation, which is crucial to improving things.

    Likewise, some people tend to say that agile can never work in an ITSM context because it is against processes, but that’s also not true. Agile speaks to individuals and interactions over processes and tools, not without appropriate processes and tools. However, if you take a step back, you see they have more in common than you might think and some changes coming to ITIL4, for example, already address how to achieve a better correlation between the three.

    Let’s look at all of them together. Usually depicted as a horizontal value stream, but looking at it as a continuous loop, makes things a lot clearer for me.

    When you start in the business with the idea, take it through the design, code, build and test phase, then move over to the release and deploy and then we get into ITSM territory with the support and monitor stages. Where you see the blue line is where DevOps focuses mostly, so this is also where you witness the creation of a new silo: support (where ideologies or frameworks come in the picture).

    Radical Transparency

    So how do we break through these silos? Radical transparency. First of all, silos have to share their common purpose, which in the end is excellent customer experience. You can change it however you want; but in essence, that is what the business, development, operations and support all strive.

    To accomplish this, we first must know what impacts the customer experience.

    Let’s break it down: There are visible and invisible effects for the customer that can either be positive or negative. Visible results on the positive side will be features; on the negative side, defects. On the invisible side, positive things include architecture and process improvements. On the negative side, technical debt.

    For excellent customer experiences, we need to focus on the visible effects. Invisible effects must be worked on, too; if completely neglected, these will become visible at some point. The most prominent and the first focus should be on the visible effect. To do that, we need insight. If we all have the same set of information, often we all come to the same result. Getting to that same set of information requires radical transparency and also sharing information and insights.

    Shared Workload Tooling

    First disclaimer: When I talk about sharing tooling that does not necessarily mean using one tool. I will specify. I mentioned before that ITSM professionals have to know what is going to happen that might affect the customer experience. They have to be aware of which new versions are coming out and which customers are going to be or are on that version. For that reason, you need to share relevant information from your release orchestration tool to your ITSM tool. For instance, in our service management tool, we can see who is on which version via an automatic integration.

    That is a flow from DevOps through ITSM. The return flow also is essential. Any feedback that comes through your ITSM process regarding bugs has to flow back as well. So, for that, you need an integration between your ITSM and bug tracking tools. I also have seen some examples within the ITSM process where a Kanban board is used to improve collaboration between development, operations and support.

    Just making visible those tasks that impact customer experience and being able to follow up on the progress made by all stakeholders is so valuable. It also gives the support team the necessary insight to be able to communicate with the customer.

    If you are working with one or multiple vendors and dependencies there, you will want to include vendors in this as well. Moreover, that might mean you even have to tie in your and their ITSM tools.

    Another thing you want to share is customer satisfaction insights. First, measure satisfaction throughout the entire chain. That includes employee satisfaction, as well. We have seen the proof that employee satisfaction has a direct impact on customer satisfaction, so make sure you have all those data available and insightful.

    Knowledge Sharing

    Next up is sharing knowledge, which requires proper documentation. Think of federating knowledge into a knowledge base to make relevant information available to the right people. Make sure you have a good understanding of each other’s processes, and the impact they have on one another is essential. As an example, we had some challenges in collaboration with one of our hosting partners. When we sat down to discuss this with them, we found out some choices we had made in the deployment and support process, were not as reliable because we had assumed things about how the hardware had been set up and those assumptions were not correct.

    Excellent documentation and communication are key. If you have things outsourced, you want to make sure your vendor is the right partner, and you have a common understanding.

    Another important thing that should be shared is dashboards, where you can have a total overview of all parties involved and the impact they might have. You especially want to have visible what the effect is on the customer experience. It is valuable to have insight into all things that need your attention, but for your shared dashboard, focus on those things that are not going well and that have an impact on the customer experience.

    When you work with one or more suppliers, you want to make sure they are also using these dashboards and actively monitoring these. For you to be sure they use that, add that in your vendor selection criteria. The only way this works is if there is that radical transparency, where all parties involved have access to your vendors, development, operations, support, business and customers.

    Takeaways 

    Hopefully, because of this article, you’ve taken some additional perspective about where DevOps and ITSM come together, why the silos have to be broken down and how working with one or multiple vendors can be especially challenging. Also, hopefully, I’ve provided a bit of background on the various best practices, frameworks and ideologies most used and how they share commonalities. Radical transparency means breaking through the silos and how this can improve your customer experience.

    Finally, to sum it up:

    1. Make sure all parties have a common purpose and are doing everything to achieve that.
    2. Share relevant information. All words here are significant: share to achieve radical transparency, applicable because you don’t want to flood people with things that are not of use to them, and sharing too much may distract them too much. Provide just the relevant information so everyone can focus on the common purpose.
    3. If you do all that, you will produce better value for your customers, all for excellent customer experience.

    — Jeroen Boks

  • Out of Date: Cloud Anti-Trends of 2019

    Out of Date: Cloud Anti-Trends of 2019

    Let’s talk about 2019 cloud tech anti-trends. A lot of those anti-trends are nothing but cloud myths, which are still widely spread and talked about.

    In this article, we’ll take a look at some anti-trends and try to understand where’s the myth part in them.

    Private Cloud Is Not Popular Anymore

    According to the IDC research organization, by 2021, companies from around the world will spend more than half a billion dollars on cloud tech. This is two times more than in 2016. At the same time, there is a clear tendency: The public cloud workloads are growing, while the situation is exactly opposite for the private cloud. Cisco said, after three years 73% of the company’s applications will move to the public cloud, while just a year ago this figure was around 50%.

    Such a shift is partly caused by cloud providers, who offer users of public clouds more and more features that simplify the development of services. For example, tools that allow creating machine learning models, manage software deployment processes and analyze logs.

    That’s why many felt the active development of a public cloud meant the final decline of the private cloud niche. During his speech at a conference last year, Anthony Lai, senior vice president of the cloud-based division of NetApp said private clouds would “die out.”

    I wouldn’t expect the private cloud to disappear completely. The statement that it will cease to exist is only partly true. The private cloud may disappear as an independent unit, but it will remain part of the hybrid IT infrastructure.

    As noted in Gartner, 72% of companies in the world use a hybrid strategy. And it is expected this figure will grow. According to analysts from Market Research, the market for cloud object storage systems will reach six billion dollars by 2023. This will definitely spur further development of the hybrid IT infrastructure.

    One reason to use private clouds is predictable business processes.

    Yes, the public cloud offers flexible scaling. The company does not have to urgently buy new hardware if the load on the services has increased dramatically (for example, the application became a success and tons of new users came in). In this case, you can simply rent additional computing power. But if the load can be predicted (say, it is static) a private cloud is often enough.

    Also, a private cloud is often used to create object storage. Sure, object storage services are also available in the public cloud and a number of companies use them. Just a couple of years ago, Netflix migrated its entire infrastructure to the public cloud.

    However, not all companies need to build a distributed infrastructure as Netflix did. There are cases when companies have object storage in a private cloud because it already has an appropriate low-cost automated infrastructure. CERN is one of them. All the data generated by the Large Hadron Collider is stored in a private cloud.

    The IT-Infrastructure Market Is Oversaturated with Specialists

    Today, companies are actively abandoning their own infrastructure in favor of cloud solutions. That is why the need for in-house specialists engaged in its maintenance, gradually disappears. The IaaS provider takes on many issues related to the management and administration of the infrastructure, and the specialists need only ensure that everything is set up correctly.

    That is also why companies often abandon their own IT departments. At first glance, it seems this will gradually lead to instability in the labor market. Experts in the field of IT infrastructure, system administrators and data center operators will have to compete fiercely for the remaining jobs. But will they really have to?

    This does not mean such specialists have become completely unnecessary. In reality, in-house specialists simply change the focus of their activities. Most of the IT infrastructure administration tasks are now on IaaS and IaaS providers are now the ones who need and are actively fighting for such specialists.

    Even when you do use third-party solution, that is what happens sometimes:

    When customers start using intermix.io for the first time, they can see the set-up and configuration of their Amazon Redshift cluster in the context of their queries and workflows.

    At that point, customers experience one common reaction: Knowing what we know now, how would we set up our Redshift cluster had we do it all over again?

    The answer is not that straightforward.

    However, sometimes it happens that migration to the cloud stimulates the company to strengthen and develop the IT department more, attracting new cloud employees. These people study the cloud ecosystem, analyze the trends and evaluate how the transfer of an application to the cloud will affect the company’s work. If the company is already working with the cloud, then cloud specialists are also exploring new features of the current platform and exploring services that will benefit a business in the future.

    Clouds Don’t Hold New Technologies

    Four years ago, IBM listed seven cloud technologies to help businesses. The list includes IaaS/PaaS, private/hybrid cloud, test and application development environments, big data processing and disaster recovery solutions, along with data storage and backup systems.

    These technologies served as a standard technology stack for a long time. It was all developed with the efforts of IaaS providers. Therefore, there was an opinion that the cloud ecosystem was finally formed and no other use cases will be foreseen in the near future.

    New Directions Still Appear: The blockchain technology was born in 2008, gained worldwide popularity 8 years later and came to the cloud about a year ago. Based on the virtual infrastructure, a new direction has started to develop: Blockchain-as-a-Service (BaaS). In this case, the client receives a virtual environment prepared for working with blockchain technology. Users can create their own applications on the blockchain (for example, for processing financial transactions) and run them in a secure virtual environment.

    Quantum Computers: the most recent, fresh cloud trend. The quantum machine uses qubits instead of bits we are used to. Unlike bits, qubits can take the value of zero and one at the same time. Thus, a quantum machine can be in several states at once. The more qubits a quantum computer has, the faster it performs the tasks assigned to it. Imagine how much faster it can brute-force passwords or simulate chemical reactions.

    Quantum computers already have practical applications. IBM has recently opened access to its 20-qubit developments via the IBM Q cloud platform. And this year, Alibaba partnered with the Chinese Academy of Sciences to deploy an 11-qubit quantum device in the cloud. Quantum Computing as-a-service (QCaaS) opened the way for experiments on the infrastructure of providers and proved the cloud industry can adapt to the emergence of new technologies.

    Another Trend Is Fogging or Fog Computing: IT-industry experts are trying to solve the problem of delay in transferring data to the cloud. In the case of fogging, server components are not located in the centralized data center, but on the periphery, close to the end-user devices.

    In such an architecture, a main, large cloud still remains but has additional networking and computing layers: cloud, fog, edge. The Cloud (cloud layer, primary) solves expensive tasks for periphery machines (edge layer, end-user): data analysis, system logging, etc. The fog layer does local calculations in real-time to generate reports and send them to the central hub. This architecture improves the performance of the entire infrastructure by significantly (up to 10 times) reducing the delay in data exchange.

    To Sum It Up on Cloud Myth-Busting

    So, as with everything, rushing into extremes doesn’t help. That’s what we figured out:

    1. Even though public cloud seems to be a mainstream option, private cloud infrastructure still makes sense for certain use cases.
    2. Most of the IT infrastructure administration tasks are now on IaaS and IaaS providers are now the ones who need and are actively fighting for Engineers i.e. market is more than far away from being oversaturated with specialists.
    3. Clouds don’t hold new technologies. Did anyone really believe this one anyway?

    — Dmitrii Borodin