devopsy.kinsta.cloud

Category: DevOps Practice

  • The Barriers to Robotic Process Automation, Part 2

    The Barriers to Robotic Process Automation, Part 2

    Enterprises are rushing to embrace robotic process automation (RPA) but, as we covered in The Barriers to Robotic Process Automation, Part 1, they’re also hitting considerable challenges along the way. While we cited prominent barriers to the effective implementation of RPA in part one, here, we will provide practical ways organizations can overcome these barriers.

    One of the biggest causes of RPA failure is the lack of executive-level sponsorship for such initiatives. When it comes to RPA success, getting management’s attention–and keeping it–is essential. One way to do this is to focus on RPA quick wins and efforts that have a measurable return on investment.

    Next, IT leaders need to understand how significant digital transformation is to AI technology.

    “An efficient digital transformation strategy will eliminate internal IT silos through integration and automation, leading to opportunities for AI projects,” said Marcel Shaw, AI expert and Federal Systems engineer at Ivanti. Shaw pointed to how organizations that interface with customers will see opportunities for chatbots that leverage natural language processing technology, thereby reducing demand on operators while enhancing the user experience.

    “IT leaders will be able to take advantage of machine learning technology that will support and enhance complex automated processes that will fulfill customer requests from beginning to end, without intervention from a human analyst,” said Shaw.

    Such benefits could make RPA an easy sale to executive leadership. Other benefits that can be used to sell RPA to leadership include cost savings, more efficient processes, improved process quality and free staff to focus on higher-value tasks.

    “The only thing holding AI back at this time is a lack of experience with the technology. AI will take many years to develop. Some estimates point to a 75-year maturity lifecycle for AI. AI technology is designed to learn. As it learns, it will correct mistakes and mature, much like a human has to learn,” said Shaw.

    Next up: The high level of investment needed for effective, automated and intelligent robotic automation.

    “There’s a great deal of catching-up to be done before most companies can even consider building out their own advanced developments with AI,” said Salvatore Stolfo founder and CTO at Allure Security. “It may be more cost-effective and faster to partner with a software vendor that has already put the time into dedicated research and development to build an AI-powered solution, and then customize that solution for your particular use case.”

    Simha Sadasiva, CEO at Ushur might agree. “Many intelligent automation solutions on the market today require a heavy IT investment. This can include hiring an outside third-party just to manage the platform they’ve invested in,” said Sadasiva.

    “Those attempting to manage the technology in-house are finding that it requires personnel with sophisticated developer-level knowledge, which is in short supply today. These issues are causing some companies who were initially interested in exploring intelligent automation solutions to abandon their projects altogether,” added Sadasiva.

    Sadasiva said one possible way to solve this is to evaluate potential zero code platforms. “These are SaaS-based intelligent automation solutions born in and live in the cloud,” said Sadasiva.

    Finally, and while perhaps self-evident is often overlooked: know what needs to be achieved with robotic process automation.

    “An organization needs to identify what the problem is that they are trying to solve with AI. This needs to be the most important pain point to them at that point in time. Inherently AI requires data to be able to train the machine to learn,” said Ben Mercer, co-founder at personalized web experience provider Personify XP.

    “The challenge for some organizations, when it comes to accessing data, is that valuable data is stored on legacy systems and difficult to extract. In cases like this, it may make more sense to train the AI on new data, than fighting with legacy systems that may delay or reduce the ROI of the deployment,” said Mercer.

    Effectively managing data, AI and the resulting automation requires staff with strong skills and experience in these areas.

    “Regarding skillset development, most of the skills needed can be learned, but you need to find different types of team members who have a great comprehension of data and database management and secondly a development background,” said Jesse Tutt, CEO at mattress company Gotta Sleep.

    Finally, chances are if an enterprise is planning to do something there’s already someone who has tried and has lessons to share. “Every automation project has likely been done by someone else already. Tips and tricks are freely available from others who have more experience so networking on Linkedin, meetups, and conferences are key,” adviced Tutt.

    — George V. Hulme

  • When Shift Right is Shift Left

    When Shift Right is Shift Left

    In some areas of IT, we utilize the idea of shift right quite a bit. Testing and security are two areas where it comes up often. The idea is that tracking late in the process–usually while the application is in production–leads us to find more insidious issues that might have remained hidden, and gives us a kind of quick reaction force to mitigate issues.

    All true, and it works relatively well. Some shops use this concept to push off necessary testing, but for the most part it allows us to keep moving while assuring the system is essentially correct. But the idea of catching issues late in this iteration might be misguided. It does do just that, but in the world of cyclical iterations, shift right is also shift left for a future iteration. The idea being that we’re catching something today that will be resolved for the next iteration. And presumably for every iteration after that.

    We often put a temporary fix to resolve the in-production issue, then list it as something to be more permanently resolved in a future iteration. Adding to a future iteration’s to-do list is the ultimate shift left. And that matters. We’re catching it late for this iteration, but early for the next one. And as a result, overall quality increases.

    Standardizing this process would be the next step. If it’s an easy fix–a minor change to code or config–it’s shifted right. If it’s a larger issue that will need to be considered in a future iteration, it is considered shifted left for that iteration. It’s an input, not just a fix. And an input that deserves a high priority if it reflects a negative impact upon the running system.

    In the daily grind, this is pretty much irrelevant. We have a problem and say, “there, that’ll keep it going. Put in an issue to make a better fix and move on.” But in the next iteration’s prioritization, it is far more relevant, “we have this massive pile of things to do, what makes this iteration?” Should include a heavy weighting for things that are limping along on a manual fix. Often, the existence of a workaround reduces the weighting, and these things don’t make the cut. But permanent fixes mean you don’t have to hustle around fixing the same old problem each time the app is deployed, and that’s greatly what DevOps is all about.

    There is an interesting dichotomy between shops/groups with a high iteration rate and those with a much slower cadence. The cost of not permanently fixing these issues is too high when you’re deploying frequently, so they tend to be handled this way already. But slower iteration shops/groups often will let these fixes slide in favor of new functionality. The burden this creates is not worth a shiny new bit on the UI. Start slating permanent fixes for production issues as highest priority, don’t add steps to deployment because you failed to resolve the more fundamental issue.

    DevOps requires speedy resolution, but long-term, any type of development effort requires adequate resolution. With iteration after iteration, you are paying a price for putting off fixes, so don’t. Shift them left in the next iteration, and spend that time of each release on something more productive.

    — Don Macvittie

  • Agile and DevOps Success Needs Continuous Testing

    Agile and DevOps Success Needs Continuous Testing

    One of the central ways to progress an agile transformation is to introduce continuous testing. With all such changes, it is cultural as much as technical. The expected outcomes are less broken builds and reduced time from break to fix, as well as a yardstick for measuring the team’s ambition for improvement.

    The Testing Bit and the Continuous Bit: Testing should be part of any agile developer’s skillset; the shift is to apply it to the whole build pipeline. Continuous implies automation, with the build featuring as a living thing whose state is always changing. The infrastructure options have expanded with the advent of cloud computing and virtual containers.

    Let’s say your teams’ developers work on components, independently buildable and testable chunks of the whole app or service. They receive stories and tasks to improve a customer journey and then add functionality to implement these. And when done, they check their work into your team’s code repository.

    Starting with One Developer, One Laptop and One Test

    The initial testing focus for a single developer (or pair) working on code will be the unit test and the coverage test. Whether writing the test first or code first, the unit test pokes the bounds of the new code and the coverage test checks that all code paths are exercised.

    As an example, maybe the developer has implemented the “customer wants to deposit money” story on an accounts component of a banking system. The tests should check what happens if a user deposits zero, negative or fractional money and the coverage confirms the tests have wandered down all possible the code paths of the new feature. In this way, coverage tests also prevent a developer from adding code without introducing the corresponding tests.

    At this point, it is worthwhile making sure tests are run as part of the build so every time the developer builds the code all the unit tests are run. This is a fairly standard thing for most build systems to do. The bottom line is the chance of checking-in breaking code is greatly reduced.

    Now the Bigger Picture

    The new code worked on your developer’s laptop, the tests passed and they checked-in the code. But in a busy enterprise team, other developers are also working on the same component, maybe even on the same file. This is no different from working on a book chapter with several authors. Your developer worked with a stable snapshot of the code. It is quite possible changes were made by others that conflict with the changes your developer just made. So how do we proceed?

    This is the purpose of the central build. The job of the central build is to check out all code for that component, build it and run all associated tests whenever new code is checked in. In today’s world, the likely path would be to create a container on a cloud server, where the component can be built from scratch, and then all the tests run against it. Traditionally, a tool such as Jenkins would watch for git check-ins and kick off a build. The result is the team always knows if the overall build is green (working) or red (broken). A developer must understand the central build is the true source of truth, not what is happening on their laptop. So, the latest check-in leads to a correctly built component. What next?

    There are three general types of testing needed to develop a software system. Tests that make sure new features work correctly and don’t break the component, tests that make sure the components still work with each other and those confirming the customer journeys your component participates in all work as expected. There would be no point in making a deposit function without a customer journey test that proves a customer can deposit cash with the system.

    The second type of testing, usually referred to as integration testing, needs an environment where all current successfully built components can communicate with each other. Today, that may mean some form of container orchestration with the system’s components placed and managed in their containers. Tests of a customer journey may involve emulating the customer’s manipulation of a website or mobile app, depending on how the application or service is accessed. These are the tests stakeholders are most interested in as they confirm what has been requested in the stories actually work. As with integration testing, these tests may be run less often.

    I Mock Thee

    Mock (or fake) services and data are a necessary part of the testing process, and usually, must be maintained and extended as part of the testing. A feature story may require a developer to spend more time developing or extending a mock service than writing the code it helps test. For example, a mock account with some transaction history is needed to test a new deposit function.

    Live environments can never be entirely simulated in test environments. Sometimes a live service uses a paid for third-party service, and it would be too expensive to use in testing. It is usually the case that the development staff are not permitted to work with live customer data.

    The Build Is Alive

    All the structures mentioned are just products of one simple attitude: “How do I know if this works?” For continuous testing to thrive, engineers should always question whether every part of the system is being tested and whether the tests are being evolved at the same pace as feature development. When tests or mocks are missing, an agile team refers to technical debt. This is as it implies, something that must be paid for later.

    When team members realize they cannot just check-in code and go to lunch without checking the central build, that is a good sign. Information radiators such as large screen displays of the build state should be used to get the extended team to understand the state of the build is everyone’s concern, throughout the lifecycle.

    — David Eastman

  • CollabNet VersionOne Picks Up $500M to Fuel Value Stream Management

    CollabNet VersionOne Picks Up $500M to Fuel Value Stream Management

    CollabNet VersionOne today announced it has picked up an additional $500 million in funding to help fuel what is rapidly becoming an arms race to drive adoption of value stream management in DevOps environments. TPG Capital, the private equity arm of TPG, signed a definitive agreement to acquire CollabNet VersionOne. Terms of the acquisition were not disclosed, but TPG pledged to invest up to $500 million to take value stream management to the next level.

    CollabNet VersionOne CEO Flint Brenton said part of the funding will be used to fuel both organic and inorganic investments in value stream management, a rapidly growing segment of the IT market that is just now coming into its own. Value stream management represents a holistic effort to align application development and delivery with strategic business objectives. In effect, IT leaders are now looking to apply advanced analytics to determine how a delay to one aspect of a development project might impact the revenue goals of the business.

    Brenton said now that more organizations are viewing themselves as software companies that make a product or deliver a service, they need tools to manage application development projects in much the same way software companies operate. Rather than building those tools, many of those companies are looking to vendors such as CollabNet VersionOne to provide those tools within a unified platform.

    CollabNet VersionOne is, of course, not the only IT vendor pursuing that opportunity. However, rather than viewing value stream management as an extension of a specific development platform, CollabNet VersionOne is pursuing an approach that makes it possible for an organization to manage software investments across multiple platforms.

    Brenton said organizations should expect CollabNet VersionOne to focus on value stream management. As part of that focus, the company will look to fill in gaps in its portfolio by investing in product development, partnering with providers of complementary applications such as providers of business intelligence (BI) tools or acquiring smaller companies.

    In effect, there is now an arms race underway when it comes to building the next generation of value stream management platforms. Much of the investments in this area will be focused squarely on advanced analytics infused with machine and deep-learning algorithms that will enable organizations to begin applying artificial intelligence (AI) to the management of software delivery.

    It’s not clear to what degree CollabNet VersionOne may one day be a target for acquisition itself as the entire DevOps vendor community continues to consolidate. As DevOps processes increasingly are embraced by enterprise IT organizations, many vendors are concluding they need to “go big or go home.” In fact, Brenton noted the acquisition of CollabNet VersionOne was initiated by TPG Capital, which was looking to find an attractive investment in the DevOps sector.

    It may be a little while longer before every enterprise has mastered the nuances of value stream management, but the one thing that is clear is there will be no shortage of options as investment dollars continue to pour in.

    — Mike Vizard

  • Why ITOps Is Moving to the Cloud

    Why ITOps Is Moving to the Cloud

    There’s been a lot of movement in the IT operations management (ITOM) business lately, from the acquisition of SignalFx by Splunk to the PagerDuty IPO, and all signs point to a Datadog IPO in the future. What’s with all this consolidation? I believe we’re seeing the rise of a future-state of ITOM; that is to say, it’s the rise of SaaS-based ITOM. And it’s easy to see why.

    In my previous consulting career as lead enterprise systems architect, our team had an impeccable record in designing and implementing well-architected hybrid infrastructure solutions. We maintained an immaculate record and near-flawless customer satisfaction record. By project sign-off, our job was always done. And yet, returning to the same solutions six months later told a different story entirely.

    Well-architected solutions are similar to human bodies: They are perfect when they’re born but need constant care and feeding. These same solutions that satisfied SLAs, exceeded expectations and transformed organizational efficiency can easily degenerate, and just like our bodies have their nervous systems to monitor, brains to send alerts and tissue to self-heal, well-architected systems need operational maintenance to keep them humming.

    The traditional approach for solving this eternal need was and still is to design and implement well-architected IT operation management (ITOM) solutions, around a well-architected infrastructure. And yet, this is a self-fulfilling paradox because the ITOM solution itself needs the same care and feeding.

    There’s a Problem On-Premise

    ITOM is a broad term encompassing application and operating system (OS) performance, alerts, log management, notification, asset configuration, incident management and more. It typically involves the purchase of a suite of on-premise point tools addressing each need, and then to develop an internal framework to help those tools interoperate in a meaningful way. While that is possible conceptually, the facts on the ground reflect a very different reality:

    • Multi-vendor tools are often not designed to work together.
    • Creating an internal logical framework that orchestrates various teams and technologies can be very complex in large enterprises.
    • It’s near-impossible to create technical integrations flexible enough to adhere to inevitable organizational and technological changes that will affect this logical framework.
    • Predicting cost-of-ownership is nearly impossible since each tool is controlled by a different vendor, and the internal integration effort is often unknown.
    • Predicting the cost of the manpower required is also very difficult, as each tool requires its own set of specialists, in addition to integration specialists to make it all work together.
    • Upkeep is often overwhelming, as vendors offload software patches, upgrades and on-premises hardware costs to the customer.

    In the face of all these challenges, the end result is often unrealized value, overwhelmed operational teams, loss of service and inability to accommodate new technologies resulting business service disruptions.

    Why Cloud? Why Now?

    It’s historically been near-impossible to build a traditional ITOM platform on-premise. Vendors typically sell a collection of white-labeled tools cobbled together by acquisitions, and this is far from a platform. The complexity and the rate of technological change make it difficult to provide consistent quality and value across the various product lines. This puts the ITOps team squarely in the middle of a pickle: How can they ride the wave of a changing environment without relying on static tool suites?

    The Future is Flexible

    Enterprise ITOps has been stretched now more than ever. There is a serious skills gap, shortage in IT workforce and ever-increasing technical complexity. Time and resources are precious, and Enterprise ITOps need simplicity and predictability along with flexibility and control. Enter SaaS ITOM. By moving the ITOM function to a SaaS orientation, the responsibilities, workloads and daily tasks can transform according to the needs of the organization:

    Keeping Up with the Business

    SaaS ITOM can keep up with technological change, and keep pace with cloud, DevOps, artificial intelligence and more. In the world of SaaS, change is an accepted constant and not an inconvenience. What’s more, SaaS ITOM is infinitely more consumable than the tool suites of legacy past, and that reduces the learning curves associated with running ITOps.

    Keeping Up with Industry Needs

    A SaaS ITOM platform will be able to deliver a framework that’s both flexible and governed and can accommodate technical and organizational complexities. This agility is a feature of modern SaaS. SaaS ITOM can also integrate features running on a single code base supported completely by the SaaS vendor, who will absorb maintenance and upgrade cycles, freeing considerable and valuable time back to the operator. All of this results in a more predictable total cost of ownership, improved service quality and more value to the business user.

    It’s not news the world of IT is moving to the cloud. It is news, however, that cloud can offer such transformational benefits in ways we’ve never seen before.

    — Wael Altaqi

  • The State of DevOps Report 2019 Is Out

    The State of DevOps Report 2019 Is Out

    The Accelerate State of DevOps Report 2019 is out like every other year. Now, let’s talk about the key findings of this year’s survey:

    • The Industry Continues to Improve, Particularly Among the Elite Performers: Proportion of high performers have tripled, now comprising 20% of all teams. This shows the improvement is real for those who really are doing the right things.
    • Delivering Software Quickly, Reliably and Safely Is at the Heart of Technology Transformation and Organizational Performance: It is evident that speed, stability and availability leads to organizational performance in terms of profitability, productivity and customer satisfaction.
    • The Best Strategies for Scaling DevOps in Organizations Focus on Structural Solutions that Build Community: High performers favor strategies that create community structures at both high and low levels of organization. Having established community of practices, having proof of concepts, helps them become more resilient and successful. Dojos are becoming more and more important.
    • Cloud Continues to Be a Differentiator for Elite Performers, and Drives High Performance: The use of cloud as defined by NIST Special Publication 800–145, is predictive of software delivery performance and availability. High performers are 24 times more likely to execute all five capabilities of cloud computing:
      1. On demand self-service
      2. Broad network access
      3. Resource pooling
      4. Rapid elasticity
      5. Measured service
    • Productivity Can Drive Improvements in Work/Life Balance and Reduction in Burnout, and Organization Can Make Smart Investment to Support It: To support productivity, organizations can foster a culture of psychological safety and make smart investments in tooling, information search and reducing technical debts through flexible, extensible and viewable systems.
    • There Is a Right Way to Handle the Change Approval Process, and It Leads to Improvements in Speed and Stability and Reductions in Burnouts: Heavy weight change approval processes such as CAB, negatively impact speed and stability. In contrast, having a clearly understood process for changes drives speed and stability, as well as reduction in burnouts. It is nice to see ITIL 4 has adopted this best practice, and the CAB there is no longer mandatory.

    The study was done on varied departments and industries.

    Software Delivery and Operational (SDO) Performance

    Over the last 6 years, a set of key metrics has been developed to measure SDO performance which provides a high-level system view of software delivery and performance and predict an organization’s ability to achieve goals. Last year, one operational metrics were added and found this measure helps delivering superior outcomes.

    This puts the focus and proves operational performance adds to the overall performance of the organization and helps in achieving the outcomes. Availability becomes an important component to recognize the outcome and co-create value. In ITIL 4 we also find this being brought out in the Service Value System. Importance of operations in adopting DevOps is very important.

    The SDO for Elite Performers as per the Accelerate State of DevOps Report 2019

    This year the report has shown two research models to help drive improvement in both improvement in performance and productivity.

    Use of cloud has been one important factor. How it is used leads to the benefit generated out of it.

    The Five Essential Characteristics of Cloud-Related Result:

    Only 29% of the respondents who said they use cloud infrastructure agreed or strongly agreed they met all the five essential characteristics of cloud computing as defined by NIST.

    The report talks about technical practices.

    Executing for maximum effect:

    • Concurrent efforts at team and organization levels
    • Continuous delivery using automation
    • Team-level technical capabilities
    • Organization-level technical capabilities

    Factors within are:

    Importance of architecture still remains important at organization level. Relook at the change process, it’s an important point highlighted in this year’s report. Use of cloud services is giving more improvement. DR testing is as critical as any other testing.

    Change management needs to be tuned to facilitate continuous delivery. It should not be heavyweight and on the other hand do not compromise risk. Points to be looked at in change management are:

    1. Implementing segregation of duty
    • Use Version Control
    • Use Infrastructure as Code
    1. Role of CAB becomes more strategic
    • Helping teams to implement capabilities that improve performance by providing guidance and resource
    • Weigh important business decisions that require a trade-off or sign-off at higher levels of the business
    • Shifting detailed code review to practitioners and automation and providing more time for strategic decisions to leadership
    • Transition from gatekeeper to process architect and information beacon
    1. Clear change process with proper delegation of authority

    Psychological safety helps in improvement of performance.

    This report talks about improving productivity. How do we improve productivity? The following factors can help:

    • Useful and easy-to-use tools
    • Technical professionals and tools
    • Internal and external search
    • Technical debts

    Finally, the approaches to improve–heatmap of DevOps transformation strategies by performance profile.

    Learn more about the Accelerate State of DevOps Report 2019 and attend the DevOps India Summit 2019 to hear the global experts.

    — Niladri Choudhuri

  • From DevOps to DevSecOps: Owning Cloud Security

    From DevOps to DevSecOps: Owning Cloud Security

    The cloud is a complex environment, with different groups managing different cloud services and environments. The most basic division is between the cloud provider, who provides the core infrastructure, and the customer, who leverages services or builds their applications using resources leased from the provider. Within larger organizations, different cloud resources, data and applications may be owned by various lines of business, each with their own budget and development resources.

    From a security perspective, this sort of divided ownership of the cloud can create severe challenges in terms of end-to-end visibility, control and compliance. Of course, the cloud is a shared infrastructure, and when it comes to security events, there will likely always be some shared responsibility. But the lack of a centralized security strategy and inconsistent security policies and standards leading to potential grey areas of responsibility can create serious security gaps, and put critical data and other digital resources at risk.

    Cloud Provider vs. Customer Responsibility

    When respondents to a recent survey by IHS Markit were asked, “who is truly responsible when there is a vulnerability or security event in their cloud deployment?” the answer reflected a lack of understanding of the basic division of responsibility. Even in the best-case scenario, where it should be clear as to who is responsible for a security event (such as a vulnerability in the cloud platform), only about half of respondents were able to identify the root cause and responsibility.

    Complicating things further, there is yet another set of tools as to which responsibility should be clear. The list of security vendors available as add-ons through a cloud provider’s marketplace is exhaustive. The cloud providers themselves also offer a range of security solutions (built into their platforms as services). And many of the cloud management platforms, hypervisors, orchestration tools and other bits of cloud infrastructure have various levels of security available.

    However, the truth is adhering to the shared responsibility model, whenever the customer applies any configuration to these tools, they become his responsibility. Nonetheless, divisions of responsibilities are not always clear for different events.

    The best course of action is to identify and apply best practices to any resource and service consumed in the cloud. A root cause analysis (RCA) playbook should identify possible threats and plot actions where possible, and plot strategies for determining the root cause of an event. This includes identifying all key stakeholders, determining which investigation tools are available for threat analysis, establishing processes for orchestrating responses and resources between all parties and digitize and map those details to actual cloud technologies. Further, any response should be automated as much as possible.

    Security Responsibilities Within the Customer Domain

    Once the primary divisions of responsibility for a dealing with a security event is worked out between providers, vendors and customers–such as determining if a security breach is the result of a corrupted service or a misconfigured system–most cloud customers still have to address the divisions of responsibility within their own organization.

    The challenge is, when it comes to cloud security, too many companies don’t have a good handle on who is responsible for what. Part of the challenge is many organizations have taken an ad hoc approach to cloud development. As a result, today’s organizations, with an average of five different cloud environments in place and two or more in development, may be hamstrung by the challenge that each cloud instance might be owned by different lines of business.

    For many reasons, these groups may be reluctant to turn over the security of their individual cloud environment to the central security team. For example, speed is a critical component of DevOps efforts, so any security implementation that interferes with their primary objective of delivering applications is going to be a problem.

    Part of the reason is traditional IT security teams rarely know much about cloud environments, and the security tools they recommend often bottleneck development. However, handing over to the DevOps teams building out cloud applications and environments is problematic as they have little to no expertise when it comes to security.

    For example, one of the biggest challenges of many cloud-based security tools is the time and skill required to properly configure and manage them, let alone adjusting those configurations to eliminate false positives, or to ensure they don’t miss critical events or drop legitimate traffic. All that requires expertise most DevOps teams don’t have.

    Shifting from DevOps to DevSecOps

    Resolving this challenge can be as simple as adding a cybersecurity specialist to each of the various DevOps teams. Once integrated, DevSecOps can help navigate the fine line of the shared responsibility model to ensure both development and security requirements are met. They can negotiate between the customer/vendor/provider responsibility model, provide strategies for ensuring consistent security across and between cloud instances, and balance protection with performance.

    Part of that is achieved through the selection, deployment and management of tools designed to meet the twin challenges of speed and security. For example, many SaaS-based security solutions, such as web application firewalls, are self-scalable, allowing web applications to grow as needed without compromising protection. Choosing the right tool can also ensure it can be easily stitched into an application transaction with minimal effort. Some of them even have the advantage of having already included deployment, maintenance, scaling and fine-tuning functions.

    The proper application of automation plays an equally critical role in this process. Checking configurations and scanning objects for malware are time-consuming activities that can get pushed to the side when other issues arise. Automated cloud security solution selected and designed by the DevSecOps team can provide comprehensive configuration assessments, dynamically secure stored data, automatically scan for public cloud vulnerabilities, identify misconfigurations that put data at risk, scan files stored in the cloud and prevent the unauthorized downloading of sensitive information.

    Building a DevSecOps Team Requires Executive Support

    DevOps has become part of C-suite and board-level discussions, attesting to the growing critical value of web applications and digital transformation as part of the broader business strategy. However, if the frequency of breaches and the growing concerns of CISOs are any indication, executives aggressively pushing for cloud solutions often have a mistaken understanding of the nature of the security risks that cloud adoption and careless DevOps programs can introduce into their organization.

    Therefore, it is critical executive leadership is educated on the risks associated with expanding an organization’s digital footprint, and as a result, its new potential attack surface. Expanding DevOps to DevSecOps enables the integration of security into new cloud environments on day one. It can also ensure RCA cloud security playbooks are developed and followed, and help introduce and manage security solutions designed to protect critical digital resources without introducing unreasonable risks. Finally, it can provide a much-needed buffer against fines and penalties associated with regulatory requirement violations should their cloud deployment become a conduit for a severe system breach.

    — Lior Cohen

  • Cloud Complexities Hinder Effective Enterprise Data Management

    Cloud Complexities Hinder Effective Enterprise Data Management

    Enterprises everywhere are rushing to embrace automation and AI to improve existing business processes and innovate. But they’re learning that if APIs are the glue they use to build their systems, data is the fuel that drives these efforts forward.

    According to data management provider Actian, however, enterprises are experiencing significant challenges when it comes to obtaining fresh data, effectively wrestling with the complexity of data and getting to the point of driving innovative initiatives.

    The Actian Datacast 2019: Hybrid Data Trends Snapshot surveyed 300 IT decision-makers and professionals in businesses with 259 or more employees. Ninety-four percent of respondents said it’s important for them to have a system in place that makes certain users keep getting data, yet 58% of respondents admitted it is only somewhat likely, or not likely, they are actually getting fresh data to their users. So much for smart, real-time decision making.

    That could explain why just 34% of respondents who said they are using data for decision-making are using it to push innovation forward, instead, the majority are using it for traditional business operations and reporting.

    “Business as usual operations keep enterprises running from day-to-day, but limiting data to operational reporting tasks means missing a key piece of the data puzzle–new insights that lead to awareness of products, markets, consumer trends, strategy and more,” said Actian in a statement.

    The costs of data management are also taking their toll. The survey found 84% of enterprises would deploy more data if they could do so more cost-effectively.

    Those costs could be driven, in part, by the complexity created with data being stored within application siloes and across multiple cloud environments.

    “Over 50% of these businesses say data complexity issues, due to siloed applications, are a top barrier to entry for accessing data and gaining effective real-time insights,” said Actian.

    Actian concludes, unsurprisingly, that enterprises must get to the point that they can exploit their data in real-time and better manage data across cloud platforms.

    “Slower decision-making is only one consequence of having to wait for data to become available for analysis,” said Jack Mardack, vice president at Actian. “There are exciting new analytics use cases, like customer-360 and hyper-personalized real-time offers, that simply don’t work with stale data”

    “Enterprises are struggling to keep up with the growing demand for business-as-usual reporting that it leaves no viable bandwidth for generating new insights. This requires that enterprises provide not only the necessary data management infrastructure to make data from all their sources available to analytics discovery, but also that they invest in and empower data teams that can pursue new insights from new data spaces,” stated Actian.

    — George V. Hulme

  • Accelerating Software Delivery Doesn’t Mean Compromising on Quality

    Accelerating Software Delivery Doesn’t Mean Compromising on Quality

    Whenever we’re pushed for time and forced to rush, we know we’re entering an unspoken deal with the devil. Whether it’s cooking the family a last-minute dinner, ironing a shirt before a wedding or writing against a strict deadline, we’re acutely aware it’s highly unlikely the output will be a fair reflection of our best efforts.

    Seldom does “hurry” equal “success”; mistakes will inevitably happen–that’s all part of the age-old mission to maintain the equilibrium between speed and quality. But what happens if the work, such as enterprise software delivery, is perpetual by its very nature and the only goal is to deliver a continuously improving experience to discerning end-users? How can such a compromise exist in such a fast-paced environment?

    It’s a challenge, that’s for sure–especially for software delivery teams at traditional large organizations. Unlike tech giants and other digital natives that were conceived around the very idea of a business model driven by incredible digital experiences, mature enterprises–your banks, insurance companies, retailers and so on–are learning on the fly, and serve and support vast product portfolios from across the business.

    That doesn’t mean they can’t compete–far from it. They just need to rethink how product-related work flows across their value streams to better empower their talented test and quality assurance teams who, with the right information available in real-time, are able to ensure customers are regularly receiving high-quality products in a timely manner.

    Trust in Your Test and QA Teams

    In most instances, enterprises can dramatically enhance the quality and speed of their software delivery without heavily investing even more resources into their test/QA organizations. Their testing and QA teams can create order out of chaos, and are in a unique position to understand and assess the health of the product’s development and delivery to answer questions such as:

    • Are all the requirements fulfilled?
    • Are their implementations defect free?
    • Are the known defects tolerable if they’re deployed into production?

     In many organizations, it’s up to the testing team to declare whether a product is ready, or if they must stop-ship. To effectively handle this responsibility, the testing team must have access to the right information at all times. It was once hoped that test management tools could be the single source of truth (at least when it came to managing defects).

    Such a dream turned out to be dead in the water because all other specialists across the software delivery value stream use their own best-of-breed tools to create and manage their work (represented as artifacts such as requirements, features, epics, stories and so on).

    And because these specialized tools do not naturally integrate and automatically flow information, testers (and QA teams) end up on time-consuming scavenger hunts to find the most current information—which dramatically undermines their ability to quickly and accurately determine the health of any product. It’s a vicious cycle.

    There’s More to Product Development than Defects

    Each specialist in the software delivery process has their own point of view on the shared artifacts. Testers need access to requirements to create tests. But the business analysts, product managers and product owners that write the requirements do not use the test management system, preferring their own purpose-built tools for developing requirements and user stories.

    Similarly, project managers use PPM tools, while IT support uses ITSM tools. And so on for each discipline across the key stages of the value stream from ideate and create to release and operation. While each of these disciplines are linked together by their desire to create and deploy the highest quality product possible, their ability to work together is hampered by the fact that their tools do not work together. And when these teams don’t work well together, it’s end users and the business that suffer.

    Knowledge-Sharing Across the Value Stream

    Getting these tools to work together can be accomplished in several ways:

    1. A Manual Process: such as logging into another team’s tool, emailing a colleague for the information or consulting a spreadsheet.
    2. Batch Data Transfer: exporting defects and requirements from one system and bulk importing them into another. Of course, artifacts loaded this way have only static information; they are not continually updated with the most current status.
    3. Automated Integration Across Toolchain: If these systems are integrated and the information in them is continuously synchronized, everyone has access to the most up-to-date information in real-time. Practitioners can even collaborate with one another via this information, all while remaining in their tool of choice.

    The third way is the only way to obtain the end-to-end visibility and traceability that is required for the vital one source of truth into the health of your software products. Only then can you access a single collection of consolidated data to analyze and optimize the quality of the software delivery process.

    By connecting these tools and synchronizing all product-critical data across the value stream, the QA process is strengthened and streamlined. For instance, when a tester creates a defect in a QA tool, it’s automatically updated in the PPM tool and the Agile Planning tool. The same is true for user stories, requirements and test plans—a free-flowing, always up-to-date stream of information that makes or breaks the speed and quality of a product’s development.

    A Single Well-Oiled Machine for Quality Control 

    True value stream integration doesn’t just enhance alignment between testers, developers and business analysts, it empowers all specialists involved in planning, building and delivering a software product. Business analysts, product managers, product owners, developers, architects, testers, system administrators, project managers, scrum masters and help desk professionals are operating within a single unified system by sharing and collaborating on the same information.

    With the test management system synchronized to other systems, the testing team gains a centralized view on a product’s health.

    Testers

    • Have access to product information in their test management system.
    • Have access to the most up-to-date information, eliminating costly rework due to inaccurate information.
    • Can collaborate on artifacts by updating and augmenting them in their tool, knowing their updates will be automatically reflected in all other systems syncing that artifact.
    • Can better leverage the features in their tools when those features rely on artifacts created by other teams—such as automatically creating tests from requirements.

    Testing Teams and Managers

    •  Gain better visibility into their products by receiving reports that pull data from multiple systems, rather than having a siloed view of their product data.
    • Get more collaborative, efficient teams. Practitioners no longer use ineffective email threads to communicate, they collaborate on common artifacts in their tool of choice.
    • Achieve a strong return on investment—each practitioner can save 20 minutes each day by automating manual processes, which can add up to millions a year in productivity savings and avoiding wasting precious human intellect.

    Moreover, with a fully integrated value stream, the QA and testing teams can institute standardized processes and tools, without being undermined by the tools and processes of other organizations. The centralized test management system can become the single pane of glass into a product’s development, providing test managers and executives with the insight into the chaos, allowing them more control and better decision-making. And, crucially, ensuring products are not only delivered on time but that they’re also continuously delivering delightful end-user experiences. No more deals with the devil, just the benefits of knowing the devil in the details.

    — Jeff McCollum

  • Challenges of Big Infrastructure Monitoring

    Challenges of Big Infrastructure Monitoring

    The growing complexity of today’s IT reality is indisputable: Software is becoming more complex, and the infrastructure it runs on is equally so. This complexity comes, in part, from adding layers upon layers to the infrastructure we manage—from bare metal servers to VMs to containers to function-based computing—and from how quickly we cycle in new technologies and how slowly old technologies are cycled out.

    Ephemeral and multi-generational infrastructure (often side by side) are the new normal. Old and new technologies co-mingle, and while new technologies may be shiny and exciting, they quickly become ordinary as the next new thing comes along.

    Here’s a sobering stat: In Q4 2017, IBM’s mainframe business grew by 32%. So even while cloud, AI and security initiatives are on the rise at IBM, mainframes sustain a bulk of their business. Old tech perseveres, and we have to find new solutions to bridge the gap between old and new.

    The hodgepodge of technologies—and the challenges they bring—are compounded further when we’re talking about big infrastructure.

    What Qualifies as Big Infrastructure?

    For the purpose of this discussion, big infra is an organization that has been around for at least 10 years. That means they’ve had success, they’ve grown and their infrastructure has grown along with them, and the technologies at play have expanded as well.

    Your organization’s exact age, size and revenue aren’t particularly material; what matters is you’ve had enough time to accumulate resources, people, practices and technologies to create what I think of as a trail mix problem—basically enough legacy infrastructure to start creating problems.

    The Challenges of Big Infrastructure

    The challenges of big infra stem from the technology itself and from the way people use that technology. As an organization grows, its teams often become larger and more siloed, and departmental goals become disconnected. Even when working toward the same goal, departments diverge in their methods for reaching that goal.

    Now, add tools and technology to the mix: Over time, teams choose tools they prefer, or build their own, which works out okay for a while until they have to collaborate with another department that uses different tools and a completely different set of practices.

    Now, add in different work styles and personality types and company politics, and everything gets tangled up fairly quickly. Trying to keep track of who’s using what, how they’re using it and what’s working (or not) is no easy task.

    One customer, for example, had one team using Chef and was all in, and were trying to get buy in from other teams as well, but there was another team that was early days Docker, hand rolling Docker hosts, and didn’t want to adopt Chef. Then there was a different team who was early days Kubernetes and didn’t want to use Docker, and yet another team using Puppet. Once they had to work with each other, things broke down—someone has to win out. You almost have to maintain a logical map of what team uses what tools as a precursor to prepare for a meeting with these business units. Not only is there no consistency in how applications are deployed, there’s also no consistency in how these different technologies are monitored.

    For example, in the past, Netflix stated monitoring accounts for 30% of their infrastructure cost, and recently mentioned they’re planning to spend $1.9 billion on infrastructure in 2019. Think about the scale and complexity involved—they’re storing and analyzing the logs, behavior and metrics for systems supporting some 125 million subscribers.

    Connecting Disparate Data Across Big Infra

    So, how can you streamline and connect all of the data spread across fragmented tools and systems?

    One solution is a data lake, a centralized repository that lets you funnel data into one place, so you have a holistic view of your infrastructure. In reality, big infra ends up spawning multiple data lakes that become siloed from one another.

    You also likely have a square peg/round hole situation—data in one format needs to be transformed to another format before it can get into one of the data lakes.

    This is where a monitoring tool becomes essential. Look for a monitoring tool designed to collect data in disparate data types and transform it into any number of formats, so you can set up an automated workflow to get data flowing from legacy systems (e.g., mainframes) alongside modern infrastructure like containers.

    An example of this idea is with Sensu, and our support of the Nagios plugin spec and Prometheus exporter metrics (amongst many other metric formats). By using a monitoring tool that works with various metric formats, companies can plug the service checks they’re already using plus data from multi-cloud infrastructure into their monitoring setup and get valuable information about their systems.

    A visualization tool such as Grafana is the icing on the cake. With Grafana, you can hook up data wherever it’s stored (e.g., Graphite, InfluxDB, Prometheus, etc.), visualize the data and display it on shareable graphs and dashboards. Using a monitoring tool as a data source for Grafana means companies can display monitoring incidents and inventory—and even mix that data with different data sources on the same Grafana dashboard. Combining a holistic monitoring solution with Grafana’s visualization capabilities gives a data-driven view of your infrastructure that you need—even if it remains a multi-generational, multi-siloed, multi-data-laked, 2,000-different-format-typed trail mix of big infra.

    — Sean Porter