Tag: continuous

  • Reimagining CI/CD: AI-Engineered Continuous Integration

    Reimagining CI/CD: AI-Engineered Continuous Integration

    In my recent article Revolutionizing the Nine Pillars of DevOps with AI-Engineered Tools, I explained that the continuous integration (CI) Pillar is about developers frequently merging code changes into a central repository. After that, automated builds and tests are run, successful builds are containerized and release candidate artifacts are signed. The main objectives of CI are to detect problems early, reduce integration issues and provide high-quality, secure release candidates. In this article, I explain how AI can enhance each step in the CI phase of DevOps CI/CD pipelines.

    Code Branching: In DevOps, managing multiple branches can be challenging, especially with large teams. This can lead to merge conflicts, confusion about where code should be committed and difficulty tracking changes.

    • AI Solution: AI can help by automatically managing branches and offering recommendations on merging. AI-based systems can analyze code patterns and project management styles to suggest optimal branching strategies, minimizing conflicts and maintaining organization.
    • AI Tool Example: Microsoft’s IntelliCode offers AI assistance for software development which could potentially help manage branches by suggesting optimal strategies.

    Code Commits: Commits need to be made regularly to maintain the codebase, but they can introduce errors. Miscommunication and the lack of a unified commit policy can lead to problems.
    • AI Solution: AI can analyze commit histories to identify patterns and recommend best practices. Additionally, AI tools can provide real-time feedback on code changes, suggesting potential improvements before the code is committed.
    • AI Tool Example: Kite is a coding assistant that uses AI to help with code changes and improvements before the commit stage.

    Merges: The process of merging code changes can lead to merge conflicts, particularly in large, complex codebases.
    • AI Solution: AI tools can help predict potential merge conflicts and offer solutions before they occur. They can analyze code structures and change patterns to predict and prevent these conflicts.
    • AI Tool Examples: Tools like DeepCode or Sider use AI to review and analyze your code to help prevent merge conflicts.

    Static Scanning: This process can be time-consuming and complex with a high potential for false positives.
    • AI Solution: AI can improve static analysis tools by learning from past scans to improve accuracy, reduce false positives, and even suggest potential fixes for issues identified.
    • AI Tool Example: DeepCode is an AI tool that analyzes your code, learns from past scans and suggests fixes for issues identified.

    Builds: Build failures can occur due to a number of issues, including poor code quality, incompatible changes, and environment configuration issues.
    • AI Solution: AI can predict build failures by analyzing historical data and identifying patterns that lead to such failures. AI can also automate the build process and manage environment configurations to reduce human error.
    • AI Tool Example: CircleCI Insights uses machine learning to analyze and predict the success of builds, helping to mitigate build failures.

    Scanning for Quality Errors and Security Vulnerabilities: This process can be slow and yield false positives, and the sheer number of potential issues can be overwhelming to manage.
    • AI Solution: Machine learning can be trained on previous scans to recognize patterns, improving the speed and accuracy of scans. AI can also prioritize issues based on potential impact, helping teams focus their efforts.
    • AI Tool Example: ShiftLeft scans for vulnerabilities in the code and uses AI to prioritize the issues found.

    Unit Testing: Writing and maintaining unit tests can be time-consuming and it’s often difficult to achieve good coverage.
    • AI Solution: AI can assist in generating unit tests based on code analysis and historical data, ensuring good coverage and saving development time.
    • AI Tool Example: Diffblue Cover uses AI to automatically write unit tests, aiming to improve coverage and save developer time.

    Containerization: Misconfiguration of containers can lead to issues such as application failures, security vulnerabilities or inefficient resource usage.
    • AI Solution: AI can help solve this challenge by automating and optimizing the configuration process.
    • AI Tool Example: Magalix can analyze historical configuration data and current application requirements to suggest optimal configuration settings for a container. It can flag potential misconfigurations or inefficiencies and suggest improvements based on best practices learned from past data.

    Functional Integration Testing: These tests often require significant resources and time to write and maintain.
    • AI Solution: AI can generate functional integration tests based on system behavior analysis, reducing the time and effort required to maintain these tests.
    • AI Tool Example: The test.ai tool uses AI to generate and execute functional tests, reducing the need for manual test case creation.

    Determining Whether a Build is Suitable as a Release Candidate Artifact: Manual analysis of build quality can be slow and subjective.
    • AI Solution: AI can analyze build artifacts, test results and other data to predict the suitability of a build as a release candidate, reducing the time and subjectivity involved in this decision.
    • AI Tool Example: Google’s Cloud Build can be leveraged with machine learning to predict the success of builds.

    Digitally Signing Artifacts in an Artifact Repository: This process can be cumbersome and prone to human error.
    • AI Solution: AI can manage the process of digitally signing artifacts, ensuring that only validated, authorized code is signed and reducing the potential for human error.
    • AI Tool Example: While no specific AI tool exists for this task, automated DevOps tools such as JFrog Artifactory or Sonatype Nexus could be combined with AI technology to automate the signing process, ensuring that only validated, authorized code is signed.

    Challenges of CI Transformation to AI

    The transition from traditional continuous integration (CI) to AI-engineered CI presents several challenges:

    Integration with Existing Systems: When incorporating AI into an existing CI toolchain, ensuring seamless integration can be a challenge. Organizations need to verify that AI tools can work effectively with current systems and don’t disrupt existing workflows.

    Skillset and Understanding: Adopting AI requires a certain level of understanding of how AI works, including its strengths and weaknesses. Teams need to be trained on how to use AI tools effectively and interpret their outputs.

    Data Privacy and Security: AI tools often rely on analyzing large amounts of data, which can include sensitive or proprietary information. Organizations need to ensure that using these tools doesn’t compromise their data security and privacy.

    Cost: Implementing AI tools can be costly, both in terms of purchasing and deploying the tools and in the potential need for additional hardware resources. Organizations need to consider the return on investment (ROI) and determine if the benefits of using AI outweigh the costs.

    Tool Maturity: Many AI tools in the DevOps space are relatively new and still evolving. Organizations need to consider the maturity and stability of the tools they’re planning to use.

    Over-Reliance on AI: While AI can greatly enhance CI processes, over-reliance on AI without proper understanding can lead to misinterpretations and errors. It’s important to balance AI usage with human oversight and expertise.

    Benefits

    Transitioning to AI-engineered CI toolchains can have several significant benefits for organizations:

    Improved Efficiency: AI can automate various aspects of the CI process, such as code review, testing and environment configuration. This can greatly reduce the time taken for these tasks, speeding up the overall CI process and allowing teams to deliver software more quickly.
    Increased Quality: AI tools can perform in-depth code analysis and identify potential issues with a level of detail that might be missed by humans. This can lead to improved code quality, fewer bugs, and more stable software.
    Predictive Analytics: AI can analyze historical data to predict outcomes, such as the likelihood of a build failure. This can help teams proactively address issues, reducing the risk of failures and improving the reliability of the CI process.
    Improved Decision Making: AI can provide valuable insights based on data analysis, helping teams make more informed decisions. This can relate to things like identifying the most suitable release candidate or deciding when and how to scale resources.
    Consistency and Standardization: AI-driven CI can ensure consistent application of rules and standards, as there’s less reliance on individual developers’ judgment. This can lead to a more standardized, reliable development process.

    Summary

    Overall, AI has the potential to greatly improve the efficiency, accuracy and effectiveness of DevOps continuous integration practices. However, it’s also important to note that these solutions require careful management and oversight to ensure they’re working as intended and improving rather than hindering processes. Organizations should start with small, manageable projects to gain an understanding and experience with AI before scaling up. They should also invest in training and support to help their teams adapt to these new tools.

  • Embracing Security by Design in Development

    Embracing Security by Design in Development

    The number of data breaches publicly reported in 2021 far exceeded the total for 2020, making 2021 a record year. With cyberattacks on the rise and bad actors and ransomware gangs demanding millions in ransom, cybersecurity has taken on new urgency.

    In addition to more sophisticated hacking techniques, the rapid pace of tech adoption during the pandemic is another contributing factor fueling cybersecurity risks. A recent survey by Forrester shows 74% of companies attribute recent cyberattacks to vulnerabilities in technology put in place during the pandemic. As companies raced to enable remote work, develop new applications and roll out new consumer-facing digital solutions, cybercriminals saw new opportunities to identify vulnerabilities and to attack. 

    Cybersecurity threats can affect every business regardless of size and every individual, regardless of who they are or where they live. Even the most protected and advanced organizations are at risk: One of the world’s biggest cybersecurity companies recently fell victim to one of the largest breaches in history. It’s time for all companies to rethink their approach to boosting security in the software development process. 

    As a nearshore development partner for businesses around the world, we have embraced a ‘security by design’ mindset to keep our clients’ builds secure. To put security first and ensure it remains at the forefront of every engagement, we adopted a parallel build systems process to improve the integrity of the code we deliver. This also reduced the overall threat of potential attacks. How did we do this? There are three key components to developing and implementing a security-by-design mindset.

    Embedded Security Expertise

    It is critical that security experts become official members of every software development team. These specialists are there every step of the way from design to delivery, finding and fixing security flaws along the development journey. Instead of being brought in at the end of the process, they are deciding on new solutions and implementing protections in real-time alongside experts in design, user experience and coding. Each plays a role and works as an integrated member of the team providing advice and suggestions during Agile cycles and sprints. 

    Predictive Analytics

    A recent survey showed that 96% of third-party applications deployed in cloud infrastructure contain known vulnerabilities. That rising risk can make it possible for bad actors to infiltrate thousands of organizations. Threat hunting tools can spot, anticipate and predict vulnerabilities in software development. Tools can automatically scan code as it is written to vet and diagnose vulnerabilities. Our team scans code while in production and reports risks and vulnerabilities and then immediately consulting with the rest of the team to work on the next secure version. 

    Continuous Learning 

    Security is changing quickly, and it’s never been more critical to ensure developers have the skills necessary to defend and protect corporate infrastructure, applications and digital solutions. With the threat landscape constantly evolving, new skills and continuous learning are required to stay ahead of the latest threats and fraud techniques. Part of our training is focused on deploying white hats, or ethical hackers, to breach our development environments and the applications we deploy. These experts then set about identifying weaknesses and helping to improve security. They also take our training to the next level and push our experts to continually learn and educate themselves on the latest threats and solutions. By upgrading their skills, security practitioners can stay ahead of cybercriminals. 

    Every day, enterprises face thousands of attacks on their systems, and malware continues to grow in frequency, volume and complexity. These attacks are not just coming from criminals looking to make a buck; there are growing fears of nation-state attacks escalating. These are often seen as the more dangerous threats to businesses. The best defense against cyberthreats is a strong offense anchored by a security by design mindset.

     

  • DevSecOps: IoT Right and Nobody Dies

    DevSecOps: IoT Right and Nobody Dies

    Dr. Suzanne Schwartz, CDRH Associate Director for Science and Strategic Partnerships at the Food and Drug Administration (FDA), recently released a blog to update us on the FDA’s role in medical device cybersecurity.

    Cybersecurity risks in medical devices are nothing new. As far back as 2012, Sonatype published warnings of security risks in pacemakers that could lead to lethal attacks. Last year, Johnson & Johnson warned of cyber vulnerabilities in its insulin pumps. More recently, our “2017 State of the Software Supply Chain Report” shared details of pacemaker programming machines that were discovered to have more than 8,000 known software vulnerabilities (see page 37).

    When it comes to software in medical and other devices, cybersecurity will be a constant threat requiring constant vigilance. The encouraging guidance I read from the FDA blog was that cybersecurity should not be limited to devices already on the market, but that security needs build in throughout the product life cycle. Schwartz remarked:

    “It is the goal of FDA’s Center for Devices and Radiological Health to encourage a coordinated approach of vigilance, responsiveness, resilience, and recovery that fits our culture of continuous quality improvement.

    “This means taking a total product lifecycle approach, starting at the product design phase when we build in security to help foil potential risks, followed by having a plan in place for managing any risks that might emerge, and planning for how to reduce the likelihood of future risks.”

    In these two brief statements, Schwartz shares views that are common in many of today’s DevSecOps conversations. Perhaps she has been listening in?

    The first view is that we need to “emphasize the performance of the entire system and never pass a defect downstream” (Gene Kim’s first way of DevOps). Schwartz recommends that device manufacturers start by building security in at the earliest stages of the development life cycle. Security cannot be an afterthought.

    Schwartz also recognizes that even when precautions are taken early in the life cycle, risks can emerge over time as new vulnerabilities are discovered. When it comes to device security, vigilance must be continuous across the development life cycle and over the product’s life in the market. As we have said for years, software ages like milk, not wine. Constant vigilance improves our ability to identify risks, initiate feedback to development teams and remediate issues in the device’s software. The faster we can address risks when they appear, the safer we can make the lives of consumers who rely on them.

    I applaud the FDA’s guidance and its proactive stance here to work with device manufactures. What we don’t want is a knee-jerk reaction to medical device security after someone has died. What we need is for security to be ingrained in the planning, design, production and maintenance of medical devices our families rely upon.

    You can read the full blog from Schwartz on the FDA Voice.

    — Derek E. Weeks

  • The 7C Holistic Approach to DevOps

    The 7C Holistic Approach to DevOps

    The DevOps methodology was first adopted in 2007 with a bigger purpose of improving organizational efficiency by bridging the gap between software release and IT operations teams. This started as the core practice of continuous integration (CI) and continuous delivery (CD) with a prime focus on better quality of releases and more frequent releases. Now, DevOps is becoming the norm of the industry and is being embraced by multiple organizations as a digital transformation with a shared goal, involving all stakeholders, to serve the customers better with higher quality product release and more frequent releases.

    DevOps has evolved beyond release and deploy and now involves product owners, project managers and all aspects of software development life cycle. This is the key reason it has evolved as a holistic approach starting from the service team interacting with the customers to product release and production monitoring. In this blog, we will explain how the 7C approach of DevOps is a natural evolution of DevOps and how it helps different organizations to perform better.

    DevOps now spans across the entire organization involving customer service, marketing, product owners, project managers, program managers, and different levels of developers, QA, release or build teams and infrastructure team. As the main focus of DevOps is to serve the customer better and faster, it is important that all the stakeholders have better end-to-end visibility of operations, planning, integration, testing, monitoring, delivery and feedback. This is the 7C approach of DevOps, as illustrated in Fig 1. This requires an efficient integration of processes and tools to automate the seamless information exchange and execution. This approach also empowers all the stakeholders to better contribute in product’s success and efficient execution.

    Fig. 1: 7Cs of DevOps

    The 7C holistic approach to DevOps also requires efficient collaboration between all stakeholders, empowering them to contribute more. The 7C approach removes all silos of information and execution.

    Startups were fast to adopt DevOps as many of them were setting up things from scratch. But the latest industry trends show large enterprises embracing DevOps primarily with the focus on efficiency and better, more frequent releases.

    The DevOps domain includes lots of tools and many of them are open source. This has provided a toolset for engineers to tinker with or experiment. This tends to have problem of its own, as too many toolsets in the flow create silos of information and execution, which creates confusion and inefficiency. This is becoming a problem and that’s why solutions are required which provide better integration and execution across multiple tools. The right solution for this will be a platform with deep integration capabilities.

    A deep integration platform for DevOps will enable seamless integration and execution across different tools, even home-grown proprietary tools, to provide a holistic solution starting from customer initiation to production deployment and feedback. This approach is significantly important for large enterprises, as this enables them to have the flexibility of preserving their existing investments on tools and processes while also allowing them to selectively introduce new tools. Rather than trying everything new, large enterprises can manage changes while primarily focusing on efficiency, automation, collaboration, better quality of releases and more frequent releases.

    This 7C approach gives same visibility to all stakeholders and everyone knows what feature is being released when. This approach also helps to move information/execution in both left and right direction with powerful traceability features, analytics and dashboards. Deep integration of information and execution also reduces dependency on other teams and promotes self-service.

    Fig. 2 illustrates a sample DevOps flow where service team uses ServiceNow; software, product and project managers are using Jira for Agile management, developers are using Eclipse as their integrated development environment, GitHub is used for source code repository, QA team uses a test management tool for test and defect management, and finally the release team uses Chef and Jenkins to orchestrate build, release and deployment. In real-world examples, teams add more tools for code analysis, security and production monitoring. All such complex flows are supported within DevOps 7C holistic approach by utilizing the deep integration platform. The 7C approach is more productive as it allows faster detection, analysis and resolution of problems.

    Fig 2. Sample DevOps Flow with Deep Integration

    The 7C approach works well with both traditional waterfall software management as well as Agile software management—mainly because of better end-to-end visibility, efficiency and collaboration.

    To conclude, 7C approach of DevOps is the natural evolution of DevOps, empowering the team to achieve better efficiency and execute better. This empowers all the stakeholders in the organization by providing better end-to-end visibility and powerful means of collaboration. Involving all stakeholders also creates a better environment of trust and collaboration. With this 7C approach, DevOps is not limited to build and release but it has spanned across the entire spectrum of software development lifecycle. This is especially highly beneficial for large enterprises as it enables them to preserve their prior investments and enhance their existing flows. This gives them leverage to selectively try new tools and technology while preserving what is working well. This is a great approach to expedite the enterprise level digital transformation.

    About the Author / Devendra Joshi

    Devendra Joshi is Sr. Director, Customer Solutions and Products at Kovair. He is a software professional with over 20 years of experience in enterprise software, social media, healthcare, SaaS companies, and EDA. He has worked with startups and large enterprises in engineering management, and product and solutions management roles.

  • DevOps Chat: Jeff Schaeffer, CA Technologies, on Everything Continuous

    DevOps Chat: Jeff Schaeffer, CA Technologies, on Everything Continuous

    I recently had a chance to sit down with Jeff Schaeffer, senior vice president and general manager of Continuous Delivery and DevOps Business Unit at CA Technologies. I have interviewed Jeff before, most recently at Jenkins World, where we did a video on our DevOpsTV YouTube channel. Jeff is a great interview who always gives good insight into what CA’s and his views of the dynamics shaping the DevOps market are.

    Jeff and I had a chance to talk about the launch of the new CD neighborhood sponsored by CA Technologies. Specifically, we discussed how CD—and, for that matter, the whole idea of continuous IT—is reshaping the way organizations do business. It is a great conversation that I hope you will enjoy.

    Below is a streaming audio of our DevOps Chat and below that is the transcript of our conversation.

    Alan Shimel: Hi, everyone. This is Alan Shimel, editor in chief, devopsy.kinsta.cloud, here for another DevOps Chat. Today’s DevOps Chat guest is Jeff Schaeffer, who is responsible for CA Technologies’ continuous delivery CD business.

    Jeff, welcome to DevOps Chat.

    Schaeffer: Thank you, Alan. Glad to be here today.

    Shimel: Good to have you here. Jeff, of course, I think the last time you and I spoke was at—it was at the Jenkins World event, and we had a chance to actually sit down on video and talk a little bit about, you know, where do you see the market, where you see the DevOps market going, what CA’s involvements are and where you guys are kind of placing your bets and moving your pieces around as this market continues to evolve.

    And we were, of course, talking offline before we started today’s podcast about kind of how you view the different waves, if you will, of maturation of companies’ life cycles with software and so forth, and you mentioned third wave, fourth wave.

    I’d love, if you don’t mind, if we could share a little bit of that context with our audience in terms of your view of the historical ebb and flow here of history, if you will, of where we are. So would you mind sharing that?

    Schaeffer: Thanks, Alan. So, Alan, as we were discussing, our—the thesis and the drivers for continuous delivery actually start kind of in the history of technology, and it’s probably never been a more exciting time for any of us to actually be involved in application development and technology in general as it’s transforming all the markets.

    And the history is useful to the extent that it helps us understand the future as well, and when we think about it, you know, we’re really in the midst of the third wave, in which businesses have disruptively adopted technology in the business.

    The first wave was back in the 1960s and it was really about the mainframes and the digitization of the paper processes in the company. I mean, it was a process disruption.

    The second wave was in the ‘90s, and it was really about ERP and CRM systems where we were automating the supply chains of companies, and again, it was a process disruption.

    Today we’re in the third wave, around the application economy, and in the context of digital transformation and the app company, you know, companies today, they promote their brands, their products, their services through their web and their mobile applications, and it’s become the primary methodology by which they engage their end users. And it is actually underpinning and enabling business model disruption, which is very different than the prior two waves of disruption.

    So when we look at it, it’s an entirely new business model. We’re all familiar with the majority of those in terms of, for example, Uber disrupting transportation and Netflix disrupting media, et cetera.

    And one of the challenges about the third wave is that—following the third wave, actually, I think is the fourth wave, and the fourth wave is gonna be around systems of insight and engagement, by which companies are able to take all of the multitude of data that they’re generating in terms of every interaction with their end consumers—you know, with suppliers and providers—and be able to analyze those data sets’ multivariate analysis and derive deep insights, and to be able to then, in turn, customize their offerings. Companies would be able to use that data to make more of the right decisions more often, to drive efficiencies, et cetera.

    And so naturally, the fourth wave—and it’s coming, but before we can actually fully benefit from the fourth wave, we actually still need to master the third wave. And in the context of the third wave, the app economy, it’s really about, how do you get these applications, these web or mobile applications into the hands of end consumers more quickly and, you know, great quality and at lower cost and factor of production.

    And so that’s really the area where companies today are focused. And, to do that, what most companies are doing is, they’re adopting agile methodologies, right? They have to increase their competitiveness and their ability to deliver this custom software. So they look to get development up to velocity, so they’re adopting agile methodologies, and then they quickly realized that the rest of the value streams are not at velocity—so testing, release, operations—which then leads them to adopt DevOps, which is where we’re gonna have a great conversation today.

    And, with DevOps, they start to adopt the culture and the practices, and then realize they need a new enabling tool chain, and so if you think about any technological disruption and being a function of organization and people being a function of process and being a function of tools and technology, they now adopted a new tool chain, which is the continuous delivery tool chain.

    And that’s an area where we at CA are focused about how do we provide the coaching around the organizational transformation and the practices in the process transformation and then, you know, from a product perspective, we’re building the next generation tool chains, building block fashions, and bringing those to market to support companies’ transformation and competitiveness in the application economy.

    Shimel: Perfect. And you know what, Jeff, that actually dovetails into, of course, just today, we announced on devopsy.kinsta.cloud that we’ve added a new CD—or continuous delivery—Neighborhood, which CA has graciously sponsored. And we’re really looking to build this into a—I don’t want to use the word portal, it’s so 1999—but a destination where people can come and get some great content and some great education around this whole CD tool chain and the CD economy, if you will, that is growing up.

    But, you know, Jeff, going back to what you were talking about, we can’t get to the fourth wave until, I think, the third wave is a little bit more solidified and a little bit more mature, a little bit more settled. In my mind, what we’re seeing is, a lot of these convergent circles starting to overlap, right? You mentioned agile, we mentioned CD, we mentioned things like API, right, which is another big area for CA.

    Schaeffer: Yes.

    Shimel: And, you know, they’re not totally independent of each other, nor are they totally dependent on each other, but you know, they kind of form the ultimate Venn diagram where the overlap is just marvelous, right? I mean, it’s just a wondrous time to be in business if you’re using—you know, you’re leveraging all of these things.

    You know, and you’ve been around, Jeff, as I have—what do you see with that?

    Schaeffer: Yeah, so Alan—exciting things, exciting things. So, a couple of quick comments, one on the Continuous Delivery Neighborhood—really excited for that. Alan, I can tell you that in that context, whenever I meet with peers across the industry, one of the things that comes out very broadly today is, there’s a big gap between conceptualization of things like agile, DevOps, and continuous delivery, and translating those into real world practice.

    Shimel: Mm-hmm.

    Schaeffer: And the opportunity to provide that education, to provide a journey to share examples of how people have adopted these and succeeded is gonna become absolutely instrumental in terms of unlocking the full potential of these methodologies and these concepts and deriving the business benefit.

    So we’re really excited with what devopsy.kinsta.cloud is doing with respect to the educational exercise, and also devopsy.kinsta.cloud and that sponsorship of the DevOps Express Consortium, which is another multi-vendor body that’s focused on this translation of concept of real world practice, and of course, here at CA, we’re happy to be a founding member of DevOps Express as well as a sponsoring member here for devopsy.kinsta.cloud and the CD Neighborhood.

    The second thing is, when we look at the change in the continuous delivery tool chains, what we’re finding is, as companies adopt agile, they start looking to have—okay, what is the prescription of the next generation software factor, software life cycle, and as they look at continuous integration to continuous test, continuous release and deployment and continuous operations, you know, how do you string these building blocks together?

    And the only way to reduce the cycle times is making both a process change but also the tooling change. And the tooling is very different today versus, say, some of the legacy providers of application development and testing tools. It’s all pragmatic, API driven, so that it’s fully automatable. There’s a lot more—as you were describing the intersection and the Venn diagrams, it’s really a function of the feedback loops. So how do I take data, for example, from operations and use that to make a smarter example in continuous testing?

    Let me share an example of what we mean by that. So, if you have a mobile application, you know, mobile application is much more difficult to manage in production than, say, a web app. First, people simply don’t update their smartphones that often, and so you end up with a long tail of versions in production environments.

    Shimel: Mm-hmm.

    Schaeffer: They’re running on different operating systems—iOS or Android and different versions of the operating systems. They’re running on different hardware platforms like an iPhone 4 or an iPhone 7, Galaxy 7. They’re running across different carriers, and there’s over 700 different carriers worldwide. And it becomes a factoral problem, and it’s unlikely that any given testing team has the ability to model that many combinations—because it’s a factoral problem—to model that many combinations in their test matrix.

    But, if you have operating data, and you know that 80 percent of the user base is on these five to seven combinations, then you can provide that data back to the QA team and proactively test those combinations and take your risk in the tail of where people have more infrequently completed combos. But, by the same token, if you can detect failures in operations, you can provide that same data back to dev, and they can focus their investigative research as they work on the current product engineering.

    So when we think about the Venn diagrams and how all of that works together in the future, it’s—not only are the format factors of the tools changing, but the feedback loops become absolutely essential in order to fully automate and to systematically improve the software life cycle process. And so, a really exciting time.

    And we’ll often hear that today, this idea of shift left and shift right, but it’s really a—you know, those things embody, that vocabulary embodies a number of concepts. One is, you know, how do I push activities earlier and earlier into the development process under shift left, so how do I incorporate, say, for example, performance testing early, and then how do I tie performance testing into operational testing, and how do I provide that feedback loop all the way back?

    So just exciting times, lots of transformational changes, and people are definitely looking for prescription and methods as how to reconfigure their software life cycles and incorporate that into their current software development methodologies, and that’s what, I think, DevOpsExpress.com will be so good at is helping provide that center of knowledge and best practices for the betterment of the industry.

    Shimel: Absolutely. Jeff, believe it or not, we’re almost up at time, but I wanted to—and I don’t mean to embarrass you, but I want you not to be shy here. CA has done a masterful job of amassing, developing—both holistically, acquisitions, et cetera—a play in this new third wave, fourth wave economy, if you will.

    Schaeffer: Absolutely.

    Shimel: Give our audience—again, don’t be shy. Why do you think CA is best situated to really leverage what’s happening out there right now?

    Schaeffer: Wow, great question, Alan. So, two, maybe three key points on that. So number one, Clayton Christensen, who is a professor at the Harvard Business School, he’s written extensively on the concept of disruption and then disruptive innovation in the market. And one of the key things that you learn about, or learn from Clayton, is that the incumbents never survive the disruption. And it’s not because they’re—you know, it’s just because of their decision making and how they’re driven by their paradigm.

    And CA, uniquely, is in a position as a company to invest in this next generation of continuous delivery capabilities. And so we’ve been putting together this next gen tool chain through a combination of organic product development as well as acquisition, and so we’re uniquely in that position in the market in terms of being a vendor of sufficient size to provide that consolidated next generation tool chain and to ensure it works together. Per kinda the research of the market and industries over time, you know, we’re in a unique position relative to the traditional incumbents and I’m excited.

    You know, we’ve pulled together the Rally Software, which brings kind of market leading agile management capabilities into the portfolio and we start with the planning process and agile management. We realize that, in the application economy driving the automation, that oftentimes in the app economy, the API is the application, but that’s also the primary means by which we exchange data and are able to integrate or automate all of the functions as well as create the next generation architectures, which are all kind of composite applications built on microservices which work through APIs.

    So we’re connected by the APIs, and so we’ve invested in their seven and brought in a really robust API management capability, and then we are laser focused in the continuous delivery space. So, how do we take and orchestrate release with release automation software and release automation continuous delivery addition, which is basically harvesting the data and providing the analytics so that we can systematically improve and continuously improve the software development life cycle.

    And then, beyond that, we have a number of kind of the next generation continuous testing tools. So, everything in terms of, we just recently acquired BlazeMeter. It’s a fantastic capability and at the heart of continuous testing. You know, it’s a commercialization of open source projects, JMeter and Taurus, which allows us to automate any open source testing tool, whether it’s the JMeter testing, which is the second most widely adopted community developed testing product, or Selenium, which is the most widely adopted testing product. Gatling, Locust, et cetera, so a host of open source testing engines.

    We add value above and beyond that. It’s all hosted in the cloud and deploys both in public and private clouds for the tests themselves, and so it’s highly automated, it’s very scalable, and it allows us to left shift the testing all the way into the integrated testing, the component testing, and the unit testing of the new life cycle processes.

    We also have put together a number of tools that improve testing efficiency around test data management, so how you can’t automate your tests unless you automate the test data, and providing test data with the right sort of attributes in order to drive the tests in a productive way. We have service virtualization software, which allows us to simulate test environments, and to emulate the test environments and abstract away things like mainframes or databases, which allows for both parallelized development. When other components aren’t available, we can simulate those or emulate them, and then in the testing practice itself, we’re able to isolate applications and put them under load testing in an effective way.

    All the way out into our operational tools, where we’re able to shift right, and provide kind of the—you know, what’s done in pre-prod gets used in production, how do we get the integration and the feedback loops back into the testing and the development processes.

    And so we’ve really been focused on putting together that next gen toolset around continuous integration, continuous testing, continuous release and deployment, and continuous operations. And again, we’re in a unique position to do it. We’re unique in the fact that it’s a disruption within the market and we’re able to put together this tool chain, and we’ve made a lot of progress in developing that tool chain up ‘til now. And we’re finding companies like—the people we’re working with across the industry adopting the CA tool chains, you know, we’re finding some significant improvements in not only the velocity but the quality and the cost effectiveness of their software development practices.

    So, for example, companies that were getting one release out every year are now able to do 30,000 deployments per month on 300 applications in both the test, the pre-prod and the prod environments. We’re able to significantly reduce the cost of handling data and increase the automation of testing, up international the 85, 95 percent range, lowering the costs and achieving ROIs, in some cases, of over 300 percent in paybacks on the investment measured in months—you know, two to three months, which is very unusual around what people would normally expect in an enterprise software vendor.

    So we’re really excited with the difference we’re making in the lives of our customers—the operating benefits, the business benefits, and then the overall, you know, the impact that we’re gonna make in the industry as a whole.

    Shimel: Fantastic. Hey, Jeff, we’re way over time. I apologize, here. I apologize to—well, I’m not apologizing to those listening, I think they’ll find it worthwhile. But we do try to keep these at 10 to 15 minutes, and we’ve gone over—but I think it was worthwhile, we had a lot of information to share.

    Jeff, continued success in leading up the CD business at CA Technologies, and we’re looking forward to more exciting news soon, and we’re actually looking forward to the CD Neighborhood on devopsy.kinsta.cloud really becoming a go-to place for people looking for information around it. So thanks very much on the sponsorship for that as well.

    We’ll be in touch again, soon. Until next time, I hope you stay well. Jeff Schaeffer, he runs the CD business at CA Technologies—thanks for being our guest on this episode of DevOps Chat.

    Schaeffer: Thank you, Alan.

    Shimel: Thank you. This is Alan Shimel for devopsy.kinsta.cloud, and we’ll see you again on another DevOps Chat.

    — Alan Shimel

  • Redefining Product Value in the Continuous Economy

    Redefining Product Value in the Continuous Economy

    The internet is now more than 20 years old. It’s older than my kids, which is a scary thought. It’s older than Kendall Jenner. It’s older than Uber, Grey’s Anatomy and the iPhone. And so now, after more than two decades of software and e-commerce, it’s high time we take stock of where consumers really are today.

    First, consumer expectations are shifting dramatically. The digital world has brought a whole host of changes in how people work, shop, interact and engage with businesses.

    Second, the notion of brand loyalty has dissolved, shattered beyond recognition.

    And, third, this utter lack and loss of loyalty puts companies at an insane disadvantage. We’re living with a “see you later” digital mindset, where it’s easy to say goodbye to something you know online for an opportunity with the newest, next-best thing.

    All of this means that companies must deliver a constant stream of software that is fluid, flexible and competitive to gain and retain customers.

    It also means that companies absolutely must understand how consumers weigh and determine product value today, because, quite clearly, buyers aren’t making product choices in the same ways anymore.

    Indeed, 20 years ago, customers chose products based on product features. Today’s modern consumers are continuously looking for new products, embracing them and then discarding them for something even more new. In fact, the willingness to switch is so strong, according to Harris Interactive, that 86 percent of consumers quit doing business with a company because of a single bad experience. And 90 percent say they would be willing to pay more for the same product if they received a more positive purchasing experience.

    This continuous process of discovery, adoption, discarding, discovery, adoption, discarding, discovery, adoption and discarding tells us—very loudly and unequivocally—that value isn’t in the actual product itself, but in our ability to continuously evolve the product.

    Shifting Consumer Expectations Require a Continuous Approach to Business

    If you can’t continuously transform the way you deploy your product, people and processes, there’s no way you can keep up with customers’ “see you later” mindset and their continuous discovering, adopting and discarding of products.

    Two decades ago, for example, you chose a car based on ideal product characteristics (make, size, etc.) and how much you could afford to spend. Your choice would live with you while you paid it off or until you could justify changing it.

    Now, software and the speed of innovation have allowed consumers the opportunity to make product choices at a greater frequency. Instead of only getting the newest features when you buy a new car every 10 years, car operating systems are updated wirelessly and provide the latest and greatest features to existing owners, without the need to buy a brand new vehicle. The car has become a software platform.

    Even the physical choice of platform (the car itself) is beginning to change. For those who wish to change their car regularly, there are many services (such as Zip Car) that allow you to rent access to a car and return it when you’re done. Or even not to drive at all, with the advent of software companies such as Uber and Lyft.

    Obviously, each product—software or otherwise—must have a set of threshold features, or else consumers won’t choose it. A car, for example, needs to be waterproof with airbags and ABS brakes. And a task app needs to allow you to add appointments and meetings to your calendar. But to be competitive today, products must now also include the ability to switch away from features and add new ones seamlessly.

    Value is Driven by Option

    This “optionality” drives product value in what has become the ‘continuous economy’ of 2016. Companies can deliver “optionality” only by eliminating barriers from their production processes.

    Companies need to adopt product development processes that are lean and high velocity to provide customers with an ongoing series of new product features and choices. That’s why you need to continuously deliver and continuously change your software.

    Keeping Pace with the Acceleration of the Continuous Economy

    If you can’t—or won’t—continuously transform innovative software products and the way you produce them, you won’t be able to compete in today’s quicksilver-fast marketplace.

    Or, to put it more broadly, software, as well as the organization and its people and processes, must be able to pivot with agility in what seems like a nanosecond to keep pace with the acceleration of the continuous economy.

    This is all very different than what we’ve experienced in the past.

    Two decades ago, an organization confronted a new competitor or change in the market, by developing and deploying new operations, organization adaptations or products to meet the challenge.

    Today, software is the primary route to market, so companies must continuously develop and deploy new software as part of an always-transforming organizational culture. If you’re unable to develop and deploy on a sustained basis in today’s continuous economy—if you only transform on a selective basis—you won’t be in business very long.

  • The Difference Between DevOps and Anarchy

    The Difference Between DevOps and Anarchy

    DevOps is about being more efficient. It’s about breaking down silos and removing traditional business obstacles that get in the way of just getting things done. It’s about enabling teams and individuals to collaborate more seamlessly.

    None of that, however, means that DevOps is about turning your IT into some sort of “Wild West” anarchy, where anyone and everyone can just push code to production. That would be insanity.

    One of the primary catalysts that started the DevOps movement was the frustration of developers trying to deal with an operational bottleneck. As developers evolved from the old-school Waterfall development method to the much faster-paced Agile, developers were able create better software more quickly. That simply resulted in a “hurry up and wait” scenario, though, because the Operations and IT infrastructure side of the house couldn’t keep up.

    Fast forward a bit and DevOps was born. Since its inception, however, there has been confusion about what exactly DevOps is or isn’t. That might be a function—at least in part—of the fact that there was no “inception” of DevOps; nobody developed a thing and then rolled it out to the world with the announcement, “This is DevOps.” DevOps was an evolution that occurred organically and out of necessity. It meant different things to different organizations and was implemented in ways that were largely subjective and often not called anything in particular. Organizations were simply struggling to address challenges and coming up with solutions that worked.

    Suppressing the Anarchy

    Although traditional IT operations were the roadblock that sparked the DevOps movement, DevOps doesn’t mean developers get to simply do their own thing. I’ve worked in the trenches as an IT admin at a dotcom startup—I’ve experienced firsthand the disastrous fallout of a developer pushing untested code onto a production server, and I’ve witnessed the wrath of executive leadership when that happens and the proverbial “stuff” hits the fan. It’s not pretty.

    No, DevOps does not mean eliminating Operations from the equation and functioning as an IT anarchy. You’ll note that DevOps is a very obvious portmanteau of developers and operations—implying that operations is half of the DevOps solution. DevOps is about developers working more closely and cooperatively with Operations—not developers bypassing IT Operations.

    There are two components to successful DevOps: culture and continuousness. First, organizations need to shed the traditional mindset of separate siloed teams and departments and let individuals focus on working together in whatever way makes the most sense for efficiency and effectiveness. Empowering individuals to do what’s necessary to collaborate and get things done is a big part of DevOps.

    The other component is continuousness. When developers adopted Agile and started producing code faster, they discovered that Operations was a bottleneck. When developers and Operations started working together more seamlessly, new bottlenecks were identified, and solutions were developed to automate as much of those tasks and processes as possible.

    The ultimate goal of DevOps is continuous delivery—to continuously deliver new solutions and features. Rather than a lawless void where developers just push code out onto production servers willy-nilly, organizations adopt continuous development, continuous testing, continuous deployment, continuous monitoring and other tools that enable their teams and those developing solutions to do so with as little friction as possible.

  • Continuous is the Engine that Drives DevOps

    Continuous is the Engine that Drives DevOps

    It has been stated time and time (and time and time) again that DevOps is primarily a matter of shifting IT culture. The tools and practices that comprise DevOps play a secondary role. But that is only partially true: In reality, the culture shift revolves around continuity, and the applications and tools that fuel continuous delivery drive the DevOps engine.

    I alluded to this point in a post in early 2015. I stated, “Everything is continuous. Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change. Continuous security results in continuous incident response … or vice versa. To top it off, all of the continuous activities continuously feed each other to drive more continuousness in some sort of DevOps Mobius strip of continuity.”

    That post last year was a bit tongue in cheek. I have since refined my position, however, and come to the conclusion that the hint of sarcasm was perhaps unwarranted. That continuous continuousness is the crux of the culture shift. It is the thing that separates DevOps from non-DevOps.

    On the Continuous Journey

    Speaking on the topic of continuous security, Andrew Storms, vice president of Security Services at New Context, explains, “Smaller incremental changes are always easier to handle than a large behemoth of a release that can include hundreds of changes. Try finding the single change in a monthly release that resulted in an unexpected security hole in your database configuration vs the release with only two changes that was just developed and released in the last few hours. Security practitioners cannot continue to rely on manual, human-controlled processes to defend against attacks. Automation is a requirement for the next evolutionary phase of security protection.”

    That sentiment extends beyond security to all of IT. When we talk about the culture shift in IT required for DevOps, we are generally talking about breaking down silos and removing the barriers between different teams. The goal of the culture shift is to shed tedious bureaucracy and red tape and empower employees to freely collaborate and cooperate to get things done. In other words, we want to eliminate barriers and roadblocks so productivity can be more continuous.

    Assuming the DevOps transformation begins at that level, it creates an environment of continuous collaboration which becomes continuous development. What follows is a natural evolution of continuity. Continuous development becomes continuous delivery, which requires continuous testing and continuous deployments, followed by continuous monitoring, which provides the metrics and feedback that take you back to the beginning of the loop.

    All of the “continuous” tools and solutions may seem to be part of the technologies and practices that come after the DevOps culture shift, and to some extent, that is true. However, they are also indispensable and inseparable from that DevOps culture shift because without all of the various gears meshing together smoothly—and continuously—what you’re left with is a more traditional approach to IT where everything comes to a halt.

    Is DevOps more of a cultural shift than a collection of tools? Absolutely. That culture shift revolves around continuity, though. Continuousness is the culture shift that drives DevOps.

  • My picks for opening day of DevOpsDays Austin

    My picks for opening day of DevOpsDays Austin

    As we wrote earlier, we are happy to be attending and sponsoring DevOpsDays Austin 2015.  The opening day morning is chocked full of great sessions with the afternoon dedicated to openspaces as is the norm.  While everything from breakfast to Damon Edwards keynote through JJ Ashgar’s session on Chef before lunch look tasty.  Here are three sessions that I will be attending that I am really looking forward to:

    michael coteThe Coming Donkey Apocalypse by Michael Cote’ –  Cote’ formally of 451 Research and now with Pivotal is going to talk about the “state of the union” of Devops. From the abstract:

    DevOps is growing in popularity and even usage in “the real world.” It seems like we’re slowly getting past unicorns only and seeing many “horses” do DevOps. Soon we’ll see the mainstream market – the “donkeys” – start to pick it up and bend DevOps to its will. This talk will go over “the state of the union” of DevOps and provide some guidance for how to prepare for the donkey apocalypse.

    I don’t know what to make of this title but it peaked my curiosity. I don’t think DevOps adoption moving mainstream is equal to moving from horses to donkeys and so it is not a very flattering title. On top of this not only don’t I think main stream adoption of DevOps means it is Donkeys adopting it, I do not think it means an apocalypse for DevOps.  In fact I think it is a good thing. On top of this, I think the idea of calling mainstream adoption, adoption of the  donkeys and them causing an apocalypse smacks of elitism and is against what DevOps is about. I don’t think it is a bad thing that more organizations will get involved and that they will “bend DevOps”. I don’t know Michael well and have only spoken to him once briefly. Everything I hear though is that he is a very bright guy and dynamic speaker and writer.  So i am interested to hear what Michael has to say in this session and whether he thinks it is a good thing, bad thing or how we can help move it along. You can’t judge a book by its cover and you sometimes you can’t judge a session by its title and abstract.

    jpaulreedFixing your Organization by Continually Breaking It by J Paul Reed – Reed is a consultant and has over a decade of experience in the trenches as a build/release and tools engineer, working with such organizations as VMware, Mozilla, and Symantec. In 2012, he founded Release Engineering Approaches, a consultancy incorporating a host of tools and techniques to help organizations “Simply Ship. Every time.” From the abstract:

    When we discuss DevOps, we often find ourselves mired in the “tools vs. culture” debate. A point often forgotten is the “amplify feedback loops” of DevOps “3 Ways” and how to build that into our cultures. In this talk, we’ll look at the importance of continually breaking your organization with experiments, some surprising examples, and how to do it in such a way that you aren’t left with a broken organization.

    I love this concept going back to Netflix and the Simian Army and have heard similar talks before on it. Looking forward to what Reed has to say and learning. Amplifying feedback loops is something I would like to see more written about as well.  I am looking forward to this one!

    josh cormanContinuous Acceleration: Why Continuous Everything Needs a Supply Chain Approach by Josh Corman – Josh is of course the CTO of Sonatype and a former 451 Research analyst too. Besides all that Josh is a friend from the security world that I know a long time. This talk sounds like a continuation of what Josh and Gene Kim presented at DevOps Connect at RSA Conference a few weeks ago. Also Ericka Chickowski wrote a good article on her interview with Josh on this last week.  Here is the abstract:

    Today we write less code & consume more re-usable open source code. Innovation accelerates & complexity grows. Complexity is the enemy of quality. Poor quality creates break-fixes & slows dev. It’s a continuous loop.
    Learn about software supply chain management strategies to accelerate delivery: scrutinize your ‘suppliers,’ minimize risk & bloat, and improve traceability, visibility & response times. 

    Josh is a great presenter. I have seen him present many topics over the years at several events, I am sure this one won’t disappoint.

    There are other good presentations including some stuff on Dockers and containers (what would a conference be without them?). Additionally the I am sure the open spaces will spawn some good stuff as well.

    I will try to update throughout the day and give those who can’t attend a sense of what is going on.

    I realize many of you can’t make Austin tomorrow, but if you are going and see me please introduce yourself and say hi.

  • The continuous continuousness of DevOps

    The continuous continuousness of DevOps

    Defining DevOps is getting easier, but it’s still an elusive goal. Part of the challenge is that DevOps is more of a culture than a specific “thing” and how that culture exists is subjective and varies from one organization to the next. DevOps is what you want or need DevOps to be to some extent, but one thing seems to be consistent no matter how you define DevOps—it’s continuous.

    If DevOps is the buzzword du jour for tech, then continuous is the buzzword du jour of DevOps. Everything is continuous. Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change. Continuous security results in continuous incident response…or vice versa. To top it off, all of the continuous activities continuously feed each other to drive more continuousness in some sort of DevOps Mobius strip of continuity.

    What does it all mean? I mean, aside from being a catchy buzzword and making that new DevOps tool or service sound sexy and impressive?

    In a nutshell it means there’s no end. You don’t really reach a point where everyone can shut down their PCs and grab a beer to celebrate the completion of the project. The project—whatever it is—is more of a living, evolving thing than a deliverable.

    There’s also no beginning—or at least no new beginning. All of the continuous activities cycle back over each other like a Spirograph drawing. Have you ever tried to find the beginning or end of a Spirograph drawing? Good luck.

    DevOps is the Alpha and the Omega, the beginning and the end, the first and the last.

    When you strip out the buzzwords what it really means is working in a more effective and efficient way. I’ve worked on software development projects the old-fashioned way. No, not Agile—Waterfall. The cascading tasks feed each other in what appears to be a logical progression. It makes sense to define requirements, design the software, implement it, test or verify it, and then kick back and go into maintenance mode…on paper at least.

    In the real world new requirements are discovered when you’re ready to implement, and testing results in new design elements. Those new requirements need to be designed and the new design element needs to be implemented and it all needs to be tested and maintained which inevitably creates new requirements, design elements, and so on.

    Waterfall was already DevOps—it was just dysfunctional because each “continuous” loop was treated as a unique and separate project. Items that were identified after the design phase were just accumulated on a list to be incorporated into the next phase or iteration. The net result was producing a somewhat stable product that you already knew was flawed, but that you’d get to fixing eventually.

    Agile development accelerated some of the feedback loops of Waterfall and resulted in faster refinements to projects, but it still had issues. The primary issue is that developers can’t go faster by themselves. Developers are just one cog in the machine and producing software faster just creates a bottleneck if operations isn’t prepared and the infrastructure isn’t ready to handle it.

    Chris Riley explained the challenge of Agile. “However, the Ops department was inherently left behind with deployments piling up faster than they could be released and customers never actually received the value they demanded. This trend ultimately gave rise to DevOps, a new-age version of Agile methodology encompassing the Dev and Ops segment to enable service delivery agility and quality.”

    Another way of looking at it is that DevOps is an evolution of Agile. I wrote recently, “DevOps is just a natural extension of Agile. It takes the guiding principles and best practices that Agile brought to developers and applies them on a much broader scale across the organization.”

    So now everything is continuous. There is no top of the waterfall that cascades through multiple layers until you reach the bottom of the waterfall. Now there’s a continuous culture of continuity. It may sound confusing when viewed through all of the various buzzwords, but when you step back it makes sense. It makes sense because that’s really the way it worked—or should have worked—all along.

    DevOps gets processes and policies out of the way to enable people to get things done more efficiently. For some reason we’ve decided that every element should be labeled continuous—which actually makes it sound a little frenetic or overwhelming. Maybe we should use “efficient” instead. That can be our new buzzword.