Category: DevOps Practice

  • 7 (More) Security Considerations for CI/CD

    7 (More) Security Considerations for CI/CD

    CI/CD orchestration has become increasingly popular in DevOps practices. It facilitates faster deployment of software and product delivery. Through the use of automation when building, testing and deploying applications, CI/CD bridges the gap between development, operation activities and teams. 

    However, there are many security considerations to keep in mind with frequent deployment. From quality automation to code history analysis, security must be baked into your CI/CD pipeline to ensure your product and processes remain safe from hackers.

    To gain insights on the top security considerations for CI/CD, I asked the speakers and sponsors for the upcoming SKILup Day as well as several DevOps Institute Ambassadors for their thoughts. In this two-part series, I will be sharing their insights. Did you miss part one? You can read it here. Here’s what they had to say in part two of this series:

    Speaker, Ravi Lachhman, Evangelist, Harness

    “CI/CD platforms are core to an organization’s engineering efficiency and typically have elevated privileges to sensitive environments such as production. Since CI/CD platforms typically will be deploying, this makes sense. Delving into the SolarWinds hack with binary artifact signing, etc., can separate security concerns into two parts. One concern would be the security around the actual CI/CD platform and one around the DevSecOps goals inside a CI/CD pipeline. With so many items shifting left, having proper role-based access control (RBAC) is crucial. With the separation of duties for regulated environments and the need for engineers to be efficient, controls that ensure only certain entities can deploy and certain entities can view pipelines is important. Around the DevSecOps movement, container/OSS scanning is still top of mind for many firms.”

    Sponsor, Basis Technologies, Eran Kriegshauser, Basis Technologies’ SAP DevOps Architect

    “The question you should be asking is ‘How do I maintain quality with the frequent deployment tempo demanded by business today?’ We need automation of quality and guaranteed rigor. For example, take promoting security configuration changes in systems, applications and products (SAP). Today, this is done by SAP transports where the change is created in an SAP development system, then imported into QA and, ultimately, into production. How do you ensure that a developer can’t give themselves extra security access through a role modification? Automated analysis and approval workflows must support segregation of duties requirements. Users can’t approve changes they create–the appropriate CI/CD process will not allow it.

    Another area is automatic peer review notification on changes to sensitive enterprise resource planning (ERP) objects. Once identified, the promotion of those object changes are halted until approved by a team lead or change and release manager.”

    Savinder Puri, DevOps Evangelist, Zensar Technologies

    “Infuse as many aspects of security as possible into your CI pipeline. CD is already too late for security—you’ve already deployed the application into an environment.

    Start from the basic static code analysis, security scanners, etc., and then keep maturing. Whatever tools you integrate into CI Server, ensure the corresponding integrated development environment (IDE) plugins are made available to developers. That way, devs can do the first-level validation within their IDE itself, and do it as they code, even before compiling. That’s the practical implementation of the often overused term ‘shift left.’”

    Vishnu Vasudevan, Head of Product Engineering and Development, Opsera

    “Now with cloud migration, emerging technologies and open source technology, companies will have to make sure they have the right tooling to measure and take action based on the security challenges born out of different development activities. 

    The focus will need to be on managing the risk that is tied to the business criticality. Each industry, domain, technology, etc., all have their own independent security goals, and you cannot have one security goal across the enterprise for different technologies and different domains. It is important to evaluate the technology being used and weigh the security requirements accordingly.”

    Stephen Walters, Sales Engineer, Everbridge

    “Inclusion! This is why the term DevSecOps came to be—because, erroneously, security was not included as part of a DevOps transformation, and the same is true for transformation of an SDLC to incorporate security.

    Security has still been treated as a non-functional requirement; an afterthought. If there is one thing we should learn, there is no such thing as a non-functional requirement. Security needs must be considered on an equal footing with all other requirements and built into the design, test, deployment and delivery of any feature.”

    Mark Peters, Technical Lead, Novetta

    “CI/CD pipelines should implement security at every step.  Tests can be automated in tooling to conduct tests of various functions at multiple levels for code accuracy, unit tests through fuzzing and integration or vulnerability testing. Most vulnerability testing in a pipeline is actually patch management, ensuring all commercial and proprietary holes are filled for release software. The first step should be using coding checks to make sure there are not any common problems with the code. While this may seem easy, if you are not managing that basic step, you cannot move ahead to the more complicated areas.”

    Pawel Piwosz, Lead Systems Engineer, EPAM Systems

    “At a high level, I would say that security is still outside the main scope of CI/CD. We should consider a few areas and different angles, here. Security should be not only implemented by CI/CD; CI/CD should be implemented in a secure way. Everything-as-code is a solution here: policy-as-code, security-as-code, drift-as-code, and if we talk about infrastructure, remediation-as-code.”

    If you’d like to learn more about the latest in CI/CD and similar topics, check out the upcoming SKILup Day schedule. Or, learn more about the benefits of becoming a DevOps Institute Member. 

  • A New DevOps Database Platform is Needed

    A New DevOps Database Platform is Needed

    The diverse number of database platforms, deployment environments and data variants needed to support the plethora of applications and services required for rapid DevOps continuous delivery lead times, coupled with the challenges of integrating database changes for application pipelines, is a growing challenge for many organizations as they strive to accelerate their DevOps transformations.

    When you stand back and look at the bigger picture, an enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. At best, it makes things chaotic; at worst, it is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

    In my book, Engineering DevOps, I explained that databases offer several challenges for CI/CD pipelines.

    In the simplest case, when a database is used solely by one application or microservice, the database pipeline can be integrated with the application pipeline to ensure its changes use the same version management system and change controls to ensure the database changes are kept in sync with the application. However, many database platforms are not as responsive to changes as the applications themselves and become a bottleneck for the pipeline.

    If a database is used by several applications, each with its own DevOps pipeline, then changes to the database must be coordinated with all applications that depend on it. Maintaining a separate database pipeline linked to the application pipelines through CI/CD toolchains, as a strategy, is overly complicated and can often introduce additional bottlenecks and synchronization issues.

    Another challenge with database pipelines is the management of data in the database during the CI and CD stages. Data in production needs to be persistent and is ever-changing, even while the underlying database code is changing. During the CI process, sample data from production needs to be used to ensure testing is realistic, and new test data must be introduced to verify new database capabilities are working.

    During the CD stages, rollbacks to prior versions of database code must avoid rolling back to versions that are no longer compatible with production data. This is a key reason why database changes must follow a disciplined approach that fully tests database changes with applications prior to full deployment. Rolling deployments, in which applications and database changes are deployed gradually in separate clusters, can minimize the blast radius in case of problems. The required coordination of separate systems adds complexity and bottlenecks.

    So, why not simply integrate database changes as artifacts within the same DevOps pipeline as the application?

    Gilad David Maayan’s September 2020 article DevOps Database Strategies and Challenges: Automation, Scalability, and More explained some of the specific challenges that arise with integration of traditional databases with DevOps.

    Database tools were not designed for integration with DevOps tools. These tools tend to be database- and environment-specific, which can make it difficult to incorporate into pipelines designed for flexibility.

    Database performance and ease of management often do not perform well in DevOps environments where multiple applications need to use the database in a short time, such as in the accelerated lead times for DevOps continuous delivery. This is especially true with larger-scale databases. If an application change must sync with a database schema, changing the pipeline can become complicated especially if there is a need to sync across multiple application pipelines to meet continuous delivery goals.

    While DevOps processes are proven for continuous delivery of applications, incorporating database changes into the DevOps pipeline is hindered by large data sets, limited database automation and security concerns. As a result, database development and testing continue to rely on outdated processes, with fixed and often shared instances refreshed from backups. A new DevOps Database platform is needed to provide highly dynamic and efficient cloud-native database solutions for modern enterprise digital transformations, and that platform must include DataOps use cases for DevOps, DevSecOps, continuous testing and governance. This new kind of data repository must be designed to integrate seamlessly with DevOps CI/CD pipelines through flexible APIs and cloud-native architectures.

    Given the points above, traditional database solutions and parallel data pipeline strategies are not sufficient to meet the challenges of enterprise DevOps transformations. What is needed is a new kind of DevOps database platform that allows the database, and database changes, to be as agile as any other artifact in the DevOps pipeline.

    Paul Stanton, VP of product management at Windocks, in his June 2021 YouTube video Windock DevOps Data Repo, said, “Imagine terabyte class database environments managed as DevOps artifacts with entire versioned databases available in seconds.” That’s what we need.

    The following are examples of the capabilities required for such a new DevOps database platform.

    • Automated database provisioning
    • Distributed repositories support
    • Simple declarative builds
    • Dynamically scalable
    • Virtual hard drive images for a wide variety of DBs – Linux, SQL, Oracle and others
    • Distributed versioned database environments working on standard file systems and storage systems delivery writeable data at the speed of DevOps – in other words, in seconds.
    • Deploys equally to all operating systems and cloud environments, Docker containers and Kubernetes.
    • Many clones per virtual machine (VM)
    • Security features including access management, data masking and encryption
    • Compatible with and easily integrates with CI/CD platforms
    • Compatible with multi-cloud (AWS, Azure, GCS)
    • Built-in data security and governance into DevOps data.

    I will explain these more completely in a subsequent article.

    What This Means

    Digital transformations are imperative for modern enterprises and organizations if they are to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many database platforms and tools that do not integrate well and that bottleneck continuous delivery goals. A new kind of DevOps database platform, in which the database is treated as any other artifact, is needed to resolve technical problems with database integration to DevOps application pipelines. A new DevOps database platform solution is needed to accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations.

  • BMC Extends DevOps Reach from the Mainframe

    BMC Extends DevOps Reach from the Mainframe

    BMC today announced it has extended its DevOps platform for mainframe environments to provide integrations with GitHub Actions, a framework for managing workflows, and HCL Launch, a continuous delivery platform from HCL Software. BMC gained the DevOps platform with the acquisition of Compuware in June 2020.

    John McKenny, senior vice president and general manager for intelligent Z optimization and transformation at BMC, said those integrations—along with existing support for DevOps platforms such as Jenkins, GitLab and VS Code—make it possible to accelerate the development of applications that include mainframes as part of a larger distributed computing environment. DevOps teams, for example, can now employ an HCL Launch plug-in for the BMC AMI DevOps for Db2 tool to automatically capture and propagate database changes across IBM Db2 databases.

    At the same time, BMC also enhanced a BMC AMI Ops Monitor for Java Environments that automatically discovers all Java virtual machines (JVMs) running on z/OS environments on a mainframe. The BMC AMI Ops Monitor for Java Environments now includes the ability to discover the application programming interfaces (APIs) and services invoked by those JVMs.

    BMC also revealed today that its Workflow WiZard tool has been accepted as part of the open source Zowe initiative launched by the Open Mainframe Project. The Workflow WiZard makes it easier to install, configure and maintain software installed on mainframes running z/OS.

    Finally, BMC revealed it has extended its privileged access management (PAM) tools by enhancing the supervisor call screener and Unix System Service (USS) data enrichment capabilities of BMC AMI Security software. Based on technology BMC gained with the acquisition of RSM Partners, this enhancement increases the level of protection BMC AMI Security software provides to thwart attacks by surfacing suspicious activity, such as from malicious insiders that may have compromised administrator credentials, said McKenny.

    BMC mainframe

    These days, McKenny noted, everything is under attack because the minute cybercriminals discover a way to compromise one platform, they almost immediately try to apply that same technique against other platforms. Mainframe environments, which tend to process an organization’s most valuable data, naturally are a high-value target, added McKenny.

    In general, McKenny said IT organizations that rely on mainframes continue to make steady progress toward embracing DevOps best practices. A recent survey conducted by Forrester on behalf of BMC found 80% of respondents said their mainframe development tools needed significant improvement to provide more value.

    The need to accelerate digital transformation initiatives, which typically require access to data that resides on mainframes, are, in many cases, forcing the DevOps issue within many organizations. Software development teams that build mainframe applications are almost always required to create APIs that make it easier for other applications to consume that data.

    Not every organization, of course, has a mainframe. However, as digital business transformation continues to evolve, the number of applications that need to consume mainframe data will steadily increase. The challenge facing mainframe application management teams is finding a way to provide that access without using traditional waterfall processes to build the required APIs.

  • Big Gains Are Made In The ‘Last Mile’ of DevOps

    Big Gains Are Made In The ‘Last Mile’ of DevOps

    A new study by mabl found a correlation between complete, fully automated test coverage and a significant increase in customer happiness and DevOps maturity. In contrast, those with moderate DevOps maturity and nearly automated testing coverage tend to experience a sharp decline in end-user satisfaction.

    Mabl recently released its 2021 State of Testing in DevOps Report, a study that highlights the state of DevOps adoption and quality testing throughout the software industry. According to Dan Belcher, co-founder, mabl, the study results clearly show that true business returns are made in the last mile of DevOps adoption.

    I recently met with Belcher to hear his viewpoint on the major takeaways. Below, we’ll explore these and other insights from the 2021 State of Testing in DevOps Report.

    State of DevOps Maturity

    Nearly one in four teams have a mostly mature DevOps methodology and automated process. Still, a significant portion, 52%, are still aspiring or striving to fully embrace a mature DevOps model. A small percentage, 11%, have a fully automated DevOps software development pipeline in place. The bulk of teams are still using manual deployment processes and disparate frameworks, limiting the potential for full automation.

    Interestingly, non-technical barriers remain the most common inhibitor to realizing a completely automated development pipeline, found the report. Out of these obstacles, reluctance to change was the most common barrier, cited by 29% of respondents. Other challenges to DevOps adoption include budgetary constraints, DevOps not being a company priority and internal politics.

    When added up, non-technical roadblocks account for 82% of the problems that hinder complete DevOps adoption. On the one hand, this depicts the cultural challenges associated with introducing DevOps in an organization. On the other, it could signify the wealth of DevOps packages and open source tooling available to manage areas like automated deployment and version control. The report found GitHub, Jenkins, Gitlab, Azure Pipelines and BitBucket among the five most commonly used DevOps Tools.

    Although it’s growing rapidly, continuous delivery is still an emerging trend; almost half of developers still deploy code less than once a month. Only 15% of teams are deploying code daily or multiple times per day. This echoes findings from CDF’s State of Continuous Delivery Report, which pegged this figure at 11%.

    Fully Automated Testing

    Testing is a critical component of the software life cycle. With the need to ensure functional applications and high-grade security, automated testing is helping teams discover more errors earlier in the development process. According to the report, the most popular test types include UI or functional tests, unit tests, regression tests, integration tests and end-to-end tests. Non-performance tests and visual testing ranked lower. Naturally, the teams that have implemented fully automated pipelines are automating more types of these tests.

    In general, it appears test automation aligns with DevOps maturity. The report found that those with fully automated testing experienced significant net customer experience gains. For example, 72% of teams that were automating regression testing indicated their customer happiness is pretty good or amazing, whereas this figure sank to 6% at companies not automating regression testing. And 80% of teams with high test coverage report high customer happiness, whereas only 30% of teams with low test coverage report high customer happiness. Nonetheless, much has yet to be fully automated—similar to the state of continuous deployment, the bulk of teams rely on a combination of automatic and manual testing approaches.

    In terms of tooling preferences, Selenium ranked as the most popular option for functional testing, with 41% of developers using it. The report also found migration toward Cypress, Smartbear and other functional testing tools. Interestingly, the report also uncovered a noticeable lack of adoption when it came to automated API testing—only about one out of three are automating their API tests. Postman, SoapUI, internal tools and ReadyAPI are among the most popular tools for running manual API tests.

    As I’ve covered before, APIs are surging in use across digital transformation initiatives. Web APIs are an increasingly common mechanism for delivering data to partners and consuming third-party software-as-a-service (SaaS). Full API testing coverage will be essential to audit the entire end-to-end digital experience. Low-code API testing is one way to reduce barriers to auditing the highly technical area of integration.

    Key Takeaways

    DevOps success is a ‘last mile’ problem. There is much variance in DevOps maturity and testing coverage. It’s pretty flat until you reach that last point, when breakthrough results occur. “Success in DevOps is a ‘last mile’ problem,” said Belcher. “The difference between 90% and 100% of the way there is dramatic.”

    Quality really matters. Though much tech discussion focuses on increasing delivery speed, code quality is just as important. “This isn’t about speed at all costs, it’s about innovation with velocity and quality,” said Belcher. “Quality engineers are just about as key to success as developers writing code.”

    DevOps success is less about technology and more about people. As the report found, the majority of the roadblocks to DevOps maturity are non-technical. “The things holding us back, as an industry, are more about leadership and figuring out our processing than introducing more technology,” said Belcher.

    Breaking down role silos is still fundamental to DevOps. Some see DevOps as instrumental to opening collaboration between roles. Simultaneously, there is the idea that new tooling dashboards will arm individual developers with complete ownership to develop, test and deploy software (known as full cycle development).

    Belcher believes that the full cycle momentum has been misinterpreted to mean developers alone have full accountability. Instead, he frames developers as a pitcher on a baseball team; leading the game, but utterly incapable without the rest of the team. Similarly, disparate roles should be working together, said Belcher. Reaching that ‘last mile’ of DevOps is not only about CI/CD or full cycle development; it’s about working together as a group.

    I’ve only scratched the surface of many findings from The State of Testing in DevOps. The report queried 600 quality engineers and software development professionals and offers loads of additional insights on the state of DevOps, testing, CI/CD and more. You can view the full results here.

  • Bringing Continuous Testing Into the Software Delivery Pipeline

    Bringing Continuous Testing Into the Software Delivery Pipeline

    For any highly competitive industry that relies on a smooth and stable user experience, it is vital that their development teams accelerate release cycles, lower operating costs and ultimately reduce risks. Continuous testing is one way companies can achieve these objectives.

    As a subset of software testing, continuous testing (CT) is valuable because it uncovers and addresses different problems at various stages of the development journey that other methods don’t. Most other testing methodologies require a large staff; CT doesn’t. CT leverages in-sprint automation, including end-to-end tests across the front and back end of the technology to cut costs and facilitate rapid delivery without adding more pressure on internal teams. Moreover, as part of a larger software delivery pipeline, CT is positioned closer to the beginning to execute tests early, often and comprehensively to realize cohesive release goals.

    CT introduces quality assurance with constant iterations of testing and actionable feedback at each stage of development to evaluate business risk coverage and ensure the best user experience and customer satisfaction. As the name implies, CT helps developers and product managers detect and fix critical bugs not only in the early stages of development, but continuously throughout the software development life cycle (SDLC), ensuring that errors in the code do not get unintentionally released into the wild.

    As software companies accelerate their release frequency, continuous testing sets a precedent for digital tools development and the quality assurance process.

    Benefits of Continuous Testing

    Continuous testing plays an integral role in the CI/CD pipeline, is part of the overall set of DevOps best practices and the information it provides is invaluable to development teams. The benefits of CT are also far-reaching, with positive impacts on the entire organization, including:

    Better Company-Wide Feedback

    Continuous testing facilitates feedback to all involved teams and stakeholders. For example, when a bug gets detected in code, ideally, the only team notified should be the team responsible for the part of the code with the bug. Without CT to automatically detect bugs and notify those responsible for fixing errors, triaging bugs could be a full team effort or, worse, a dedicated team’s main function. This approach is very inefficient and wastes the time of the engineers whose code is bug-free. The feedback provided from CT cuts down on these unproductive scenarios and allows stakeholders to make immediate and informed decisions.

    Higher Quality Code and User Experience

    A company that effectively uses CT should expect higher quality code and a more stable user experience. Stability measures application health and user experience; it is negatively affected by session-ending events like crashes. With CT, businesses will give their development teams better awareness of product quality and performance while also significantly improving overall business metrics by fixing broken apps faster.

    CT Breaks Down Siloed Teams

    Not only will CT enhance quality assurance, it similarly works to improve the development of digital tools through the breakdown of siloed teams. When teams become siloed, they inadvertently keep processes and knowledge to themselves, resist cross-functional collaboration and work in isolation toward their own goals. Predictably, separate groups fail to share insights or ideas, leading to stagnation and decreasing innovation. However, by leveraging CT, siloed teams can become more collaborative in development, testing and operations, increasing overall business efficiency.

    CT Enables Objective Evaluation

    Finally, CT enables an organization to pause and objectively assess if it is aligning efforts with its specific business goals. As mentioned before, CT uses automation to provide timely feedback on business risks, reducing the chances of a digital application or tool failing at launch. Likewise, dev teams and managers can more easily meet their business goals and optimize the value of a release. Additionally, objective assessments are helpful for critical decisions – as CT gives actionable data for decision-makers to support their choices.

    Today, consumers have come to expect error-free experiences within their software applications, with the most stable of these applications driving conversion, engagement and retention. Applying a continuous testing model to a software development pipeline will ultimately result in applications being delivered faster and at higher levels of quality than ever before, with an enhanced end-user experience. As companies combine agile development with continuous testing, they will optimize each aspect of the development life cycle and their larger business goals.

  • Lessons Learned on a Low-Code Journey

    Lessons Learned on a Low-Code Journey

    The popularity of low-code/no-code tools are on the rise as organizations chase the promise of creating ‘citizen developers.’ The idea of using these tools to empower non-technical developers and/or to free development teams from writing code line by line is certainly compelling.

    About a year ago, a friend reached out to ask if I would critically review a book he was writing about citizen development. At the time, the expression was new to me but after reading and reviewing the book, I was intrigued. It challenged me to embark on my own citizen developer discovery. I want to share several applicable lessons I learned along the way.

    Intrigued: Citizen Developer Tools Versus Dev Teams

    The curious part of me was trying to understand if these were tools that would remove the need for expensive development teams. The previous, well-intentioned attempts had fallen short of their promise—think Computer Aided Software Engineering (CASE) from the 1990s as an example.

    Around this time, I discussed the topic with the CTO of a progressive client. His business users would not wait for technical resources when they needed apps to address persistent problems. He was quick to point out  that low-code was preferable to shadow IT. The architecture for these low-code/no-code tools meant database administration was in his team’s hands. He was supportive of business units building small tactical pieces of functionality with a clear business case, and he supported them where possible through training and allocating resources to their projects for data integrations and other functions.

    The Challenge: Write an App

    Encouraged, I decided to challenge myself over one weekend to see if I could write a commercial application. I had studied programming languages like Fortran and C in the 1980s, but I’d never worked as a developer and had never written commercial code.

    Start of Day One

    I selected a low-code tool at random. I would recommend, however, that a citizen developer engage the IT team rather than choosing at random, as most low-code/no-code tools have an underlying programming language. If your organization is a Java shop, for example, then it would make sense to pick a tool that has the same language—in this case, Java—at its source.

    I purchased a six hour training course on YouTube for $20. This would prove to be a wise decision. Learning the tool I selected was not straightforward, but the training course allowed me to pause the tutorials, rewind and repeat to ensure I was understanding concepts and could follow along with the ‘how to.’

    First task: write ‘Hello World.’ I recommend that citizen developers start here. If you can do this, then it’s more than likely you can build something more complicated.

    Three hours later (of which two were a system error) … it was done.

    low-code

    Clicking on the green button produced the screen below.

    low-code

    At this point, I still had more questions than answers. The GUI for the development tool was point-and-click within a Windows directory structure. There was no command line editor. I was even more encouraged. Could it be this straightforward?

    A note: while it took less than an hour to complete the ‘Hello World’ task, I encountered a Windows system error which pointed to a server port error. The vendor’s help desk didn’t operate over the weekend, but their documentation was adequate; it took a further two hours of effort to fix the problem, which arose because my Windows Surface Pro was both a client and a server.

    Would an ‘average’ citizen developer have been able to overcome this problem? I’m not sure. I recommend when trying low-code/no-code tools that you also trial the help desk, even if you need to pay for it. While many software tool companies try to force software engineers to work through wikis to solve problems, citizen developers may need more hand-holding.

    The next step was to build an application. Most non-technical people tend to describe applications from a graphical perspective and as business functions. This is very much in keeping with how I worked for a further two hours, at which point the ‘Home’ web page looked like this:

    low-code home page

    Using data from our in-house spreadsheets, I created the data model below with relative ease. I believe a citizen developer would benefit from asking an in-house database expert to review this ‘design’ to save time later when maintaining the application.

    low-code design

    This work was undertaken at the same time as I was building a second tier of pages to amend data. So, for example, the automatically generated customer page which allows a user to search, add, edit and delete customer records looked like this:

    low-code edit page

    This was completed in four hours. Once the table was created with the correct columns, the creation of the web pages was automatic. In fact, it became repetitive and I found it was easy to lose my concentration and make errors.

    I was learning to save every small change to ensure that the live web and mobile application builds were working. A professional developer would see this as a form of continuous integration for the citizen developer.

    I don’t recommend having teams of citizen developers working together, at least at the outset, as it introduces branching and versioning and the concept of ‘breaking a build.’ Professional developers would need to be consulted at this juncture.

    End of Day One

    I had a real sense of achievement at the end of day one and was eagerly anticipating day two. Citizen developers should note that, at this point, I had produced what developers would call a prototype. It would be possible, upon reaching this point, to approach IT to assign a developer to finish the job. If IT does not have capacity, then keep going.

    Start of Day Two

    Having designed screens and a database, the next task was to add business logic.

    low-code logic

    Creating flows of work with rules was a more engaging task, but still accomplished via drag-and-drop or point-and-click. Boolean logic operators like if- then- else allowed me to quickly build logic and attach it to visual elements on the webpage.

    low-code flow

    This is an event registration flow where data is accepted from a web page and, based on a true or false response, treated differently depending on that response.

    As before, the first workflow took some time to learn. There were many ways to both trigger an action and direct the subsequent work. After a further half day, I had produced enough working code that I felt I probably couldn’t learn a whole lot more and the application was roadworthy.

    low-code mobile app

    From there, creating the mobile version of the app was as simple as pushing a button. At the outset, I had set a switch for mobile deployment. The web and mobile editions were always in sync, and the GUI was adjusting automagically for the deployment platform. Citizen developers should check on in-house standards for web and mobile app deployment configurations and ensure their selected tool supports them.

    End Day Two

    Lessons from My Discovery

    Organizations that want to adopt low-code or no-code tools will need to ensure IT support and training for business users that want to self-service their app dev needs. I recommend early engagement and frequent retrospectives with IT colleagues.

    Business functions should consider ‘easy’ and tactical opportunities for piloting.

    Allocate an uninterrupted period of time to focus on the task. Learning a new tool over a weekend was a really good idea, too. There was zero context switching, apart from my wife dropping in with coffee for a chat.

    I recommend citizen developers try to be a little bit Agile for these engagements. Using a ‘Plan, Do, Check, Act’ methodology means that you are not ‘all in’ on this—you are simply taking small discrete steps to see if you can solve problems more efficiently than you would if you instead waited for IT.

    Professional developers should ensure that any low-code/no-code tools produce code that can be imported into their standard IDE and that the citizen developer’s code is not locked into the tool.


    As a DevOps Institute Ambassador, O’Reilly helps empower DevOps Institute community members with the SKIL framework to advance the Humans of DevOps with skills, knowledge, ideas and learning. Learn more about O’Reilly’s work in DevOps here.

  • Survey Finds Biggest DevOps Obstacles are Organizational

    Survey Finds Biggest DevOps Obstacles are Organizational

    A survey of 600 IT professionals published this week by mabl, a provider of a test automation platform, finds the biggest barriers to adopting DevOps practices are more closely related to organizational issues rather than any technology.

    The survey identified non-technological issues as the main inhibitor to DevOps by a wide margin (82%). Only 11% of respondents said their organizations has fully implemented DevOps practices, while another 24% said they were most of the way toward achieving that goal.

    The top two barriers to adoption are slow processes and speed of adaptation (29%), followed by budget and funding (21%). Only 18% identified technology limitations as an issue.

    Nearly half of respondents (46%) said they have been able to implement continuous integration, while more than one-third (36%) said they have also been able to achieve continuous delivery. Just over a quarter (27%) said they have also achieved continuous deployment. However, only a third of respondents said they deploy new code at least once per week. Almost half still deploy less than once per month. The number of teams that deploy new code daily or multiple times a day (15%) is roughly equivalent to those that deploy code on a quarterly basis (16%).

    The most widely used DevOps tools are GitHub (34%) and Jenkins (33%), followed by GitLab (21%), Azure Pipelines (20%) and Bitbucket (19%).

    In terms of testing, the most widely automated tests are user interface (UI)/functional and unit tests at 42% each, followed by regression at 41% and integration at 35%, respectively. The most widely used functional testing tool is Selenium (41%), followed by Cypress at a distant second place (14%). The most widely employed tool for testing application programming interfaces (APIs) is Postman, followed by SoapUI (21%) and homegrown tools (17%).

    Dan Belcher, co-founder of mabl, said that while it’s clear most organizations still have a long way to go before they fully embrace DevOps best practices, it’s now more a question of degree rather than if. As organizations realize the extent to which they are dependent on software today, the focus on both quality and ensuring timely delivery will increase. The stakes involved in software delivery these days have never been higher, noted Belcher.

    Nevertheless, the survey makes it clear how resistant to change organizations are. Some organizations have been building software the same way for decades. The only thing more difficult than convincing IT and business leaders there is a fundamental need for change is driving that change all the way down to individual developers and IT operations teams.

    Competitive pressure, however, will inevitably force the issue. If rivals are delivering better application experiences that create a business advantage within the context of a digital business process, an organization will have to adapt. Otherwise, their fate will be, over time, to fade away or be acquired. Equating DevOps with that existential threat may be a challenge, but unless that connection is made it’s only a matter of time before organizations that aren’t embracing DevOps best practices will slide toward irrelevance.

  • Leveraging the Cloud for Mainframe Access

    Leveraging the Cloud for Mainframe Access

    Now more than ever, organizations around the world are looking to extend their existing mainframe investments as part of larger digital transformation initiatives. As core systems of record and reliability, mainframes continue to run many organizations’ most important applications and the idea of replacing these systems comes with greater risk than reward. While organizations are considering modernization of their mainframe investments, it’s important to remember that maintaining mainframe access and offering new ways for their users to work with the data and systems is a vital part of the larger project.

    Leveraging the Cloud for Mainframe Access

    With the recent increase in remote work, it’s no surprise that more companies are looking to leverage the cloud for mainframe access. This has many benefits from increasing scalability, availability and performance, to reducing application costs and improving security. Implementing modern mainframe access as part of a larger cloud project ensures that an organization’s modernization journey is not only transformational but also secure.

    If making the decision to leverage the private or public cloud for mainframe access, the integration of a secure host access solution ensures teams have access to the systems and applications needed to do their jobs, wherever they may be located. Deploying this secure access in a cloud environment is made easier by host access administration being integrated and centralized with the corporate user directory. Along with simplifying provisioning and management, centrally managed host access is, most importantly, more secure than other forms of access.

    The extension of enterprise security to the mainframe is sometimes overlooked in a transformation, but it is an integral part of the larger process of strengthening data and system security. In a perfect world, this can best be done through the integration of a host access solution with the existing identity and access management (IAM) system. If this is possible, the authentication credentials that are already in use for authorizing or denying access to the mainframe can be leveraged successfully.

    There are numerous benefits to an IAM integration with regard to access, data, stronger passwords, application security settings and system configurations. With IAM, the benefit of least privilege access means that users can access only the mainframe systems and data that are needed to do their jobs. Additionally, application security settings and session configurations can be changed from a central point by using the user directory. Finally, the stronger credential requirements of the IAM integration get rid of weaker eight-character passwords, strengthening security as a whole.

    Host Access Improves User Experience

    When it comes to host access, having a secure and centralized solution not only improves the mainframe access experience for the user, but also for the administrator. With zero-footprint secure access, users can be moved to a more secure web with higher security standards if needed. Security is also stronger with enterprise credentials that leverage mainframe session authorization to automatically sign them in. The integration of a host access solution gives administrators the ability to quickly deploy secure browser-based sessions, as well as providing web interfaces with no Java required, eliminating workstation dependencies as a result. This is also paired with the use of containers to enable deployment to countless users and support for mobile devices. For these reasons, a host access solution should be a common component in any cloud journey.

    The mainframe and its data, applications and user access remain vital for organizations and it is important that they remain protected and connected. With security being an ever-evolving issue, keeping these core systems secure while ensuring employees are able to access the systems and data they need should always be a priority. In conclusion, organizations looking to leverage the public or private cloud for their mainframe will have more success if they also strengthen their data and system security as part of the process.

  • Continuous Testing Practices Gap Assessment

    Continuous Testing Practices Gap Assessment

    Continuous testing is an essential part of DevOps. To get the maximum value from continuous testing, it is important to use recommended best practices, inclusive of people, processes and technologies. A gap assessment is a great way to efficiently evaluate an organization’s practices for DevOps and determine a strategy for improvement. Gap analysis provides valuable input for formulating a strategy and a roadmap to improve continuous testing.

    My SlideShare, Gap Survey, Assessment and Analysis For Continuous Testing, explains how my continuous testing gap assessment tool greatly speeds up the gap survey, gap assessment and gap analysis process for continuous testing.

    It is important to clarify a few definitions for this blog because I have found that different people and organizations use these terms differently. In this blog, and in my consulting work, I use the following definitions.

    • Gap survey is a discovery tool used to collect information about the current state of practices for a topic such as continuous testing.
    • Gap assessment is a process to determine differences (gaps) between current practices and recommended practices for a topic. Gap assessment is also used to refer to the entire collection of activities in this list.
    • Gap assessment workshop is a meeting in which the gap assessment is validated prior to gap analysis.
    • Gap analysis is the process of determining priorities for reducing gaps found during a gap assessment.
    • Gap assessment tool is software that helps perform gap surveys, gap assessments and gap analysis.

    I define and use a seven-step process for gap assessments.

    The steps are as follows:

    1. Pick a topic to analyze. The same processes and tools can be applied equally well to almost any topic, such as continuous testing.
    2. Determine recommended practices for the topic. Something that is worth doing a gap assessment for, such as continuous testing, will normally be fairly complex with tens if not hundreds of practices. Categorize the practices into groups (I call the groups “pillars”) to make the list of practices more understandable and meaningful. To do this requires some research and discovery work. For example, I have a database of practices and pillars for DevOps, continuous testing, DevSecOps and SRE as well as some other topics.
    3. Enter the practices from step two into a gap assessment tool. The tool can be a simple survey tool you create, a commercial survey tool or something custom-made.
    4. Determine who needs to be included in the gap survey.
    5. Collect data using the gap survey.
    6. Perform a gap assessment.
    7. Perform a gap analysis.

    For the remainder of this blog, I will explain in detail the seven steps for continuous testing.

    Step One: Pick a Topic to Analyze

    In this blog, the topic is continuous testing.

    Step Two: Determine Recommended Practices for Continuous Testing

    For an example continuous testing assessment, I used the 15 practices categories described in this three-part series:
    Continuous Testing Practices Part 1
    Continuous Testing Practices Part 2
    Continuous Testing Practices Part 3

    The 15 practice category pillars are:

    • Continuous testing tenets
    • Leadership
    • Culture
    • Test strategy and plans
    • Test management
    • Test automation
    • Test tools
    • Test infrastructure management
    • Test scripts
    • Test results
    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    Step Three: Enter the Practices from Step Two Into a Gap Assessment Tool

    A gap assessment tool, pre-loaded with sample continuous testing practices, can be found in a file called “Continuous Test Practices Assessment Tool v2. It can be downloaded for free from one of the resource pages found on my website Engineering DevOps.

    You can edit the practices categories and add/delete practices from each category if you prefer to make changes.

    Step Four: Determine Who to Include in the Gap Survey

    For the gap analysis to be comprehensive, people in roles that are affected by continuous testing practices need to be surveyed, or at least represented, to ensure their perspectives are included. The following are example roles that typically are included.

    • Business leaders – because they influence culture and budgets for continuous testing.
    • Developers – because they need to design in accordance with continuous testing practices.
    • Project owners – because they influence product work priorities.
    • QA testers – because they need to test in accordance with continuous testing practices.
    • Ops – because continuous testing practices affect operations.
    • Security – because continuous testing affects security practices.

    The survey can be conducted for an individual application, a group of applications, or all the applications in the enterprise. However, it is important that the gap assessment and gap analysis be performed on the organizational segment that is being targeted for improvement.

    Step Five: Collect Data Using the Gap Survey

    A gap survey should allow each surveyed person to enter an importance level score, a practice level score and comments for each practice. All this information is essential for the gap assessment and gap analysis. In the gap survey included within my gap assessment tool, survey respondents are asked to score the importance of each practice as one of 0=Not relevant, 1=not Important, 2=Nice to have, 3=Important, 4=Very important or 5=Critical. Practice level choices are 0=Not sure, 1=Rarely, if ever, 2= Sometimes, 3=Most of the time, 4=Always or 5=We are really good at this. Comments that are relevant to qualify or explain the scores entered for each practice should also be entered, especially if there is any doubt or ambiguity.

    Step Six: Perform a Gap Assessment

    The gap assessment process requires all the practices scores collected from the surveys to be collected and assembled to calculate an aggregate set of scores. Gap scores are calculated using a formula that weights each practice level score with the corresponding importance level score. A visual representation helps to identify practices areas and individual practices that have the highest, most important gaps.

    No matter how professionally written the practices and score definitions are, it is not unusual for some people to misunderstand and to enter scores that they would not have otherwise. For this reason, it is important to ensure that the data collected is validated before conducting the gap analysis. The preferred approach is to conduct a gap assessment workshop with key representatives from each role that participated in the survey. During the workshop, the scores for each practice that have a high deviation between survey responses are discussed and, if necessary, adjusted.

    Step Seven: Perform a Gap Analysis

    The gap analysis process involves extracting the high gap practices and tagging and ranking each of them against solution categories determined by a consultant or topic expert. The results of the gap analysis indicate where solution strategies and implementation roadmaps need to be focused to reduce the most important gaps.

    What This Means

    Those who are serious about DevOps will do well to learn from others and be serious about using recommended continuous testing practices. This blog explained a gap assessment approach, which uses my 15 categories of continuous testing practices. My free Continuous Testing Gap Assessment Tool facilitates performing a gap analysis for continuous testing and outlines priorities to improve DevOps practices for an organization. For those that want to dig deeper on this topic, additional definitions and contextual information related to the way I do gap assessments are clarified in my book Engineering DevOps.

  • Continuous Testing Practices – Part 3

    Continuous Testing Practices – Part 3

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help with understanding what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part three, which covers the following categories of continuous testing practices.

    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    You can read part one here and part two here.

    Security Tests

    • Credentials and secrets needed to operate any test are vaulted.
    • Customer sensitive data in application databases used for testing is obfuscated from misuse by people involved in testing.
    • New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
    • Security tests for each DevOps pipeline stage are automated and may be selected automatically according to predefined criteria.
    • Test results that indicate possible security concerns are tagged for security analysis.
    • Dynamic or interactive application security tests exercise the application to check for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
    • Attack patterns, abuse cases and tests are built according to application module profiles.
    • Unit, functional and integration tests around—and especially outside—boundary conditions are tested. Tests include error handling, exception handling logic and negative tests.

    Pre-Flight Testing

    • Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance, to ensure the changes will not break the integration branch.
    • Requirements for testing are included in agile backlog planning procedures.
    • Pre-flight tests include unit testing, static code analysis scanning, functional, regression and performance tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested, for use in later stages of the pipeline and subsequent regression tests.
    • Pre-flight tests, conducted prior to integration, include unit testing, static code analysis scanning, functional, regression, performance and security tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested for use in later stages of the pipeline and subsequent regression tests.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance to ensure the changes will not break the integration branch.
    • Test-driven development (TDD) is used to ensure tests are created ahead of the code to be tested. TDD is most often applied for unit level tests but may also be applied for functional testing. A key advantage of TDD is that it ensures working tests are available for use in all pipeline stages. Without using TDD, there is a tendency for test creation to fall behind the code. Another advantage of TDD is that tests are created without the bias of knowing how the code was written, because it hasn’t been written yet.
    • From a testing point of view, microservices introduce the requirement to validate the contract of each service with any other services that use it. The independence of the microservice or any interservice dependencies need to be well tested. Performance and reliability considerations of operating services over a network need to be verified. If the application changes, affected microservices, or any dependent group(s) of microservices also need to be regression tested.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • A strategy ensures that changes to a database, or changes to an application that uses a database, perform as expected, during each stage in the continuous delivery pipeline.

    Integration Testing

    • Software changes are tested incrementally by a cross-functional team over all stages of the continuous delivery pipeline.
    • During the code commit stage, assessments are conducted on the committed code to validate that the pre-flight test results satisfy acceptance criteria for integration into the integration branch.
    • During the build stage, a select group of tests are run on each build to ensure the integrated build satisfies build integration acceptance testing criterion.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • API contract testing is used to test microservices interfaces.
    • Test code and data for each application are stored and maintained in test containers so they can be managed using the same container creation, orchestration and monitoring capabilities as the application.

    Pre-Production Testing

    • After the integration and build stage, functional, performance, regression, customer acceptance and security tests are conducted to ensure the images resulting from the build or set of builds satisfy functional, performance, security and delivery assessment criteria.
    • Testing during deployment include dark launch strategies such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Release assessment criteria are defined and use test results to decide whether a release meets release acceptance criterion.

    Test In Production

    • Testing during deployment uses strategies that minimize the impact of failures such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Automated tests verify each deployment.
    • Chaos engineering is used for testing the resiliency of applications, deployment infrastructure and pipelines.
    • Tests for disaster recovery and restoration of services, infrastructure and pipelines are conducted periodically.
    • Fire drills on applications, pipelines and infrastructures are used to test procedures for detection and recovery of applications, pipelines and infrastructures.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer Continuous Testing. This article is part of a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s Continuous Testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.