devopsy.kinsta.cloud

Category: DevOps Practice

  • How to Secure C-Suite Buy-In for Your DevOps Efforts

    How to Secure C-Suite Buy-In for Your DevOps Efforts

    You could be forgiven for assuming DevOps is the modus operandi only for big modern enterprises like Tesla and Uber, but this is not the case. DevOps builds on earlier approaches, such as Agile development and Lean methodology, by emphasizing the importance of collaboration between development and operations teams and the use of automation to streamline the software development process.

    So, while the term “DevOps” is relatively new, the principles and practices that it encompasses have been in use for many years.

    In a traditional waterfall model, the software development process is split into distinct stages, with each stage being completed before moving on to the next one. The development and operations teams work in separate silos, with little interaction between them until the end of the development process when the finished product is handed over to the operations team for deployment and maintenance.

    Benefits of DevOps

    In contrast, a DevOps model emphasizes collaboration and communication between the development and operations teams throughout the entire software development life cycle. The goal is to create a culture of shared responsibility for the success of the project, with both teams working together to ensure that the software is developed, tested and deployed quickly and efficiently.

    DevOps also emphasizes automation, with the use of tools and processes to streamline the aforementioned project tasks. This automation helps to reduce the likelihood of errors and allows the teams to focus on higher-level tasks.

    Overall, the key differences between DevOps and a traditional waterfall model are the emphasis on collaboration and communication, the use of automation and the focus on shared responsibility for the success of the project.

    In short: Speed, time to market, agility and cost savings.

    What’s Driving DevOps Transformation?

    Any company, regardless of size, can release faster with DevOps. Look at the agile FinTech success stories; look at the generative AI success stories. In terms of cost savings too, it’s not ‘How much am I paying for 10 people, 20 people, 100 people…?’ From a DevOps standpoint, cost means penalties avoided from failing to release on time, project delays and project poor quality.

    Overall, development practices are evolving. Some clients call DevOps ‘zero rework’ development, others refer to it as ‘shift-left development.’ Whichever term you prefer, the context is the same: They want faster release and to reduce duplication of work. The way to achieve ‘shift-left’ or ‘zero rework’ would be through DevOps transformation. All of the individual pillars of DevOps concern shifting left, causing them to release faster and achieve their cost optimization.

    However you refer to it, DevOps is mainstream because of its scalability benefits, as well as the agility to react to unforeseen situations during development, QA or operations. This equates to competitive advantages and future-proofing.

    Challenges of Implementing DevOps

    Every business is different, but many of the challenges are the same. One of the main challenges is understanding your company’s specific blend of pain points on its road to DevOps transformation. Common challenges include:

    Vision – DevOps is not a top-down initiative. The role of executives in DevOps is to define a vision; if the vision is not clear, you won’t have buy-in. You can have the best tools and technology but if you don’t have buy-in, then it won’t work.

    Investment – DevOps is a change in every aspect of people, processes, technology and culture. You can’t just bring in this change with zero investment. It’s not unusual for costs to go up for the first couple of years after moving to DevOps because you’re transitioning from manual testers and normal developers to fungible team members that come at a premium. The cost advantage shows up after at least a year as rework goes down and the quality of release incrementally increases, leading to a better customer experience. So, it’s self-funded from the efficiencies it will bring in … but only if you persevere.

    Celebrating Success – Too many people try to boil the ocean with DevOps transformation. This leads to frustration, misaligned teams and, eventually, poor results. If you start small with DevOps transformation, it soon shows its superiority as a business model. This inspires your leadership and your executive teams to see the benefit of DevOps and buy into it by osmosis rather than taking the word of that executive with a big idea and a big purchase order.

    Culture and People – This is more than just a technology play. In fact, technology is the easy part. People, processes and culture all need to transform to implement DevOps. You need to move from a project to a product-based methodology. From a people perspective, you can’t bring in manual testers or manual validators to do DevOps. Any successful DevOps implementation has to get this right.

    How to Implement DevOps: The Hub and the Spoke

    Again, start small. You could start with only continuous integration and then move on to add continuous deployment. If you start over aspirationally, you are more likely to fail and go back to the old ways. As you expand DevOps gradually, you can follow the hub and spoke model.

    In the hub and spoke model, the hub is a lean team responsible for setting up the tools, automation and best practices. The hub is responsible for defining what is to be executed. The spoke, on the other hand, is responsible for execution. The spoke is the extended team and their individual projects which they are responsible for executing.

    The hub is also responsible for enablement by defining what processes, best practices, tools and pipelines should be used. Teams in the spoke will use the pipeline to promote their goal from development to QA, from QA to staging and from staging to production.

    Next-Gen DevOps

    The evolution of DevOps continues even for early adopters, especially with the introduction of AI and generative AI. How do you use AI to validate your requirements? How do you use AI to decide what infrastructure is needed and when? How can you use AI to optimize your test sets? Next-gen DevOps is about smarter DevOps and intelligent DevOps.

  • How Test-Driven Methodologies Reduce CI/CD Lead Time

    How Test-Driven Methodologies Reduce CI/CD Lead Time

    Modern development environments, in which rapid continuous delivery is facilitated by automated continuous integration/continuous development (CI/CD) pipelines, require thorough and automated testing in development prior to integration. This article explains how test-driven development (TDD) and acceptance test-driven development (ATDD) methodologies reduce CI/CD lead times.

    Lack of test automation in development impacts CI/CD lead time for subsequent integration, delivery and deployment stages of a CI/CD pipeline in the following ways:

    • Increased defect rate: Without automated testing during the development stage, more defects and issues are likely to be introduced into the codebase, increasing the time spent on debugging and fixing issues in later stages.
    • Slower feedback loop: Without test automation, the feedback loop between code changes and test results is slower, making it more challenging to identify and resolve issues quickly. This delay can impact the overall lead time in the CI/CD pipeline.
    • Reduced confidence in releases: When testing is not performed adequately or isn’t automated, there is less confidence in the quality of the software being released. This may lead to more manual testing and validation, further increasing lead times.
    • Inefficient use of resources: Without test automation, developers and testers need to spend more time on manual testing, reducing the time available for developing new features or fixing existing issues.
    • Increased merge conflicts: When test automation is not performed early in the development stage, the risk of merge conflicts increases as different developers work on the same codebase. Resolving merge conflicts can be time-consuming and may cause delays in the integration, delivery, and deployment stages.
    • Decreased stability and reliability: Inadequate test automation during the development stage can lead to less stable and reliable software. This may require additional time and effort to address issues and ensure that the software is production-ready.
    • Inconsistent test environments: Manual testing may not always be conducted in a consistent environment, leading to potential discrepancies between development, staging, and production environments. This can cause unexpected issues and delays in the pipeline.
    • Hindered collaboration: Lack of test automation may reduce visibility into the quality of the codebase, making it more difficult for team members from development, QA, security and operations to collaborate effectively and identify issues early.

    Although CI/CD pipelines help catch issues and ensure code quality through automated builds and tests, having a strong foundation of test automation during the development stage helps minimize the potential for issues in the pipeline and ensures a smoother CI/CD process.

    There are several reasons why many development teams do not automate tests prior to integration, despite the benefits of doing so.

    • Lack of awareness or understanding: Some development teams may not be aware of the benefits of test automation or may not fully understand how to implement it effectively in their development process.
    • Insufficient skills or experience: Test automation often requires specialized skills and knowledge, and some development teams may lack the necessary expertise to implement it effectively.
    • Time constraints: Developing and maintaining automated tests can be time-consuming. Teams working under tight deadlines might prioritize feature development over creating automated tests, believing that it will save time in the short term.
    • Limited resources: Test automation tools and infrastructure can require significant investments in terms of cost and resources. Some organizations may not have the necessary budget or resources to invest in test automation, especially for smaller projects or teams.
    • Resistance to change: Some teams may be resistant to adopting new practices, particularly if they have been successful with their existing processes. Changing established workflows and learning new tools can be challenging and may encounter resistance from team members.
    • Incomplete or rapidly changing requirements: In situations where requirements are unclear or frequently changing, development teams may struggle to create meaningful automated tests, as they may require continuous updates to stay relevant.
    • Overemphasis on manual testing: Some teams may place a greater emphasis on manual testing due to a belief that it provides more accurate or thorough results. This perception can lead to a reluctance to invest time and effort into developing automated tests.
    • Fear of increased maintenance: Automated tests, like any other code, need maintenance and updates as the application evolves. Some teams may be concerned about the additional effort required to maintain a test suite, particularly if they have limited resources.

    The amount of test automation that should be done in development prior to the integration stage of a CI/CD pipeline depends on several factors, including the complexity of the application, team resources and the specific project requirements. However, here are some general guidelines to help you determine the right balance:

    • Unit tests: Developers should aim to automate most, if not all, unit tests during the development stage. They are typically fast to execute and provide quick feedback, allowing developers to catch issues early and minimize the number of defects that make it to the integration stage.
    • Component and integration tests: While unit tests focus on individual components, component and integration tests verify the interaction between different components or services. These tests should be automated as much as possible to ensure that any issues arising from component interactions are identified and resolved quickly.
    • Static code analysis: Automating static code analysis checks during development helps identify code quality issues, security vulnerabilities, and potential performance bottlenecks early in the process.
    • Code coverage: Strive for a high level of code coverage with your automated tests. However, it’s important to remember that code coverage is just one indicator of test quality and should be complemented by other testing strategies.

    Striking the right balance between automated and manual testing is important. Automate tests that are repetitive, time-consuming or prone to human error. Manual testing should be reserved for exploratory testing, usability testing and other areas where human intuition and judgment are necessary.

    Test-driven development (TDD) and acceptance test-driven development (ATDD) are development methodologies that emphasize creating automated tests before writing the actual code, helping teams ensure code quality and catch issues early prior to integration.

    Test-driven development (TDD) is a software development methodology that emphasizes writing tests before writing the actual code. The primary goal of TDD is to create clean, maintainable and bug-free code by catching issues early in the development process. TDD follows a short iterative cycle consisting of three main steps: Write a test, make the test pass and refactor the code. The developer starts by writing a test for a specific feature or function, which initially fails. Next, the developer writes the minimal amount of code necessary to make the test pass. Finally, the developer refactors the code to improve its structure and readability while ensuring that all tests still pass.

    Acceptance test-driven development (ATDD) is an extension of TDD that focuses on defining and validating the high-level requirements and acceptance criteria for a feature or functionality before development begins. ATDD involves collaboration among developers, testers, and business stakeholders to ensure that the application meets the needs of its users. The process begins with writing acceptance tests, which are derived from the user’s perspective and based on the acceptance criteria. These tests are written in a human-readable language to facilitate clear communication among all team members.

    Once the acceptance tests are defined, the development team follows the TDD process to implement the feature, using the acceptance tests as a guide. As the team develops the code, they also create unit tests and other lower-level tests to validate the implementation. The acceptance tests are executed as part of the development process and serve as a measure of progress. When all acceptance tests pass, it indicates that the feature or functionality meets the defined acceptance criteria, and the development task is considered complete. By using ATDD, teams can improve collaboration, ensure that the application meets user expectations and reduce the risk of miscommunication or misunderstanding requirements.

    Hurdles to overcome when transitioning to TDD and ATDD include:

    • Mindset shift: Adopting TDD and ATDD requires a change in mindset, where testing is considered an integral part of the development process rather than an afterthought.
    • Skill development: TDD and ATDD require specialized skills in writing effective tests and implementing test automation. Development teams may need to invest time and resources in acquiring these skills.
    • Time and resource allocation: Transitioning to TDD and ATDD might initially lead to a perceived increase in development time, as writing tests before code can seem slower. However, the long-term benefits of improved code quality and reduced debugging time should be considered.
    • Collaboration and communication: ATDD relies on effective collaboration and communication among developers, testers, security, operations and business stakeholders. Establishing a culture of open communication and collaboration is crucial.
    • Tooling and infrastructure: Implementing TDD and ATDD may require investments in new tools or changes to the existing development infrastructure to support test automation and continuous integration.

    In the context of TDD and ATDD, the roles of quality assurance (QA), security and operations teams are crucial for ensuring a well-rounded and robust software development process. Their involvement helps to address various aspects of the application, leading to a more reliable and secure product.

    • Quality Assurance (QA): QA team members collaborate closely with developers in both TDD and ATDD methodologies. They can help define test cases, create test plans and provide input on acceptance criteria. In ATDD, they play a particularly significant role in developing and validating acceptance tests, ensuring that the application meets the defined requirements and user expectations. QA team members may also be involved in setting up and maintaining test automation infrastructure, analyzing test results and identifying areas for improvement.
    • Security: Security professionals contribute to TDD and ATDD by identifying potential security vulnerabilities and ensuring that the application is developed with security best practices in mind. They can help define security-related test cases and help to embed security into the application, reducing the risk of vulnerabilities and improving the overall security posture.
    • Operations: Operations teams are responsible for managing the infrastructure and deployment processes that support the application. In TDD and ATDD, they can collaborate with developers to ensure that the application is designed and developed with operational concerns in mind, such as scalability, performance and maintainability. Their involvement also helps to promote a DevOps culture where development and operations teams work together to ensure the application’s success in production environments.

    A practical roadmap for teams to transition to TDD and ATDD while overcoming the hurdles mentioned earlier could look like this:

    Build awareness and buy-in:

      Start by educating the team on the benefits and principles of TDD and ATDD. Obtain buy-in from team members and stakeholders to ensure a smooth transition.

    Provide training and skill development:

      Organize workshops, training sessions or pair programming exercises to help team members develop the necessary skills for writing effective tests and implementing test automation.

    Start small and iterate:

      Begin the transition by applying TDD and ATDD practices to small, manageable parts of the project or a pilot project.

    Foster a culture of collaboration:

      Establish regular meetings or workshops for developers, testers, security, operations and business stakeholders to collaborate on defining requirements and acceptance criteria. Encourage open communication and shared responsibility for test creation and maintenance.

    Set up the necessary tooling and infrastructure:

      Allocate time and resources to select, set up and configure these tools and ensure they are properly integrated into the development process.

    Monitor progress with metrics:

      Monitor key metrics, such as code coverage, test execution time, defect rates and lead time to track progress and identify areas for improvement.

    Continuously improve and adapt:

      Regularly review the team’s progress, gather feedback, and adjust the process as needed.

    Summary

    TDD and ATDD methodologies contribute to reducing CI/CD lead times by automating tests during development. This improves code quality, minimizes defects and promotes collaboration among team members from the outset of the development process. By writing tests before the actual code and focusing on meeting acceptance criteria, teams can catch and resolve issues early, leading to fewer delays and reduced time spent on debugging and fixing errors during the integration, delivery and deployment stages. These practices also help ensure that the application aligns with user expectations and operational requirements, resulting in a smoother and more efficient CI/CD pipeline.

    Anyone who is passionate about modern quality engineering processes for building high-quality software to participate is encouraged to join new ShiftSync Community from Tricentis to engage in discussions about this topic and other development methodologies.

  • Despite DevOps, Software Supply Chain Security Challenges Persist

    Despite DevOps, Software Supply Chain Security Challenges Persist

    A survey of 397 IT, cybersecurity and application development professionals conducted by Enterprise Strategy Group (ESG) found that while most respondents work for organizations that have adopted DevOps practices, multiple software supply chain security issues and DevSecOps concerns have yet to be addressed.

    The survey, conducted on behalf of Data Theorem, a provider of a platform for securing application programming interfaces (APIs), the survey also found 82% of respondents have adopted DevOps, with 51% reporting they make extensive use of DevOps workflows (51%). Nearly three-quarters (73%) are deploying code weekly and 31% are deploying code daily.

    However, securing DevOps workflows remains a work in progress, with 41% reporting security teams lack visibility and control of development processes. Other issues include new builds deployed to production with misconfigurations, vulnerabilities and other security issues (40%), developers skipping security processes (39%), software released without going through security checks and/or testing (38%), lack of security process consistency across different development teams (38%), security team can’t keep pace with release cadences (34%) and developers’ reluctance to work with security (29%).

    API security is also becoming especially challenging, with 92% of respondents dealing with at least one API security incident in the last 12 months. Well over half (57%) have dealt with multiple API security issues in the last 12 months.

    On the plus side, the survey also found that more than half (54%) of teams responsible for securing APIs are involved with development as soon as or before the APIs are published. A full 97% said developers either have a good (22%) or high level (71%) of API security knowledge. A total of 89% reported their organization provides formal API security training to their development teams. More than three-quarters (78%) can remediate an API vulnerability within a day, and 39% reported they can do so within hours. A total of 60% reported a dedicated budget has been specifically allocated to API security, with another 30% identifying it as a discrete item within their cybersecurity budget. Two-thirds (66%) said spending on API security would significantly increase in the next 12 to 18 months.

    The use of APIs is only going to increase in the months and years ahead. Well over one-third (36%) of respondents are using APIs across all their applications, with 50% expecting this to be the case in two years. Another 44% reported most of their applications are using APIs., with 32% expecting this to be the case in two years. More than three-quarters (76%) of organizations reported that they have an average of 26 APIs per application and 53% have more than a quarter of their APIs facing the internet.

    More than a third (35%) are updating APIs daily while another 40% do so weekly. A full 61% are using APIs to connect microservices.

    Data Theorem COO Doug Dooley said those APIs are now being targeted by cybercriminals that routinely test the weaknesses of every API after each deployment and update. The most common types of API attacks reported by survey respondents included exposure of data (32%), account takeover (31%), denial of service (DoS) attack (31%), attack on misconfigured API (30%), fake account creation (28%), API injection attack (28%), ransomware (27%), data breach (26%) and content scraping (26%).

    Despite these issues, however, nearly three-quarters (74%) of survey respondents said they believe their organizations have a robust API security program in place, with 22% reporting they have some process in place. Organizations are using a mix of approaches including API security tools (59%), web application firewalls (WAFs) (57%) and API gateways (50%), distributed DoS mitigation (48%) and bot management solutions (42%).

    Challenges included managing multiple tools (29%), visibility into and control of APIs (28%), inventorying APIs (27%), inconsistent adoption of APIs (27%), API security testing (27%), data governance/exposure (26%), discovering and remediating misconfigured APIs (26%), keeping pace with threats (26%) and accurately tracking API usage (25%).

    Significant API security concerns included API authentication issues (41%), account takeovers (39%), identifying and tracking third-party APIs (38%), using tools not purpose-built for security (38%), shadow/undiscovered APIs (38%), abusive behavior bypassing app security tools (38%), denial of service attacks (38%), data governance and/or data exposure issues as a result of insecure APIs (37%), abusive behavior bypassing bot management (37%) and outdated/unneeded APIs, also known as zombie APIs. (33%).

    The type of vulnerabilities that are of the greatest concern are sensitive data exposure (34%), access control vulnerabilities (31%), API business logic flaws (31%) and DDoS attacks (30%).

    More than 79% of respondents, however, are relying on manual testing and review processes to ensure APIs don’t expose sensitive data (79%), with 63% having automated alerting capability for when sensitive data is exposed.

    Overall, it’s apparent that more resources as being applied to securing software supply chains in general and APIs specifically. However, given the amount of software being built and deployed, it may be a while before organizations see the return on those investments.

  • OpenAI to go Open Source — Elon Musk was a ‘Huge Idiot’ ¦ Mojo Risin’

    OpenAI to go Open Source — Elon Musk was a ‘Huge Idiot’ ¦ Mojo Risin’

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Sources say it’s an open source future for OpenAI, and improving AI perf with Mojo. (more…)

  • GitHub Adds Accessibility Improvements for Developers With Disabilities

    GitHub Adds Accessibility Improvements for Developers With Disabilities

    GitHub today revealed it added several capabilities to make it easier for developers with disabilities to access its code repository platform. The new capabilities were announced to coincide with Global Accessibility Awareness Day.

    Capabilities highlighted include a navigation tool and a search tool that can be used with a screen reader that GitHub provides.

    In addition, GitHub’s contribution graph has been updated to remove barriers for keyboard-only and screen reader users.

    Finally, GitHub has made site-wide color contrast improvements to the light and dark themes presented by Primer, the GitHub Design System.

    Ed Summers, head of accessibility for GitHub, said there are more than 1.3 billion individuals with a wide variety of disabilities that could, with access to the right tools, write code. That community today, however, is underrepresented as a percentage of the larger IT community, he noted.

    However, as tools such as GitHub Copilot—a generative artificial intelligence (AI) tool that helps developers write better, more secure code—is paired with GitHub Copilot Voice—a speech interface for GitHub Copilot that is available in technical preview—the GitHub platform becomes more accessible to developers with disabilities, he said.

    Generative AI coupled with a speech interface that eliminates the need for a keyboard is lowering the barrier for entry for every potential developer. But for individuals with disabilities, it represents a tremendous advance, noted Summers. Anyone who understands how code should be constructed can build an application, he added.

    It’s not clear what impact generative AI will have on the global shortage of developer talent that has persisted for decades but it’s clear the pace at which applications can be developed is about to dramatically increase. In addition, many organizations have embraced working from home in the wake of the COVID-19 pandemic. That shift makes it easier for many individuals with disabilities to become part of the workforce because they no longer need to commute to an office.

    Generative AI platforms are fundamentally changing the way humans interact with machines. Instead of requiring a developer to create a level of abstraction to communicate with a machine, it’s now possible for machines to understand the language humans use to communicate with each other. Via a natural language interface, developers will soon be asking generative AI platforms to not only surface suggestions but also debug applications and write the code used to create an application.

    At this point, like it or not, the generative AI genie is out of the proverbial bottle. Just about every job function imaginable will be impacted to varying degrees. In the case of DevOps teams, the ultimate impact should involve less drudgery as many of the manual tasks that conspire to make managing DevOps workflows tedious are eliminated.

    It may be a while before the impact of generative AI is fully appreciated, and there will certainly be more advances to come that will be applicable to individuals with disabilities. In the meantime, however, many members of a DevOps team may not even realize a member of the team has a disability as the writing of code continues to be increasingly automated.

  • Infrastructure-as-Code Pitfalls in Platform Engineering

    Infrastructure-as-Code Pitfalls in Platform Engineering

    In the context of cloud operations, the last decade was ruled by DevOps and infrastructure-as-code (IaC). But what is true DevOps? Is it developers running their own operations? Is it operators learning development skills to automate infrastructure provisioning using infrastructure-as-code (IaC) tools like Terraform?

    Though DevOps is widely understood to incorporate both of these, software developers are far from fully understanding or operating infrastructure. IaC is purely a DevOps tool. When an organization claims to have automated its infrastructure, they mean automation for the efficiency of the DevOps teams, not automation for developers. Developers are still left waiting on DevOps for days for even small infrastructure changes.

    While there were efficiencies gained by the current DevOps model, there is an increasing acknowledgment that we require a better approach to enable developer self-service. This has led to the platform engineering discipline.

    The high-level goals of platform engineering are:

    • Developer self-service for significant parts of infrastructure updates without DevOps subject matter expertise.
    • Built-in security and compliance controls.

    Infrastructure-as-Code (IaC) Limitations in Platform Engineering

    A DuploCloud survey of 40+ enterprises that have made substantial investments in platform engineering showed that the prevalent approach is for DevOps teams to build a DevOps platform with IaC as a core underlying technology. They create templates for the organization’s use cases and publish them in a CI/CD pipeline or a self-service catalog.

    Here are the top reasons why platforms that build on top of IaC are failing these platform engineering goals:

    1. IaC templates are rigid and not flexible enough to meet changing developer requirements: It is true that DevOps teams can anticipate cloud infrastructure topologies to some extent and have IaC templates for those with a few customizable parameters. But in a microservices world, there are thousands of other workflows and topologies possible based on changing application needs and security controls. Large data centers and edge infrastructure make this even more complex. A manual approach, relying solely on DevOps personnel to constantly build and update myriad combinations in static scripts simply can’t scale.
    2. Scripting tools can’t build life cycle management: In cloud operations, people-triggered changes are only a subset of possible use cases. Many asynchronous operations need to be continuously performed. These range from detection of configurations, from desired state to reverting, or complex configuration scenarios where individual components have to be set up asynchronously and brought together later. It could be as simple as a certain component going down and needing to be restored. IaC is a script that runs when triggered to completion. It has no active life cycle to operate continuously in the background.
    3. Inability to build a concept of an environment: In any orchestration system, users have a concept of the environment they want to build. When they log in to the platform and navigate to their environment, they expect to update as well as view the state and aspects of resources in that environment, be it the provisioning status, metrics, logs, faults, audit logs, compliance posture, etc. They may want to perform debugging functions, such as restarting services, SSHinginto a VM or accessing a resource’s cloud console (S3 Console, for example) using access control boundaries within that environment. For example, in Kubernetes, you choose the namespace to be the environment and management software like Rancher on top of K8S to provide these functions. If we must replicate this same concept across a broad infrastructure-level platform, we cannot do it using only IaC. We need something with a scope that spans all cloud operations, like Kubernetes. Terraform is a configuration updating and management software, not an orchestration system.

    Many platform teams have tried to work around these problems with point solutions by using a disparate set of jobs for certain aspects of life cycle management, building a thin UI shim on top of cloud accounts for visualization of resources while redirecting to other systems like Datadog for logging, metrics and alerts. But for most of the use cases, the DevOps team is still very much in the operations workflow. This completely defeats the concept of developer self-service as well as continuous compliance goals.

    Learning From Other Successful Platforms

    Two recent examples of successful cloud platforms are Amazon Web Services (AWS) in the context of infrastructure-as-a-service (IaaS) and Kubernetes for container orchestration. These are distributed system implementations using higher-level programming languages like Java and Go. You can’t build such complex systems using scripts and jobs.

    Building a true DevOps orchestration platform requires a systems design approach. It also takes expert systems engineers and many years to build and mature, like it did for K8s and IaaS in the Public Cloud.

    Conclusion

    Since platform engineering teams need systems design expertise that is likely outside of the scope of the typical DevOps role, we need the software development function to intervene with distributed systems programmers and many years of investment. Most organizations trying to build in-house platforms don’t have or don’t want to make this type of investment, as it may not be their core business. Megascale tech companies like Facebook, Uber, Netflix, etc. will likely build an in-house solution, as they have the talent and scale, but for the masses, it is likely that the solution will be an off-the-shelf product from an ISV.

    The industry has solved IaaS with public cloud, container orchestration with Kubernetes and observability with platforms like Datadog. The platform engineering category is heading in the same direction.

  • What’s the Difference Between DevOps and Platform Engineering?

    What’s the Difference Between DevOps and Platform Engineering?

    The DevOps space is buzzing with a new trend—platform engineering. Platform engineering has been gaining steam over the last few years but has really peaked in interest lately. In fact, according to a study conducted by Puppet Labs, 51% of organizations have adopted platform engineering or plan to do so in the next year. But what exactly is platform engineering, and how is it different than DevOps?

    Below, we’ll explore the nuances between these two concepts. We’ll review what DevOps and platform engineering are and see how they’re similar and dissimilar. We’ll also outline the benefits of platform engineering and suggest some best practices to make the most of this emerging discipline.

    What is DevOps?

    DevOps is a broad topic, but it generally involves efforts to increase software release efficiency and frequency. Some of the hallmarks of the DevOps discipline include reducing siloes between teams and increasing collaboration. DevOps is often directly associated with various tools that automate the process of moving code from a local environment into production, also known as continuous integration and continuous delivery (CI/CD). Monitoring, incident management and configuration management tools also support the DevOps process.

    “DevOps reflects more of a philosophy and a way of working,” said David Sandilands, senior solutions architect for Puppet. DevOps enables more of a shift left mentality. Instead of having operations teams “guarding the gates,” developers work in partnership with Ops. In doing so, groups can decrease build time significantly, he said. This style helps engineers incrementally and rapidly release new features, replacing old waterfall development cycles.

    The ultimate goal of DevOps is to automate the software life cycle and shorten the feedback loop. Thus, DevOps takes a life cycle approach to shipping applications more efficiently, streamlining phases such as build, release, configure, monitor, plan, create and verify. Microservices architecture and containerization also often play a crucial role in enabling DevOps.

    What is Platform Engineering?

    Platform engineering is all about developing a self-service internal platform that serves the needs of developer consumers. Although DevOps opened things up and empowered individual developers with advanced automation, it left organizations with scattered tooling and processes, especially at large enterprises. For example, various infrastructure-as-code (IaC) configurations might exist for similar functions within the same organization but be unknown to engineers in different departments. Not to mention, overseeing a growing DevOps pipeline means added responsibilities around continual updates and security.

    Platform engineering places more control in the hands of a central platform team rather than individual developers. These teams typically treat the platform as a product. They do so by evaluating the needs of developers across an organization, building out internal self-service tools that suit the needs of most consumers and maintaining the platform.

    The 2023 State of DevOps Report found that this self-service approach typically handles areas like deployment, operation of infrastructure, monitoring production applications and security and compliance. Other capabilities, such as abstracting Kubernetes or overseeing secrets management, may also play a role in these efforts.

    Benefits of Platform Engineering

    • Improve developer experience: Platform engineering can enhance the experience a developer has with their software build and release systems. A better developer experience can do loads to improve morale and create a more productive engineering atmosphere.
    • Standardize DevOps practices: By bringing together common DevOps approaches into a shared platform, organizations can begin to standardize with reusable build processes and automated infrastructure. This can unite an organization on common runtimes, observability methods and deployment processes. This can streamline the introduction of DevOps practices to an organization.
    • Secure the DevOps pipeline: Various vulnerabilities exist within tooling that comprises modern CI/CD chains and cloud-native open source software. By outsourcing DevOps responsibility to a shared platform team, the organization can coalesce around more secure practices and avoid overlooking important updates or holes.
    • Supply additional guardrails: Platform teams might also assist in the acquisition of new tools as well as outlining and enforcing policies for software infrastructure. While this may come at the cost of customization, additional guardrails can help avoid shadow IT and governance around internal development. Letting platform teams handle it could make maintenance easier across an organization.

    Platform Engineering: Under the DevOps Umbrella

    I don’t see platform engineering as replacing DevOps. Instead, I view platform engineering as falling under the umbrella of DevOps since platform engineering takes DevOps processes and makes them more reusable and self-service. Both use a lot of automation and share many of the same end goals.

    However, there are nuances between the two approaches that reflect the recent trends in the IT market. With DevOps, the market was more interested in shifting responsibility left and empowering individual developers to oversee the entire software life cycle. Now, as sprawl and shadow IT emerge, the pendulum seems to be shifting back toward a bit more centralization and governance with platform engineering.

    Platform engineering is typically more of an in-house development initiative. So, what are some best practices IT divisions should remember to get the most out of platform engineering? Hope Lynch, senior director of platform at Cloudbees, shared some great strategies to consider that revolve around treating the platform as a product. Doing this involves accepting feedback from developers, not controlling everything and always maintaining a focus on developer experience.

    Hopefully, by following these tips, organizations should have a better grasp of the emerging discipline of platform engineering and have the confidence to begin introducing platform-based concepts within their own environments.

  • DevOps Security: Your Complete Checklist

    DevOps Security: Your Complete Checklist

    In a fast-paced environment like DevOps, your security strategy needs to be even more agile. While moving fast is great for delivering application features and functionality, speed shouldn’t come at the cost of security.

    DevOps security incorporates security strategies directly into your DevOps pipeline, creating a culture of pre-secured software deployments that work seamlessly for the end user. However, this requires a complete shift in your operational policies and security mindset. You need to start viewing security as an integral part of your DevOps lifecycle instead of a separate department.

    Here’s a checklist of DevOps security strategies for your development teams to deliver secure and reliable applications faster:

    Your Complete DevOps Security Checklist

    1. Analyze your development process for security hazards.
    The first step in implementing a DevOps security strategy is finding and fixing loopholes in your existing development process.

    For instance, continuous delivery and development are key in a DevOps environment. However, if each change doesn’t go through a code review, it has the potential to break your application.

    Deploying changes without passing them through a security checkpoint is like driving at high speed on a road with many curves. Each curve (change) is a potential hazard, and the danger increases when the driver does not know a car is coming from the other side (bug).

    Therefore, you must turn the spotlight on your development process and find those curves.

    2. Create a common goal between development and security teams.
    Traditional roles of developers and security teams might look like they are at crossroads — Developers write code, and security teams find fault with their code. However, in a collaborative environment such as DevOps, security and development must work hand-in-hand to deliver a secure and seamless experience to end users.

    Your organization should have a common goal of integrating security directly into the development process. Therefore, encourage open and regular communication between development and security teams. Further, establish joint milestones and review them to ensure there are no silos between both teams.

    Once these teams get used to this collaborative approach, your deployments will become faster because fewer things will break.

    3. Integrate security procedures in the development phase.
    Automation is the lifeblood of DevOps security. Integrating scanning tools into development workflows is critical to your entire process.

    For example, automated tools can scan the code, network and infrastructure to identify security vulnerabilities and provide recommendations for remediation. With this, development teams can fix a few security concerns themselves before passing them on to the security team for a final check.

    4. Add observability to your pipeline.
    One of the biggest challenges of the distributed nature of DevOps operations is visibility. How do you efficiently monitor a large number of operation nodes spread across several different locations?

    Observability helps you do this by using logs, metrics and traces—i.e., the outputs of your application to determine internal application health. With observability tools, you can see much deeper into the operations of your systems and identify the exact details of any security concerns. You can also easily identify potential bugs and fix them before they cause problems.

    5. Make changes in small, incremental units.
    It is infinitely easier to test code in small units and deploy them as minor but incremental changes than handling large, unwieldy blocks of monolithic code. Deploying gradual changes means you can resolve vulnerabilities faster.

    6. Scan and secure your entire DevOps pipeline.
    A large part of the security hazards in your DevOps process comes from unsecured containers and operations nodes. According to a study from ThreatStack, 94% of organizations say containers represent a security risk.

    However, the risk is not limited to containers.

    Other services like APIs, third-party tools like Docker and imported code pose a potential security risk to your processes. This makes it necessary to scan all your operations nodes periodically for vulnerabilities. And once you’ve cleared them, remember to lock the door behind you by requiring strict credentials for access to your administrative dashboards.

    Ticking Items Off Your DevOps Security Checklist

    If you implement most of the security guidelines on this checklist, your organization will have a solid security foundation. However, DevOps security is not a one-time process.

    You will need to continuously integrate security protocols into each part of your DevOps life cycle. Furthermore, you will need to re-train your employees and invest in new technology that makes your security pipeline smoother, such as observability.

    However, since this entire process makes your applications more secure and compliant while shipping them to market faster, investing in DevOps security is well worth it!

  • Standardize: It’s Not the Where. Sometimes it’s Not the What

    Standardize: It’s Not the Where. Sometimes it’s Not the What

    In our industry’s attempts to follow best practices that marketers assure us we must have or we are going to lose to our biggest competitor … No, wait, we’ll lose to that startup no one ever heard of … No, hold on, we’ll lose to foreign competition … No, wait … (okay, I’ll stop. You get the picture), many of us have gotten completely unreasonable in our IT approaches.

    Like asking a vendor, “Where can I run your application?” Well, the more “where” options the better, because every organization has different needs. But a vendor has to start somewhere, and you can tell when they get to a certain size because they don’t want to talk about it. “We’re SaaS,” was the en vogue thing to say six months ago; now we’re back to “containers” or even “cloud,” thanks to a whole slew of cloud-first offerings out there. But what all of these vendors really need to be able to say (and a disappointingly small number of them actually can) is, “We run on containers, so we don’t have to care where you deploy us.”

    Those are the types of scenarios that forward-looking enterprises need to start anticipating—the broadest, easiest-to-implement, longest-lasting solutions that can be found. The days of “devs picked this DB no one ever heard of because it was 0.0000001 microseconds faster on a hundred-line query,” need to be gone. Standardize on highly usable tools across the DevOps toolchain from libraries approved for use in apps to deployment technology.

    A big part of this approach right now is my pretty common refrain: Advanced automation. The more a system can do, the more time staff has to do other important things. All forms of testing are finally within our grasp because of advanced automation technology ranging from Kubernetes programmability and controllers to ‘AI’ filtering of results (Side note: What most IT folks currently call AI is not; we need to stop calling it that and give it a more accurate name. Machine learning is better; advanced automation is more to the point, IMO). But whatever you call it, it enables massive pre-release testing. From functional to security, it offers the ability to make these tests useful by speeding up testing and results processing so staff need only look at issues considered critical. No one cares where the test engine runs, just that it can run tests wherever the application resides.

    The same is true for a whole lot of other ‘whats.’ Vulnerabilities are probably the best example. Tools are starting to look for them in an automated fashion. Very few organizations have a comprehensive list of which vulnerability databases a tool must support. Any given organization might have one or two databases in their list, but most just require that the functionality be available.

    So don’t get bogged down in the details, and don’t select five different tools to do the same job on five different teams. Start standardizing again. Oh, standardization won’t last when something evolutionary comes along in the space, but it will leave you less clutter to clean up and more specific expertise in-house.

    And that leaves you more time to keep wowing the business. Keep those systems running smoothly and keep cranking out more useful stuff for the organization. Most orgs show their appreciation these days, but some still do not. In those cases, we thank you for them. We know what you’re going through—we are going through it, too, and yet the lights are still on. Keep rocking it.

  • Boomi Brings Generative AI to Integration Platform

    Boomi Brings Generative AI to Integration Platform

    Boomi announced today it is applying generative artificial intelligence (AI) to enable organizations to automate integrations via a natural language interface.

    Boomi CEO Steve Lucas said Boomi AI makes it much simpler for professional and citizen developers alike to create integrations at scale using generative AI platforms to invoke the low-code engine that Boomi already provides to create integrations between thousands of applications.

    Those generative AI capabilities will complement a wide range of AI capabilities based on machine learning algorithms that are already embedded in the Boomi platform, he added.

    Generative AI’s Impact on Integration and Automation – Steve Lucas, Boomi from Techstrong.TV on Vimeo.

    Boomi has been widely used to drive business process integration via an integration platform-as-a-service (iPaaS) environment it delivers as a managed service. The low-code platform the company created significantly reduces the need to rely on professional developers to write procedural code using Java, for example, to create those integrations.

    Generative AI capabilities make it possible for almost anyone of any skill level to create those integrations across multiple types of application programming interfaces (APIs), noted Lucas.

    That capability also makes it possible for Boomi to notify end users when, for example, a data attribute in an application has changed and ask them if they would like to apply that change across their entire application environment.

    In addition, Boomi AI can also interpret the intent of a prompt to better orchestrate processes in addition to providing an audit history in natural language that explains how decisions were made and what outcomes occurred as a result. As part of that effort, Boomi algorithms are also trained to avoid biases and unfairness with established ethical guidelines.

    Boomi’s AI engine is trained using anonymized data collected from approximately 20,000 customers that have created more than 200 million integrations using the Boomi platform. Those integrations have been exposed to third-party large language models (LLMs) that the company is using to train Boomi AI, said Lucas. Boomi is not storing any customer data to train its AI models.
    DevOps teams, of course, spend hours creating integrations that should be much simpler to create, test and validate with the rise of generative AI. In fact, the number of integrations being created to drive a wide range of digital business transformation initiatives should substantially increase.

    In the meantime, DevOps teams would be well-advised to start evaluating which tasks they manually perform today that could instead be automated by a generative AI platform. Many of the routine integrations that DevOps teams create today using low-level APIs will be increasingly automated by end users. The ultimate goal should be to identify more complex tasks that previously DevOps teams would not have attempted simply because there wasn’t enough time.

    It’s still too early to determine what impact generative AI will have on the way IT is managed, but DevOps teams that are typically committed to automating as many processes as possible should be at the forefront of adoption. The challenge is going to be simply keeping track of the rate of change that might soon be enabled by these platforms.