Category: DevOps Practice

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Monster.com

    Citibank
    Tampa, Florida
    Application Development Technology Lead Analyst
    $154,000 to $179,000

    Dice.com

    ZealTech
    San Jose, California
    DevOps Architect
    $150,000 to $200,000

    SimplyHired

    Musicnotes
    Remote
    DevOps Engineer
    $85,000 to $100,000

    Indeed.com

    Snowrelic
    Remote
    DevOps Engineer
    $82,486 to $173,694

    LinkedIn

    Oak Ridge National Laboratories
    Oak Ridge, Tennessee
    DevOps Engineer
    $78,000 to $137,000

  • Tools and Strategies to Drive a Developer Culture of Collaboration

    Tools and Strategies to Drive a Developer Culture of Collaboration

    Agile methodologies such as DevSecOps have instilled collaborative habits in many development teams. But developers still struggle to work effectively with other stakeholders in the development process. And hybrid work environments have made collaboration that much more challenging.

    In fact, across departments, employees cite collaboration issues among the chief difficulties of hybrid work, recent research shows. Workers experience deficits in team collaboration, working relationships and connection with the organization’s culture.

    A key problem for developers is connecting information silos as they interact with quality assurance (QA), site reliability engineering (SRE), product management, and lines of business. Developers need to span their own data chasms, too, as they use a wide range of siloed technology tools. In fact, 68% of developers encounter a knowledge silo at least once a week. It’s no wonder they say their two biggest challenges are fragmented information (46% of respondents) and lack of tool integrations (45%).

    There are enough stumbling blocks that your development teams probably aren’t achieving a true culture of collaboration. A culture of collaboration means that collaboration is the default mode of work–with cross-functional teams aligned, team members engaged and productive, and projects consistently achieving desired outcomes.

    As much as collaboration is built on human relationships, technology can help, especially in hybrid work environments. In particular, a collaboration platform purpose-built for technical and operations teams is fundamental to building a developer culture of collaboration.

    Automation + Integration = Collaboration

    Computers are very good at performing repetitious tasks quickly, accurately and consistently. Humans are not. They are, however, well-suited to creative activities and qualitative decisions.

    An effective collaboration platform will automate workflows so that software handles rote tasks. Team members are then freed for deep conversations that lead to better decisions and outcomes.

    Playbooks are a great way to leverage automation to improve collaboration, and your collaboration platform should include them. Playbooks are essentially digitized checklists that, with a single click, route workflows through predefined steps to complete a task.

    For example, it’s not uncommon during development for a code issue to be identified that needs to be escalated for resolution. Typically, this involves redundant texts, emails and phone calls to identify the right decision-makers, plus multiple manual steps for reviews, approvals and status updates across stakeholders and conversations.

    That can all be automated in a collaboration-platform playbook. A single click can launch an automated process to escalate the issue to appropriate team members. Workflows take place in the background, quickly and accurately, while developers continue with their collaborations.

    But developer collaboration isn’t only hampered by manual tasks. It’s also held back by a lack of integration across software tools. Developers, QA specialists, business analysts and other stakeholders use many purpose-specific technology tools. These range from Google Docs to Microsoft Excel, from Figma to Jira to GitLab. So many standalone products used by so many teams result in capability and data silos.

    A useful collaboration platform will unify these tools under a single pane of glass. That way, all team members can continue to work with software that’s purpose-built for their needs, yet all can benefit from integrated access to tools in one context. Combining disparate capabilities in a common context lies at the heart of good collaboration.

    Secure and Collaborative

    Of course, any development project involves intellectual property (IP) and other sensitive data that your organization must protect. It’s imperative that your collaboration platform provide strong security and access control.

    General-purpose collaboration products typically are cloud-based, meaning software-as-a-service (SaaS) offerings hosted by the vendor. That doesn’t always meet the needs of industries such as finance and the public sector that must comply with strict regulatory requirements for data protection.

    A collaboration platform for technical and operations teams should offer an on-premises option. That way, you can retain ownership and control of sensitive data.

    This level of security does more than simply safeguard your IP. It also encourages collaboration because team members know they can communicate and work together freely without running afoul of internal policies or external regulations. Built-in access controls should ensure that each team has access to the data it requires, but only to the data it needs to contribute to the project.

    Digital Tools for Analog Connections

    Ultimately, effective team collaboration is built on personal relationships. An effective collaboration platform will encourage human interaction through virtual check-ins, get-togethers and other digitally enabled means of “analog” connection. That could be as simple as providing a rich palette of customizable emojis that allow for personal expression and help to reveal the person behind digital communication.

    At a more sophisticated level, project retrospectives should be built into your collaboration platform to evaluate process and performance for continual improvement. Retrospectives should be blameless so all team members can benefit from learnings without feeling censured. A playbook can provide predefined guidance on meeting purpose, scope, timeframe, budget and so on. Approached this way, retrospectives provide an opportunity for team-building and strengthening of personal connections.

    Hybrid work environments for developers will continue for the foreseeable future. So will the strategic importance of developer collaboration. A purpose-built technical and operational collaboration platform can help ensure that development teams perform in a culture of collaboration and drive successful application development outcomes.

  • Balance Software Quality, Dev Speed and Cost With CI/CD

    Balance Software Quality, Dev Speed and Cost With CI/CD

    In many cases, success in the business world relies on software, be it a compelling mobile app, desktop application suite, interactive website or even an embedded system for the IoT. Because of this need, companies with mature software development processes enjoy a significant competitive advantage over those without. Part of fostering this maturity involves crafting quality software with high velocity but low costs.

    Modern software development methodologies, like Agile and its variants, play a key role in crafting applications in an effective fashion. As more companies adopted an Agile approach, the movement also fostered DevOps, a management structure combining software engineering and network administration teams while leveraging automation to build apps more quickly. In many cases, the ultimate goal for these adopters involved achieving the programming Holy Grail of continuous integration and continuous delivery (CI/CD).

    So, let’s take a closer look at continuous integration and delivery and how it helps companies successfully balance quality, speed, and cost in their software development projects. In a competitive global economy, adopting this approach remains a crucial differentiator, whether or not a business also fully adopts DevOps. In either case, understanding the benefits of CI/CD helps your software company achieve its business dreams.

    A Closer Look at DevOps

    As noted earlier, DevOps somewhat grew out of the Agile movement, but it isn’t a software development methodology in and of itself. Instead, it involves a company merging its software development team and its network administration staff largely responsible for helping the process of building applications. It increasingly leverages modern technologies like containers and virtualization for development, QA and production servers and automation for builds and testing processes.

    This organizational structure and mixture of tools combine to help companies build high-quality software with high velocity and at a lower cost. A mature community grew around DevOps, including tools, best practices, process experts and more. While DevOps originally served as the sole domain of large enterprises, now businesses of all sizes involved in software development benefit from adopting this approach.

    Defining Continuous Integration and Continuous Delivery (CI/CD)

    Continuous integration effectively automates the critical software development processes of builds and testing. It especially makes sense for shops using multiple programmers, as it works with the team’s source code version control system to ensure all programming changes are merged and remain in sync whenever a developer checks in their code.

    Any check-in typically kicks off a process where unit tests and a build are automatically run to ensure those changes don’t “break the build.” In pre-DevOps days, this tended to be a highly manual effort, leading to inefficiencies in any software development project. In short, the team only discovered any integration issues after a manual test. Automation now plays a crucial role in making this aspect of application building significantly more efficient. CI remains a critical best practice in any DevOps shop.

    Continuous delivery shares many similarities with continuous integration, especially its use of automation and the latest software development tools. However, instead of performing builds for internal developers and the QA team, CD focuses more on delivering a software product for release to a user community, customer or the public at large. Again, this is another part of the software development process that led to inefficiencies in the days before DevOps and automation.

    In terms of software development for mobile apps, CD ends up being the last part of the development process before the app gets delivered to the App Store or Google Play. It also comes into play with any fixes or feature improvements after the app is on the market. A key aspect of engendering goodwill among your customers directly relates to the speed of app improvements. Adopting a CI/CD approach plays an important role in making app changes with high velocity.

    CI/CD is the Heart of Successful Modern Software Development

    Simply stated, if your company wants to be a successful software developer, adopting both continuous integration and continuous delivery remains crucial. Following an older, manual approach causes too many bottlenecks and inefficiencies to thrive in a highly competitive industry. No matter how great your startup’s idea or app design is, failure to deliver it with high velocity ultimately results in failure. Once again, this same concept applies to fixes and feature enhancements after the app’s launch.

    At the same time, you don’t need to fully embrace the DevOps organizational structure by creating a separate team made up of software engineers and network administrators. At many startups with a small headcount, your developers might handle many of those network tasks, including managing virtual servers and containers. Hiring a network admin or two might be an unneeded luxury.

    In the end, if you want to build your startup’s app in the most efficient manner, adopting the CI/CD approach makes perfect sense. It provides speed that allows your team to effectively balance quality, speed and cost, resulting in a great app without overspending. It’s the right approach for any successful software project.

  • Microsoft Power Platform Integrates ChatGPT: What This Means

    Microsoft Power Platform Integrates ChatGPT: What This Means

    Microsoft recently announced it integrated ChatGPT into its low-code Power Platform developer suite. This follows the news of ChatGPT-powered Bing search and Microsoft’s pledge to invest billions into its partner company, OpenAI.

    ChatGPT, the large language model (LLM), has reached viral status due to its ability to generate impressively complex textual outputs given simple prompts. And now, enterprises are hoping to cash in on its power. Microsoft’s inclusion of ChatGPT in its Power Virtual Agents and AI Builder tools will likely signal more native inclusion of AI models within other low-code development environments throughout the industry.

    However, not all is peachy in the world of generative AI. Users have compiled a laundry list of ChatGPT failures, ranging from simple arithmetic errors and inaccurate information to social biases and even one attempt to dissolve a journalist’s marriage. AI seems sure of itself, but the “hallucinations” it produces can be mind-bogglingly off-base. If such inaccuracies persist within production use cases, it could lead to false suggestions that confuse users and even harm a company’s reputation.

    Below, I’ll consider the implications of this new paradigm of AI-driven development, exploring the benefits and potential risks of incorporating LLMs into low-code development frameworks. We’ll also highlight how this may upset the competition and consider things leaders should keep in mind as they seek to leverage this new technology.

    Benefits of Incorporating ChatGPT

    Low-code development platforms (LCDPs) excel at abstracting complex functionalities into usable components. They typically offer drag-and-drop capabilities and reusable templates to enable citizen developers and professional programmers alike. And there are many potential benefits of incorporating ChatGPT into such a low-code environment. For example, as a feature within Power Virtual Agents, ChatGPT can be used to create chatbots that connect to public company resources and internal company data. This could help users seamlessly develop a contextually-aware AI trained on a company’s documents.

    Utilizing AI within low-code can spur development with deep inferences and guidance. Such AI assistance can drastically accelerate development efforts through enhanced auto-completion and intelligent insights, quickly generating boilerplate code and making suggestions more customized to the task at hand than your typical static templates.

    GPT-based chatbots leveraging AI trained on a company’s documents could be incredibly more valuable than an off-the-shelf model without context. In the realm of customer support, an in-house trained AI could be beneficial in summarizing company information and automating manual support activities. Providing a low-code means to generate such chatbots also brings advanced AI to a broader range of users. All in all, this could advance the pace of development, especially for mobile interfaces, automated workflows and contextually-aware chatbots.

    Implications For the Low-Code Space

    In the wake of ChatGPT, the market has been scrambling to respond. Tech giants have debuted their own generative AI to mixed reviews. And some, myself included, are left pondering how low-code and AI will coexist in this new era. Will natural language-driven code generation replace the need for programming entirely, whether it’s traditional or ‘codeless’?

    This remains uncertain. But what appears most plausible is that these recent developments will raise the bar throughout the software industry and augment how LCDPs are built and used. AI has the power to enhance LCDPs in a few ways, including augmenting the developer experience, training bespoke ML models and building more intelligent end-user experiences.

    Yet, with Microsoft funneling billions into AI research and development, it could be challenging for mid-size low-code platforms to keep pace without their own AI-driven workflows. Thus, the LCDPs that do not integrate robust AI of their own may lose out on new subscribers. Plus, engineers hoping to create AI models that run on internal datasets may prefer working with a larger cloud technology suite in which their data is natively stored.

    Downsides of AI-Infused Programming

    ChatGPT and other generative models are absolutely impressive, but the results can’t be 100% trusted. And at the time of writing, ChatGPT usage within PowerApps is still experimental, which is indicative of ChatGPT and generative AI in general—still in the experimentation phase.

    This doesn’t bode all that well for programmers relying on its outputs, given the potential for inaccuracies. Because although the results of ChatGPT sound authoritative, they are generated from knowledge scoured from the public web, which often contains bugs, errors and inefficiencies.

    The outputs of ChatGPT may even suggest entirely nonexistent features! This is, unfortunately, a current issue with geocoding API provider OpenCage. ChatGPT routinely recommends users integrate OpenCage’s “phone lookup service” for certain prompts. But … OpenCage doesn’t even offer this feature. The team has received so many complaints from angry developers that they even had to issue a statement explaining what happened.

    With AI code generation, the focus shifts from hardcoding toward creating and organizing prompts and debugging errors. And although it’s giving programmers newfound agility, it doesn’t solve all software development problems—namely, the DevOps side of the equation. There will still be friction in deploying code and piecing together an optimal architecture. Also, due to rising software supply chain vulnerabilities, code automation that loops in third-party dependencies must require careful security forethought. There’s also the chance of breaking changes and ongoing maintenance hurdles with any software.

    Another important element is that low-code requires governance to be secured correctly. No-code business users may not have the security oversight to understand the implications of spinning up new services. And when you add AI into the mix, the technical implications escalate. AI can result in ethical violations, and bots have been found to communicate anger and irrational ideas when poked and prodded.

    Are We All Just Hallucinating?

    ‘Hallucinating’ is when an AI model confidently spouts nonsense or inaccurate information. And although this is the case with some models currently, they are actively accepting feedback and being retrained. Over time, their outputs will improve. But for now, engineers should be aware that they are still under experimentation. As such, enterprises should use caution and test new AI innovations on internal processes first. And AI adoption must be governed carefully to avoid misconfigurations and abuse.

    AI is here, and the future is now. But although it’s dismantling some coding barriers, developers still have their work cut out for them. This reality is a win for low-code solutions offering standard software delivery pipelines and centralized collaboration features. If LCDPs can maintain pace with AI and embed it into their workflows, both sides should fare pretty well in this new era.

    Surprise — this entire article was written by ChatGPT. Just kidding! None of it was. I’m just tired of seeing this “gotcha” moment at the end of news segments…

    This post was written by a human.

  • New Relic Adds Module to Surface Code Metrics

    New Relic Adds Module to Surface Code Metrics

    New Relic today added a free CodeStream module to its observability platform to give developers access to metrics and telemetry data that will enable them to write higher-quality code faster.

    Peter Pezaris, group vice president and general manager of strategy and experience for New Relic, said this capability is part of an effort to shift observability left toward developers to optimize code as it is written and after it is deployed.

    CodeStream surfaces metrics and telemetry data within the integrated development environment (IDE) developers use to write software. Telemetry metrics are displayed as a line of text above the code or in the debugging tool used to troubleshoot an application. Alternatively, developers can view metrics via an observability panel at the service level. They can also view performance metrics such as error rates and response times for related services.

    That approach eliminates the need to wait for IT operations teams to surface issues that usually don’t manifest until long after the code in question was originally developed, noted Pezaris.

    The overall goal is to provide development teams with frictionless access to observability data at every stage of the software development life cycle, he added. That analysis capability will result in faster mean-time-to-detection (MTTD), mean-time-to-resolution (MTTR) and, ultimately, shorter development cycles, said Pezaris.

    Organizations should also be able to reduce technical debt faster by making it easier to identify issues that, for example, adversely impact application performance, he noted. That activity can be especially challenging because developers often inherit code written by someone else that often lacks clear documentation. CodeStream makes it simpler to onboard a developer to any application modernization initiative that has been launched to reduce technical debt, said Pezaris.

    A recent New Relic survey found only 27% of respondents have achieved full-stack observability, and only 5% claimed they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints, the survey found.

    On the plus side, the survey also found nearly three-quarters of respondents said C-suite executives in their organization are advocates of observability, and more than three-quarters of respondents (78%) saw observability as a key enabler for achieving core business goals. However, more than half (52%) of respondents said they experienced high-business-impact outages once per week or more, and 29% said they take more than an hour to resolve those outages.

    A full 82% of respondents said they use four or more tools to monitor the health of their systems. Only 7% said their telemetry data is entirely unified and only 13% said the visualization or dashboarding of that data is entirely unified. Nearly half of the respondents (47%) said they would prefer to have a single, consolidated observability platform, but only 2% said they currently used one tool for observability.

    Regardless of the approach to observability, it is clear that development and IT operations teams need to be looking at the same metrics and telemetry data at the same time to collaborate effectively.

  • Zenhub Brings Issues Management to Project Management Platform

    Zenhub Brings Issues Management to Project Management Platform

    Zenhub today announced it is adding an issue management module that makes it simpler for stakeholders to collaborate with developers on application development projects without having to log into a GitHub repository.

    Fresh from raising an additional $10 million in capital, Zenhub co-founder Aaron Upright said Zenhub Issues is a plug-in for a project management application the company developed for GitHub environments to better streamline software development life cycles. The issue is that many application stakeholders, such as marketers, are not especially pre-disposed to logging into a repository for managing software components, he said.

    Zenhub Issues provides an alternative means for collaborating with developers that doesn’t require stakeholders to log into GitHub, he said. In addition, organizations no longer need to acquire additional GitHub licenses for those users, noted Upright. Many of those end users wind up with read-and-write access to software components, which creates unnecessary cybersecurity risks should their credentials be compromised, he noted.

    In some cases, organizations will also be able to eliminate the need for the project management applications many of those stakeholders have adopted because they can track issues via Zenhub Issues, he added.

    Many organizations’ entire business strategies now revolve around accelerating the rate at which software development projects can be completed. Many elements of those projects, however, can’t go forward until a specific component upon which it depends is completed, so project management has become a critical function. Investors are holding business leaders more accountable for software delivery timetables that have direct implications on revenue generation, so that pressure naturally falls on DevOps teams.

    It’s not surprising that there aren’t a wider range of end users of project management applications as various stakeholders seek to keep closer tabs on how updates to schedule impact their planning cycle. Everyone from the sales team to the board of directors now wants more visibility into the rate at which software development projects are being completed. The challenge DevOps teams face is that software development is an inexact science. Application development projects can just as easily fall behind schedule as they can leap ahead depending on how well code is written.

    Regardless of how application development projects are managed, the relationship between DevOps teams and the rest of the organization has fundamentally changed. Unfortunately, expectations are not always realistically aligned with available development resources. A project management application that is widely employed across an organization should make that apparent to all interested stakeholders. In theory, at least, that should result in either the hiring of additional developers or a reassessment of the business strategy.

    One way or another, project management applications designed for managing software development projects now play a pivotal role in business planning. The challenge is finding a way to populate those applications with the data needed without requiring developers to help enter data that can already be easily pulled from the DevOps tools and platforms. Otherwise, instead of being viewed as a benefit, the entire project management process becomes one more tedious task that development teams are not going to prioritize over writing code.

  • IT Teams Need to Do More With Increased Budgets

    IT Teams Need to Do More With Increased Budgets

    A survey of 600 C-suite and IT decision-makers published today found the majority of organizations are trying to make increased IT budgets stretch even further as costs rise and the number of workloads deployed continues to increase steadily.

    Conducted by SoftwareOne Holding AG, a provider of software and IT services, the survey found 93% of respondents expected IT budgets to increase, with 83% planning to invest more in application modernization. However, 83% also noted they are under pressure to stretch budgets even further as part of an effort to increase agility (46%), improve customer experience (45%) and improve security to support remote workforces (45%). A full 92% said they would be increasing investments in security.

    The core issue that IT leaders are wrestling with appears to stem from increased cloud costs. The survey found the primary causes of those increased costs are inflation-related price hikes passed along by cloud service providers (41%), workloads exceeding planned capacity (40%), transformation initiatives that are cloud-dependent (40%), fewer cloud discounts (37%), lack of visibility into and management of cloud costs (34%), overprovisioning (34%), repercussions from previous lift and shift efforts (34%) and shadow IT (30%).

    Craig Thomson, senior vice president of cloud and application services at SoftwareOne, said that technical debt is having a significant downstream impact across organizations. For example, the survey found 72% of respondents reported that digital transformation initiatives are falling behind, with 38% noting the accumulation of this debt stems from rushed cloud migrations throughout the pandemic. Just under a third (31%) failed to optimize workloads before commencing the migration. More than a third (38%) admitted their organization miscalculated the cloud budget.

    Progress is being made, however, with 45% of respondents reporting they have improved transparency and control of cloud costs. A full 80% planned to increase their investment in financial operations (FinOps) to achieve this, and 39% said they would use cloud-native tools to reduce licensing costs.

    Those efforts suggested IT leaders are being very deliberate in their approaches to reining in costs while at the same time embracing best practices, said Thomson. IT leaders have finally started to realize the cloud is not so much a destination for workloads as much as it is an operating model, he added.

    The challenge is that cloud computing environments are becoming more complex to manage as organizations deploy cloud-native applications on Kubernetes clusters and serverless computing frameworks deployed alongside legacy monolithic applications that were lifted and shifted to the cloud. Organizations will need to invest more in automation to make up for an IT talent gap that prevents organizations from accelerating the rate at which they can build and deploy cloud applications, noted Thomson.

    On the plus side, it appears organizations are willing to allocate budgets to achieve their IT goals. The issue is that, as more organizations become more dependent on software, they are asking IT teams to do even more with the resources at hand. The degree to which that can be achieved, of course, is almost wholly dependent on how much of the technical debt that already exists in the cloud can be retired.

  • Tips For Securing CI/CD Pipelines

    Tips For Securing CI/CD Pipelines

    Most development teams want to increase the pace of their software delivery. As such, continuous integration and delivery (CI/CD) has grown in importance, helping push code from build to production as seamlessly as possible. CI/CD pipelines often loop in many elements and may comprise a diverse stack of tools, automations and various languages. But because of this, the attack surface can be large. And as gaps in the CI/CD toolchain emerge, so do security concerns.

    In recent years, we’ve seen several high-profile cases where CI/CD environments were compromised. For example, take the Circle CI breach, in which a bad actor accessed Circle CI production systems. Others include the compromise of the SolarWinds build system and the Codecov breach. CI/CD security issues are alarming because so much control and automation are given to a pipeline. Unauthorized CI/CD access could be leveraged to publish malicious code or steal sensitive information.

    I recently met with Daniel Krivelevich, co-founder and CTO of Cider Security (acquired by Palo Alto Networks), to better understand the risks facing modern CI/CD. Krivelevich is a seasoned CI/CD security expert and co-author of the Top 10 CI/CD Security Risks framework, which OWASP adopted in late 2022. According to Krivelevich, there is a general lack of knowledge and visibility into these risks. “The CI/CD ecosystem as a whole is full of holes and opportunities for attackers,” he said.

    Below, we’ll review some of the risks that CI/CD pipelines face and offer general ways to mitigate some of these concerns. Hopefully, with greater security awareness around CI/CD threats, both tooling providers and users can begin to diffuse the potential of attacks against their systems.

    The Top Concerns Facing CI/CD Security

    The OWASP Top 10 CI/CD Security Risks framework covers the most impactful risks within modern engineering systems and gaps in CI/CD. It was informed by analyzing many attacks in recent years and was compiled with insights from numerous experts. Krivelevich hopes the framework will spark discussions around the mindset defenders should have when they’re looking to protect the modern engineering ecosystem. The top 10 risks are as follows:

    • CICD-SEC-1: Insufficient Flow Control Mechanisms
    • CICD-SEC-2: Inadequate Identity and Access Management
    • CICD-SEC-3: Dependency Chain Abuse
    • CICD-SEC-4: Poisoned Pipeline Execution (PPE)
    • CICD-SEC-5: Insufficient Pipeline-Based Access Controls (PBAC)
    • CICD-SEC-6: Insufficient Credential Hygiene
    • CICD-SEC-7: Insecure System Configuration
    • CICD-SEC-8: Ungoverned Usage of Third-Party Services
    • CICD-SEC-9: Improper Artifact Integrity Validation
    • CICD-SEC-10: Insufficient Logging and Visibility

    Before an organization can dissect these individual risks, however, they must overcome an even bigger problem—a critical lack of understanding regarding the CI/CD ecosystem and the many elements under the hood. Since CI/CD is not traditionally viewed as a security risk, this can result in insecure default configurations, insecure processes and systems not adopting a secure-by-design approach.

    “There’s a big gap between the level of understanding and opportunities attackers have and the knowledge defenders have over what attackers can do,” said Krivelevich. “Organizations traditionally have not been proactive enough in understanding this, unlike other elements that are more traditionally recognized as part of the attack surface. The main gap is a mindset shift.”

    Tips to Secure CI/CD

    So, knowing the above risks, what are some best practices to secure CI/CD pipelines? Krivelevich shared some tips to keep in mind.

    Bring CI/CD under the security umbrella. First, realize that CI/CD should be scrutinized. This will require an awakening in the community and more proactive effort than before. Krivelevich encouraged development teams to place it on their roadmaps and think about it.

    Map the attack surface. Before implementing security measures, you need to know what you’re protecting. Therefore, it’s a good idea to map your CI/CD environment and produce an inventory of assets. Cataloging the total surface area of dependencies and integrations is necessary to understand risks and helps mitigate CICD-SEC-10: Insufficient Logging and Visibility.

    Keep this updated regularly. Since the cadence of change is fast, teams will want constant visibility into their ecosystem and what’s under the hood. This may even lead to shadow IT, as per CICD-SEC-8: Ungoverned Usage of Third-Party Services. Therefore, Krivelevich recommended keeping a regularly updated inventory. This means continually mapping control systems, container registry artifacts, plugins and all other assets used as part of CI/CD.

    Protect the software supply chain. Many building blocks make up the modern software delivery process. But, some dependencies may be abused or contain malicious code, as per risk CICD-SEC-3: Dependency Chain Abuse. Dependencies within the CI/CD process may be prone to a host of open source vulnerabilities.

    Safeguard CI/CD credentials. CI/CD tools “possess secrets to most sensitive crown jewels in an organization,” Krivelevich explained. Therefore, access tokens and secrets related to CI/CD tools should be protected to avoid costly credential leaks. A zero-trust approach to CI/CD could help avoid CICD-SEC-2: Inadequate Identity and Access Management.

    By creating a virtual heat map of areas to address, organizations will be more informed and better equipped to assess risk. And according to Krivelevich, by implementing these best practices, defenders have a lot of power to raise security expectations in the CI/CD space. “When defenders and users become more diligent about security, the requirements from vendors will increase.”

    CI/CD is the Beating Heart, Don’t Let it Break

    “The engineering ecosystem and pipelines themselves are the beating heart of engineering,” said Krivelevich. And as this system becomes a larger part of what makes a business tick, it will be more heavily relied upon for mission-critical tasks and retaining agility.

    Yet, many building blocks comprise the contemporary software delivery process, and CI/CD pipelines can be a risky area due to the number of systems they touch. Further complicating things is the fact that the delivery ecosystem wasn’t traditionally viewed in security terms. This combination of many opportunities and limited security oversight makes CI/CD lucrative for attackers.

    Rectifying the top 10 risks is a relatively easy gap to address, said Krivelevich—what’s not so easy is the mindset shift required to take CI/CD security seriously. Organizations that invested deeply into microservices and rapid engineering processes will most likely be more susceptible to attacks, he added, since they typically use a diverse array of tools and a complex, integrated delivery process.

  • Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Intel optimizes Linux multithreaded networking, data centers in space (again), and more DevOps toolmaker layoffs. (more…)

  • Raise Those (Feature) Flags

    Raise Those (Feature) Flags

    I’ve written about feature flags before, but I think it’s time for a solid bit of advice: “If you are not yet using feature flags for DevOps, it is past time to reconsider.”

    I say this because feature flags have been a slow-growth item, with a few spikes here and there in usage, but mostly on the development side. While many of you are using feature flags to do dynamic toggling of features–definitely an advanced use case, but still very development heavy–few organizations are using them for deployment options–blue/green without separate code, environment changes for different targets (deploy this to the cloud provider, but do this for local installation type stuff), etc.

    Feature flags are starting to be supported by DevOps tooling for this precise use case, and it is glorious. While the ability to say, “run this code,” and if the new code fails, change something like an environment variable to say, “stop running that bit,” is huge. We have enough time with this process that the industry is well aware of the need to clean up as soon as the correct code options are known, and that is the biggest weakness; it makes spaghetti code that must be cleaned up. But the benefits are well worth a bit of cleanup at the end of a project, even if far too many of you don’t bother.

    Now we can use them in deployment, and in full life cycle toolsets, we have them available for both–so a set of source code can be gated behind a feature flag and deployment can use the same feature flag. This enables the ability to say, “If our FF environment variable is ‘TestDeployment,’ connect to ‘Test Database’ … then use the same variable to say, “If our FF environment variable is ‘TestDeployment,’ spin up the container with test data instead of real data.” This simple example is something that could be (and has been) done manually, but doing it this way makes it part of the process. Automated testing can then follow without the need for human intervention, and from “commit” to “test results,” IT can focus on other things instead of spinning up the right containers and making sure the right DB is accessible.

    After all, we are in a time of massive automation, which should not worry us. There are globally too many IT positions and too few people to fill them; automation should be exercised as much as possible to address that simple fact. And there is a good chance that you are already using tools in the DevOps toolchain that support feature flags–a growing number do support them. If not, it is easy to research and find a good tool to do so and while it doesn’t offer as much functionality, a stand-alone feature flag product will still offer more options than not having any.

    And keep rocking it. The applications you wrote years ago are still powering the enterprise, even if you are at a different organization today. We leave a legacy of applications behind us that is a testament to quality. Keep leaving quality, and use all of the tools you reasonably can.