Category: DevOps Practice

  • Database Observability: How to Circumvent your Weakest Link

    Database Observability: How to Circumvent your Weakest Link

    Today, IT pros field an increasing number of assets they need to monitor to secure systems and understand what’s happening in their database and applications. This is further complicated as organizations adopt hybrid IT strategies and IT pros are tasked with managing tech stacks that span across on-premises and public clouds. 

    To cut through complexity and create end-to-end visibility into their IT systems, organizations must enact an observability approach.

    Monitoring Vs. Observability

    When it comes to observability, many think it means the same as monitoring. Whereas monitoring is a verb; observability is a measure of how well the internal states and conditions of a system can be inferred from its external outputs. 

    When monitoring systems are set up, dashboards are built around certain assumptions that an IT pro may expect will happen. When monitoring databases, teams are building multi-level component IP systems. The algorithms must process that data, and the user interface must interact with it. 

    However, to make that system highly observable, performance information and other kinds of metrics across all kinds of different endpoints must be incorporated. Especially since IT pros must deal with microservices, mesh, IoT or containers orchestrated through Kubernetes.

    How to Monitor Effectively 

    In the past, IT pros would typically monitor a SQL Server or even other database platforms like PostgreSQL or MySQL. In this new era of observability, on the other hand, IT pros need to do something different with the front-end code with regard to the different kinds of algorithms processed against that data set. Typically, developer teams would write the code, but there wasn’t anything written in that code to send out debug signals.

    Today that has changed. To make sure our systems are fully observable (because we can’t anticipate all possible problems!), we must log different steps to keep track of the many moving pieces. Organizations must collect as much of everything as possible. The more data, the more information organizations have about what’s happening in their application, from the database through the virtualization or containerization layer to the virtualization host or the orchestration to the bare metal. If organizations can get visibility into those elements, we can enable our systems to be fully observable and more easily diagnose any problem or anomaly that happens.

    An Observability Approach

    When using the observability approach, high-level dashboards are handy. They can help IT pros roll up everything that’s happening and get to a root cause analysis if there is a problem. However, sometimes there’s not a specific problem; there’s just a general performance issue. With observability platforms, teams can quickly discover, map and understand dependencies in ways that make it easier to achieve and maintain service level objectives (SLOs). 

    As IT teams increasingly distinguish between merely monitoring IT environments and achieving observability, they can proactively surface anomalies using machine learning algorithms and other forms of data science. And the depth and insight that can be applied to managing IT is increasing dramatically. Observability is quickly becoming table stakes for any modern IT organization. 

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Dice.com

    Capgemini
    Chicago/Remote
    DevOps Architect
    $160,000 to $200,000

    Indeed.com

    Northrup Grumman
    Melbourne, Fla
    Deputy DevOps Integration and Test Manager (Systems Engineering Manager 2)
    $123,900 to $185,900

    LinkedIn

    TikTok
    Mountain View, Calif
    Software engineer – infrastructure
    $103,000 to $187,000

    Monster.com

    Hackajob
    Remote
    DevOps Engineer
    $100,000 to $200,000

    SimplyHired

    Crate & Barrel
    Northbrook, Ill
    DevOps Engineer
    $100,000 to $140,000

  • The Future of FinTech and the Cloud

    The Future of FinTech and the Cloud

    The integration of FinTech continues to revolutionize the way businesses interact with their consumers. What started as a viable solution to eliminating the need to carry physical currency has now become a multi-billion-dollar industry.

    Basically, FinTech is a catch-all term used to describe software, mobile applications and other integrated technologies that improve and automate traditional forms of finance for businesses and their consumers. The financial infrastructure-as-a-service (IaaS) focuses on enabling FinTech companies to quickly facilitate highly secure transactions among an internal network while improving user experience. The complex (yet efficient) solution helps streamline fiscal transactions while simultaneously eliminating unnecessary steps throughout the transaction process. Ultimately, making financial transactions more affordable and accessible.

    The integration of FinTech continues to bolster unprecedented growth within the business sector. The industry’s rapid growth can be attributed to a shift in consumers’ lifestyles towards more accessible and affordable solutions. As a result, FinTech has been able to solidify a prominent role within business operations, however, needs to remain vigilant in adapting infrastructures to satisfy ever-changing consumer and industry demands.

    With this goal in mind, FinTech companies have begun to innovate new solutions which include embracing the cloud within its infrastructures. With new integration, companies will have to determine whether using fintech-cloud computing entities will ultimately be an asset in achieving longevity and lucrative business growth or not. 

    FinTech Meets Cloud

    FinTech companies are rapidly turning to the cloud to provide businesses and consumers with effective solutions. According to a research study, 87% of enterprises plan to accelerate their cloud migration by 2025. So, why is cloud computing becoming imperative for FinTech companies to integrate?

    1. Agility: For businesses to compete, they must stay ahead of the competitive curve by reacting quickly to market changes. Cloud services satisfy this need by enabling companies to accelerate its scaling process. Businesses can scale their capacity up or down quickly to meet customer demand in real-time. As a result, providing a more cost-efficient solution.

    2. Competitive Advantage: Within the financial industry’s competitive landscape, it can be quite difficult for smaller companies to gain significant market share. Likewise, larger corporations struggle to retain customers as internal legacy IT practices hinder reaction rate times to changing customer demands. The cloud provides both entities with a better way to compete by offering a quicker alternative to meeting customers’ demands.

    3. Enhanced Customer Experience: Millennials and Gen Z have transformed the banking industry by starting to view finances through the lens of technology. Cloud computing offers FinTech companies the opportunity to promote enhanced customer experience with consumer-centric web applications and improved efficiency metrics. Ultimately, providing the next generation of consumers with an accessible, agile and much more manageable solution that adheres to their technology-savvy lifestyle.

    4. Cost-Effective: Businesses can incur significant infrastructure costs while developing and deploying FinTech products for a large customer base. Cloud adoption can help cut down costs by limiting usage, promoting elasticity of its products and adhering to an all-inclusive price.

    5. Amplified Security Measures: With the ecosystem of hackers and cybercriminals continuing to evolve, companies must remain proactive in mitigating the potential risk of data breaches and cyberattacks. FinTech applications with weak security measures can compromise customer data and other private information. Consequently, leaving the door open to exploitation by malicious operators. Hybrid cloud architecture provides a more secure framework by allowing companies to build their own solutions in the cloud infrastructure they own, rather than in a shared environment within a shared network. As a result, they can be proactive and help eliminate potential data security risks.

    Limitations of FinTech: Enhancing Cloud Integration

    Cloud computing integration continues to provide FinTech companies the opportunity to accelerate business growth. While there are a number of FinTech companies that have managed to achieve great success while using these infrastructures, other companies have failed.

    Any company that invests in FinTech (or any other public cloud service) is susceptible to potential disadvantages. It’s important that companies begin to identify these potential challenges and actively try to avoid them.

    Here are potential limitations to consider:

    1. Refusing to Use Existing Tools:  Change is difficult to manage. Nevertheless, it’s important that companies continue to embrace new technology as they emerge. If FinTech companies refuse to build platforms using current technology, they run the risk of an influx of user discrepancies. On the contrary, if they decide to scale up, they run the risk of hindering the agility of the entire company. By using existing and open source technology, companies can proactively fix their problems and promote continuity in their infrastructures by eliminating outdated data, obscure patches and massive inefficiencies.

    2. Ignoring Scope and Capacity: Using FinTech engineering requires an abundance of capacity planning to ensure an effective solution is created. Without planning, companies run the risk of creating infrastructures that cannot accommodate the volume of metrics used. This can result in killing dashboard performance and making troubleshooting issues impossible. It’s important to plan at every stage regardless of what providers are used.

    3. The Desire to Implement More than Required: Amid a fast-paced and competitive industry, FinTech engineers are constantly implementing new innovations to solidify their competitive advantage. It’s important companies continue to innovate and improve their products; however, operations should not stray too far from their original scope of work. Businesses should work to eliminate unnecessary additions within their projects that could result in wasting a considerable amount of time and resources. Keep simple tasks simple.  

    4. Plan and Evaluate: Businesses need flexibility, independence and a cost structure to thrive. To achieve these marks, it’s necessary for companies to take the time to plan and evaluate each offering to determine which solution accommodates their business needs. Although switching providers might seem doable, doing so can result in a more complicated and expensive solution. Setting long-term objectives before deciding on a provider could make that eventual move simpler for your team.

    The integration of FinTech and cloud computing services continues to impact the technology, finance and business sectors. The question of whether a FinTech-cloud computing structure is a more viable solution in businesses remains. By planning and evaluating each financial solution, businesses can determine which FinTech service is the most feasible for optimizing business objectives.

    FinTech companies will always continue to provide safe and seamless solutions for their customers, however, it’s the companies who can capitalize on efficiency that will rise to the top. As FinTech and cloud integration continues to revolutionize customer-brand solutions, we will become closer to unveiling the future of FinTech and the lasting impacts it has on society.

  • Exploring the Emerging Role of FinOps

    Exploring the Emerging Role of FinOps

    Cloud overspending is becoming a top-of-mind budget item for the modern CFO. To direct resource utilization in the cloud, some organizations are turning to increased automation and more diligent cloud procurement processes. As part of this momentum, new roles are emerging to oversee cloud financial operations.

    Amid rising cloud costs, FinOps is emerging as a helpful discipline. It is not always a specific role but more often a culture that blends engineers, finance and business areas to participate in dynamic cloud cost management. But getting all these players on the same page isn’t always easy — new findings say influencing developers to actually make optimizations is a top challenge.

    The FinOps Foundation recently released The State of FinOps 2022 report, a deep exploration into the role and the state of FinOps adoption across 1,000 companies worldwide. Below, I’ll highlight the key insights from the report to understand current trends and consider how organizations might implement similar tactics to better operationalize their cloud impact.

    Who Is Practicing FinOps?

    What type of person is involved in FinOps? Well, folks come from diverse backgrounds, spanning cloud financial management, IT or DevOps or consultants. Many are also new to the practice in general—the average career tenure is roughly three years. FinOps tends to be a well-paying field—45% of practitioners report salaries of over $100,000 annually.

    FinOps also tends to be associated with larger enterprises—45% of respondents came from organizations with 10,000 or more employees. In terms of industry, it is most common in financial services and IT companies, although studies show the practice is proliferating in many other sectors too. Geographically, FinOps is most adopted in primarily adopted in North America and Europe.

    Team Dynamics

    The average team size is five people, although many feel they’re a “one-person show.” Interestingly, more mature adoption tends to involve a larger team, upwards of 20+ members for established teams. There’s also a notable increase in FinOps within organizations managing between $50 million and $500 million of cloud costs.

    FinOps must interface with many levels of the company, sharing projections with both development and executive leadership. Most FinOps teams report directly to the CTO or other C-suite members. This makes sense, as an executive directive was found to be the most common way to promote a FinOps culture, followed by organic and grassroots adoption.

    Technology and Tools

    In general, FinOps tends to be achieved through cloud awareness and dedicated tooling. The most mature practitioners are “all-in on cloud.” The largest subgroup (24%) runs 1,000 to 5,000 computing instances in the public cloud, hosting technology like virtual machines, containers and serverless functions. Simply put, the average FinOps practitioner has a lot on their plate.

    In terms of tooling, there’s not one cloud cost management suite to rule them all. Teams use an average of 3.7 tools. A variance of tooling here is likely due to rising hybrid multi-cloud architectures and the need to manage costs separately within each cloud environment.

    As such, FinOps tooling includes a mixture of native cloud provider tools and other middleware. Surprisingly, after AWS Cost Explorer, the second most common tool is a homegrown tool. After that, Cloudablity, CloudHealth and Azure Cost Management rank as popular primary tools.

    Top Challenge: Collaboration

    The top barrier has to do with interdepartmental collaboration. Getting engineers to take action on cost optimization was the biggest challenge, cited by 30% of respondents. Other common challenges included accurately forecasting cloud spending (29%), convincing the organization to adopt FinOps (22%), enabling automation (20%) and reducing waste or unused resources (19%).

    Developers in the flow likely don’t want to be burdened by new cost optimization rules, so FinOps may encounter an uphill battle trying to affect low-level change. It will probably take a combination of shift-left IDE-based cost projection tooling and a decree from upper management to jumpstart this change.

    Besides getting everyone on board, FinOps is often tasked with chargeback models—48% of respondents report using chargeback to report cloud costs, the report found. Yet, correlating cloud costs to a single department or team is challenging with so much shared infrastructure. As such, shared costs remain a challenge; so much so that many practitioners don’t even split them up.

    There is also the need to resolve unexpected anomalies and increases in cloud costs, and it can be hard to distinguish between the two. As a result, 53% of respondents said it takes days for their FinOps teams to respond to unexpected cost increases. In the future, finding ways to shorten this timeframe will help save the company money.

    Next Goal: Getting Engineers to Take Action

    The vast majority (94%) of organizations are wasting money in the cloud, according to the 2022 HashiCorp State of Cloud Strategy Survey. Idle or overprovisioned resources can unnecessarily increase the cloud budget, making the need for FinOps evident. Investing in cloud optimization is also another tactic to meet sustainability pledges.

    FinOps is an emerging role that can help forecast spending and reduce waste. But the practice does require time and resources to reach an ROI. For example, a mature team only has a 5% deviation comparing forecasted spend and actual spend, whereas a less mature team can have as high as a 20% deviation. To improve, teams will require automation for evaluating and communicating cloud projections and more established anomaly management best practices.

    But above all, organizations must grapple with the number-one problem: Getting engineers to take action. Perhaps this is due to the fact that most respondents share cost data with technical teams monthly. Instead, more frequent cost transparency would match the pace of agile development and might empower more engineers.

    The State of FinOps 2022 surveyed 1,056 respondents on their FinOps roles. The yearly study is conducted by The FinOps Foundation, a Linux Foundation program. For more details, you can read the entire report here.

  • DevOps World 2022: Developer and Security Links Protect Your Supply Chain

    DevOps World 2022: Developer and Security Links Protect Your Supply Chain

    Ever since the SolarWinds attack back in December 2020, software supply chain attacks have been top-of-mind for any company that builds software. The idea of endangering not just your organization by being attacked but also your customers really sharpens your focus.

    Another complicating factor is that applications use a combination of open software components and homegrown code to deliver applications. This software supply chain means your software can be compromised through no fault of your own but instead based on the components you use. The Log4j attack is an example of a component creating vulnerabilities in any software that uses it. So how do we defend our software? We need to think differently about where security starts and how security concepts can be integrated into software earlier in the development process where it interfaces with the rest of the technology infrastructure.

    Building a Secure Pipeline

    Modern software development is built on the CI/CD pipeline. It allows the application security testing and software composition analysis to happen way earlier in the integration process (shift left), and that allows you to identify and fix issues before the code gets anywhere near customers. Sounds simple, no? Just run your code through a pipeline, integrate security testing, identify issues, fix them, and deploy secure software.

    As with most things, what sounds simple in theory can be challenging to implement in reality. First, shifting left requires a collaboration between developers and security teams. Finding the security issues don’t matter if they don’t get fixed. To increase the likelihood of these issues getting fixed, the security team needs to provide context to the developers regarding the urgency of the flaw or vulnerability and some guidance on the best way to fix the issue. To be successful, security folks can’t just drop a report with hundreds of defects and vulnerabilities on the developers and expect that anything productive is going to happen. If this scenario reminds you of getting a report from your vulnerability scanner with hundreds of issues you won’t fix, well, that’s because it’s very similar.

    Second, security must add value to the development and architecture process by offering up infrastructure-as-code (IaC) templates and a toolchain that makes it easy to implement secure application stacks. Maybe that looks like a template library managed by a cloud security Center of Excellence, which can provide these tools across all of the application teams leveraging efforts.

    Combatting Supply Chain Attacks

    Finally, you can implement a security champions program, training a developer (or two) on each team about security issues and they can act as a security emissary to the developers. This provides scale, ensuring that the security team doesn’t become a bottleneck to fixing the security issues.

    Given the sophistication of the software, the attack surface of the code, and the velocity of the development process, combatting supply chain attacks requires tight and ongoing collaboration with the development team and shifting left to catch the security issues earlier in the integration process.


    Want to learn more about this and other related topics? Join CloudBees for DevOps World 2022, held in Orlando, Florida, at the World Marriott Center. Use discount code DW22 when registering and get access to all the fantastic offerings, including keynotes, sessions, training and other interactive activities. We can’t wait to see you there for the DevOps Remix!

  • How’s Facebook Work? They Don’t Know! | Cali. Pay Law | NASA RISC-V Launch

    How’s Facebook Work? They Don’t Know! | Cali. Pay Law | NASA RISC-V Launch

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Facebook engineers admit they’ve no idea what Meta stores (or where), California requires job ads quote salaries, and RISC-V will power future NASA spacecraft. (more…)

  • Federal Agencies Share DevSecOps Guidelines

    Federal Agencies Share DevSecOps Guidelines

    The National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA), and the Office of the Director of National Intelligence (ODNI) have published a set of DevSecOps best practices based on the Enduring Security Framework (ESF).

    Developed by a public-private cross-sector working group led by NSA and CISA, the ESF framework was created in the wake of the breach of the SolarWinds software supply chain that led to malware being distributed downstream to a number of organizations that relied on the company’s IT management platform.

    The Securing the Software Supply Chain for Developers set of best practices published by NSA, CISA and ODNI takes the ESF a step further. The guidance walks organizations through the processes required to develop secure code, verify third-party components, harden build environments and deliver code.

    The best practices provided are the latest dividend of an executive order issued by the Biden administration that requires every federal agency to review the security of its software supply chains. The expectation is that enterprise IT organizations will conduct similar reviews using the best practices and frameworks defined by the federal agencies that specialize in cybersecurity.

    The challenge, of course, is embedding those best practices within existing DevOps workflows. In theory, more responsibility for cybersecurity is shifting left toward application development teams. However, the level of cybersecurity expertise that exists today among application developers is limited at best. Cybersecurity was an elective in most developers’ training and education programs, so it should come as no surprise that few of them ever attended a cybersecurity class.

    There’s a clear need to improve cybersecurity training for developers, but the real work involves automating cybersecurity processes within the workflows managed by the tools developers employ to write code and the platforms that DevOps teams use to deploy applications, said Mitch Ashley, principal analyst for Techstrong Research, an arm of Techstrong Group that is also the parent company of DevOps.com. “Security needs to be part of the toolchain, workflows and how the software is designed,” he said.

    Ideally, organizations should appoint a security champion that will work with development teams to consistently implement a set of DevSecOps best practices, added Ashley.

    Of course, the security champion doesn’t eliminate the need for cybersecurity professionals. Rather, it reduces the pressure when there is a chronic shortage of cybersecurity professionals. In the absence of any formal cybersecurity review during the application development process, it’s simply too easy for developers to inadvertently introduce malware into a software component or misconfigure a cloud infrastructure service. In either case, cybercriminals are increasingly finding it easier to compromise software supply chains at multiple points.

    It’s not clear how much the federal government’s guidance might impact the overall state of application security. However, one way or another the level of security scrutiny being applied to how applications are built and delivered is only going to increase in the months ahead. The challenge is finding a way to deliver secure code without necessarily slowing down the rate at which applications are currently being built and deployed.

  • Study: Real-Time Data Has A Transformative Effect

    Study: Real-Time Data Has A Transformative Effect

    We all know that data is the heart of the modern digital business. And real-time data is arguably the next evolution—having up-to-date data can empower energy savings and optimizations to reduce cost. Real-time data processing also aids fraud detection and improves health care responsiveness.

    Especially in DevOps, access to real-time data is necessary to reduce incident response time and inform ongoing SRE objectives. Leveraging performance data can also empower future decision-making and open new revenue opportunities. In essence, real-time data can have a transformational effect on the operations of a digital business.

    DataStax and ClearPath Technologies recently released the 2022 State of the Data Race Report. The study found that the majority of data leaders are using real-time data to create additional value. The report also suggested that working with cutting-edge real-time data technologies increases developer experience, which is good to maintain amid ongoing talent shortages.

    Below, I’ll review the key findings from the study to infer how organizations can effectively use these technologies and how they can mitigate roadblocks that arise throughout the process.

    The State of the Data Race

    Simply put, customers expect real-time experiences. And it’s not only the end consumers—developers are also increasingly accustomed to building in this modality. This is evidenced by the recent rise of asynchronous, event-driven architectures. The survey depicted how fundamental real-time data is to modern software, finding that 78% of respondents agreed that it is a “must-have,” not just “nice-to-have.”

    The investment appears to be paying off. A full 71% of all respondents said they can directly tie revenue growth to real-time data. Not only does it seem to have a transformative impact on revenue growth, but it enhances productivity, too—66% of real-time data-focused companies agreed that developer productivity has improved. Naturally, the report found that developers are the ones most likely to be working with real-time data.

    Although modern databases and real-time data processing tools are more readily available these days (and open sourced), the maturity of data handling is not homogenous. Based on the classifications of this report, “data leaders” made up only 10% of respondents.

    Roadblocks to Value

    The road to data maturity has a few obstacles that could limit adoption. First is data complexity—33% said this is one of the biggest barriers to leveraging real-time data. This was closely followed by controlling data costs (32%), data accessibility (30%) and available skillsets (28%).

    Most notably, a skills gap continues to be a commonly cited hindrance throughout industry reports and across technology niches. Interestingly, a lack of in-house talent rises slightly higher in organizations with a high real-time data maturity. Those surveyed in this category said that their top barrier is finding the appropriate skills in their business unit.

    Cybersecurity is also a top concern—51% say security challenges are a big concern when it comes to the use of real-time data. As the report surmised, the business-critical nature of real-time data raises its worth as a commodity and area for attack. Furthermore, amid new data sharing regulations, IT departments are now under more pressure than ever to uphold data security and avoid the leakage of sensitive personal information.

    Unlocking Real-Time Data

    This area has a lot of potential; 86% of developers at organizations with a strategic focus on real-time data said that the “technology is more exciting than ever,” the report found. Real-time data is beginning to be applied at the edge, where AI/ML can be trained to give a more responsive advantage to modern applications. Yet, organizations must prioritize its adoption and mitigate the roadblocks mentioned above to unlock the benefits of real-time data.

    Prioritizing real-time data will take more than tools—it will take vision. According to the study, those who created more value with real-time data initiatives were found to:

    • Have clear product owners
    • Have business unit accountability of data
    • And have department representatives working together in a cross-functional team.

    As you can see, these foundational goalposts have less to do with technology and more with lucid leadership and collaboration. That being said, an investment in the appropriate tooling is undoubtedly required to achieve real-time data.

    For example, this tooling might encompass NoSQL databases optimized for real-time or Apache Cassandra for high throughput. The transition to real-time will likely require streaming and messaging technologies as well. Less obvious, however, is an API layer to sit between the applications and the database. This element could greatly help open data accessibility in a standardized fashion.

    The State of the Data Race: Final Thoughts

    Working with real-time data can be a differentiator and have a transformative impact. As a result, most developers and leaders see real-time data as an exciting area. Yet, organizations must reduce data siloes to open up data access to realize its benefits. It will also take collaboration between data scientists and business divisions to construct meaningful outcomes.

    The State of the Data Race report surveyed 500 leaders across various industries. Above, we’ve summarized the key points from the survey. Readers are encouraged to pick up a copy for more nuanced data points and further commentary.

  • Supply Chain Security: Has the Next SolarWinds Already Happened?

    Supply Chain Security: Has the Next SolarWinds Already Happened?

    More than two years after the now-infamous hack of the firm SolarWinds, the incident is very much in the foreground of conversations about cybersecurity defense, threat detection and incident response. As evidence of that: The SolarWinds incident was the subject of five separate talks and panels at the recent RSA conference in San Francisco. Dozens more talks addressed issues related to supply chain security that–directly or indirectly–invoked the SolarWinds incident. 

    That’s because the SolarWinds attack raises important—if not existential—questions for both software development and DevOps teams and the companies running the software. First: If sophisticated cybercriminals can insert malicious backdoors directly into application code from reputable developers, what organization can afford to blindly trust software? Second: If it is now the responsibility of software teams  to spot sophisticated, methodical compromises of their development pipelines, what is the best and most reliable means of doing that? Finally, if it is also now the job of software consumers to monitor for—and detect—compromised binaries from otherwise reputable vendors (like SolarWinds), how should they go about doing that given ongoing staffing and technology constraints? 

    The answers to these questions are matters of debate within DevOps, software development and information security communities. What is beyond debate is that the SolarWinds incident was not a fluke. In the months since that attack, attackers—both criminal and nation-state-backed—have leveraged software supply chain security blindspots to target development organizations as means of gaining access to sensitive IT environments. While we haven’t yet heard of another incident of the scale or importance of SolarWinds, it is entirely reasonable to assume that the next SolarWinds-caliber compromise could already be unfolding, but has yet to be discovered and/or publicly acknowledged. 

    Software Supply Chain Security Weaknesses

    SunBurst, the malicious implant in SolarWinds software, endures as a topic of interest and conversation for information security professionals and for DevOps and development organizations precisely because of the ways it exposed glaring weaknesses in the security of development organizations and software supply chains. As noted in my analysis of the hack in December 2020 and FireEye’s analysis of the same compromise, the SolarWinds hackers gained access to the company’s development environment months before they actually launched their attack. During that time, they closely studied the underlying source code and mimicked SolarWinds style and nomenclature as they added malicious code and integrated it into the SolarWinds Orion application via a compromise of the company’s build system.  

    Though carefully planned and conducted, the SolarWinds attack wasn’t foolproof. The attackers left clues, such as their additions of native Windows functions, use of cryptography, compression and more. The attackers obfuscated these using compression and Base64 encoding designed to fool simple threat hunting rules that looked for their plaintext variants and other suspicious strings. Still, the obfuscation repeated throughout the implanted code was enough to send up a red flag to anyone who bothered to look closely. That left the attack open to discovery. 

    The problem is this: Very few software publishers have the staff, expertise or tooling to comb through massive codebases to spot a handful of sketchy obfuscated strings or analyze large binaries for deviant behaviors before deploying them. As evidence of this, ReversingLabs sponsored a survey of more than 300 IT professionals at software-driven enterprises. It found that 87% were aware that software tampering, like what occurred in the SolarWinds attack, could lead to enterprise security breaches. About 40% of those surveyed said CI/CD toolchain exposures posed a risk to their organization. Still, barely a third of respondents (37%) said they had the means to detect such software-tampering attacks. 

    Threats to Software Development Organizations Grow

    Attackers are aware of this weakness within the software development pipeline, which is why threats to development organizations are metastasizing. Just in the last year alone, we have seen a series of attacks on development infrastructure and software supply chains. In one April 2021 incident, attackers were successful in pushing malicious, unreviewed code directly to the PHP main branch, ultimately resulting in a compromised, formal version of PHP being pushed to all PHP websites. That same month, there was a breach at the firm CodeCov, which involved unauthorized access to and modifications of the company’s Bash Uploader script. An investigation of that incident ultimately concluded that the alteration of a single line of code in that script by an unauthorized third party enabled the attackers to (potentially) export information stored in the users’ continuous integration (CI) environments. 

    More recently, ReversingLabs observed proof-of-concept and “white hat” activities testing so-called “dependency confusion” attacks, in which attackers set up malicious packages in public repositories like GitHub that squat on the names of private, non-public repositories and sport high version numbers. Misconfigurations in package management systems can result in these malicious components being imported in lieu of the legitimate, privately managed namesakes. As ReversingLabs found, a number of malicious modules discovered on the npm platform were used to target German technology, manufacturing and entertainment firms. It was later revealed to be the work of a white hat firm conducting red team exercises on behalf of those firms. Red team exercises aside, RversingLabs research also detected other, malicious packages in npm, as well as similar attacks targeting Ruby Gems and Python Package Index (PyPi). 

    And that doesn’t even count issues like the Travis CI flaw that exposed (potentially) millions of credentials to developers’ accounts on GitHub, Amazon Web Services and Docker Hub. Lapses like that can fuel supply chain attacks, giving malign actors the ability to insert malicious code into otherwise legitimate open source or proprietary code repositories. After reading such headlines, and without the means to check code intent, how can developers trust their software to update? And how can they do so while at the same time being forced to frequently update their dependencies to reduce or eliminate critical software vulnerabilities?

    Dev Teams Need New Tools

    For software development organizations struggling to meet delivery deadlines and satisfy customer expectations, reorienting to address amorphous software supply chain threats and attacks is a bridge too far. The same is true for enterprises struggling to staff security teams and simply keep on top of targeted phishing attacks or threats to public-facing IT assets. Too often, these entities only find religion about software supply chain attacks after they have occurred. 

    What software development organizations need is a way to secure the CI/CD pipelines and release packages, ensuring that their software builds are free of code tampering and other supply chain risks introduced through build systems. That should include the ability to detect compromises by monitoring for malicious software changes and enforcing security controls and failure conditions on software build processes to ensure that you can release your code with confidence.

    On the customer end, organizations need a way to detect malicious behaviors and unauthorized software changes in compiled binaries before they are released into production, while also monitoring for risks like exposed secrets or exploitable software vulnerabilities that might lurk in compiled code. 

  • Tips for Better Governance – Techstrong TV

    Tips for Better Governance – Techstrong TV

    David Williams, VP of Porduct Strategy at Quali, and Charlene discuss David’s top tips to better governance in DevOps. The video is below followed by a transcript of the conversation.

    Charlene O’Hanlon: Hey, everybody. Welcome back to Techstrong TV. I’m Charlene O’Hanlon, and I’m here now with David Williams, who is the VP of product strategy over at Quali. David, thank you so much for joining me. Good to see you.

    David Williams: It’s good to see you, too, Charlene. It’s good to be here.

    O’Hanlon: Excellent, excellent. So, lots of stuff going on in the world of DevOps these days, but one of the things that we have been hearing a lot more noise about is governance in DevOps. And especially with just the insane amount of automation that’s going on to help kind of speed up the processes, obviously, it can be pretty easy for things to kinda go off the rails. So I wanna kinda get a sense from you as to what you see happening in the space with respect to governance in DevOps, and how we can kind of maybe move away from the herding cats mindset when it comes to governance.

    Williams: Yeah. It’s been a journey on DevOps. DevOps, when it was introduced many, many years ago as a concept immediately aligned itself with Agile, so it became only the ability to get things out as continuously as possible, and also as fast as possible. And so what happened is that at that time, things were very much managed. It was small teams doing lots of work. As enterprises started kicking in, with DevOps become a serious strategic way of actually delivering applications to their consumers, the amount of people got involved grew. 

      The amount of teams that were involved grew. Even though the DevOps philosophy was small teams, fast moving, application componentry going out into the ether as fast as possible, it stopped being that quite a while ago, and started becoming much more mainstream. So unfortunately, DevOps grew up a lot, and one of the things that became an issue was governance. And governance from that perspective wasn’t really just about security – y’know, SOC 2 compliance, and making sure that your secrets are held and managed efficiently across the pipeline – 

      but it was also around, how do you understand what people are doing? And what is the interaction that they’re having? And why are they having that interaction? And basically it’s a fragmentation challenge, more than anything else. DevOps pipelines are fragmented, and so how do you get control without inhibiting innovation? And how do you get visibility, having lots of teams that wanna innovate in their own areas, but not necessarily give you that full end to end? So that’s the challenge that has emerged. 

      And I think it’s going to continue to do that for quite a while to come, because the applications are getting bigger and more contention-based in regards to infrastructure sharing. So governance is gonna be the thing.

    O’Hanlon: It’s interesting, because I think of it a lot like a company itself growing out there. When you have two or three people working in a company, it can be very difficult to get a lot of achieved without everybody kinda going all in. But as a company grows, you’ve got more people who are doing the work, but maybe they’re not necessarily talking to each other, and not everybody has all the information they need. So you kind of have to kinda build a technology, or at least a process, around being able to get the information out and helping everybody be on the same page. 

      And it kinda sounds like the same thing is happening in the DevOps space with respect to just having a greater understanding. Not just an understanding of the process, but just, I guess, a better way of enabling the process from an understanding and a communication perspective – as well as just making sure that all the right things are getting done at the right time. So what do you guys propose? What do you see that you believe could make a real difference?

    Williams: You used two words that always make me smile when I talk DevOps. I used to be an analyst, talking to a lot of DevOps practitioners, and the two words that I used to run for the hills were “process;” y’know, that was one thing. And so they wanted collaboration, but they wanted to be able to collaborate without being governed in any way. So I think that that process word is one, and anything to do with “abstraction.” So abstraction’s not a good thing. It takes you away from the core. 

      So what I’m saying to you is that any solution that enables you to get greater control on governance and understanding has got to be able to do it without impeding people’s ability to do their jobs. And it’s gotta be something that is subsumed within their normal breathing in and out. It can’t be something that’s another job to do. That defeats the whole point of, anything that’s to do with lean practice is about removing stuff. We’re the software industry, so the way that we make things less complex is by adding something, y’know; that’s the normal inclination. 

      I think DevOps, when you actually use it, whatever you add into the mix has got to be subsumed within your normal day-to-day activities. So it can’t be yet another thing to do. For example, security has gotta be there, but not intrusive. Why I came to Quali was because I’d spent a lot of time with companies where infrastructure, the ability to provision into the cloud, do all the things, was becoming a bit of a mystery. People would assume infrastructure. 

      So the governance around what you’re deploying and how you’re provisioning your infrastructure not only became fragmented through your pipelines – Because you plan something, and then you code it, you build it, you release it, and test it, et cetera. Each one of those is a very defined area. It might be one person; it might be many teams. But each one of those is a very specific thing you have to do. But wouldn’t it be cool if you could subsume testing into your code? Wouldn’t it be cool if you could consume compliance while you’re coding? 

      So you’re coding, and it’s automatically looking at configurations and modifications, and logging that appropriately without you having to add another layer. And I think the same with infrastructure. I think that the companies I’ve worked for before really didn’t know what was going on, and I worked for DevOps companies that were trying to work out, “Okay, so the infrastructure was there; now it’s gone,” or “It’s there, and it’s modified.” 

      Every commit that a coder does is, y’know, a point in time. “So what has happened between that commit and the next commit?” And so all these stages of infrastructure, how do you build that in? So at Quali, what we looked at is providing a method with which you can let people use infrastructure as code practices, configuration management tools. Do what you need, but it gets automatically subsumed into a model. And the model is what you rate on. That’s what applies the policies around access and et cetera. 

      But it should allow the coders, the release managers, the testers to do what they need, but also it provides sort of like a control plane on top that enables you to talk and be governed, but without that Big Brother type of thing looking at you. So I think that’s the magic about DevOps, is if you add something in, it’s gotta be something that doesn’t get in the way of speed. It’s gotta not provide another barrier. It’s gotta do these things. It’s a hard problem to solve, but if you approach it in a way that assumes that people are going to do things, they’re gonna break stuff – 

      People do that; that’s what they do. They do their own stuff. They don’t like being told what to do. So if you can provide as subliminal a level of control over infrastructure that says, “I know how you’re doing it. I’m now going to provide that same capabilities that we used in build to help the testers understand what to build.” So now you’re providing some consistency across, using the control plane. So you’re using your models, and you’re moving them across. But it subsumes what they already use, so you can add it in. 

      Your infrastructure stack and the way that you deploy it is implemented in the model that has policy, access control, secrets integration. And so that’s really the philosophy, is to make sure that we can provide the end-to-end capabilities, accelerate the delivery life cycle, but not impede the practitioners from what they do, but provide the business with visibility which they do not have today. So I would just follow up by saying cost isn’t a governance thing, but unless you govern it, and you understand why you’re doing something, then how do you associate a cost of the cloud, if you can’t associate it with a pipeline? 

      At the moment, cost is associated with instances, and you’re having to group that together. What if you could provide the model with a value while you’re doing it, the application that you’re delivering on? And if you could do that, then you’re subtly applying what they do at the practitioner level with an application, with an owner, with a cost. So that is a way that governance actually comes in with the access, the controls, without being intrusive. We can basically provide all the capabilities there, and enable the business to get better visibility control over all sorts of things, including, for the first time, cost on the end to end. 

    O’Hanlon: Yeah, that makes a whole lot of sense. And in my mind – please tell me if I’m wrong – but it also means that the developers don’t really have to think about, how shall I say? They don’t actually have to learn more about something that would – If they’re moving from, say, a regular kind of a developer motion into more of a DevOps motion, it doesn’t require them to kinda add on a level of expertise that they might not need otherwise. This kinda does it for them, and kind of fills in the blanks for them, and also kinda helps other organizations or other teams within the organization understand the flow a little bit better, which makes a whole lot of sense. 

      And I think that may be part of kinda the governance conversation as well, is if we all have a greater level of understanding about what is happening at any particular time, we can better manage it.

    Williams: Yeah. I mean, secrets management, people hear a lot about that. They don’t quite know what it is when you talk to them about secrets management. I mean, something like a password or a username would be used maybe throughout a process, and it’s a shared piece of knowledge that you don’t wanna expose. So you keep that consistent throughout your environment, but keep the access controls consistent throughout whoever uses that access throughout the pipeline. One thing that we’ve learned is that DevOps isn’t a thing; it’s an interpretation of a thing. 

      The objective is very clear for people, but how they implement it can differ, depending on organization, application, design, et cetera. So what we also have to do is make sure that people with lesser skills – And I don’t mean lesser skills in regards to infrastructure; infrastructure can be cloud provisioned. But what if they have to manage all the networking components and the database instances, the things that aren’t inherent within a cloud-provisioning environment? 

      So we’ve gotta make sure that as developers of different skill levels interact with the infrastructure, we can fill the gaps for them, so they can do what they need to do, but if there’s networking, et cetera that needs to be managed, then the model should be something to give them the guardrails with which to implement that in support of the application requirements to that part of the pipeline. So it is something that we’re working on quite heavily, to make sure that we can get those guardrails in place.

    O’Hanlon: That’s great. And I think about all of the things that have kind of expanded DevOps, if you will, beyond the original parameters, including the implementation of DevSecOps, and the adoption –

    Williams: Yeah.

    O’Hanlon: – and DevNetOps, and all of these things that have kinda piled on top of DevOps. 

    Williams: [Chuckling] Yeah.

    O’Hanlon: It can be very, very difficult for the DevSecOps community, or the DevOps community, especially developers, to kinda have to manage all of that knowledge. So having those guardrails in place, I imagine, is going to be very well received by a lot of organizations, simply because we’re also talking about the dearth of talent in the developer community these days.

    Williams: Yeah.

    O’Hanlon: So anything that an organization can do to help ensure that there is a level of governance around the way their DevSecOps tools and practices and culture and everything is run, I think is a huge benefit. David, thanks very much for walking me through what you guys see happening with DevOps and governance. And hopefully this is an issue that is going to get better as we move forward. To your point, we do tend to throw technologies at problems, and hopefully this is one that will stick, if you know what I mean. 

    Williams: Yeah.

    O’Hanlon: So thanks again for your time. I do appreciate it.

    Williams: Right. Thank you, Charlene. It’s been a pleasure. 

    O’Hanlon: All right, everybody. Please stick around. We’ve got lots more Techstrong TV coming up, so stay tuned.