devopsy.kinsta.cloud

Tag: compliance

  • CloudBees Further Extends Jenkins CI/CD Reach

    CloudBees Further Extends Jenkins CI/CD Reach

    CloudBees announced it is expanding the scope and reach of its Jenkins open source continuous integration/continuous development (CI/CD) platform into the realms of compliance and serverless computing frameworks.

    Announced at the DevOps World | Jenkins World conference in Nice, France, a new plugin for Jenkins adds support for the AWS Serverless Application Model (SAM) defined by Amazon Web Services (AWS). That model spans AWS Lambda functions, application programming interfaces and Amazon DynamoDB tables and is natively supported by AWS CloudFormation. The AWS SAM plugin supports both classic FreeStyle jobs and Jenkins Pipeline to make it possible to read and deploy serverless applications.

    In effect, CloudBees is making it possible to treat AWS Lambda as a natural extension of any CI/CD pipeline. Longer term, serverless computing frameworks will play a larger role in the cloud-native instance of CloudBees as well.

    In the meantime, CloudBees is also extending its CI/CD platform to address compliance issues that arise from identity management. CloudBees Core is now integrated with the LDAP-based directory Google makes available within its Cloud Identity service. This capability makes it possible to centralize user management policies by using the same Google Cloud Identity or G Suite credentials to connect to CloudBees Core.

    Christina Noren, chief product officer for CloudBees, noted governance and compliance are becoming much bigger issues within best DevOps practices as more responsibility for security continues to shift left toward developers.

    The CloudBees community continues to grow, even as the number of CI/CD rivals continues to expand. At the conference, CloudBees executives revealed the Jenkins community has grown 24 percent in the period Sept. 30, 2017, to September 30, 2018, reaching 192,623 active installations. That number only applies to installations that permit usage information to be transmitted to the Jenkins project, so this number is not representative of the total Jenkins user population. All told, it’s estimated there may be anywhere from 15 million to 23 million developers globally relying on Jenkins. CloudBees also noted the combined number of defined Jenkins Pipeline and other jobs increased during this same annual period, from 10,509,706 million to 20,664,054 million, with Jenkins Pipeline jobs demonstrating growth of 309 percent from last year.

    Noren said one of the primary reasons for that continued success is that unlike other CI/CD platforms that provide a highly opinionated framework accessible only as a cloud framework, the Jenkins environment makes it possible to deploy a more flexible framework on-premises or in a public cloud.

    Of course, there are still millions of developers that have yet to be exposed to any type of CI/CD framework. But as DevOps practices continue to become the primary method through which application development and deployment is increased, it’s now only a matter of time before most developers are working within the context of a CI/CD platform.

    In fact, it’s arguable that those organizations that don’t soon embrace CI/CD platforms will be overwhelmed by rivals capable of updating applications at rates that are increasingly being measured in days and hours versus weeks and months.

    — Mike Vizard

  • Chef Extends Scope of Compliance Automation

    Chef Extends Scope of Compliance Automation

    Chef has extended the reach of its InSpec compliance automation framework by adding a plug-in architecture that makes it easier for developers to programmatically generate compliance controls that can be applied to automation frameworks. The Terraform configuration management framework developed by HashiCorp is the first framework supported via the plug-in architecture. InSpec previously was integrated with the open source Ansible automation framework.

    Julian Dunn, director of product marketing for Chef, said the plug-in architecture supports multiple communications protocols to extend it to multiple frameworks beyond the Chef IT automation framework.

    InSpec 3.0 also makes it easier to describe new resources that can be tested in addition to a stable application programming interface (API) that enables DevOps teams to modify how tests are conducted. Improvements to a packaging mechanism for profiles will also make it easier for developers to more easily iterate on InSpec profiles with dependencies.

    Finally, InSpec 3.0 adds a key value-based description interface to enable more fine-grained reporting as well as de-duplication of controls that satisfy one or more compliance regimes. This allows organizations to create their own custom metadata categories.


    In general, Dunn notes that application development and deployment, thanks to the rise of DevOps, is moving at a pace that exceeds the ability of those tasked with compliance to keep pace. As a result, responsibility for compliance needs to shift left along with other aspects of cybersecurity, said Dunn.

    Dunn conceded it is still early days in terms of making that shift. But as more organizations discover that the rate at which applications can be deployed is being held up by compliance audits, the more incentive there is to enable developers to programmatically insert the appropriate controls into their applications. Once those controls are in place, InSpec then generates human-readable code that serves to reduce the amount of time required to pass an audit before and after an application is deployed in a production environment, said Dunn.

    Of course, passing a compliance audit doesn’t mean the application is secure. But it does at the very least provide a baseline for what’s required to make an application nominally secure. In fact, the path of least resistance to any shift toward embracing DevSecOps might be automating the processes for putting compliance controls in place.

    While there are hundreds of regulations requiring various levels of compliance, it turns out most of them require the same types of controls to be in place. Once organizations begin to programmatically embed those controls within their applications, the speed at which those applications are certified to comply with multiple regulations should increase. There will be less reason for developers to skip a compliance control to meet an application development deadline when that control can be integrated automatically.

    None of this means that people with risk management backgrounds will be participating in DevOps processes anytime soon. But it does means that the output from DevOps teams won’t be stacked up waiting to pass a compliance review because all the applications in front of it are missing one key control or another.

    — Mike Vizard

  • Don’t Forget to Automate Cloud Security

    Don’t Forget to Automate Cloud Security

    Cloud has been around long enough, and we’ve had enough breaches stemming from unsecured cloud (mostly data objects, but other items, too) that we should know better. I cringe whenever I see a security headline along the lines of “… Unsecured S3 Bucket.” Would you publicly expose a server in your data center and not lock it down? No.

    So, what is the problem? Frankly, “cloud” now consists of so many separate-but-integrated offerings that it can be daunting, even improbable, for security staff to lock it down. Add in dozens or even hundreds of instances/storage locations/routers, and you have a separate data center with different requirements—at least, that’s what it essentially is for your security team. Then add agile and DevOps, spinning up new infrastructure and changing the parts at a rapid pace, and you have a fast-moving, complex data center environment. Yet, if you are a typical organization, your security staffing hasn’t significantly changed, nor do you have requirements for securing the physical infrastructure. In short, same group of individuals, a ton more work to do.

    Before it was purchased by VMware, I had the pleasure of working with CloudCoreo, and truly liked what I saw. The problem in highly automated environments, be they DevOps or just automated, is that a lot of parts are moving as the system is maintained/updated. Some of those parts are changing security settings in a complex (data center-size complex, in many cases) environment. For security teams to keep up and be part of DevOps—or just keep up with automation—quite often corners must be cut.

    That’s bad for all of us—security staff, the company and customers. If anything, when the cloud environment grows complex, security needs to be more cautious, not less. And yet, the number of data leaks/exploits that involve “misconfigured” security settings in the cloud says we have a problem.

    So what to do? One option is to check out security control and auditing software from the likes of RightScale, Dome9 or Evident.io. (DISCLAIMER: I have not used any of these tools. They were chosen because each shares attributes with CloudCoreo, which I have used.)

    The ability to scan and apply policy to a broad array of cloud artifacts—from users to back-end storage—is essential in allowing security to keep up with DevOps. There is just too much change in a fully automated cloud-based environment to expect security to keep up without the same level of tools as developers and ops get. “Shift left” doesn’t stop misconfigurations, but scanning can detect them, and some tools can even auto-fix those misconfigurations.

    In the end, protecting systems and data is what the security team does. Bringing them along on the DevOps grand tour and offering them the same type of automation opportunities as the rest of IT is benefiting from is mandatory. It is not “would be nice to have” to give security the chance to keep up; it should be a requirement.

    And you might still have rogue S3 buckets that security doesn’t know about that might be misconfigured, but any that are paid for by the corporation will be available to security and those automated tools and, thus, will be locked down.

    And if security isn’t enough of a reason for you to want to implement one of these tools, “Come for the compliance, stay for the security.” Most solutions offer auditing that helps with compliance.

    — Don Macvittie

  • Has DevOps Caused the Re-emergence of Shadow IT?

    Has DevOps Caused the Re-emergence of Shadow IT?

    Shadow IT is back, and this time, it’s in the DevOps toolstack

    “One of our clients conducted an audit and found more than 10,000 Jenkins instances across their IT landscape!” This remark from a tooling vendor that helps enterprises centralize their continuous integration infrastructure really took me by surprise. For starters, I was impressed how the company had unleashed creativity at this scale—if the number of Jenkins instances in an organization is an indicator of DevOps. Clearly, its employees have taken the saying, “Don’t ask for permission, ask for forgiveness,” very seriously.

    However, there are certain risks with this approach, especially security risks if teams haven’t considered the long-term implication of their decisions. For instance, how do you handle another major security flaw such as the OpenSSL Heartbleed bug? You may have to send hunting squads to track down all the instances of Jenkins (in this case) across your IT estate—including the Linux server hidden away under the desk of the engineer—and patch them. If you are lucky, you can audit all the virtual machines on your chosen cloud where all your tooling instances reside. And if you are really lucky, the entire DevOps tooling infrastructure would have been set up in a DevOps way, using infrastructure-as-code principles with automated patching capability.

    Consider the nightmare scenario for IT compliance, security and governance teams: The spectre of shadow IT is amplified due to poor adoption of DevOps! Shadow IT in the application stack has always been a big headache for CIOs; the use of IT systems within an organization without the approval, or even the knowledge, of corporate IT has never gone away. Shadow IT has been historically associated with business users who made the purchase, but did not have the expertise to maintain and upgrade the software to make sure there were no risks to the wider organization. Much has been written about the problem in the application stack and how DevOps ways can help resolve the issue.

    However, with the proliferation of DevOps tools in a localized way, the shadow IT problem is now manifesting itself in the tooling architecture of enterprises. With the increasing adoption of DevOps practices and tools in the enterprise, teams have started installing various DevOps tools in a localized and autonomous way. This is leading to the shadow IT problem in the tooling landscape of enterprises. It’s slightly different than the problem that originated in the application stack, but with similar risks and challenges. Access to cloud in the enterprise has made it easier for teams to set up their own localized DevOps tool sets. A sizable program of work in its full DevOps glory can easily use dozens of tools and need to consider aspects of security such as:

    1. What is the patching policy ?
    2. Are your DevOps tools critical to the business due to the dependency on them for the build and deployment of critical  services that the business requires?
    3. How resilient is the tooling infrastructure ?
    4. Is there enough knowledge in the enterprise to handle this change?

    Essentially, there are four risks of shadow IT due to DevOps tooling, based on a pertinent classification by Computer Weekly:

    1. Software asset management: The risk of discovering unapproved and unlicensed software is huge and the CIO could be potentially jailed for sanctioning usage of unlicensed software.
    2. Compliance with industry and government standards to demonstrate quality to customers.
    3. Lack of testing and change control: Managing cycle of change and releases becomes complicated when there are dependencies on third-party SaaS.
    4. Configuration Management: IT teams rely on the veracity of information in the configuration management database CMDB and shadow IT compromises it.

    Organizations typically respond to the problem of shadow IT with rationalization, migration or even sunset of the targeted applications—most of which are reactive measures. In the context of DevOps tools, an extreme way to deal with the problem is by taking control of all tools with an “iron fist” and shutting down everything—at the risk of killing off innovation in the enterprise. In the short run, teams do achieve higher levels of productivity by deploying tools in the way that suits their business context and experience of the engineers in the team. However, over time I find that the teams are overburdened by the maintenance overhead of DevOps tooling and maintaining a tooling stack in addition to their software application stack. At times, if they are early adopters in their organization, they tend to be victims of their own success: by supporting other teams with their tooling solutions, almost becoming a tooling platform provider internally, and diverting their attention away from their core purpose of delivering value to their business partners.

    One pragmatic way to resolve this is to add an element of centralization or “platformization” of the tooling infrastructure.

    Autonomous teams understandably fiercely resist any attempt to centralize services in the enterprise. A few useful ways to gain a balance between innovation and control include:

    1. Provide a responsive tooling platform: “You can have any color as long as it’s black” approach to DevOps tooling will only lead teams to finding their own unique solutions.
    2. Anticipate tooling needs and build them: Think ahead, track the trends and procure the tools that you expect teams to achieve. For example, if a few teams are already playing with docker and containers, then think a step ahead and look at Kubernetes to manage the containers.
    3. Focus on user experience: Think about the ease of subscribing to tools, installation, billing … the entire life cycle
    4. Offer help to transition to the tooling platform.
    5. Run an internal communications campaign on the tooling capabilities that are available to teams.

    Ultimately, a well-thought-out DevOps approach and implementation can actually help resolve the challenge of shadow IT that organizations face, instead of accentuating the problem.

    — Aditya Vadaganadam

  • Software Liability Goes Global

    Software Liability Goes Global

    This month, France turned up the conversation on software liability for manufacturers who place known defective software components in their products. But, they are not the first.

    Software Liability in France

    According to Lukasz Olejnik, lawmakers in France just suggested a desire to “put the security liability in hands of product suppliers. In other words, making companies responsible for the security of products they put on the market—for as long as the products are commercially available.”

    He continued: “The strategy then mentions that one of the solutions could be to release source code and documentation after an end of support date. The strategy itself mentions taking this discussion to the international level. This will be interesting.”

    You betcha, Lukasz.

    (If you can read French, you can find the 166-page proposal here).

    Software Liability in United Kingdom

    The need for improved cyberhygiene in the UK reached new heights in 2017 following large-scale ransomware attacks on its nation’s hospital system and an increased focus on software liability. According to the UK’s National Cyber Security Strategy 2016-2021 report, “Businesses and organizations decide on where and how to invest in cyber security based on a cost-benefit assessment, but they are ultimately liable for the security of their data and systems. Cyber attacks are not necessarily sophisticated or inevitable and are often the result of exploited–but easily rectifiable and, often, preventable–vulnerabilities. In most cases, it continues to be the vulnerability of the victim, rather than the ingenuity of the attacker, that is the deciding factor in the success of a cyber attack.”

    Shedding a spotlight on the cyberhygiene and software liability, Britain’s Information Commissioner’s Office (ICO)—the country’s data regulator, said a hacker exploited a well-known security flaw on a Gloucester City Council website months after the vulnerability had been widely reported on and updated component versions had been made available. The ICO fined the Gloucester City Council £100,000 in June 2017 for not preventing a cyberattack exploiting the OpenSSL Heartbleed vulnerability.

    Software Liability in the United States

    Last August, Sens. Mark R. Warner (D-VA) and Cory Gardner (R-CO), co-chairs of the Senate Cybersecurity Caucus, along with Sens. Ron Wyden (D-WA) and Steve Daines (R-MT), introduced bipartisan legislation called the Internet of Things Cybersecurity Improvement Act of 2017.

    According to a fact sheet released at the time, “While ‘Internet of Things’ (IoT) devices and the data they transmit present enormous benefits to consumers, the relative insecurity of many devices presents enormous challenges. This legislation is aimed at addressing the market failure by establishing minimum security requirements for federal procurements of connected devices.”

    The proposed legislation requires vendor commitments:

    • To ensure devices don’t contain known security vulnerabilities when shipped;
    • To ensure proper disclosure of new security vulnerabilities discovered within their devices; and
    • To prepare remediation plans for any IoT device where known vulnerabilities have been discovered.

    While the legislation is clearly aimed at consumer protections and privacy, it also focuses on quality, safety and regulatory standards applied to every other major manufacturing industry (thou shall not ship products with known defects).  The legislation specifically calls for vendors selling IoT devices “to provide written certification that the device does not contain, at the time of submitting the proposal, any hardware, software, or firmware component with any known security vulnerabilities or defects.”

    Software Liability in the European Union

    When the innovation race is being run without proper oversight, getting to the finish line safely will require greater (and faster) care. That’s set to be a major challenge for organizations developing software under the forthcoming EU General Data Protection Regulation (GDPR) in May 2018.

    Article 32 of the GDPR states that organizations must “implement appropriate technical and organizational measures” to “ensure the ongoing confidentiality, integrity, availability and resilience of processing systems and services.” When combined with Article 25, which mandates that data protection measures be implemented “by design and by default,” it’s clear that privacy and security must become ingrained in every element of IT infrastructure.

    If you fail to follow these rules and known software vulnerabilities end up inadvertently helping hackers steal sensitive consumer data, you could be on the hook for seriously big fines: up to €20 million, or 4 percent of global annual turnover, whichever is greater.

    Software Liability in Germany

    In July 2017, a consumer advocacy group in Germany has filed a law suit against a retailer in Cologne that sold an inexpensive smartphone made by Mobistel. The Mobistel model Cynus T6 was sold in Media Market stores for just 99 euros. Sounds like a great deal, right? Not so much. You see, the phone’s software came with 15 critical and known security vulnerabilities that were not disclosed to the consumer at the time of purchase.

    Instead, these security flaws were later identified by investigators from the Federal Office for Information Security (BSI).  Unfortunately for Mobistel and Media Market, consumer advocacy groups tend to fight back when manufacturers and retailers sell products to consumers without disclosing “essential information” such as known security defects in a smartphone.

    Although the complaint is at an early stage, it points to the possibility that companies manufacturing software applications could be held liable for selling defective products to consumers—in exactly the same way that auto makers have long been held liable for defective parts in cars.

    Think You Aren’t Shipping Vulnerable Software?

    Think again. According to Sonatype’s “2017 State of the Software Supply Chain Report,” 80 percent to 90 percent of a modern application is now assembled from open source and third-party components. Therefore, “It is incumbent upon development organizations to practice good hygiene when consuming open source components. Teams with suboptimal hygiene inevitably consume open source components with critical vulnerabilities. Last year, 5.5% (1 in 18) components downloaded from internet-based open source repositories contained known security vulnerabilities.”

    What does this mean for you? If you have not been paying attention, the chance that software your organization is building contains a known software vulnerability is nearly 100 percent.

    The Writing is on the Wall

    Organizations failing to manage software supply chains, who are unwittingly releasing vulnerable applications into production, will face increased liability due to gross negligence.

    The writing is on the wall. Are you prepared?

    — Derek E. Weeks

  • Automation Levels the Playing Field for Continuous Compliance

    Automation Levels the Playing Field for Continuous Compliance

    At this point, I doubt there is any company doing business anywhere that doesn’t fall under at least one compliance framework. One universal truth of compliance—in intent, if not in practice—is that it is a process, not an event. Checking boxes and passing an audit is a requirement of most compliance mandates, but the real value of compliance lies in maintaining that level of visibility and security all the time—which is why continuous compliance is important.

    The Need for Continuous Compliance

    Hopefully I don’t need to continue to make the case that compliance is an ongoing requirement. Because it is something that needs to be consistently monitored and maintained, it also makes sense to automate the process as much as possible.

    In a report titled, “How to Avoid Compliance and Audit Concerns When Using DevOps,” published in November, Gartner stated, “Auditors favor the consistency and traceability of automated systems that have strong logging capabilities and transparent auditable controls.”

    A recent survey of more than 1,500 users conducted by Chef found that 74 percent of cross-functional application, infrastructure and security teams do exactly that, though: They assess software for compliance manually prior to production. Once violations and vulnerabilities are discovered, half remediate manually instead of automating the process. Manual processes result in teams’ detecting and remediating security issues in days (31 percent) or weeks (19 percent), instead of hours (18 percent).

    “At scale, manual reviews or out-of-band security tools bolted onto the DevOps pipelines won’t be able to keep pace with new velocity of change,” said Derek Weeks, VP and DevOps Advocate at Sonatype. “We saw this first with paper-based open source governance policies in the application security realm and it’s exciting to see similar approaches applied to infrastructure.”

    Andrew Storms, VP of Product at New Context, agreed. “Infrastructure automation and configuration management are key requirements for securely storing and transmitting data. … In DevSecOps, bringing security into the development process early and often helps ensure more cost-effective data security.

    “Think about this: By 2020 there will be some 20 billion IoT devices connected to the internet,” he continued. “Most organizations can’t even handle automating security and compliance of their existing infrastructure. The growth demand is far outpacing human abilities. Organizations simply cannot afford to rely on the methods of treating our infrastructure as cute and cuddly pets—and spending weeks producing evidence for security audits is not going to happen.”

    “Rather than force cumbersome security processes on top of the DevOps pipeline, security and compliance professionals need to adopt a new mindset of integrating controls as code,” Weeks added. Codifying controls initially improves the performance of the checks, but, over time, serves to expand the breadth of security and compliance practices as more controls are coded.”

    Speaking a Common Compliance Language

    Effective communication is a critical part of the process. “If you were ever in doubt on the importance of testing, then you should ask NASA about how they lost the Mars climate orbiter,” Storms said. “Years of development and likely some millions later, the space probe burned up because two software components weren’t using same units to communicate.”

    Chef just released InSpec 2.0, which advances the capabilities of compliance automation. Mike Vizard wrote, “InSpec provides a mechanism to automate compliance management using a set of declarative tools that don’t require anyone on the IT team to possess programming skills.”

    The leveling of the playing field by removing the need for programming skills is crucial for DevOps organizations, noted Julia Dunn, Director of Marketing for Chef. “One of the main challenges to DevSecOps (getting developers, security and operations to work together) is communication: not only seeing each other’s point-of-view but having a common language to collaborate on requirements,” she said. “InSpec is a framework that allows for those requirements to be expressed unambiguously in something all three groups can understand. This is in contrast to the previous state, where security throws lengthy scan reports over the wall at operations, or developers toss insecure code at security folks for manual inspection.”

    Jon Williams, CTO of niu Solutions and a Chef InSpec user, has seen the benefits of such a technology. “InSpec has helped us unify our compliance, security and DevOps teams and streamlined audits, reducing the thousands of staff hours usually required by as much as 95 percent and eliminating duplication of effort and data throughout the process,” he said. “It has given these teams more control over compliance policies and enabled business units to be more active in maintaining their own environments. Most critically, it allows us to continually monitor for audit compliance, ensuring desired state and eliminating change drift between nodes.”

    Regardless of what technology you choose, it is important you have a process or framework in place that enables the various parties involved to speak the same compliance language. It is also crucial to automate as much as possible to provide continuous compliance rather than just passing a compliance audit and assuming everything will stay that way.

    — Tony Bradley

  • Chef Advances Compliance Automation

    Chef Advances Compliance Automation

    Chef today expanded the reach of its InSpec platform for automating compliance management with an update that adds support for Amazon Web Services (AWS) and Microsoft Azure public clouds, as well as integration with additional third-party tools. In total, InSpec 2.0 adds more than 30 additional capabilities, including support for Docker containers, Microsoft IIS and NGINX web server and PostgreSQL databases software.

    In addition, InSpec results now can be exported as JUnit format for integration into continuous delivery tools such as Jenkins. Compliance profiles can now also be pulled from Chef Automate.

    Plus, InSpec 2.0 runs 90 percent faster than InSpec 1.0 on Windows and 30 percent faster on Linux, according to Chef.

    Julian Dunn, director of product marketing for Chef, said the primary goal of InSpec is to enable DevSecOps teams to address compliance issues before an application gets deployed in a production environment. It’s not feasible for any DevSecOps teams to remember all the compliance requirements that must be applied when any piece of software or hardware should be configured, Dunn said.

    InSpec provides a mechanism to automate compliance management using a set of declarative tools that don’t require anyone on the IT team to possess programming skills. Dunn conceded it’s still early on in terms of incorporating compliance into any DevSecOps initiative, most of which are nascent efforts themselves. But it’s now only a matter of time before compliance shifts left along with security, Dunn noted.

    Security becomes even more pronounced in the age of the cloud because developers are now routinely deploying workloads on top of infrastructure that is not often managed by an internal IT operations teams, Dunn said. In fact, there have been a recent raft of security breaches stemming from the fact that data was left exposed on a public cloud such as AWS.

    There’s always been a fierce debate concerning the degree to which compliance encourages security. Advocates say that without some basic compliance requirement, the level of security inside most organizations would be more abject than it already is. Others contend that compliance requirements only encourage organizations to implement the bare minimum of security required; thereby creating a false sense of security that cybercriminals easily exploit. Regardless of viewpoint, however, compliance requirements are not going away anytime soon. In fact, major security breaches usually are not accompanied by hefty fines that are based on the number of compliance requirements that were ignored.

    Developers don’t always appreciate all the nuances of compliance when under pressure to deploy and update applications faster than ever. Organizations today need to find ways to enforce compliance mandates without slowing down the rate at which applications are developed. That’s simply going to impossible to achieve without increased reliance on automation. Today, unfortunately, most of what passes for compliance processes don’t begin until long after an application is deployed in production, which makes fixing those oversights once discovered a much more expensive proposition than if they were discovered much earlier in the application development life cycle.

    — Mike Vizard

  • Survey: Not Much Compliance Progress in DevOps World

    Survey: Not Much Compliance Progress in DevOps World

    A survey of 1,500 customers conducted by Chef illustrates the lacking state of DevSecOps in the enterprise today, finding that nearly three-quarters of IT organizations still manually assess whether applications comply with various regulations before being deployed to production. Worse yet, half the respondents also manually remediate any issue once it arrives, which takes days or sometimes weeks to accomplish.

    In fact, according to the survey, more than half the respondents (52 percent) have applied automation to fewer than 25 percent of the servers they manage.

    Manual processes conspire to make IT organizations less agile at a time when the pressure from the business to deploy applications and associated updates faster has never been higher.

    Julian Dunn, director of product marketing for Chef, said compliance—much like security—needs to move much further left into the DevOps process as a subset of DevSecOps. After all, most compliance regulations are trying to set a bare minimum standard for IT security.

    The first line of IT security defense is detecting vulnerabilities before they go into a production environment. Whenever a vulnerability gets exploited, most organizations incur the initial injury. But after the cybercriminals have come and gone the regulators show up, demanding explanations. Those explanations usually don’t do much to mitigate the fines and penalties that are assessed, especially in highly regulated industries.

    Chef has been making the case for applying the same constructs it developed to automate configurations to compliance for some time. Most compliance issues typically stem from one setting or another that wasn’t configured properly, noted Dunn.

    The issue is that most compliance officers inside organizations are unaware that many of these issues could be automatically resolved, and DevOps teams are too focused on daily operational issues to make anyone aware of what’s now possible. But now that DevSecOps is becoming more mainstream, many of the individuals involved are starting to realize they are dealing with the same issues time and again, said Dunn.

    Dunn added that, much like IT security professionals, compliance professionals who typically reside in a risk-management function need to be brought into the application development process earlier, but without slowing down the pace of application development. However, given the current level of manual remediation occurring, earlier participation mostly likely will reduce the level of manual work occurring right before the application is supposed to be going into production.

    When it comes to compliance, most IT organizations are their own worst enemy. Compliance requirements typically are spelled out in arcane language in documents that not many read, much less comprehend. Asking developers to make sense of those requirements is not the best use of their time. Besides, few of them would interpret the requirements correctly. An automated approach to compliance is not only going to create a better application, but it also should reduce a lot of unnecessary aggravation for all concerned.

    — Mike Vizard

  • Why DevOps Principles Fit Well in Highly Regulated Industries

    Why DevOps Principles Fit Well in Highly Regulated Industries

    DevOps is best suited to digital start-ups with minimal regulation and few restrictions, right?

    Initially the DevOps movement stemmed from the agile, innovative, born-on-the-web companies that were able to change their way of working quickly. However, the principles and practices are completely transferrable to all organization types. This includes those at the other end of the spectrum, where processes are very tightly controlled and regulation dictates a significant part of the business investment.

    Why Highly Regulated Companies Initially Resisted DevOps

    When DevOps first emerged, the major concerns for companies following strict regulations were to maintain their security controls and governance processes. Deploying changes more frequently was viewed as a risk to security and the governance controls that were firmly in place.

    Another reason for not moving toward a DevOps way of working was the “segregation of duties” requirement, making it impossible for developers to deploy into production—thereby not being able to “do” DevOps.

    Related Content:

    Break the Rules and Win at Digital Transformation

    Understanding is Key to Achieving DevOps Culture

    ‘Is DevOps Really Worth It?’

    There were established, formalized processes in place that satisfied all the regulatory needs. Some organizations saw making any changes to those processes as more work than what would be saved by automation. This meant that the processes stagnated and hindered future change.

    Despite this, the majority of executives now believe that DevOps is a crucial ingredient in making sure that their company can adhere to new regulations and stay ahead of their competitors in compliance adoption.

    Why DevOps is Actually Better for Highly Regulated Industries

    DevOps can be your biggest ally when it comes to regulation and compliance—as long as you follow a few simple rules:

    1. View auditors (regulators) as stakeholders in the DevOps journey.

    By collaborating with the auditors in the same way as other stakeholders, you can engage with them early in the process and ensure details get agreed upon without costly change further down the line.

    Getting buy-in from auditors on any technical solutions at an early stage reduces the likelihood of change requests, missed deadlines and non-compliance. By working closely with auditors, you will ensure that the company has a strong understanding of the level of compliance needed for each different aspect or component of their systems. This helps avoid the waste of being compliant in areas which are not necessary.

    Sometimes you will only be able to clearly interpret a regulation through close collaboration with auditors. This avoids misunderstanding and unnecessary rework. In some cases, working closely with the regulatory body can highlight aspects that had not been addressed previously, developing a feedback loop with them. This evolving feedback means that the company will end up with a solution that is satisfactory.

    Close collaboration with auditors increases trust in the processes which have been developed in your organization.

    1. Codify compliance requirements and policies.

    Good DevOps processes will increase the amount of audit trails and governance in the process, while enabling fine-grained traceability throughout the entire delivery process. Compliance documentation can be enforced more easily as part of projects. There is generally less need for process documentation, as so many tools now automatically generate any documentation required. By using collaborative sharing tools we can also reduce the problem of different document versions.

    1. Automate your delivery pipeline end to end.

    Automation gives reliability, repeatability and traceability. Automation means we have a predictable outcome and fewer manual processes, which is good for auditing and tight control.

    By having fewer manual tasks, we can also reduce the possibility of manual error and missed processes. Automated environment provisioning improves the quality of testing and gives a high level of confidence in the change.

    With end-to-end automation and process orchestration, controls are written in and embedded to processes and systems. This helps to reduce liability. Systems are designed to reduce the risk of individuals making errors, forgetting processes intricacies or even taking malicious action. This also helps to develop a blameless culture. Bullet-proof processes means that security protocols now become built-in. Rather than see DevOps as a threat to security, it is now being viewed as the best way to mitigate risk. It also becomes a way of enforcing security, audit and compliance requirements.

    Automation in all areas allows you to quickly evolve with changing regulation requirements and last-minute additions, helping you to stay ahead of your competitors who aren’t able to do that. Future regulation is likely to require reporting on the lowest-level processes, and the only way to ensure that processes are being followed reliably every time is to introduce automation and DevOps—a culture of ongoing improvement.

    What is Happening Now

    Some of these companies in highly regulated industries are actually starting to lead the curve in terms of innovation and DevOps adoption now. They have seen just how powerful DevOps can be and are creating an agile business that can meet regulatory requirements and evolve with customer needs, making their business much more competitive.

    — Bronwyn Davies

  • DevOps Chat: Chef Habitat Project Continues to Mature

    DevOps Chat: Chef Habitat Project Continues to Mature

    When Chef Habitat was released a little more than two years ago (good devopsy.kinsta.cloud story on Habitat here and a good DevOps Chat on it here), many people in the DevOps community wondered where Chef was going with Habitat. It was a really ambitious vision, but could it meet its lofty goals?

    Habitat is now made up of many different packages that provide different functionalities. There is, of course, much more to Chef Habitat. This DevOps Chat will tell you all about it.

    Joining us on this DevOps Chat are Jaime Winsor, a co-author of Habitat, and Tasha Drew, senior product manager of Habitat, and Blake Irvin of SmartB, a Habitat user. Between the three of them, there is a ton of great information here. If you are interested in Chef Habitat or would like to find out more, this is a great place to start!

    As usual, the streaming audio of our conversation is below, followed by the transcript of our conversation. Enjoy!

    Audio: Chef Habitat

    Transcript: Chef Habitat

    [su_pullquote][su_custom_gallery source=”media: 118811″ target=”blank” width=”250″ height=”250″]Blake Irvin
    Blake has spent over a decade working in systems and software engineering automation, on platforms ranging from on-premise metal to containers-as-a-service, BSD and Solaris systems to embedded Linux sensor platforms. He lives and works in Berlin, Germany, where he’s Senior Operations Engineer at smartB, GmbH, an energy-efficiency and sustainability startup.[/su_pullquote]

    Alan Shimel: Hey, everyone, this is Alan Shimel from devopsy.kinsta.cloud here for DevOps Chat, and we’ve got a really great chat lined up for this episode. It is all about Chef Habitat, and we’ve got some really-we actually have three guests on the show today, so we’re gonna move it along to keep it moving.

    Let me first of all introduce my two Chef guests or from Chef. One is Tasha Drew, who’s a Product Manager for Chef Habitat, and one is Jamie Winsor, an engineer with Chef and one of the co-authors of Habitat. Jamie, Tasha-welcome to DevOps Chat.

    Tasha Drew: Thanks so much. Happy to be here.

    Shimel: Thank you.

    Jamie Winsor: Yeah, thanks for having us, man.

    Shimel: Hey, it’s my pleasure. And then, joining Jamie, Tasha, and myself today, all the way from Berlin, is Blake—and I hope I’m not gonna mess this up—Irvin. Blake, are you there?

    Blake Irvin: Yep.

    Shimel: Okay. Thanks. Hey, Blake, I appreciate you taking the time to join us. Blake, you’re with, is it BusyBee, is that what’s the name of the company?

    Irvin: It’s called smartB.

    Shimel: smartB.

    Irvin: Yeah, and the B essentially stands for business, because we are building, because we focus on providing energy efficiency analysis for businesses—small to very large businesses and the buildings that they operate.

    [su_pullquote align=”right”][su_custom_gallery source=”media: 118812,118813″ target=”blank” width=”200″ height=”200″]Tasha Drew
    Tasha Drew is the Product Manager for Habitat. She previously launched Chef Automate’s analytics capabilities, was Head of Product at Rentlytics, was responsible for product for Engine Yard Cloud, and spent several years in various data center endeavors on behalf of Lockheed Martin. Tasha holds an M.S. in Management from RPI and a B.S. in Computer Science from USC. When not in front of a computer, Tasha prefers to be cruising around Napa or at the beach with her dogs—all while having intensely deep thoughts about how to save application developers from tedious rework via the wonders of application automation with Habitat.
    Jamie Winsor
    Jamie Winsor is a lead engineer at Chef Software and the coauthor of Habitat, an open source project built upon distributed system protocol Butterfly to provide a self-healing, self-configuring, stack-agnostic, frictionless abstraction for running applications—regardless of their complexity—to software developers. Jamie has been a software engineer in the video game industry for 10 years, focusing on networked application servers on such titles as League of Legends, Lord of the Rings Online, and Dungeons and Dragons Online. One of Jamie’s responsibilities in his game development tenure was to bring what we today know as DevOps into the daily lives of the other developers on his team, which Jamie accomplished by building, evangelizing, and teaching methods to his peers. He draws on that experience today in building Habitat, as he helps enable all software developers, regardless of their experience, bring their ideas to life without investing in the details of operationalizing an application.[/su_pullquote]

    Shimel: Wow, that’s really good stuff. Jamie and Tasha, you both work for Chef, no need to introduce who Chef is to our DevOps audience, but Blake-welcome. And we have you on today because we’re gonna talk a little bit about Habitat, and Blake, if you don’t mind, I wanted to start with Tasha and Jamie and talk a little bit-just a little background. Maybe we have some folks on here who are not familiar with Habitat.

    So, Jamie, you’re a co-author. Why don’t you take that one, right? What’s Habitat, for people who may not be familiar?

    Winsor: Yeah, sure. So, the Habitat project is a number of pieces of software, and the overall goal is to, I hope, redefine what application automation means by focusing on the application instead of the infrastructure. In the past, we have been building computers in service to host our applications, and what Habitat does instead is it says, “Let’s focus on what the applications are that we need to run our application, what software we need down to a bare minimum, and then provision the machine around it,” and Habitat runs exactly the stuff that you need.

    It does so with a packaging system and a build system and then a process supervisor. The process supervisor runs the packages, the process supervisor communicates to other supervisors using a gossip protocol to share and spread information about what software they’re running, which allows us to enable things for free, just by packaging your software with Habitat, such as service discovery, leader—follower failover, rolling configuration updates, rolling software updates—and it’s all coordinated using the gossip ring.

    Finally, the piece that we built, the last piece of Habitat is simply called Builder, which we can also chat about today, which we just launched about a month ago. And Builder is a hosted Software as a Service build service that’s currently in preview mode. It’s free, and it builds your software that’s packaged with Habitat and hosts it as well in release channels, which allows you to have supervisors subscribe and then receive automatic updates using a complete CI/CD process of promotion between an unstable branch to a stable branch and so on. That’s basically it.

    Shimel: Yeah, no, more than basically-that was a great description and great stuff there, Jamie.

    Tasha, now, Habitat was released as an open source project by Chef, and it’s probably about two years ago now, right?

    Drew: Yeah, yeah.

    Shimel: And so, Tasha, tell us kinda what’s been the latest with Habitat.

    Drew: Sure. So, basically, we have been working nonstop since the initial reveal of Habitat about two years ago, just adding functionality and capability across both that Builder system and that process supervision system that Jamie’s been talking about.

    We’ve also been working on the deployment piece of the overall integration between those two which, in Habitat’s world, is highly agnostic. So, the idea is, use whatever tools you need to deploy those artifacts and supervisors that you’re using with Habitat. And so, we’ve been working with partners to have a really great cloud foundry ecosystem story, Kubernetes ecosystem story, AWS and Azure story, and then Terraform as well.

    So, the idea is, we have been making it easier and easier for people to come get started using Builder, and that’s why we’ve released Builder in free preview mode, it’s because we just want to have this very low friction way for people to get started with Habitat and start rebuilding all of their workloads, and then deploy using your favorite technologies and then start seeing the benefit of the supervisor in Builder, communicating between the two.

    Shimel: Absolutely. And as you mentioned, Builder is in free preview right now. You know, some people look at things like Habitat and some of the other great open source projects Chef is kind of shepherding, and they say, “This is all great, great, great, but how—you know, where does, how do they stay in business? Where do they make money?”

    So, Tasha, can you talk a little bit about sort of where Chef sees commercial opportunity with Habitat?

    Drew: Yeah, absolutely. So, for our enterprise customers, having on prem solutions that integrate with all of our open source products in order to have that really strong control plane management layer is super important. And so that’s, with Chef Automate, something that we continue to invest and see incredible success with.

    And then, with Habitat, Habitat becomes the technology that continues to support enterprises while they want to move fast to deliver applications to production at scale. And so, having this part of the technology suite be adopted by our core set of enterprise customers will continue to be part of our strategy, but then we also have Builder to enable smaller shops to just get started—really low friction point, really easy to use. And how we monetize Builder itself over time will be something that we investigate as we see how people adopt and use the technology.

    Shimel: Excellent answer. Great, Tasha. So, Blake, I didn’t wanna leave you too long in the cold. Let’s bring you into this now. First of all, Blake, I don’t think I asked you, and I don’t know if our audience is savvy to it, but what exactly is your role at smartB?

    Irvin: So, I—we have a team of sort of multidisciplinary, polyglot engineers at smartB. We do all kinds of things because we’re a small shop and a small company, still. But the focus that I have—and this has been my focus and my interest, sort of, for the last, it’s actually been a little bit over 10 years is on all manner of operations engineering.

    So, what I used to do would be something more like traditional systems administration, and now I would say I’m more of a software engineering operations specialist. In other words, I’m focusing on improving and streamlining the process of software engineering and less focused on things like racking servers.

    Shimel: Sounds like you’re doing DevOps to me, man. [Laughter] But Blake-

    Irvin: Yeah, you could say that.

    Shimel: Yep. So, Blake, smartB is using Habitat?

    Irvin: Yeah, that’s true.

    Shimel: Let’s talk about why.

    Irvin: I would say, for the moment, about-sorry?

    Shimel: I was gonna say—why? What are you using it for?

    Irvin: Well, we’re using it for almost everything. We have one—yeah, we have one production service that is not yet what I called Habitized or wrapped up in the Habitat package, and we’re just waiting for some work in the community to be finished before we can do that. But yeah, like, 85, 90 percent of our services are running under Habitat, using the Habitat supervisor, and we’re using as much of that workflow that Jamie and Tasha were describing as we can at the moment.

    And that’s actually been a pretty good process for us, I think, because the way that Habitat’s designed, the adoption process can be incremental. So, when Tasha was referring to giving Habitat users the freedom to pick and choose how they want to deploy or how they want to go about adopting, that’s definitely something we’ve leveraged a lot to make our story good for ourselves.

    Shimel: Got it. And so, Blake, I mean, when did you guys adopt, start with Habitat?

    Irvin: So, when I saw the launch video, which was, I think, Adam speaking in Seattle-Adam Jacob, the Chef CTO. I was sort of, I was watching it originally just out of curiosity and then, as the talk went on, my mouth started hanging open further and further, I think. Not because I thought it was the most groundbreaking thing that I’d ever seen, but more because I couldn’t-I was surprised to see how many of the check boxes that I had in my mind from 10 years ago around this kinda stuff, all this kind of operations work, how many of those check boxes were getting checked off by the project as the architecture and the behavior of the tools was described.

    So, we really started looking at the feasibility of adoption within weeks, I think, of that launch video.

    Shimel: Wow.

    Irvin: And started doing the rollout probably a few months afterwards. So, we’ve been doing it like a very slow, incremental rollout to, you know, ensure that we don’t disrupt any of our current software development process or our current production environments. That’s what’s been going on. It’s been a continual process for, I’d say, like eight months now, but just done very slowly.

    Shimel: Very cool. So, Blake, you know, I have a long history of working in open source-related projects and stuff, and I’m always curious, you know, as a consumer, if you will, or a user of this open source project, have you guys—what have you contributed back? Have you contributed any code, bug findings, suggestions for new features? How are you giving back to the community, here?

    Irvin: Yeah, sure. I think the thing, since we’re a really small team, what we’ve really focused on is trying to use the tool in the way that it’s been designed to be used and then giving feedback to the community and/or to the core maintainers whenever we find bugs or places we want to see improvements, and just keeping that conversation really open and fluid, which has been great. It’s one of the reasons that I’ve continued to use Chef software stuff for—not quite my whole career, but a sizable majority of my career. It’s just that, I think, as an organization, Chef believes that community is very important, and in fact, perhaps, one of the most important things about building a company. And compared to other experiences I’ve had with other open source groups, I think that’s been a real win for everybody involved.

    So, in terms of code contributions, I think mostly what we’ve done is put out a lot of plans, which are the sort of blueprints for packages that Habitat uses when you set up like a set of dependencies for your application. So, we’ve definitely published a lot of those back into the community. I think you can find them if you look on the Builder site under the smartB origin, you’ll see all the packages we’ve published. And we’ve also published some stuff back into the core plan side, just the ones that the Habitat team maintains.

    Shimel: Got it, got it. So, Jamie and Tasha, does Blake and smartB sort of represent the kind of prototypical Habitat user?

    Winsor: I’d say so. You know, with Habitat, you can pick and choose what pieces you want, and it’s been great to watch Blake and smartB grow from—back to the packaging manager and the supervisor and gradually moving toward adopting Builder. They’ve been great to work with in the community. They gave us a lot of great feedback. You know, they’re users, so one of the things that is really important is, you build a feature or you build—you know, you fix a bug, and then you immediately get the feedback from the community, and they’ve been unbelievable to work with.

    I’d say that Habitat would work for any team that ships software, though. So, they’re absolutely a typical user, because they ship software, right?

    Shimel: Mm-hmm. Cool beans. Guys, we’re almost at the end of time, unfortunately, leading up here over 15 minutes. But Tasha, for people who may want to, you know, go explore Habitat themselves, where can they go to get information?

    Drew: Yeah, we would love them to visit our website, Habitat.sh. There, you can see, we have a quick, you know, just kick the tires, 10-minute kinda step-through demo. We have more in-depth tutorials if you want to start just playing around with both the run and build time features of Habitat, and our community is very active on our Slack channel, which you can find a link there to join, but it’s just at Slack.Habitat.sh. So, as you get started, if you wanna just jump in that Slack channel, the entire core team is there, our community is there, and we seek to be super welcoming to new people. New questions are very welcome in the general channel. So, you can just hop in and get support and ask whatever questions you have.

    Shimel: So, it’s Habitat.sh?

    Drew: Yeah, yeah.

    Shimel: Excellent, excellent. Guys, listen. Thank you very—all three of you, Blake Irvin, Jamie Winsor, Tasha Drew—thank you so much for being our guests in this episode of DevOps Chat. Continued success, Blake, with you guys in your bigger business as well as using Habitat, and Jamie and Tasha, with you guys continuing to develop and make Habitat, you know, an indispensable tool for companies around the world. Alright?

    Drew: Thanks so much.

    Winsor: Thank you.

    Shimel: Alright, everyone, have a great day. This is Alan Shimel for DevOps Chat. We’ll hopefully see you soon on the next chat. Until then, take care, everyone.

    — Alan Shimel