devopsy.kinsta.cloud

Tag: regulatory compliance

  • Overture Maps’ Challenge to Google ¦ Frontier Model Forum’s AI Safety Shtick

    Overture Maps’ Challenge to Google ¦ Frontier Model Forum’s AI Safety Shtick

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: A big-tech consortium goes toe-to-toe with Google Maps, and a big-tech consortium tries to fool AI regulators. (more…)

  • Can Regulatory Mandates Secure Software Development?

    Can Regulatory Mandates Secure Software Development?

    Software companies have a long history of delivering incomplete and insecure products. This happens for a couple of reasons. Fast time-to-market has always been a business priority, taking precedence over security, especially as DevOps has become the norm in software development methodologies. And software, by nature, is easy to update and change, so some defects or oversights could be allowed to slide through with the thought that “We can always fix it later.”

    Unlike car manufacturers that put a premium on including safety features upfront (or face the heavy hand of regulators), software makers have typically emphasized speed to market as a way to stay competitive.

    But the other shoe often drops on that approach in the form of costly recalls, extensive patch programs, cybersecurity breaches, and other issues. The fix-it-later mentality in software has become increasingly challenging in recent years with the emergence of connected/IoT devices that are harder to keep current and have a greater impact. Connected closed-circuit surveillance cameras, for example, are frequently hunted and exploited on the internet to spy on people. Internet of things (IoT) devices operating at dams or nuclear power plants can now be hacked (as seen at the Bowman Avenue Dam in New York, or three utility companies in Ukraine hit by BlackEnergy malware). And, across the board, the cost of data breaches continues to rise.

    A White House executive order (EO) from 2021 is also putting pressure on software manufacturers, DevOps and DevSecOps professionals to improve quality, with a mandate for U.S. federal agencies to enforce secure software development through their contracts and acquisitions as part of efforts to secure the software supply chain. The core difference between this effort versus previous attempts such as FISMA, is the visibility into the software development process.

    The attention being paid to secure software development is a step in the right direction, but ultimately it’s still up to the industry rather than regulatory bodies, to change the mindset toward building secure software from the ground up.

    Technology is Faster Than Mandates

    The EO represents a shift in the landscape, mandating not just secure products, but secure development practices. Although it applies officially to federal civilian agencies and their contractors, federal regulations and other government mandates often become global standards, as has happened with the General Data Protection Regulation (GDPR), the EU’s privacy and security law.

    But regulatory mandates aren’t enough. They often move too slowly to be a real solution, especially in a software development environment that is moving steadily towards more agile, fast-paced environments where continuous integration and continuous delivery (CI/CD) is the norm. The timelines for new regulations typically require a two-year window for implementation; if a company already has an applicable product in use, it could be up to five years before they’re required to be in full compliance, such as in the payment industry. By that time, the key elements of the industry best practices have already changed. Simply put, technology moves too fast for regulatory bodies to keep up.

    What’s more, the regulations themselves could ultimately fall short if they are perceived to have a high cost burden of compliance. This is specifically true if the technology used in the software development life cycle can’t keep pace with the needs of compliance. In support of the EO, the National Institute of Standards and Technology (NIST) has released Draft Special Publication (SP) 800-218, and, as is routine, accepted open public comments. The feedback from industry consortiums in these situations often includes considerations about the costs of implementing new regulations and can sometimes lead to softening of some aspects, arguing, for example, that the new regulations put an unfair burden on small and mid-sized companies. The final set of regulations can sometimes leave the door open for ways to address compliance without addressing the issues that led to the regulation in the first place.

    For example, the California Consumer Privacy Act (CCPA) is widely hailed as the nation’s strictest set of privacy protections, yet it still has its weaknesses. The most commonly cited weakness is that the law makes it extremely difficult for consumers to take advantage of the new regulations when they feel they have been wronged.

    Industry Must Make it Happen

    Ultimately, it is up to software manufacturers and the industry to improve the security of their software. They should adopt best practices to build secure procedures into the design, development, testing and validation of new software, as well as its deployment and use. NIST’s Secure Software Development Framework is one example of guidance, being drawn from recommendations from groups like BSA, OWASP, and SAFECode. Industry groups, such as the International Organization for Standardization (ISO), could adopt standards for development and certification similar to the universally accepted UL standards for electrical products.

    The rising demand for secure development requires a substantial shift in attitudes toward how software is made. A lot of these things can be more easily implemented if organizations invest in the adaptation of new technologies faster.

    The threat landscape has become too sophisticated and pernicious for the problem of insecure software to be ignored. Fix-it-later is no longer a valid option if organizations want to protect their networks and, especially, their data, which has become the lifeblood of operations in every sector. While regulatory mandates help, the job ultimately falls to the software industry itself and its consortium of like-minded people bridging the gap between developers and DevOps teams of competing companies to ultimately deliver what’s best for the consumers.

  • Rethinking Your Approach to Ops Governance

    Rethinking Your Approach to Ops Governance

    The “You build it, you run it” approach has changed the mindset of modern organizations that are increasingly adopting DevOps practices. No longer completely beholden to the requirements of operations, engineering teams are continually acquiring more autonomy to deliver rapid innovation. At the same time, more organizations are adopting platform teams to accelerate development, empowering teams with easier access to services without requiring them to understand the underlying complexity. While this approach is helping teams build, scale and ship software faster, it’s also resurfacing previous areas of tension around operational governance. 

    Organizations are generally choosing between two governance approaches: Mandated or paved path. Each has a different perspective on the appropriate level of autonomy to grant development teams and where governance should sit within the development life cycle. But there’s a third approach—the “trust and verify” approach—that can provide both high levels of autonomy for development teams and the sophisticated governance that operations teams need to ensure compliance and security.

    What is a Platform Team and Why are Organizations Adopting the Model?

    A platform team provides a foundation of self-service APIs, tools, services, workflows and support that accelerates the delivery of digital services and products. Platform teams operate similarly to any R&D team; employing customer support, operations, development, SRE, product managers and other personnel. They use research, customer feedback and usage data to iteratively improve their services.

    Rather than each development team reinventing the wheel, the platform team builds shared services that shield developers from the underlying complexity of infrastructure and enable self-serve access to services. As engineering resources become even more in-demand and scarce, organizations are using platform teams to optimize resources and share outputs.

    Two Platform Governance Approaches: Mandated Vs. Paved Path

    Organizations today typically are using one of two approaches to platform governance: Mandated or paved path.

    A mandated approach solves the governance problem by mandating that all engineering teams use the platform. However, this approach puts the onus on the platform team to keep up with the changing demands of engineering. Organizations must invest in a platform team that can keep up with engineering requirements so they still feel like they can innovate and use the best tools.

    The second approach is a paved path, popularized by Jeff Lawson of Twilio. This allows for engineers to go outside the platform if they are willing to take on the SLAs the platform provides and are responsible for all the costs of maintaining their own alternatives. Lawson said in this McKinsey interview, “What you do is you create the incentive structure for teams to take the paved path. But if they really have to go a different route, you make it possible for them to do that.”  

    While the paved path approach offers more autonomy for engineering teams, it requires teams that veer off the path to maintain the integrity of their software. Without guardrails, operations must put full trust in teams to keep their services secure and compliant. It can often be difficult for operations teams to maintain visibility into the services and infrastructure that fall outside the core platform.

    So, given that a mandated approach restricts autonomy and a paved path can make compliance more difficult, what is the middle ground?

    Bridging the Gap Between Mandated and Paved Path

    First, it’s important to understand that governance is not synonymous with command and control. Instead, operations teams should think of governance as the practice of maintaining visibility and providing best practices. Rather than a mandated platform approach, organizations can choose an approach that offers more autonomy to engineering teams (similar to paved path) while also maintaining operational governance. This is a trust and verify approach. 

    Using the trust and verify approach, organizations should focus their efforts on technology, not people, to maintain compliance. The goal would be to have a platform that could easily connect to the entire toolchain any product team uses for complete visibility and context. In this way, operations can gain governance without adding friction for product teams. This requires a top-down approach within organizations: I&O leaders investing in engineering resources or an off-the-shelf platform that can enable this technology. Both engineering and operations can flourish in this modern environment. Approaching governance from a trust and verify perspective will expand access to innovation while ensuring compliance and security standards. 

    Governance Should be Over Technology, not People

    Creating a bridge to a trust and verify approach means placing the responsibility for governance on technology rather than people to maintain detailed audit trails of human and machine actions tied to identity and reduce the potential for human error through automation. This approach provides autonomy for engineering teams while also ensuring governance through visibility and process controls embedded into workflows and tooling. 

    Shellie Miller, senior manager, technology governance and compliance at Hulu, believes technology can support DevOps practices. “To have a successful DevOps organization, we don’t need to have command and control or have this clear line of separation between dev and operations. It’s putting the responsibility on the infrastructure, the platform and the tooling and the policies and the processes that you have to enable those self-serve capabilities,” Miller said. 

    Mitigating Human Error Through Automation

    This investment in technology should focus on automation and process controls that fit within teams’ current processes. “What auditors are really trying to do is mitigate human error,” said Miller. “Teams should really focus on automating controls, building those security practices and those best practices into the platform. That way, dev teams are just subscribing to that offering.” 

    Changes to applications, systems, data and code—one of the biggest culprits with regard to risk—is where automation and process controls embedded into platform offerings can make a huge impact. “From an engineering perspective, constant change is needed to support rapidly changing consumer demands,” Miller added. “However, from an audit perspective, these activities increase risk of material impact to viewer experience, revenue and/or financial reporting.”

    Investment in Governance Comes From the Top

    While technology holds the key to better governance and compliance standards, there’s an investment that must be made, whether that’s investing engineering resources in building automation or investing in an off-the-shelf platform that can enable automation.

    “Leadership and organizational backing is absolutely key,” said Miller. “You can’t get to compliance or governance or any kind of operational maturity, for that matter, when you’re doing a bottom-up approach.”

    Building a compliance and security program around technology and not people means having the proper skills and mindsets. A bottom-up approach could create friction, especially due to a lack of education and understanding of the why and how. A top-down approach ensures there are resources to both implement the technology and to foster a culture of understanding and education. 

    The most common question I get asked is, ‘Why can’t we trust our people?’” Miller said. “Compliance requirements don’t generally start with the thought that our people are being malicious. We should run an organization of trust, and through automated monitoring and governance, we can verify and govern without being intrusive and disruptive. Lack of training and the demand on teams is what drives the perception that there’s no trust. Security and compliance is intended to mitigate risk and prevent threats to the systems that enable the business.”

    Building on a “Trust and Verify” Platform Approach

    When it comes to governance, technology is our greatest asset. Of course, this investment can be substantial for many organizations that don’t have the engineering resources to build in-house. Today, new open source or off-the-shelf platforms enable operations teams to build self-service capabilities that provide guardrails against human error, an audit trail of human and machine actions and visibility across disparate tools and services. 

    No-code solutions that also provide a developer platform for deeper customization enable teams to build out these core requirements of governance without consuming considerable engineering resources. The door to a future where operations teams can provide more autonomy to development teams while maintaining governance is now open. Every organization has the ability to attain a “trust and verify” approach and meet the needs of both dev and ops.

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

  • Tackling Audit Compliance as Code

    Tackling Audit Compliance as Code

    Do the words, “It’s audit time!” make your stomach sink? If so, you’re not alone. Assisting with evidence collection for compliance audits around PCI DSS, SOC-2, ISO 27001, NIST and HITRUST is a drain on DevOps teams’ time and resources at companies of all types and sizes – time and resources that could be better spent driving innovation.

    The good news is, vendors have taken notice, and are delivering new tools and platforms that use automation to eliminate repetitive manual tasks and dramatically speed the audit process. This is entirely in line with the DevOps philosophy of tackling various IT tasks and requirements “as code” to remove tedium and improve efficiency and orchestration. In fact, as this movement to improve the compliance process picks up steam, I believe it’s entirely likely the next evolution is the use of open APIs and online communities where engineers can build and share software to automate a wide variety of evidence collection for compliance.

    But before we can get there, let’s understand where we are today and assess new opportunities going forward.

    Audit Support: DevOps’ Favorite Task

    As keepers of cloud environments, DevOps teams typically spend hundreds or thousands of hours each year gathering evidence for compliance audits. This tedious, manual labor takes significant time away from their primary job responsibilities. Large enterprises often have multiple, overlapping audits each year, with little break from the grind of evidence collection. Companies that make heavy use of the public cloud face additional challenges; the increased complexity means that evidence collection is more difficult, and the fast pace of software development and change management in the cloud (compared to more traditional monolithic application updates) makes it difficult for compliance to keep up. This contributes to an overall lack of clarity and confidence in the compliance process at the leadership level.

    Audit Prep as Code

    When I ran a cloud engineering team at a previous job, we realized the vast majority of repetitive, manual labor involved in evidence collection could be automated to save time and free up resources for other business needs. This automation also improved the completeness and accuracy of the evidence gathered, ensured that important areas weren’t overlooked and allowed for more regular assessment to prevent “drifting” out of compliance in the fast-paced cloud world. While automation helps with multiple aspects of compliance audits (such as evidence-to-control mapping and cross-walking evidence across common control frameworks), it’s the automation of evidence collection that primarily impacts DevOps.

    In this scenario, automated collectors crawl your public cloud environments and connected SaaS systems to rapidly gather technical evidence such as specifics on user access, encryption of data at rest, key management, network segmentation, firewalls, vulnerability scan reports and more. The results are transformed into formatted reports ready for internal review, analysis and auditor review/approval. This, alone, can save thousands of hours annually for typical enterprises facing multiple audits. Even better, the data is considered more complete and accurate from a compliance perspective, because it includes time stamps and other metadata to ensure capture and processing integrity.

    As mentioned, this is well aligned with the broader “X as code” mantra that underlies many DevOps automation initiatives – and why not? What could possibly be a more repetitive, manual task than capturing, renaming, organizing and describing screenshots?

    The Scripting Curse

    When confronted with evidence collection, an engineer’s first thought is usually, “I can write a script for that!” Scripting is a form of automation, but it has some limits in this scenario. First, scripting and doing the necessary QA shouldn’t be your day job; there are better things to spend time on. Second, given the explosion of cloud infrastructure and SaaS apps, there can be a lot of scripts to create. In fact, it’s arguably a never-ending task. Finally, while you understand your scripts, who’s doing the next audit? If it’s not you, are you going to explain and train someone else on all the work you’ve done? Scripting elements of evidence collection is an improvement over doing it manually, but is still only one piece of the larger puzzle.

    Compliance is a Community, not an Island

    So what’s the best way to automate evidence collection? Multiple vendors are tackling the challenge, from cloud platform providers to established GRC vendors and cloud compliance startups – and new tools have already begun to hit the market over the last few months.

    But as discussed, a primary challenge is that there is a lot of complexity in most cloud environments, and tools that approach the puzzle from only one direction – say, a particular cloud platform – still leave a lot of potentially relevant systems untouched. In fact, when you take all of the variations possible with cloud infrastructure into account, this might, ultimately, be a challenge that is best tackled at the DevOps community level.

    Consider this: adherence to industry standards for data handling, privacy and protection is not a competitive issue. We are all better off if everyone shares a basic commitment to complying with these IT security and compliance standards. Viewed from that perspective, the more we can lower the barriers to achieving this compliance, the broader the adoption will be.

    What I hope to see is a DevOps community that coalesces around automating audit readiness. At the engineering level, this could entail sharing of custom collectors. For platform and SaaS vendors, it means ensuring open APIs that allow capture and extraction of relevant data. And the compliance tool vendors must design their collection engines, evidence libraries and usage policies such that customization and sharing is possible. Similarly, automation tools must not only pull data from standard systems, but support pushing data from custom systems or legacy infrastructure. In my view, this bottoms-up approach is the best way to stop wasting DevOps’ time on audit prep, while still allowing enough autonomy to do what works best for their specific environment.

    In this future, then, perhaps an upcoming compliance audit won’t produce a collective groan from the DevOps team – or at least, generate a quieter one.

  • How to Automate HIPAA Compliance with DevOps

    How to Automate HIPAA Compliance with DevOps

    DevOps can help make HIPAA compliance more achievable

    Automating the provisioning of HIPAA-compliant server infrastructure enables compliant hosting service providers with the ability to provision and deploy infrastructure-as-code with minimal human intervention. Providing the automated process is thoroughly tested, revised and updated, it can offer a healthcare organization huge flexibility when enforcing a HIPAA-compliant server or serverless infrastructure within a private, hybrid or public cloud offering.

    Organizations that practice centralized operations, also known as DevOps, have a unique team who have the desirable skills of software development, IT operations and quality assurance (QA). The DevOps objective is to create a performance-based IT solution within a greatly reduced provisioning lead time using technical server blueprints within a continuous delivery environment.

    DevOps within a healthcare organization must be able to meet the stringent administrative, physical and technical safeguards required by HIPAA. In all cases, the required safeguards must be met, but with DevOps, it can be significantly easier to meet the addressable requirements of HIPAA.

    Planning

    Each healthcare organization can present differing business needs which they need a HIPAA-compliant service provider to resolve. It is not always possible to provide a cookie-cutter solution for every healthcare organization, often one design does not fit all; however, it is possible to create a baseline of operations using automation.

    A great deal of planning must be completed with all relevant stakeholders to not only understand the technical solution but to gather critical information about unique aspects of the architecture and then plan ways to automate the delivery process.

    It may not be possible to automate every workstream, but getting the DevOps team involved in the early planning stage will allow thorough and accurate playbooks to be created.

    With advanced planning, significant gains can be made in defining the networking security layout for the healthcare clients virtual private cloud, work out what isolated cloud resources can be defined and make decisions about potentially leveraging network layer protection services such as DDOS protection or a software-defined web application firewall.

    Provisioning

    The next step to creating the infrastructure-as-code is to write the automation playbooks that will provide the infrastructure. There are several provisioning applications that can be used, such as Terraform, Chef, Puppet, Google Deployment manager or AWS Cloud Formation, and playbooks will need to be written (or converted to) a compatible language such a python, YAML or JSON.

    The machine-readable code interacts with the service provider’s API front end; this may be an on-premises VMware vSphere cluster or a public cloud provider such as AWS, GCP or Azure. The API interprets the code and uses the descriptor commands to provision and deploy infrastructure in a cloud environment.

    A DevOps engineer will select the cloud provider they using and then define all the environmental variables to use to create the infrastructure-as-code. Connection variables are usually configured first and will define the cloud provider locations, regions and availability zones, as well as the encrypted account passwords or IAM configuration to interact with the provider API.

    Once the connection variables have been initiated with the provisioning software of your choice, they are used throughout the project life cycle. The next stage is to define the HIPAA infrastructure environment, including storage, network and compute resources. This will include provisioning resources and setting private and public networking.

    At all stages of the design it important to define where ePHI data will be stored and processed. The process must include step-by-step documentation and a customer responsibility matrix. At this stage, it is imperative to incorporate technical safeguards such as server access control lists (groups), introduce activity logging and provide auditable controls.

    Server hardening policies must be created for servers, active directory, workstations or providing least privilege authorization accounts at multiple levels, such as user profiles, that enable or restrict access to individuals or third-party systems. Preapproved software is typically installed at this step as well.

    Next, the inclusion of boot volume encryption and encryption of disk objects and storage volumes must be defined, this is easily achievable on mass provisioning of servers, simply providing a volume ID and an encrypted $TRUE statement.

    Throughout the playbook designing phase, testing is conducted to close loopholes, refine automated network configuration and create shared VPN tunnels with passwords that automatically cycle. The playbooks set the properties which make an organization HIPAA-compliant, and can also be used to create secured bastion servers if VPN is not an option.

    Playbooks allow DevOps teams to create rotating security keys, SSL certs and log aggregation and filtering. This can also be configured to trigger automated messages that can be emailed in case of any errors or failures in the automation process.

    Continuous Delivery

    When all the playbooks are streamlined and accurate, they can be used to define a healthcare clients service catalog, which enables the use of standardized templates to be used within the HIPAA environment. This may include server templates, container configuration, storage templates or any predefined software application.

    Security tools can be used to track and inventory resource usage and any system changes, track all user activity and any API activity requests.

    Monitoring tools can be used such as Cloud Watch, Nagios or Nimsoft to monitor server resources and application usage, these monitoring tools can also pick up errors or predictive errors and alert against predefined thresholds.

    Many organizations choose to use serverless workloads such as Docker or Kubernetes. These massively scalable solutions also bring about a huge benefit that aids the DevOps continuous delivery approach of automated update rollouts—essentially, applications can be isolated over differing pods, allowing the underlying hardware to be updated or versioned with zero downtime.

    DevOps is changing the automated delivery of HIPAA-compliant solutions. It is important to remember that DevOps does not have all the answers for HIPAA, especially many of the administrative regulation. However, it does empower managed service providers with the ability to provide tried and tested automated solutions that meet or exceed the technical and physical safeguards of HIPAA compliance.

    — Moazzam Adnan Raja