Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: A big-tech consortium goes toe-to-toe with Google Maps, and a big-tech consortium tries to fool AI regulators. (more…)
Most organizations are not fully compliant with the European Union’s (EU) General Data Protection Regulation (GDPR) before it takes effect in May, analysts report.
While DevOps certainly gives organizations the mean to turbo charge the compliance process, stakeholders outside of IT departments often are failing to greenlight this necessary project ahead of the deadline, due largely to uncertainties about the regulation.
Not being in compliance could hit organizations hard, with fines up to 4 percent of their annual revenues or €20 million.
The Cloud Security Alliance (CSA) and Netskope, for example, recently released the results of a survey of more than 1,000 respondents about their organizations’ GPDR readiness. The survey revealed a full 83 percent were not very prepared for GDPR. While 85 percent had taken steps for compliancy, only 31 percent had well-defined plans for meeting the provisions of GDPR.
The GDPR’s “right to erasure,” (53 percent) “data protection by design and by default,” (42 percent), and “records of processing activities” (39 percent) were deemed the biggest stumbling blocks for compliancy, according to the study.
A separate survey of 700 network security and IT professionals by Enterprise Strategy Group (ESG) showed only 11 percent of firms were completely ready for GDPR ahead of the May 25 deadline. Thirty-three percent said they were prepared to publicly disclose data breaches within the GDPR-mandated 72-hour period, and 33 percent and 44 percent percent of the firms surveyed were mostly or somewhat prepared, respectively.
Most organizations ESG surveyed also reported they are investing in GDPR, reflecting the seriousness in which they are regarding the mandate. While only 10 percent said they had made or were planning to make major investments in GDPR, 63 percent said they would make or had made incremental investments.
Analyst firm Varonis made similar findings after completing a survey of 500 IT security experts in the United States and Europe in October, out of which 200 were in the United States and the rest of those surveyed were equally split between the UK, France and Germany.
Varonis also found 60 percent in the EU and 50 percent in the United States faced serious challenges before they could be GDPR-compliant. However, 70 percent of the firms represented were also convinced GDPR compliancy would give them a competitive edge.
Much of the uncertainty about GDPR reflects how organizations often fail to understand where exactly their data is located. While not specific to GDPR, Varonis found many organizations lacked fundamental data-protection protocols. Those surveyed revealed, for example, 21 percent of their organizations’ electronic folders were “open to everyone.” The survey also revealed 54 percent of the firms’ data was stale. Effectively tracking data extending from on-premises servers to across the cloud was often cited as an issue, making it difficult to track the exact origins of customer data and to determine whether it might be subject to GDPR protection or not.
“We live in a hybrid world in which data isn’t stored entirely on-premises or completely in the cloud. The hybrid model creates additional challenges and potential risk from a security and data governance standpoint,” Ken Spinner, vice president of global field engineering at Varonis, said. “You can’t assume your data is safe in the cloud. If your cloud environment isn’t secure, your data won’t just be in danger of being exposed to your entire organization—it could be accessible to the world.”
A lack of understanding about what GDPR compliance involves is often cited an impediment to making necessary changes for compliancy. Much has been reported, for example, about whether EU citizens, residents, or simply those who are only temporarily on European soil are protected by GDPR. How the EU will impose fines against non-EU organizations also remains murky. According to ESG data, less than one-third of the respondents of the organizations that took part in its survey fully understand all the GDPR’s requirements.
These questions, of course, largely need to be resolved by legal counsel. Meanwhile, many legal departments are waiting for more guidance from the EU about uncertainties mentioned above, as well as other remaining questions about the GDPR’s scope.
In other words, DevOps is ready to do the necessary, while legal and other stakeholders who fully understand what needs to be done must pull the trigger first. Only then can DevOps, consisting of IT operations, QA, InfoSec and development teams take over. When that happens, DevOps’ role will remain critical for making compliance more agile, automated and, ultimately, reliable.
“DevOps can assist companies in complying with regulations like GDPR by ensuring that key regulatory requirements are part of the initial design criteria, and taken into account when applications are updated,” Varonis’ Spinner said. “GDPR, manual methods of enforcing data policy aren’t realistic solutions because large organizations could have millions of files. DevOps can support GDPR requests with processes and solutions that ensure personal data that falls under the new regulations is protected and used appropriately.”
The General Data Protection Regulation (GDPR) will have an effect on how Google Analytics collects and processes data and how DevOps uses the tool—at least, that is what is known for sure at this time. These changes also should almost certainly impact DevOps for any organization that relies on the tool to collect and analyze user data. It is also known DevOps teams will have to reconfigure their Google Analytics implementations, even if an organization has no users based in the European Union (EU).
However, while Google has recently offered some guidance about modifications to Google Analytics that impact DevOps, the company has yet to clarify some key points about changes ahead of when the new mandate takes effect May 25. In the immediate, here are some changes Google has communicated so far and actions DevOps will need to take ahead of the regulation’s deadline.
The GDPR’s “right to be forgotten” provision offers users options to demand the deletion of personal data third parties store and share. To meet the terms of this provision, Google Analytics will allow organizations to manage how long their users’ data remains on Google’s servers. In this way, Google Analytics will automatically delete user and event data older than the date specified. Organizations can specify that the data remains on Google Analytic servers for 14 months, 26 months, 38 months and so on, for a maximum 50 months. It is also possible to opt for data to not expire automatically. Google Analytics says it will not delete information based on aggregate user data.
Google Analytics also will allow organizations to delete data for individual users in an EU country that can be used to identify them under the GDPR’s “right be forgotten” clause. This data will include common identifiers that Google Analytics stores—including standard Google Analytics first-party cookies, user ID or App Instance ID.
However, Google Analytics has yet to specify details about how organizations will be able to delete user data. While the process will be animated, Google says more information “will be available shortly,” presumably ahead of the GDPR deadline.
Google says it is also drafting a new “EU User Consent Policy” ahead of GDPR. This policy that customers will have to agree covers organizations’ “responsibilities for making disclosures to, and obtaining consent from, end users of your sites and apps in the EU.”
Another key modification Google has yet to reveal is the interfaces Google Analytics will offer that will allow organizations to make changes to how their users’ data is stored. Whether DevOps will be able to automate these processes or not will depend largely on whether Google Analytics will offer a GUI, an API or both.
“If Google Analytics offers a GUI with no API, there is not much a DevOps team can do to automate it outside the existing functionality, since you need a person to do that. But if it’s an API, DevOps teams can weave the controls into their existing frameworks,” said John L. Myers, an analyst for Enterprise Management Associates (EMA). “If both are available, then you’ve got the best of both worlds.”
Additionally, an API would allow the retention and deletion tools to work within existing DevOps frameworks and to become automated as part of business processes, Myers noted.
DevOps will obviously need to be prepare for GDPR in other ways outside of the scope of changes to Google Analytics or face fines up to 4 percent of their annual revenues or €20 million ($25 million), whichever is greater. Indeed, GDPR applies to all organizations worldwide that collect data from users in EU countries. In fact, it is often incorrectly reported the new privacy mandate applies to just EU citizens, while instead, the regulation concerns data collected from any user who is physically located in an EU country, regardless of nationality. Regardless, it is also obviously impossible to determine whether user data protected by GDPR including cookies, names, email addresses, social media posts and other information belongs to a user with an EU IP address who might just be visiting the region or is a full-fledged EU national.
Given the far-reaching provision of GDPR that places strict controls on how third parties store and share data and protect their online privacy, GDPR likely will fall under an urgent DevOps project category, requiring the input of the change advisory board (CAB), as well an organization’s IT operations, QA, InfoSec and development teams. The degree to which GDPR will change code deployments and processes will depend on the organization’s business model, of course. GDPR compliance should also require review of qualified legal counsel.
Meanwhile, Google Analytics says its tools can help organizations to comply with GDPR for data collection policies. These features include cookie settings, privacy controls, data sharing settings, data deletion on account termination and IP anonymization.
However, as mentioned above, Google still need to provide DevOps with more clarification about certain changes to Google Analytics affecting organization ahead of the May 25 deadline. Many in the DevOps community are also very likely waiting to learn about other ways DevOps can use Google Analytics for GDPR compliance.
During my career as a technology leader I have worked on PCI security, lawsuits, federal and state compliance, foreign market launches and acquisitions. All of these have regulatory, compliance and legal aspects to them, and over time I’ve come to realize we are bad at dealing with such things—really bad. At the heart of being so bad at this is something I call “binary risk management.” Binary risk management can, and often does, paralyze an organization.
Binary Risk Management is a failure to recognize the probability associated with a risk and treat all risks as if they are destined to happen. If there is no risk at all, proceed. If there is any risk, no matter how small the probability of its occurrence, halt!
We have all heard something like this: “Legal says we have to replace all the images that have any text in them across our entire French site because you’re not allowed to have any English on a site in France.” I actually heard that. It turned out not to be the case, but that did not stop me from wasting quite a bit of time working on this project. I was scared. The fines were said to be “hefty” and legal said it, so it must be true. Right?
I have wasted months on work that never had to be done. I have seen a CEO need to ask a compliance department’s permission for lower-level employees to have an informal social get-together. I’ve seen an acquisition almost fall apart because of a non-existent risk (the risk was similar to your chances of catching AIDS from a toilet seat). I have many stories and I’d love to hear some of yours in the comments section.
In all these cases, the outcome of the risk materializing instead of the probability of it actually happening was the focal point of conversation. In some of these cases the probability of occurrence was, in fact, zero because the whole thing was misinterpreted to begin with, so paralyzing are specters of lawsuits and fines.
Here are a couple of rules of thumb—tenets, if you will—for dealing with regulatory, legal and compliance issues. These tenets safeguard against wasting time and money that could be better spent helping our customers and making my company more secure, compliant and safe.
Legal and compliance teams often are not compensated on business outcomes, so, for them, choices on preventing risk can be either A: Take a risk, or B: Take no risk. They often choose B. Forcing a critical examination of the wording of regulations in question, an exact understanding of the consequences, how they can be softened and a real analysis of the probability of the outcome of an event can save your company a lot of money. The money saved can be directed to achieving better outcomes for your customers.
I’d love to hear some of your examples of binary risk management. Please share in the comments.
I’d also love to hear from those legal and compliance minds that do get it (I have met a few) on other techniques for being safe and effective while avoiding paralysis that comes of binary risk management.
Consider how much money your enterprise spends on audits. Smaller firms not subject to external regulation should consider how much money is spent on quality assurance—or the lack of it. A business that suffers from a customer-facing quality incident will, at minimum tarnish the brand of its services or at the high end, lose market share altogether. Larger firms that fail audits can suffer from brand tarnishing as well as high fees, not to mention further scrutiny by regulatory agencies.
Failure to perform or failure to follow proscribed processes inevitably leads to a black eye in front of customers. The entire discipline of governance is premised on avoiding both. Whether your governance practices are efficient or not—and whether your “process” as it sits today leads to quality or not—is not the subject of this review; it is how DevOps and automation can shine a spotlight on both and radically reduce the costs you spend today to achieve them.
The spotlight of DevOps begins with consistency that leads to predictability. Automating any given kind of process or technology the same way every time begins to give insight on any “holes” from things that do not work (or work well). If all that changes in your deployment process, for example, are pointers inside a configuration file and the kind of testing executed against this version of the code, you have achieved a build-once, deploy-many state of deployment automation. Operations staff who support the deployment of change or maintain the environments in the software development life cycle (SDLC) begin to isolate any issues faster because they become familiar with “how” a given kind of technology is automated to deploy. This reduces the total number of deployment errors and, in tandem, reduces the length of time it takes to fix them. Any “process” problems or weaknesses in your governance model become apparent quickly and can be fixed quickly. Limiting variation is the medicine to keep them away in future.
The spotlight of DevOps next turns to transparency. Offering your entire enterprise visibility or “read only” access to your entire DevOps tooling suite has a profound impact on the transparency of change. As an example, consider that I am a humble accountant in your firm who has submitted an agile story to improve our billing system. I may or may not, given the nature of my job, be co-located near the programming team who will fulfill this request. If our firm specializes in accounting software then perhaps I will co-locate, but if I am only requesting a change to an ancillary or support system, likely not. Without direct access to the programmers through traditional agile co-location, my visibility into the progress of change is limited to the tracking system I submitted the request within (assuming you have one, and don’t use email to do this).
If I know a programmer in MIS, perhaps I ask him at lunch how my change is coming along. But if I have read-only access to the testing systems, or to the release orchestration (RO) tooling, I may get a much better up-to-the-minute view of exactly where my change is, how it is doing from a quality assurance (QA) perspective, and when it is scheduled for release into production. Based on this news, I may make decisions about how we bill our customers in what time frame to increase overall revenue to the company. I would notify my boss of same (who also has access, but pays less attention), and of any changes that impact the new timeline I create. The marketing team (who also has access) begins to work with public relations (PR) and the advertising guys on customer announcements, etc. etc.
The net impact of having universal transparency is that audiences who are not traditionally notified or interested in the inner workings of change movement now make decisions based on it. Consider that prior to this systemic access, and the reliability of DevOps to ensure delivery, the only news you got before was from “the IT guy,” who was often wrong. It does not take long before IT loses credibility from being wrong about delivery timelines, and these other audiences begin to pay less attention (if at all). At that point, an “I’ll believe it when I see it,” attitude emerges. And this same attitude infects internal auditors because of what they hear and see from disaffected internal customers who have been misled repeatedly by the failed manual methods of the past (pre-DevOps).
Processes that are disinfected over time by the consistency of DevOps begin to change the internal company thinking about the delivery of innovation—namely that it can be counted on as accurate. Transparency offered to all audiences of a company increases the visibility into all aspects of innovation delivery and better ideas emerge, both about what to do with completed efforts and how to improve the innovation process itself. And before you know it, your governance and QA processes become efficient, reliable and predictable. Variation is eliminated or greatly reduced, and the costs of operation and innovation begin to decline.
What is rarely a matter of focus in most companies is the secondary ability to automate the compliance history of a given application. Most firms are content (translate, only able to) work with their testing tools to produce logs about a given version of an application now residing in production. A test log does show an auditor the majority of the quality related information about a given version of an application. But the log is a highly limited view of what has taken place. What is needed is the full context of innovation. Enter compliance automation.
Consider for a moment, that requirements drive the train. Why are we doing the innovation we are doing? Is this a bug-fix, a new feature, a client request, a regulatory requirement? Being able to trace the code itself back to the requirements system that drove the need adds significantly more context than is possible from mere test logs. Next, think about the number of attempts that finally resulted in this version of the application. An automated build tool (part of our DevOps suite) has this information. At this point you could evaluate how many builds and deployments took place in development before a worthwhile candidate emerged to traverse the remainder of the SDLC. An automated deployment tool (part of our DevOps suite) can trace the number of environments this version of the app was deployed into for testing and how long the overall process took to get something moved from development to production.
And finally, testing is often done from a variety of tools, with results compiled in a variety of result repositories. This information may be available from a solid RO tool that tracks the gated progress of this version of the application, but if it’s not, it still should be sitting in each of the testing tools used throughout the process. Depending on the level of inherent DevOps tool integration (i.e., using a vendor suite) much of this information may “flow” from one tool to the next. But if not, the information does still exist in the logs produced by each of the DevOps tools.
It may be well worth the application of internal programming resources to put together a tool that gathers all this information (automates the collection and presentation of) for audit purposes. Imagine the reduction in cost of supporting an audit that can gather a simple push-button report (or dashboard) of the entire life cycle of a given version of change. At that point, you have the entire context for a piece of innovation. You know what drove it (requirements). You know what it took to create it (build and deploy). You know the entire history of testing it (testing tools). And, finally, you have the entire governance process used in its deployment to production (RO tooling). But instead of examining multiple tools to collect and present it, you have created a push-button system to do so.
The investment in adding automation to the compliance history of innovation pays off in more than one way. It is not just the ease with which history can be recited. It follows the same principles as DevOps itself. Automation introduces consistency in how audit information is gathered and presented (reducing variability from one audit to the next). Automation introduces transparency if the entire company is given “read only” access to this new system. Automation shows auditors we are not afraid to learn from our mistakes and proud of our accomplishments. And in effect, it reduces the need for audit altogether.
Typically, regulatory bodies investigate and audit firms where they are uncertain of how the results were obtained. An automated system like this one fully eliminates that need. Transparency as could be offered here, may convince auditors that attention is not warranted here. Keeping in mind the spotlight of DevOps has already improved quality and streamlined process, add to that full transparency of audit materials and the confidence of auditors regarding your company may increase significantly. Routine becomes the only reason to audit, rather than suspicion.
Invariably the “new” will sometimes result in an error that your current systems and processes (automated or not) will not catch. But this truism is recognized, and accounted for, in how DevOps enables innovation. And if you have this kind of automation in place, it can be demonstrated in how we fixed what we fixed (whether governance or QA). Your strategies in DevOps should help you achieve this level of confidence. (If you need help, feel free to contact me, I am currently looking.)
To continue the conversation, feel free to contact me.