devopsy.kinsta.cloud

Category: Sponsored Content

Self-Service Workload Protection for DevOps Teams
Everyone (or almost everyone) in the DevOps world likes to talk about the importance of security. But too often, these DevOps security conversations end with generic recommendations such as, “Follow the OWASP Top 10.”

Don’t get me wrong—reference frameworks such as OWASP are great resources for helping to guide DevOps security. But it’s usually far from obvious to many DevOps teams how to bridge the gap between the abstract theory of these frameworks and actual practice. That’s especially true given that many DevOps teams spend their days putting out fires in fast-moving CI/CD environments and have little time to step back and operationalize security best practices, even if they know they should.

In this article, I’d like to present one real-world strategy for implementing the DevOps security best practices outlined by organizations such as OWASP. That strategy is what I call self-service workload protection. The term refers to making security not just something that DevOps teams recognize as important in the abstract but an easy-to-use service that anyone on the DevOps team can operationalize just like any other process within the CI/CD pipeline.

In other words, we need a self-service approach for offering workload protection through an accessible library of policies and tooling that DevOps teams can leverage on demand. Let’s take a look at how we can achieve that in practice.

Understanding Self-Service Workloads

To understand how self-service workloads work, we need to know the common parameters that define a secure and properly configured application deployment. With that information, security teams can automate the process and offer a complete set of reusable and adaptable tools for DevOps service teams to run.

The common parameters usually boil down to the following factors:
- Making sure that all required compliance and security scans run at least once for every deployment. Those scans need to use preventive actions and block insecure deployments.
- Making sure that for any new vulnerabilities found, we add them to the list of checks. (For example, we should be able to see updated reports for new issues as they’re discovered.)
- Performing audit reports and continuous security monitors over the development and production sites to capture security issues.
- Establishing sane and pragmatic secure default policies and authorization profiles.
The above parameters should be captured and packaged as a self-service solution and used during the normal flow of operations—for example, CI/CD pipelines, validation and compliance checks, creating and uploading a new image to a container registry, etc. This means the security team will have to spend some time to make sure the application of workload protection policies is a team effort.

For more tips on automating security, check out Symantec’s recent webinar on security and compliance for DevOps.

How to Make Security Better Via Self-Service Workload Protection

From the perspective of the security team, the main goal is to make security accessible to everyone in the organization. Not only that, but the tooling and resources they offer need to be documented and easy to use. The DevOps team should be able to plug in security controls without knowing their ins and outs, though they need to have the means to let the security team know if something goes wrong. Let’s explore how we can make this process better.

Shift-Left Security Through CI/CD Pipelines

Shift-left security is the main focus point of the security team, and it’s also the most tricky to get right. By adding more security checks and audits earlier in the development process, there’s a risk of stepping into other people’s ponds (which may degrade their workflow experience).

A better approach is to give more responsibility to the DevOps team and let them determine the best possible integration point. Mistakes will be made, but important lessons will be learned. Gather feedback from the process and, if required, iterate over the agreeables. That way, everyone will benefit. If the appropriate steps are taken, the security team will have a way to apply its security checks during the development process, and the DevOps team will have its infrastructure protected (at least from known risks).

Test automation and continuous integration points are the most obvious solutions, as they are the most automated parts of the system. The security team can go the extra mile and offer reusable policies and deployment steps that can be added on each deployment type. For example, they could offer a container image that is bundled with security checks and frameworks that test common security issues. The DevOps team can simply enable the checks via the CI/CD dashboard and during the build stage the check would act as a gateway if a security check fails.

In addition, the security team can share infrastructure policies and secure default configuration via infrastructure-as-code tools such as Terraform using remote state. This could be enabled during infrastructure provisioning by the DevOps team, without configuring anything. Sharing state this way often improves collaboration between departments.

Pre-Baked Secure Images

Developers use VMs and containers to package their apps for the DevOps team to deploy them to production. Usually, those images host crucial configuration and services that are specific to that application. What’s more ideal than to have the security team prepare a list of safe-to-use images that are preconfigured with secure defaults and scanned against vulnerabilities? Developers just have to specify the required image by ID so that the DevOps team can use it for all deployments.

Tip: Symantec’s CWP product is integrates with Jenkins to enable simple, automated vulnerability scanning during the CI/CD process.

For example, tools such as Packer are used to create identical images for AWS compute instances, and features such as seccomp profiles are enforced to Docker images. With Packer, you can deploy consistent images to the cloud and integrate vulnerability scanning into the process to ensure the deployed images meet GSO guidelines. There is a small caveat here, though: If developers decide to add an external library that needs more capabilities and they don’t inform the security team, then the application might misbehave in production and the DevOps team would be caught unaware. So, it’s important to have a good communication channel and do a security assessment for each new dependency or change introduced to the system. This keeps the self-service agreement working without interruptions or misunderstandings.

Conclusion

Letting the DevOps team manage its own self-service affairs when it comes to security controls and compliance requirements is the next major step toward a more productive business model. There are additional costs, but the end results have long-term advantages.

By having a preset repository of tooling, configuration policies and pre-baked images accessed from secure portals, teams can integrate security from the start while maintaining the benefits of accelerated development. And by doing that, security and DevOps teams can improve their collaboration.

That’s great news. What’s even better is that we can use a tool that is suited for all the above tasks—Workload Security Protection from Symantec. If you want to learn more about the platform, see how it can help secure your cloud with a free trial.

This sponsored article was written on behalf of Symantec.

— Theo Despoudis
June 27, 2019
Ensure Your AI is Working for Your Data Team
AI tools hold great promise for DataOps. They can help teams identify the best apps to migrate, identify optimum cloud infrastructure and even validate decisions. But how can an IT team ensure the AI tools it uses are actually advancing its DataOps goals?

A hard pill to swallow is that even the best-designed algorithmic decision-making and AI tools are prone to errors. Fortunately, this is unlike errors in software development, which are largely caused by development. AI errors emerge from poorly defined project goals and the organization’s tendency to treat the technology as if it’s magic. Without solid requirements, model monitoring and regular check-ins with reality, it is almost impossible for AI to deliver.

This is why audits must be a mandate when introducing any AI tool that greatly influences business decisions. However, before the audit is conducted, ensure that the following points are met:

Ensure the AI Tool is ‘Auditable’

This step ensures that organizations bring in the right AI tools to their arsenal.

An AI tool is auditable when it is:
- Transparent: When AI tools are used to enhance the decisions a business makes, stakeholders must be able to peer into the system to ensure the legitimacy of the algorithm decision. Ensure the AI tool you choose has a logging system that makes it transparent as to how the system is making a decision, what factors most influence it and how confident it is in that decision. This allows decision-makers to trust in the system, as well as creates a feedback loop for the developers to take in real-world data and enhance and fine-tune the system to your needs.
- Built with best practices: Just as security engineers follow best practices when securing a network, AI models and tools built with best practices in mind fare better when in use. As a general rule of thumb, all AI models must keep detailed access and process logs, incorporate failback mechanisms and generate regular health reports.
Set Up Markers for Success

Don’t approach an AI audit as though it’s a sandbox question. If teams are clear as to what an AI tool must achieve and how to measure its success, only then will an audit be able to judge the performance of the tool.

It is helpful to set up markers that describe success. This could be the number of cases the AI model has rightly predicted, the time it takes, the number of edge cases it is able to handle and the number of times the AI decision collided with a human decision.

The Audit

Once the AI tool has been integrated into the organization and is in active use for a defined period, audits must occur. They help you answer the following questions:
- Is the algorithm suitably transparent to end-users?
- Is it likely to be used in a socially acceptable way?
- Might it produce undesirable psychological effects or inadvertently exploit natural human frailties?
- Is the algorithm being used for a deceptive purpose?
- Is there evidence of internal bias or incompetence in its design?
- Is it adequately reporting how it arrives at its recommendations and indicating its level of confidence?
(HBR: “Why we need to audit algorithms”)

Assess the Algorithm’s Fitness for its Purpose

From a use-case perspective, AI algorithms in modern software applications tend to fall into two main categories. One is offering insights that help humans make decisions—this is what AI-driven analytics is all about. The other is allowing software applications to undertake automated actions entirely on their own, as AIOps tools increasingly do.

A basic step in auditing your algorithm is to determine which type of use case it is supposed to support, then evaluate how well it achieves it. This is important to do regularly, because AI use cases can drift as applications evolve and the original purpose of an algorithm might change over time. (It’s common for algorithms to be implemented originally as a way to offer insights, but for applications to be updated with the expectation that the algorithms can support automated decision-making.)

Compare and Contrast Algorithmic Decisions Against Human Decisions

Though some of the inner workings of an AI model are a black box to even data scientists, the result of those inner workings is not. An AI audit must set up a process and team that regularly and actively monitors the decisions AI makes and compares them against decisions that a human would make.

This is useful in identifying early on whether the AI tool has internalized biases. With an AI system, input greatly affects learning. For example, if a tool used to forecast purchase orders is incorporated by the organization during peak sales season, the tool will assume that this is the organization’s baseline performance. This causes erratic predictions in the long run.

Active Experimentation

Though an audit should uncover any bugs, biases or security concerns of an AI tool, it also could be used to test system variables. An AI tool is generally built on assumptions of the problem borders and trends. However, assumptions are very fragile in reality. The baselines of reality could change, causing the AI to produce questionable results.

An audit also must be able to revisit AI tools with new assumptions, annually or biannually, to uncover any new parameters that could help improve the accuracy of the tool.

Interviewing Users of the System

Ultimately, an AI tool is put into place to either reduce the workload of humans or enhance it. Daily users have vast knowledge and perspective into not just how the tool works but also its limitations and quirks.

Conclusion

More often that not, the mystique of an AI model is used as an excuse for erratic results and poor performance. By building a robust and transparent audit system, your organizations can ensure your AI tools are actually working for your teams—and if not, it can at least uncover rare scenarios and edge cases and help mitigate them.

This sponsored article was written on behalf of Unravel.

— Swaathi Kakarla
June 27, 2019
How IT Engineers Can Learn DataOps
To seasoned IT engineers, DataOps may not sound very difficult. After all, if you work in IT, you know all about managing databases, provisioning data storage, moving data around and so on. How hard could DataOps be, really?

The answer is, “Harder than you might think.” Even if you work frequently with data in the context of ITOps, DataOps probably requires you to learn some new skills.

Those skills are not impossibly hard to master; any competent IT engineer can learn to do DataOps. But it requires learning some new tricks and—most important—some new ways of thinking about data and how it is processed and managed.

What is DataOps?

For the purposes of this article, DataOps refers to data operations in general. It encompasses all the processes an organization must perform to make use of data—from collecting and storing data to managing and transforming it, to analyzing and archiving it, to optimizing the performance of the data that drives applications.

ITOps and Data

ITOps engineers certainly interact with data on a regular basis. However, the type of data-related work they do is limited to specific types of tasks, such as the following:
- Setting up and managing databases.
- Provisioning data storage infrastructure.
- Load-balancing and monitoring data transfers over the network.
- Configuring access control policies for data.
- Managing data backup and recovery routines.
- Managing and aggregating log files.
ITOps might occasionally do other things with data, too; the above is not an exhaustive list. But in general, the work that ITOps does in relation to data falls short of the broader set of tasks that DataOps entails. ITOps does not typically play a major role in work-related data analysis, data transformation, data quality control or data monitoring and optimization.

Extending IT Skill Sets for DataOps

We live in a world where data is becoming increasingly more important to running businesses and applications. Making sense of data is critical for making informed business decisions. At the same time, data is critical for powering applications that rely on machine learning and artificial intelligence. More generally, it’s essential for monitoring and managing application environments.

That is why it is more important than ever for ITOps engineers to extend their work into the realm of DataOps.

As noted above, although making that jump requires gaining skills that IT engineers don’t typically use in their IT work, it’s not as hard as you might think. Consider the following strategies for helping to extend ITOps skills into DataOps work.

Think in Terms of Data Performance, Not Just Data Availability

ITOps engineers are used to making sure that data remains available. They troubleshoot disk failures, network connection failures and other issues that could make data unavailable.

It’s easy to extend these skills into DataOps if IT engineers focus not just on data availability, but also on data performance more broadly. Instead of thinking just in terms of whether data can be accessed as intended, think about whether data can be accessed quickly enough, whether it meets data quality requirements and if there are any bottlenecks that threaten the ability of data to do its job.

Systematize Data Operations

Because managing data is not IT engineers’ main job, it’s easy for them to think of data-related tasks as one-off or ad hoc requirements. They might check databases for corruption only when an issue arises, or wait until a storage system starts running out of space to think about how to optimize it.

DataOps requires a more systematic and consistent approach to data management. Data performance optimization and management should be performed continuously, as part of the daily work that IT engineers do to support application delivery in general.

Optimize Infrastructure for Data Operations

When it comes to planning and setting up infrastructure, IT engineers usually hold the keys to the kingdom. They decide which infrastructure to stand up and how to architect it.

Traditionally, data operations were not a primary consideration when planning new infrastructure or optimizing existing infrastructure; instead, the focus was on application deployment. But by making data operations a key focus as well, IT engineers can ensure that the infrastructure they control is ideally suited to meet their company’s data operations needs, too.

Make Data Monitoring a Part of APM

IT engineers usually own application performance management, or APM. But traditionally, their focus on APM workflows is about optimizing the performance of software, not the data that software depends on.

Given the inseparability of application performance from data performance today, that approach no longer works. Consequently, IT engineers need to think about data performance management as an integral part of APM. In addition to identifying software and infrastructure problems, their APM tools and workflows should alert them to data quality and performance issues.

Conclusion

ITOps and DataOps are distinct disciplines, but they increasingly depend on each other. For that reason, it’s important for IT engineers in many cases to extend their skill sets beyond traditional ITOps and contribute to data operations, too.

Fortunately, doing so doesn’t require going back to school for a degree in data engineering. In many ways, IT engineers already have the foundational skills for data operations. They simply need to extend their thinking and toolsets to support data operations.

This sponsored article was written on behalf of Unravel.

— Chris Riley
June 25, 2019
Building Continuous Compliance into DevOps

“Continuous” is one of the operative words of DevOps. A major goal of DevOps is to make all processes continuous, which means having them proceed smoothly and constantly to avoid delays or pauses within the software delivery process.

You’ve probably already heard of the importance of continuous integration and continuous delivery. But the importance of continuousness doesn’t end there. It extends to every component of the software delivery process, including compliance.

To illustrate the point, let’s take a look in this article at what continuous compliance means and how to achieve it.

Defining Regulations and Compliance

Regulations and compliance are two sides of the same coin. Regulatory bodies put in place regulatory laws and policies that are aimed at the benefit of the broader economy and end consumers. Organizations under the purview of these regulatory bodies must put a compliance framework in place to meet regulatory requirements.

Here are three of the most widely applicable regulatory requirements that organizations commonly need to adhere to today:

HIPAA

HIPAA, or Health Insurance Portability and Accountability Act, is applicable to healthcare organizations. It concerns the handling and protection of PHI data (Protected Health Information). The costs for mishandling healthcare data are steep for users. According to research done by Accenture, 26% of healthcare consumers were subject to data breaches, and it cost each victim an average of $2,500. Stats such as this make a strong case for HIPAA compliance.

PCI DSS

The PCI DSS (Payment Card Industry Data Security Standard) is a regulation for the financial services sector. It governs the use and handling of users’ financial data, such as credit card information and bank account information.

GDPR

The GDPR (General Data Protection Regulation Act) went live in the European Union in mid-2018. It defines policies related to how organizations handle data of users in the European Union. This includes giving customers the right to erase all their data from a company to ensure the organization doesn’t have indefinite possession of the data and to ensure confidentiality of users’ private information.

Why Continuous Compliance?

Cloud-native computing brings new complexities to how applications are delivered and how they handle user data. For starters, the scale of data flowing through the system at any given time is now much higher. How organizations and their applications manage this data is critical to compliance.

Audits can be outdated the very next day when dealing with such dynamic systems. For example, when new releases are deployed continuously, it’s easy for one release to expose an endpoint to a third-party application and leave sensitive data accessible to outsiders.

The regulatory requirements themselves change every few months. It’s important to keep track of these changes and adapt your systems to still be compliant. In light of this, it’s necessary to adopt continuous compliance as part of any DevOps workflow.

Achieving Continuous Compliance

Two Goals: Security and Speed

It’s easy to think of compliance as being about simply meeting regulatory requirements. However, this is far from true. When done properly, compliance also empowers your organization to operate at a pace that encourages innovation. Every organization that adopts DevOps wants to be secure and move fast at the same time. Continuous compliance is the way to achieve both these goals.

Compliance from Day One

Usually, compliance is an afterthought. However, continuous compliance requires that DevOps teams start their design and development with compliance in mind. This includes having clarity on the exact requirements, down to the fine details. With this in mind, the team can move forward to build applications that meet these requirements. To have this kind of clarity, the various teams, including business leaders, need to speak a common language and have common terms to describe the different parts of compliance.

Continuous Testing

Testing is essential to compliance. In the DevOps world, testing has undergone numerous changes. The good thing is that these changes all support continuous compliance. First, testing has shifted to the left of the development pipeline. With testing happening earlier, it’s easier to catch errors and vulnerabilities before they reach production. Second, tests are increasingly automated. This makes them predictable, measurable, repeatable and compliance-friendly.

Automated Compliance

Your compliance verification and enforcement process should be as automated as possible. Toward this end, integrating compliance checks into the CI/CD pipeline is a best practice. For example, make vulnerability scanning part of the CI/CD process by integrating it with your CI server so that checks are performed at build time automatically.

The Right Data Hierarchy

All data isn’t equally important. Some data is more sensitive. This is why you need to organize and categorize all the data in your systems. Once categorized, you can define policies to govern different types of data. This includes encrypting data at the appropriate level—and not just in transit, but also when data is at rest.

Track Access Control

In the effort to achieve control over compliance, identity and access management (IAM) plays a critical role. It helps define who is allowed to access data and what changes they’re allowed to make. When assigning access, it’s important to assign it according to teams or roles and not according to individuals. This makes IAM policies scalable and easier to control. Putting in place streamlined IAM policies negates the need to share passwords and other sensitive information. Every user can access only the data they need, and not more. This is the principle of least privilege in action.

Track Change History

Beyond IAM policies, it’s essential to maintain a change history for all users and parts of the system. The best way to implement this is to use log data as the source of truth. It delivers deeper visibility than mere monitoring events and help to troubleshoot incidents. Logs have always been a cornerstone of good compliance, and in today’s cloud era, they continue to play a key role.

Conclusion

Achieving compliance is challenging in any environment, but when you embrace DevOps, compliance presents special difficulties due to the speed at which workflows move and the importance of making processes continuous. Implementing continuous compliance takes effort, but the rewards justify it. Not only will your organization stay on top of regulatory requirements, but it will be better positioned to move quickly and innovate.

To put continuous compliance into practice, you need the right tool for the job. Symantec Cloud Workload Assurance (CWA) can help. Learn more about CWA features in this data sheet, and sign up for a free trial to see how CWA can help secure your cloud.

This sponsored article was written on behalf of Symantec.

— Twain Taylor

June 25, 2019
Building a Security Feedback Process for DevOps
The last few years have seen some major slip-ups in the security space among all major cloud providers, resulting in uncertainty and speculation. That’s understanding; cloud security is an extremely complicated subject as enterprises build and deploy applications faster than ever before to keep up with business requirements. Most of the security issues that occur in the cloud today are around compromised accounts, identity theft, malware apps and, of course, data breaches, which are business-critical and showstoppers, in many cases.

Given the dynamic environment of DevOps and the nature of the cloud, it’s a complex problem to address and fix. The most effective way is to embed security, or what is usually talked about as the shift-left of security, in the IT pipeline. If DevOps is the culture of collaboration to reduce risk by the size of change, DevSecOps is the automated security control built into each point to build a sort of self-defense mechanism.

DevOps is not about tools; it’s about philosophy and process. One of the most important processes to include to succeed at DevOps is a feedback loop that reinforces constant collaboration.

This is certainly true when it comes to security-related information. In a healthy DevOps organization, security problems that are detected at any point in the delivery pipeline are communicated quickly to all stakeholders, who can, in turn, respond accordingly.

In this post, we will look at the steps involved in building a security process that will ensure the right information reaches the right people quickly and that teams collaborate without duplicating efforts.

Automate Your Way to Better Feedback

Automation is the first step, and a no-brainer when it comes to DevOps. Without automated security tools for code analysis, configuration management, access control, vulnerability management and secrets management, it is very difficult to scale security in a dynamic DevOps environment. Manual efforts in the cloud are doomed to fail in many cases, as environments change rapidly. Hence, automate. The recommendation is to automate every security control and the entire feedback process as much as possible.

Manage Roles Right and Control Their Accesses

Identity and access management has become the linchpin in cloud security. Since it’s very easy to give people the wrong access, it becomes important to figure out who can access what resources. Access to information and resources must be secure, yet easy to obtain. All access should be given to a group rather than an individual. Mapping security access to the organization’s hierarchy is an effective way of establishing a communication loop. The right information reaches the right group of people. It is important to be able to answer questions such as the following to ensure the communication loop is efficient:
- Who can delete this VM?
- What are the resources a certain user has access to?
- Who has service accounts access?
Tools that show a relationship between users, groups, roles, permissions and resources help answer these questions.

Encrypt Secrets, Manage Their Keys

Another important step to building an effective communication cycle is controlling access to application-level secrets. Security can be improved by tightening information flow between relevant groups, which can be accomplished through creating and controlling access to secrets using principles of least privilege. Access keys to those secrets can be kept in a centralized secrets management tool or a key management tool. The rotation of these keys can be automated as often as required.

Designing for Failure and Proactively Managing Vulnerabilities

Though security personnel may cringe at the mention of the word “failure,” designing for failure means thinking ahead and having a plan B. The cloud is dynamic by design and things happen at an incredible pace, often resulting in issues of elasticity, configuration and cloud power.

Context becomes an important parameter for managing vulnerabilities because there are multiple vulnerabilities at any given time. Automated vulnerability management and configuration management solutions are available to take the complexity out of protecting cloud environments. Automation capabilities and continuous monitoring provide deep insight, intelligent recommendations and continuous protection of the cloud, thus ensuring effective management of information in the feedback cycle.

Log Everything — Everything

Just embedding security to shift left is not enough to ensure security. It is important for all relevant collaborating groups to know the status when some code is deployed. The primary source of visibility and feedback comes from logs. Effective logging strategy is the paramount attribute of security design. Logging should be enabled everywhere possible: data access logs, identity logs, admin activity logs, logs within the cloud environment, OS, network platforms, etc. Logs, though cumbersome and volume-heavy, are non-intrusive and a critical visibility tool. You need to know what’s going on before deciding to do something with the event itself, which brings us to the next step: monitoring.

Monitor Those Events

Testing departments often seem paralyzed with too many flags, which prevent them from prioritizing critical issues. Having log and event data is an important first step, but the absolutely necessary next step is to relay them into an automation strategy that will separate the unimportant, voluminous issues from business-critical priorities. Automated monitoring for events triggered makes the feedback cycle more effective. Otherwise, logs can become noise—a copious amount of noise.

Alerts and Notifications

We embed security in the code. We manage access and roles. We turn on the logs and monitor them. An event occurs. Now, who do we alert? An effective alert notification policy will ensure that notifications are sent to the right group of individuals, depending on the notification filters—the level of criticality and relevance of the issue type. Notification policies specify the conditions that trigger a notification that will be sent to the recipient. Notifications can be sent by email, SMS, voice, push messages, etc. A recipient can then acknowledge the alert, which indicates to other users that someone is working on the issue, thus avoiding duplication of efforts.

Conclusion

To conclude, automating security controls and feedback cycles enables a productive DevOps ecosystem by exposing vulnerabilities early in the product life cycle. Automation, streamlined communication and appropriate response to priorities ensure availability, network connectivity and resilience. At the same time, they reduce the possibility of a data breach and duplication of efforts. At the heart of everything is the ability to deliver code in a secure and reliable way to meet ever-evolving business needs with agility.

To see how Symantec can help maximize security in a DevOps environment, check out a free trial of Symantec Cloud Workload Assurance.

This sponsored article was written on behalf of Symantec.

— Twain Taylor
June 20, 2019
The Taxonomy of DataOps

Data Operations, or DataOps for short, is one of those IT buzzwords that lots of people use, yet few can define precisely. Like the cloud or DevOps, DataOps doesn’t make sense until you sit down and really think through what it entails.

To that end, let’s take a look in this article at the taxonomy of DataOps—or, in other words, which IT disciplines, areas of expertise and types of work make DataOps happen. I won’t explain everything about how DataOps works (that would require more than one blog post), but I will offer a primer on how DataOps workflows break down.

What Is DataOps?

Before delving into the taxonomy of DataOps, let me first define at a high level what I’m referring to when I use the term “DataOps.”

I use DataOps as a shorthand for data operations of all types—from data collection and storage to data management, monitoring, analytics and performance optimization.

I mention this because when some people hear the term DataOps, they think only of the practices and ideas described in the “DataOps Manifesto” and similar writings, which focus on extending the core principles of the DevOps software delivery philosophy to data operations. I’m a big believer in the “DataOps Manifesto,” but because it is targeted at data scientists rather than the IT community at large, I think it presents a narrow image of what data operations entails.

To me, data operations encompasses the work performed by anyone who works with data in any way. While data scientists and data engineers may perform the bulk of the heavy lifting related to data transformation and data analytics, developers and IT engineers also have important roles to play in data operations. When I use the term DataOps, I’m thinking of them, too.

The Taxonomy of DataOps

With that clarification out of the way, let’s explore what goes into DataOps.

Data Collection

You can’t have data operations without data—and so, data collection is a core part of DataOps.

Data collection refers, of course, to all of the processes by which you accumulate data and store it somewhere, for some length of time.

It’s important to note that data collection can be both passive and active. Much of the data that organizations store and analyze these days is not data that someone goes out and collects by hand. It’s the information produced through the everyday operations of these organizations—their data exhaust, so to speak.

Data Storage

Data storage may seem simple and straightforward. After all, how much do you really need to think about the storage of data? Don’t you just pick a storage location and put your data there?

Well, no. Data storage is much more complex than that, which is why data storage is a distinct discipline within DataOps.

Effective data storage requires finding the right type of storage infrastructure—the one that will deliver the best balance between data performance, availability and cost. It also involves defining data retention policies to determine how long data will be stored. It encompasses data backups, too, which are important for ensuring that the data you intend to keep in storage actually will be available to you if the unexpected happens.

Data Management

Data management is a broad discipline that involves all of the tasks required to keep data available and accessible once it has been collected and stored. It entails work such as transforming data from one format to another to make it compatible with a given set of tools. It also involves tasks such as moving data between storage locations and identifying and resolving bottlenecks within the overall DataOps pipeline that are slowing down performance.

Data Monitoring

In the IT world, the word “monitoring” usually refers to the monitoring of applications or infrastructure for problems. But you have to monitor your data also if you want to do DataOps effectively.

Data monitoring includes tasks such as identifying data quality problems (such as missing data or data formatting issues) that will make your data less reliable or more difficult to work with. It also means checking for signs of failures, such as a failed database or infrastructure crash, that could cause a rupture in your overall DataOps pipeline.

Data Analysis

Everyone (or everyone’s boss, at least) wants to “turn data into dollars,” as Gartner put it. Data analysis, or the process of seeking insights within sets of data, is key to making that happen.

Sophisticated data analytics is the only discipline on this list that is usually performed by data scientists alone, not the rest of the IT team—although that is changing somewhat as more and more simplified data analysis and visualization tools enter the market that makes it easy to make sense of large data sets without having a Ph.D. in statistics or knowing how to program in R.

Data Security

Data security is a big topic, especially in the present age of high-profile data security breaches. Keeping data secure requires a range of processes and tools, from setting the right access control policies for data to monitoring for signs of unauthorized access and scanning data to root out malware or other threats that could exist within data sets.

DataOps is Everyone’s Ops

You might have noticed that many of the tasks described in the DataOps taxonomy above typically are performed by employees of the IT department (or, in some cases, programmers), not data scientists or engineers who specialize in data operations.

This is an important point to drive home because when many people talk about DataOps, they assume that every organization has a team of data experts on staff who handle all work related to data, from collecting and storing it to analyzing and securing it.

In reality, many companies don’t have dedicated data operations teams. And even if they do, the general IT department still plays a huge role in tasks such as configuring tools and platforms for data collection, provisioning data infrastructure and keeping databases running healthfully. Security engineers, too, are key for things such as enforcing data access control and monitoring for signs of data security problems.

Thus, whether your company has a dedicated data operations team or not, DataOps is the responsibility (at least in part) of everyone who does anything related to IT. It’s everyone’s Ops.

This sponsored article was written on behalf of Unravel.

— Chris Tozzi

June 20, 2019
Unravel: A System Architect’s Guide to Data Operations
Big data is an older fashionable term that, when combined with agile methodology, AI and ML and analytics, has become a powerful trend once again in data operations (DataOps). If you’re a systems architect, it pays to understand the philosophies, benefits and tools to power DataOps and improve your data-driven applications. This article serves as a guide to define enterprise architectures that support the needs of an organization.

Inside Statistical Process Control (SPC)

The DataOps Manifesto defines a set of key principles to guide systems architects and implementation teams. In summary, they focus on the customer and analytics, driving positive change through a continuous improvement process that relies on both teamwork and automation to help drive efficiency and overall data and system quality. This is based on statistical process control (SPC), which is a quality-driven approach to using data and associated statistics to monitor, control and improve systems and processes. Quality, in this sense, translates to maximum application uptime, increased usefulness to the customer and reduced waste in terms of delivery time and resource usage.

Data is king in SPC, and most data-driven application architects go through a set of phases when they design systems. These phases include:
- Data Discovery: Knowing the sources of data used by applications, as well as identifying new sources of data about your applications and their environment. This includes performance statistics and other systems and process measurement data.
- Data Relationships: From the data and their sources, knowing the relationships and hidden value in your data is the next phase. This is the realm of big data and analytics where you search for business meaning buried in the data to be used to expand your offerings to customers while also enhancing and improving the customer experience.
- Data Monitoring: This means you want to ensure system uptime by monitoring your data-critical systems (e.g., data feeds, databases, analytics and so on), but it also includes automation of the first two phases above.
In line with SPC, performance data is used to ensure that your systems and their processes are stable, with the uptime and responsiveness needed to meet customer demand. To achieve this, the reach of DataOps extends from the data scientists and business analysts within your organization who solve business problems with data and algorithms, through to the database administrators and the developers who work with that data (e.g., ETL) and build applications from it.

From here, DataOps gives you insight into value discovery, turning these into new methods and tools, with an emphasis on continuous refinement. Tools and techniques will help you automate data mining, the discovery of potential performance improvements, identifying areas to create more efficiency and lower costs and prepare you for the future in terms of business and architectural needs.

Leveraging DataOps: Automation Is Key

Putting feedback loops in place helps to automate the entire data flow, from business intelligence, where tools are used to help data analytics processes, to data science, where new mathematical and architectural approaches are used to drive deeper understanding data, to developers and operations staff, where the data is aggregated into applications and further measured to drive new value back through this process.

Automation is used to build efficient feedback loops without overburdening your IT staff. In turn, automation tools are used not to replace your analysts, data scientists, developers or other IT personnel, but to enhance their jobs and make them more efficient at what they do. As data production and consumption continue to grow, your organization’s ability to discover hidden value in this data and drive it through to real system improvement will be a key competitive advantage.

AI-Driven DataOps

While data visualization is important to effective DataOps, displaying graphs and pictorials can go only so far. Gaining actionable insight into metrics is what drives improvements in performance and efficiency. To remove the potentially error-prone manual approach to achieve this, organizations are leveraging an AI-driven approach to DataOps. This helps discover hidden dependencies between applications and their resources (e.g., databases and other data sources) along with their users and their needs, providing unified visibility and operational intelligence to optimize your entire ecosystem.

AI can help you dive deeper into the effectiveness of your data, going beyond operational efficiencies to identify improvements to code, overall system architecture and the configuration tools and frameworks in place across the cloud or data center. Going further, AI-driven DataOps can help you optimize your consumption of cloud-based resources or identify key components to migrate (with recommended changes to deployment) when moving to the cloud.

The Big Picture: Cloud, Microservices, Serverless and DataOps

As a systems architect, you need to maintain a big-picture view of the data, systems and processes used within your organization to best serve your customers. This means you need to understand the entire application ecosystem to communicate and simplify your system and data operations. Tools help provide and maintain a 360-degree view into the performance, interactions and dependencies between components in your software systems and data workflows, along with all of the storage, compute and network infrastructure that supports them.

To enable the value in DataOps and drive visibility of the data and interactions within your systems, adopting newer architectural approaches can help. For instance, breaking apart monolithic applications using a microservices approach helps drive real value throughout your technology ecosystem; it helps improve the makeup and velocity of your development teams, improves the benefits of operational efficiency, drives a test-driven approach to development (since these services are more easily verifiable) and helps enable cloud deployment.

Other approaches such as the use of containers and serverless architecture help remove system dependencies from implementation, reducing the impact of deployment changes on your applications’ code. With all of these changes, the data orchestration and workflows without your applications (including your customers’ usage of this data, in some cases) become more apparent and measurable.

Call to Action: Leverage Tools and Frameworks

DataOps helps operations staff, app developers and enterprise architects reduce the complexity of delivering reliable application performance. Whether you’re new to DataOps or have been on board for some time, knowing the tools and frameworks to help drive both your architectures and operational control is key to success. Tools such as those from Unravel (see Figure 1) provide monitoring solutions that map dependencies between applications and their data sources, and all of the components in between.

Figure 1: DataOps tools such as the one from Unravel should support your existing implementation and technology choices, while supporting the three phases of DataOps work.

The tools you use should automate the discovery of value and optimizations to be made, where possible, from the viewpoint of business owners, IT operations staff and the data scientists and developers defining and building the systems. All of this should be achievable regardless of the platforms, frameworks, languages and cloud providers you use to enable your applications and users. Finally, make sure your DataOps approach and tools support your organization in the future, with support for continuous improvement of overall architecture as well as data processes.

This sponsored article was written on behalf of Unravel.

— Eric Bruno
June 18, 2019
4 Things Developers Should Know About Security in the Age of DevSecOps

If you’re a developer, most of your experience when it comes to security probably centers on designing and writing secure code. You know how to prevent buffer overflows, architect your microservices in a way that helps mitigate the impact of a breach and otherwise churn out secure application code.

But the fact is that today, knowing how to keep the application itself secure is not enough—even if you are a developer. We’re living in the age of DevSecOps, and developers everywhere are being asked to play broader roles in security at all stages of the application delivery pipeline, from code implementation through to production release and management.

With that reality in mind, here’s a look at four things developers should know about security today.

What is DevSecOps, and What Does It Mean for Developers?

Before diving into four critical things for developers to know in the age of DevSecOps, let’s take a look at what DevSecOps means.

You’ve probably heard of DevOps, an approach to software delivery that emphasizes constant collaboration between developers and IT teams.

DevSecOps extends that concept by adding security to the equation. For organizations that have embraced DevSecOps, the entire software delivery team becomes responsible for helping to secure the application. Security is no longer the exclusive domain of security engineers.

In practice, DevSecOps includes work such as integrating vulnerability scanning into the build process using tools such as Packer and ensuring that images used for deployments meet GSO guidelines. Applying OWASP’s “Security by Design” principles is also a good best practice.

What that means for developers (to put it simply) is that being a security-minded developer means more than just writing secure code. Developers must now understand security challenges and best practices at all stages of the software delivery pipeline, starting with the design and implementation of code—developers’ traditional domain—and continuing through with testing, staging, deployment and post-release management—all of which traditionally were not areas in which developers played much of a role.

This isn’t to say that developers now have to bear primary responsibility for all of the things that happen after code is written. They don’t. Their main job is still to write code. But developers should have a solid understanding of the security implications of everything that happens once they hand off code to other members of the software delivery team, to ensure they are doing their best to write code that will stay secure as it travels down the pipeline.

DevSecOps Security Lessons for Developers

If you’re a developer, the DevSecOps trend means you should now have some expertise in the following:

Microservices Security is Difficult (and Developers Can Make it a Little Easier)

Microservices architectures, which have become very common because of the agility they provide to applications, create extra security challenges that were not present in monolithic architectures. The main reason why is that microservices add many more moving pieces to an application. As a result, there are more potential security vulnerabilities within the application, and detecting those vulnerabilities can be more difficult because there is more activity to monitor.

Developers can help to address this challenge, and one way is to think about security monitoring complexity when they are deciding how many microservices to create. If developers can reduce the total number of microservices in an application without sacrificing agility, they can simplify security monitoring and enable a more secure application overall.

Developers also can make sure the microservices they write log security-related information sufficiently and clearly. You don’t want to make it more difficult for your colleagues who are monitoring an application to keep track of security events because the application doesn’t log authentication information in sufficient detail, for example.

You Can No Longer Count on Firewalls

In the days when most software ran on-premises, firewalls were a handy way of isolating applications from most security threats.

Today, organizations face a string of cloud security challenges that didn’t exist when firewalls were designed. Firewalls can still be helpful in some circumstances, but in many cases, you can no longer neatly segment applications or services off from the public internet, because you need the internet for services that are spread across a hybrid infrastructure to communicate with each other.

For developers, the lesson here is that it’s especially critical to design and write code that can resist threats that originate from the internet. Even if you don’t expect a particular part of the application to be exposed to the public internet, assume it might be, either because deployment patterns change in the future or the component is exposed without anyone realizing it.

To put this another way: Developers can’t and shouldn’t expect the IT team to be able to erect firewalls that will keep applications secure. More of the burden has shifted to developers to write applications that can resist network-based security threats because the IT team can no longer deploy firewalls like it used to.

Shift Security Testing Left

DevSecOps goes hand in hand with the mantra of shift-left testing. The latter term refers to the practice of performing tests earlier in the software delivery pipeline, rather than waiting until just before deployment to do the tests.

Shift-left testing can apply to any type of software testing, including security tests. For developers, taking advantage of shift-left security testing is a useful way of identifying security flaws within newly written code early, when they can be fixed with less effort because they have not yet been integrated into the broader application code base.

Security testing at later stages of the delivery pipeline is important, too, especially because you won’t be able to detect all security flaws until you can monitor all of your microservices as they interact with each other. But the time developers have to spend rewriting code to address security issues will be reduced greatly if their organization takes advantage of shift-left security testing.

Don’t Be Afraid to Ask for Compliance Help

A final point worth mentioning relates to compliance. Compliance is becoming ever-more challenging. And so is creating applications that meet the security requirements of compliance frameworks.

In the past, developers were often left on their own to figure out how to address compliance needs. Given the spirit of collaboration that DevOps and DevSecOps champion, however, developers should no longer feel like they have to go at it alone when it comes to interpreting and addressing compliance requirements. And indeed, they shouldn’t. If you are a coding expert, you’re probably not an expert in figuring out what complex compliance frameworks require.

The point here is that developers should not be afraid to ask for help—from legal experts, management or the IT team (which is more likely to have some experience with the compliance issues that arise once the software is in production) to make sure that the applications they design and write meet compliance requirements.

Conclusion

The developers’ main job is still writing code. But by having a greater awareness of the types of security risks that can arise within the code as it is pushed down the delivery chain, developers can do much to improve the security of the applications they write. They might even save themselves some time and headaches by reducing the number of complicated security issues they end up having to address by rewriting code.

This sponsored article was written on behalf of Symantec.

— Chris Tozzi

June 18, 2019
The DevOps Security Stack
The whole idea behind DevOps is automation. DevOps automates the development process by eliminating or minimizing human error. If an error occurs, the applications fail quickly so the errors can be resolved promptly on a smaller scale. DevOps and cloud-native applications complement each other perfectly.

Even though DevOps offers a more efficient and faster way to develop and deploy applications, security remains an issue. On paper, DevOps and cloud-native applications might be more secure than their traditional counterparts, but the reality is not that simple. With the newer, more complex, highly distributed infrastructures common to DevOps come new challenges that need to be identified and addressed.

This article walks through those challenges to highlight what it takes to secure a DevOps workflow. I won’t focus on specific tools because DevOps is not about specific tools; rather, I’ll focus on the components of DevOps workflows that present special security challenges and explain how to address them.

The Vulnerabilities You Just Can’t Ignore
1. Careless use of recycled code
One way DevOps improves developer productivity is by letting developers recycle code from previous projects or open source repositories such as GitHub. But without proper screening and sanitization, these pieces of code can pose threats to an application.
1. Multiple platforms make the attack surface larger
When deploying code in the cloud, you can’t use traditional security services. You need to rethink your security strategy. DevOps teams leverage various platforms, from serverless computing to microservices to multi-cloud in their applications. There’s no single solution to secure cloud-native applications. These applications are highly distributed and have various components, so organizations need a multifaceted approach to better secure these applications.
1. The dynamic nature of cloud-native applications doesn’t help
Cloud-native applications are continuously integrated and deployed, which makes it difficult for security teams to identify and mitigate risks. Traditional security tools can’t match the sheer velocity, scale and dynamics of cloud-native applications, leaving them useless.
1. Secrets need to be secured
When following the DevOps approach, security professionals need to create privileged accounts and login details and share them over business networks automatically. With serverless applications and the cloud, security teams don’t have a traditional location such as the operating system to execute security policies. A host of useful data, such as keys, storage account credentials, secrets, database passwords and embedded passwords, are stored in repositories. It’s crucial this data is kept extremely secure, as attackers can use this information to cause some serious damage.
1. Containers can leave your applications more vulnerable
Since the microservice approach involves a distributed infrastructure, it exposes your system to the network, which makes it easy for attackers to sneak into the system. Containers can be spread across different systems, which makes the threat landscape even larger. Containers are highly replicable, so if there is a vulnerability in one microservice, it can get replicated every time the source code is used.

Using open source software in containers can make way for an increase in vulnerabilities. DevOps teams should make sure unauthorized containers are not used in production. These containers can be used by attackers to launch an attack on the entire application.
1. Hosts aren’t automatically safe
Even if containers are properly secured, the hosts they run on are vulnerable to new threats and day-zero attacks. Container runtime should be up-to-date to avoid risks that can lead to container escape. This can help attackers to take control not only of a container but also the host operating system. So, it is important that hosts are given appropriate protection.
1. Kubernetes clusters need additional security
Kubernetes is a widely accepted container orchestration platform. However, it’s difficult for security to be enabled at multiple layers. Kubernetes uses APIs to manage containers, and recently, it was discovered that K8’s API server is quite susceptible to attacks—attackers can make use of the API server to perform malicious activities or install malware in the application.

Running multiple workloads on one cluster is not a good idea. Sensitive workloads should be run on a dedicated set of machines to avoid attacks through less secure neighboring applications. There’s also a need to secure sensitive metadata, which can be stolen and used to change privileges in a cluster, thereby paving the way for unauthorized changes in an application.

DevSecOps: The Future of DevOps Security

In traditional applications, security holds less priority, and security policies are always employed after development is completed. But with DevOps, security can’t be an afterthought. Companies depending on the DevOps approach need to make security their top priority. One of the ways to make sure security is in the foundation of a DevOps implementation is a DevOps security stack. To employ the DevOps security stack, IT teams no longer should be walled off from the DevOps team.

Security experts’ involvement can help them gain insights into what the application is and how it’s supposed to function so they can identify risks and take steps to mitigate them. This DevSecOps approach has become popular. DevSecOps is a collaboration between the development, security and operations teams who work together to ensure security is at the root of an application, not just on the surface.

Symantec Cloud Workload Protection: A Modern Solution for Modern Problems

Most organizations use hybrid cloud storage comprised of workloads distributed among public clouds, private clouds and on-premises, which can lead to an increased attack surface and vulnerabilities.

Symantec Cloud Workload Protection (CWP) helps organizations monitor and protect their workloads, no matter where they reside. With CWP, organizations don’t have to look for multiple products to meet their many security needs. CWP offers a single console to monitor and manage security across various platforms. It offers automatic discovery of workloads across AWS, Azure and Google Cloud, and visibility into security postures and software, which enables automatic workload monitoring and protection. With continuous delivery workflows and malware prevention, CWP is essential for modern software development.

Sign up for a free trial of CWP.

— Twain Taylor
June 13, 2019
Shift Left Without Fear: The Role of Security in Enabling DevOps
It’s the age of DevOps, and we all want to shift left, which refers to the idea of performing processes earlier in the CI/CD cycle. That includes security checks and audits. By starting security operations earlier in the delivery pipeline, it becomes easier to find problems, and teams have more time to address them before pushing code into production.

The challenge, of course, is building a shift-left security operation that allows you to perform security operations early without slowing down delivery speed. Let’s take a look at how to square that circle.

Defining Shift-Left Security

A popular term in DevOps culture is “shift left.” But what does shift left mean? Shifting left refers to the effort by a DevOps team to implement measures to ensure application quality at an early point in the software development life cycle. In the case of application security, this means implementing measures to ensure security concerns are taken into consideration while the application is being developed, rather than at the end of the process.

The overall goal of shift-left security is to identify and resolve potential vulnerabilities earlier in the development life cycle, when these issues are less expensive to fix. Doing so will help to mitigate the risk potential security concerns pose to both the delivery schedule and your end -users.

DevSecOps: Integrating Security Into the Development Process

Let’s be honest: Faster and more frequent development cycles aren’t going away. Slowing delivery speed to analyze and bulletproof the application at the end of the development process is no longer a viable format for implementing security. DevOps and CI/CD need to be embraced and enabled by the security staff.

For security teams to enable the DevOps process, collaboration is key. Through this collaboration, the process of DevOps transforms into something known as DevSecOps. In DevSecOps, security personnel work closely with the rest of the DevOps organization to help those who create and deliver the application to gain a better understanding of what is needed to address security concerns as early in the development life cycle as possible.

The following tips serve as a jumping-off point for implementing this collaboration:
- Make security the responsibility of the entire DevOps team: Impress upon the development and operations staff that security is now the responsibility of the entire organization. Simply addressing potential vulnerabilities post-development will eat up too much time and increase the risk that something will be missed. Buying into this process is an essential part of implementing successful collaboration between the security team and the rest of the organization. An example of this would be impressing upon developers to code with security in mind.
- Facilitate regular discussions about application security throughout the development process: It is the responsibility of the security folks to keep security considerations at the forefront of a developer’s mind while they develop new features. Use the agile process to your advantage. Backlog meetings can involve security folks to ensure the creation of user stories includes consideration for security concerns. Your daily standup meetings are another place to include security personnel. This will provide security staff with an opportunity to speak up when the work described by the developer requires—or might require—considerations related to security that have yet to be addressed. Doing so will help to ensure the ball is not dropped, allowing for potential issues to be rectified before they even exist.
- Use test automation and continuous integration to your advantage: When it comes to addressing security concerns throughout the development process, continuous testing is your best friend. And, in cases where it makes sense, the DevOps team should integrate automated test scripts for the detection of security and into their continuous integration process. Such tests can include developer-written scripts to test for vulnerabilities in the code base and tools for static code analysis. Another part of your continuous testing strategy could involve the use of tools for infrastructure monitoring.
The data produced by these strategies can then be analyzed by security folks. And from this analysis, recommendations can be made to the DevOps team regarding the path forward to close potential gaps in security before they ever reach production and can be exploited.

Understanding the Role of Security Personnel in the Development Process

Security folks are often not coders, and development personnel are often not security experts. And while it may be frustrating at times, the solution to managing this disconnect is in the collaboration described above. Situations will arise where developers need to develop with various technologies to build a quality product. It’s possible that these technologies will lend themselves to security concerns that require security expertise. The job of security personnel shouldn’t always be to fight back against the use of these technologies. The security team should act as facilitators—take what development and operations staff intend to do and foresee the potential impacts that innovation will have on security—then take these potential impacts and provide insight into what needs to be done to reduce the risks.

Conclusion

In the past, security often was considered as an afterthought in the development of an application and its infrastructure. The movement toward faster development cycles and more frequent application releases has forced development organizations to think about security differently. Nowadays, so much can change in an application in a short time. This has forced DevOps organizations to integrate security into the development pipeline to make security a continuous consideration in the development life cycle. With this integration, the responsibilities of security personnel have shifted. Security teams now need to be enablers of the DevOps culture, empowering DevOps processes with security in mind.

To see how Symantec CWA can help you shift left without fear, sign up for a free trial to see how it can help secure your cloud.

— Scott Fitzpatrick
June 11, 2019