devopsy.kinsta.cloud

Tag: data

  • How Logs Can Empower Your CI/CD Delivery Pipeline

    How Logs Can Empower Your CI/CD Delivery Pipeline

    Change is the currency of software success. This need to change and grow is the basis for continuous integration and continuous delivery (CI/CD). Each year, new industries embrace this philosophy of constant change, but without the necessary data underpinning their decisions, every deployment is a risk. When you enrich your software delivery with log data coupled with best practice log management, you mitigate this risk and enable rapid, safe change in your organization.

    Logs are often the most common source of information about an application’s behavior. They provide context as well as result, opening up far more information than a simple count metric will. Despite the immense insight that they can offer, organizations regularly overlook them. Many leave their logs in files on servers, far away from the CI/CD pipeline that needs them. It doesn’t have to be this way; with some best practices, every part of your business can collect, analyze and utilize the hidden knowledge within your logs.

    Add Life to Log Data With JSON

    Logging in an unstructured format dramatically increases the complexity of detecting patterns in your logs. By logging your application output in JSON, you open up the possibility of analysis of all of your logs. This gives you a top-down view of your entire system while still maintaining readability. This empowers your CI/CD pipeline by allowing you to query and filter your logs, which then enables you to focus in on the problem and precisely diagnose any unwanted side effects of your latest change.

    Create Actionable Alerts

    If alerts are defined and channeled correctly, have context and can be interpreted easily, they will be actionable, add context and offer greater value. It is difficult to provide context when a single metric exceeds a threshold, but log lines are more sophisticated. They can include additional information to provide context and allow us to pinpoint the source of the issue fast. This is an essential capability for a company embracing CI/CD.

    Prioritize Your Logs

    Structured logs often come attached with a severity, which indicates how seriously we should investigate an event. An INFO level log signals business as usual, but an ERROR demands immediate attention. Once your logs have these tags, you can make intelligent, automated decisions that will automatically respond to unwanted changes in your system.

    Benchmark Each Version to Understand ‘Normal’

    “Normal” is a complicated idea when working in a microservices architecture. It may be safe to ignore a minor slowdown of a single service, but an unfortunate combination of events can lead to disaster. Benchmarking enables us to see these disasters well before our users. Benchmarking is the act of recording what the normal behavior of an application is. After each deployment from your CI/CD pipeline, compare your new behavior with your previous benchmark. If something is out of the ordinary, logs provide the information you need to act decisively. They provide an outstanding baseline signal for your benchmark.

    Analyzed Logs Can Level up Your CI/CD Pipeline

    Building a CI/CD pipeline is not the most difficult part. Neither is the deployment of new features. In fact, the greatest challenge facing any organization wanting to deliver their changes via a CI/CD pipeline is observability. With a best-practice approach to the preparation, curation and analysis of application and system logs, we can overcome this challenge and confidently change at a pace that propels us to the forefront of our market.

  • Consider DataOps for a Competitive Edge

    Consider DataOps for a Competitive Edge

    DataOps seeks to eliminate existing barriers between people, technology, tools and data

    It’s no secret that COVID-19 has put the economy under enormous strain and future economic prospects are uncertain. A smart way to give your organization a leg up against competitors is DataOps. Adopting a DataOps culture makes the best use of your data—it helps to optimize the data powering your business because it successfully monetizes and systematizes your approach to working with data. It will help to get your new products to market faster and also accelerates business agility.

    For example, a DataOps approach can help a data scientist who has spent weeks developing a new machine learning model that is complete but not ready to go into production. She will need to set up data stream monitoring and other basic requirements before she can put it in place.

    This type of challenge is routinely faced by DevOps teams and the best practices they implement help avoid potential setbacks to quickly put their hard work and innovation to use. Applying the same development practices to the deployment of data-intensive services and applications significantly speeds up project completion times.

    Why DataOps?

    DataOps puts the principles and powerful DevOps practices such as automation, continuous delivery and quick feedback cycles to work in a data environment. Its strategic focus is to bring people closer to data, by delivering proven tools that deliver data to people with big-picture business knowledge. DataOps also removes complicated engineering and IT processes that stand between data and those with business knowledge to give projects an important and meaningful business context that greatly improves outcomes.

    The goals of a DataOps approach are to eliminate the friction that’s holding business stakeholders, developers and data scientists back and to increase project productivity and business outcome success. Automating and streamlining processes help people to quickly and easily move from ideas to development to production without lag time.

    DataOps also helps take concepts into production by fitting data projects into familiar, standard CI/CD development workflows and plugging them into any existing framework to deploy applications.

    How to Put DataOps to Work With GitOps

    GitOps is becoming a core component of DataOps practices and puts them into action. It allows data engineers to define data processing applications and pipelines as configuration and manage them through Git and mature CI/CD pipelines. This brings with it higher levels of standardization, security, governance and, as a result, an accelerated rate of delivery. Adopting GitOps in the data management world can pose problems for some organizations working with powerful and fast-changing technologies such as Apache Kafka and Kubernetes, but the automated processes it brings turbo-charge delivery time.

    Cloud migration is another way in which a GitOps approach makes a difference for organizations that want to take advantage of managed services in Microsoft Azure, AWS or Managed Streaming for Apache Kafka (MSK). GitOps helps simplify migration of Kafka clusters used for streaming to the cloud service of an organization’s choice—simply define the desired state, step back and GitOps takes over.

    GitOps’ pull-based approach to deployment, in which an operator watches Git and applies the desired state, creates several benefits. One benefit is not needing to poke holes in production environments to provide push-based actions from software such as Jenkins, which has numerous common vulnerabilities and exposures (CVEs). With a GitOps pull-based approach, attack surfaces are reduced, enabling solution migration across technology and clouds.

    A Better Process Results in Better and Faster Outcome

    Real-time business is here to stay and there is huge pressure on businesses to change fast. As the saying goes, “You snooze, you lose.” With the right tools and strategy backed by a good communication plan, DataOps will work to your advantage, bringing the benefits of a DevOps culture.

    Faster time to market via increased productivity is one advantage of a DataOps approach. Putting people closer to data, together with the right tools, helps you empower business users and developers so they can add new features and capabilities to any deliverable, and deploy and monitor it quickly.

    When working at a faster clip, don’t overlook the importance of governance and security. Build governance into the DataOps process to ensure deployment of the correct applications in the right way. And, maintain business continuity without disruption to existing infrastructure and processes.

    DataOps seeks to eliminate existing barriers between people, technology, tools and data. By bringing business and technology imperatives closer together, you gain invaluable context and a better ability to focus on driving strategic business outcomes that matter.

  • How To Build a Business Data Map

    How To Build a Business Data Map

    Recently on devopsy.kinsta.cloud, Anais Baig provided a brief overview of the privacy process and capabilities that developers need to think about to comply with new global privacy laws such as Europe’s GDPR and California’s CCPA. Top of the list for complying with these laws? The need to build a business data map.

    It’s no exaggeration to say that without a comprehensive, accurate data map—or, in other words, a complete inventory of the personal data the business possesses at any given time—it’s impossible to comply with these modern privacy regulations. Developers will understand better than most the difficulty in creating and then maintaining this record.

    But it can be done, and today we’re going to dive into the requirements and some of the technical processes that go into crafting a data map.

    What’s in a Data Map?

    In simple terms, a data map shows the personally identifiable information (PII) your business possesses, why you possess it, where it lives and how it’s being used. To answer these four questions, you’ll need to audit the databases in your business. Be sure to include those that are part of third-party SaaS applications as well as your own proprietary data systems; figure out the fields of information contained in each. Then you’ll need to coordinate with your team to understand the business purpose for which each of those data points is being used.

    The bulk of the work that developers typically do is that first step: auditing the business data system and documenting the database structures. At the very least, these are the things that you’ll need to be able to say about every field of personal information you possess:

    Source: How was it collected?

    Use Case: What is the business purpose for which it is being used?

    Storage: Where is it being stored for additional processing, transformation or record-keeping?

    Access: What business units have access to the data?

    Retention: What is the schedule for deleting this data or renewing the user’s consent to possess it?

    Consent: Was this data obtained through explicit customer consent?

    If you’re a small business, sometimes a data map can be “visualized” with a simple spreadsheet, but it very quickly uses its utility as data infrastructure scales. Larger businesses often find it very useful to construct a data map like an actual map, with a visual representation of the data relationships that exist in the business. In this form, a data map most often contains nodes and links to show how different systems that contain any personally identifiable information link together in your organization.

    Here’s one built by Ethyca:

     

    As you can imagine, without automated tools it takes a good bit of grunt work to get to this point of elegant visualization. For those just getting started mapping data in their business, we recommend a hybrid mapping approach.

    The Hybrid Approach to Mapping

    The hybrid approach means using an automated baseline process to identify and record the individual personally identifiable information that your organization is processing into one central repository, and then using manual label analysis data analysis to annotate and verify entity relationships across applications in your stack.

    Label analysis involves using a data attribute’s name to decide which data dictionary element the attribute identifies. For example, an attribute named ‘email_id’ can be identified as ‘person.contact.email-address’.

    When the attribute’s name alone is not enough, the name of the entity to which the attribute belongs is included in the analysis. For example, the attribute named ‘name’ in an entity named ‘customer’ can be identified as a name of a person, whereas a similarly named attribute in an entity named ‘city_master’ can be identified as a name of a city.

    Sometimes a group of attributes occurring together can be identified easily. For example, attributes with names ‘lat’ and ‘lng’ can be identified as latitude and longitude of a location. Similarly, attributes with names ‘f_name’ and ‘l_name’ can be identified as parts of a person’s name.

    Data analysis lets you manually confirm the relationships posited in your label analysis by using actual data from your databases to test.

    The presence of certain values or patterns in the actual data can confirm certain mappings. For example, if the data for an attribute follows ‘foo@bar.com’ pattern, it can be confirmed that this attribute captures an email address.

    Range values captured in attributes of different tables can be checked to see if those attributes are referring to the same piece of information, such as a user_id, that can be used for linking the tables with specific relationships. Similarly, the absence of values in an attribute can confirm the opposite.

    If you’re not using an off-the-shelf data mapping solution, the label and data analysis processes can be arduous. But if done correctly, by the time you’ve completed the exercise, you’ll have a comprehensive understanding of how the data in your business links together, and you’ll be ready to start answering the compliance-related questions about each PII field you hold. You can also begin thinking about processes for maintaining the map’s accuracy as your data inventory flexes and evolves.

  • Grafana Labs Update Aims to Simplify Observability

    Grafana Labs Update Aims to Simplify Observability

    Grafana Labs today released an update to the open source Grafana project that adds a viewer that simplifies tracking the path of a single request through a distributed system.

    Version 7.0 also adds a plug-in for Amazon CloudWatch Logs along with support for Apache Arrow, a unified data and plugin framework that promises to make it easier to build plug-ins for the analytics and monitoring tools that are widely employed in DevOps environments to provide observability.

    The latest release also adds a range of automatic rendering options for a variety of formats, including tables and non-time-series charts alongside panel plugins for maps, pie charts and gauges.

    Ryan McKinley, vice president of applications for Grafana Labs, said Grafana has gained traction in DevOps environments in part because it makes it easier to query and analyze multiple types of data without having to pull it all into a central repository. All queries are launched against data wherever it resides, he said, noting by chaining a set of simple point-and-click transformations, users will be able to join, filter, rename and calculate all in the same platform.

    That approach makes it easier for DevOps teams to set up observability platforms using any mix of tools and services without much help from internal IT operations teams. That approach is now also starting to gain traction outside of DevOps environments as business analysts look for easier ways to also analyze data from different classes of applications, McKinley added.

    While observability has always been considered a core tenet of DevOps, achieving it has been challenging for most organizations. Simply setting up all the underlying platforms required has proven too challenging for many IT teams. Grafana has emerged as a tool that enables DevOps teams to visualize relevant data within the context of a custom-built observability platform.

    It’s not clear to what degree Grafana might emerge as a de facto standard. However, during an economic downturn, it’s not uncommon for IT organizations to rely more on open source tools to reduce costs. Rather than having to lay off an analyst or developer, many organizations will first look to reduce their reliance on commercial software licenses. Many of those same organizations have been wrestling for years with a fragmented approach to analytics software that often can yield conflicting insights.

    McKinley said one of the benefits of Grafana is that it reduces the cost of developing different models that enable organizations to explore different potential scenarios. Too often organizations are wedded to specific models that might not reliably reflect a new business trend simply because the cost of creating a new model is too high or takes too much time.

    Whatever the analytics path forward, the one thing that is clear is that as IT continues to become more complex thanks in part to the rise of microservices, DevOps teams will no longer be able to function without some form of observability platform being readily accessible. The only real question now is how best to go about building and maintaining that observability platform.

  • New Relic Extends Reach and Scope of AI Capabilities

    New Relic Extends Reach and Scope of AI Capabilities

    New Relic this week announced it has extended the capabilities within its New Relic AI module to include support for additional data sources and provide access to more advanced analytics.

    Integrations with Splunk, Grafana, Prometheus and Amazon CloudWatch are now provided along with support for a number of incident management platforms, including ServiceNow, OpsGenie and VictorOps. New Relic AI previously provided support for the incident management platform from PagerDuty.

    DevOps teams can also surface more details about anomalies, including attributes that caused spikes, related signals to investigate root cause by looking at what happened around the anomaly and view upstream and downstream dependencies.

    Guy Fighel, general manager and group vice president for New Relic, said these and other capabilities being added to extend the capabilities of a monitoring platform that as a software-as-a-service (SaaS) platform provides a natural focal point applying machine learning algorithms to the massive amounts of data required to inform an AI engine.

    Other new capabilities being added to the platform include the ability to infuse correlations with decision logic to drive custom decisions and greater transparency into why and how correlations are performed.

    New Relic AI has also been enhanced to reduce the number of extraneous alerts that might be generated in addition to providing insights into the rate of total incidents correlated, total correlated issues, rate of noise reduction and the most common reasons why incidents are correlated. New Relic AI also provides access to a configurable set of filters to specify a grace period for each issue priority.

    Fighel said the goal is to make it possible to continuously evaluates telemetry data for anomalies and then share those insights via existing collaboration tools. To accomplish that goal, New Relic AI ingests multiple data types, including alerts, logs, metrics and deployment events.

    New Relic doesn’t see AI replacing the need for IT teams anytime soon. Rather, Fighel said AI will augment existing IT staff in a way that enables them to determine the root cause of any issue faster. Monitoring tools accessed via the cloud make an ideal platform for infusing AI into a service where data already reside, said Fighel. In contrast, other platforms require IT teams to aggregate massive amounts of data to enable an AI model to be trained.

    There’s naturally a high degree of skepticism when it comes to applying AI to IT management, but Fighel said it’s only a matter of time before AI capabilities are taken for granted. In fact, as IT environments continue to become more complex and there is an increased focus on driving down IT costs in the wake of the COVID-19 pandemic, it’s now only a matter of time before AI technologies become more widely adopted.

    In the meantime, DevOps teams should expect many of the rote tasks they need to perform today to become increasingly automated. That might make some members of the IT team uncomfortable, but before long most IT professionals won’t really miss aspects of their jobs that today are little more than pure drudgery.

    — Mike Vizard

  • Six Steps Every Business Should Take to Protect Its Data

    Six Steps Every Business Should Take to Protect Its Data

    Data may or may not be the new oil but it is the growth engine of today’s digital age. Data about a company’s customers, products, services and sales is worth a lot to the company itself, of course, but also to its competitors, as well as to business-savvy hackers. This is why every business, regardless of size, needs to protect its data. 

    Once created, data never really dies, and it is used in multiple ways. In fact, the more data is used for making decisions, the more valuable this asset becomes to you and everybody else.  

    Protecting something of such value requires a different strategy–we need to start from the data itself. This may look complex but here is an easy way of thinking about that process.

    What Data Do You Have?

    Knowing what sensitive data your company is generating and storing sounds like a no-brainer, but many businesses don’t have a good handle on the different types of data they create, copy, use or store. You may not realize, but some of your data might include personally identifiable information (PII), healthcare data and financial data. As you are a custodian of such data, this requires special attention not only to protect your users, but also to satisfy compliance regulations. If you do not know your data, you cannot quantify your risk, and hence cannot prioritize your limited resources and time.

    Where Is Your Data?

    If you don’t know where your data is, you cannot protect it. And many times, even IT staff aren’t exactly sure where sensitive data resides. Your data could be in databases, file systems, applications or archives. For a hacker, these are all ripe targets, not to mention the four, eight or more copies of the same data distributed across less protected test, development, disaster recovery and backup systems. In addition, fragments of your sensitive data often get distributed across dozens of interlinked or even disconnected systems. If that sounds odd, think of your own personal information. Some of those bits may sit in email, Word and Excel, along with text messages, sticky notes and notebooks. Now, extrapolate this to a company of thousands of employees, and recognize that hackers need to get access to only one of those sources.

    Who Has Access?

    It is important to understand who your users are, and what access they have. Many companies give too much access to too many people, and worse, they do not know about how many keys have been distributed across the company, and to whom. Instead, companies should limit access to as little data as possible, and assign rights on a strict “need-to-know” basis. The fewer the people with limited access, the lesser the chance that they copy, share or contaminate that data. Even if a hacker were to take over their account, your loss would be limited.

    Who Touched Your Data?

    Understanding where your data is and who can access it is a great starting point, but you need to monitor actual data access to detect anomalies and unauthorized usage. Given that threats and vulnerabilities are always evolving, you may not be able to stop all of them, but detecting intrusion can allow you to intervene and limit the loss. Companies should audit users who log on to their systems, and track what they are doing, especially if they have administrative access to those systems or have broad access to sensitive data. It’s not that you do not trust them, but blind trust without verification can be exploited—especially by hackers masquerading as them.

    Why Keep Doors Open?

    A bank with broken windows or weak locks on side doors invites intruders. It is critical to lock down your systems by reviewing your configuration settings, patching your systems, closing unused services/ports and locking down critical data. Hackers today have sophisticated automated tools that can quickly scan systems for gaps and vulnerabilities. If they are going to map out your IT systems, don’t you want to scan them and fix them before they come probing? The more exploitable the holes in your system, the faster a hacker’s ROI.

    Why Use Real Data?

    It certainly is easy to test applications against real data, but doing so creates copies of data that your organization was already having trouble protecting. One simple way to reduce risk is to use fake-but-real-looking data for testing and development purposes. This means systematically replacing all occurrences of real social security, credit card numbers, email addresses, financial information and health data with unrelated random data. Using fake data not only limits exposure to the testers and developers, but also mitigates damage should someone gain access to the test system.

    Protecting data isn’t just about setting up the firewall or installing anti-virus software on your endpoints. To most adversaries, these are just speed bumps. Protecting data requires starting from the time of creating data, discovering what you have, knowing where it is stored, enforcing access rights and tracking its usage.

    As custodians of the valuable data that never dies, we need to have a strategy around completing these tasks in a scalable and repeatable manner. Status quo is not an option in an age where breaches are the stuff of near-daily headlines.

    We hope you never become a target of any attacks, but in case you do, understanding your sensitive data and being proactive in protecting it can make the difference between a quick recovery and a drawn-out nightmare.

    — Vipin Samar

  • Why Enterprise Survival Depends on a Context Driven Approach to Data

    Why Enterprise Survival Depends on a Context Driven Approach to Data

    Enterprises are sitting on an enormous amount of data that could be fueling their automation journey. Much of the information needed to run a business is existing data. However, much of an organization’s data is inaccessible and unsearchable. This so-called “dark data” is hidden away in paper forms, spreadsheets, emails and literally billions of PDF documents. This dark data is locked away and mostly unusable by the systems and AI required to power the autonomous business of the future. Enterprises are trying to achieve 21st-century outcomes while being shackled to 20th-century data practices. 

    I recently read a report from NewVantage Partners on a survey they did concerning how big data and AI were driving digital transformation. While 92% of the survey respondents said they were increasing investment in big data and AI, there were some interesting findings that suggest organizations are not realizing the ROI they had hoped. For instance:

    • Only 31% have a data-driven organization.
    • Only 28% stated they have a data culture.
    • 53% are not treating data as a business asset.
    • 38% have not realized any measurable outcome from their investment.

    Now, there’s a whole list of reasons and hurdles that are preventing organizations from realizing better value and outcomes from their efforts to become data-driven. But, could the underlying approach to becoming data-driven be flawed? Could organizations be struggling because they continue to apply a legacy mindset and approach to data acquisition and storage? Could these challenges be a contributing factor in 70% of digital transformation projects failing?

    Just take a look at the growing number of stories, market reports and surveys on digital transformation and you’ll see a graveyard of failed projects, lack of success and high turnover rates. The current approach to digital transformation is not working. 

    In our push to become data-driven, we capture data and attempt to transform it into computable bits and then stuff the data into rigid structures. This legacy approach to capturing and representing information outside the human brain results in a proliferation of single-use data that doesn’t scale with the business. We always lose some of the meaning, relationships and context. Despite all the advancements in technology, we’re still trying to model the information based on how we’ve historically stored data. We’re still producing flat data.  

    In the autonomous enterprise of the future, and the digital workers that will run them will need rich, contextual data. The true power and value of digital transformation will only be unlocked by the adoption of a semantic data-driven approach. The future of work will be built on digital workers powered by machine learning, artificial intelligence and knowledge graphs. It’s not enough for an organization to be data-driven anymore–they need to become contextually-driven. We need to start capturing the meaning to achieve a better understanding of the data.

    “I liked the idea that a piece of information is really defined only by what it’s related to, and how it’s related. There really is little else to meaning,” said Sir Tim Berners-Lee, the inventor of the World Wide Web.

    Context is all about the meaning. Context is more than just a piece of data. Context is the background in which an event, task or process takes place. It is the deep, multi-dimensional understanding of data and the relationships between the entities. Understanding the context can lead to accelerated insight through AI and machine learning. To be competitive, enterprises need more than data; they need context to enhance productivity and efficiency. But our current approaches to capturing and storing data have historically failed to capture this valuable context. As the amount of data grows exponentially, organizations will realize semantic data is key to survival.

    — Kevin Harbauer

  • Micro Focus Lays DevOps Foundation for RPA

    Micro Focus Lays DevOps Foundation for RPA

    Micro Focus has announced the general availability of Micro Focus Robotic Process Automation (RPA), which sets the stage for employing DevOps processes to construct highly automated business processes.

    Thanks to the rise of artificial intelligence (AI) and digital business transformation initiatives, interest in RPA has increased significantly. However, rather than merely using RPA to build bots to automate a narrow set of static processes, RPA is now leveraging machine and deep learning algorithms to optimize a wider range of dynamic business processes and adjust more easily to changes within an application.

    Travis Greene, senior director for IT operations management ay Micro Focus, said new capabilities such as object recognition technology included within Micro Focus RPA enables the platform to detect and automatically adapt to changes to the user interface.

    Other new capabilities now include the ability to visually or programmatically create workflows using a tool that records screen actions, the ability to more easily combine graphical and programmatic workflows spanning multiple platforms and the ability to deploy hundreds of thousands of bots as needed. A centralized dashboard manages the bots, all of which have unique IDs and encrypted, role-based credentials.

    As RPA and other forms of AI continue to be integrated and embedded with applications, Greene said it’s a matter of time before DevOps practices are extended to RPA platforms. Organizations will seek to train bots continuously as more data is made available, which in turn will drive updates to applications that are dependent on those bots. In most cases, depending on the complexity of the task, the bots will be built using a mixture of application programming interfaces (APIs), command-line interfaces (CLIs) and graphical tools. Organizations will need to define the processes to determine how and when each organization will opt to invoke one interface instead of another, either in isolation or combination, added Greene.

    Historically, organizations have been able to apply RPA to static processes with mixed success. With the rise of AI, the scope of those efforts has become more ambitious. Business leaders are too inclined to believe that just about every business process can be automated using a combination of AI and RPA. However, AI and RPA still are confined to relatively narrow use cases involving highly predictable outcomes.

    That’s changing, however, and RPA and AI are becoming more pervasive across the enterprise. DevOps teams will need to find ways to update both their applications and RPA/AI models without breaking one or the other. Most of the current RPA/AI models are highly sensitive to any change made to the application environment they are monitoring. And, unfortunately, the data scientists who are largely responsible for constructing those RPA/AI models don’t yet tend to have much of an appreciation for best DevOps processes. There will, however, come a day soon when these two worlds will collide.

    — Mike Vizard

  • What is Cloud-Native Workload Protection?

    What is Cloud-Native Workload Protection?

    We’re living in a cloud-native age. That means that many of the paradigms that worked in the days of on-premises hosting no longer suffice.

    Chief among them is security. To thrive in today’s cloud-native world, organizations need to rethink their approach to workload protection and bring it up to speed with cloud-native environments.

    In this article, I walk through what that means by explaining how cloud-native computing changes the calculus of workload protection and security.

    The Rise of Cloud-Native Computing

    More and more organizations are shifting from on-premises hosting to cloud-based environments built using shared infrastructure. As a result, applications and infrastructure these days exist in more dynamic and complex environments that contain a lot of moving parts and are dependent on a slew of external resources (which, in turn, may rely on other compute and infrastructure resources). Thus, we have to think in terms of an entire workload (instead of just our application code) for our applications to run effectively on the cloud.

    This new paradigm is enabling the concept of “cloud-native workload” to gain traction. We can think of this as a distinct capability that we can run on a cloud instance—in other words, all of the resources required to make an application functional, which could be anything from a web server to a database, to network resources, to the data that needs to be fed in (and of course the application code itself). A lot of these workloads likely would run on containers.

    These discrete sets of workloads running on cloud platforms necessitate a new way of mitigating risks and protecting our application and its dependencies. Cloud-native workload protection is a security category that is fast becoming very relevant among security-minded folks and distinguishes itself from application security. Applications are only part of a broader context of workloads (which may include things such as deployment and monitoring), and we must think of security in terms of a workload. Any means of protecting these workloads and mitigating risks and attacks against them is what cloud-native workload protection refers to.

    Although cloud environments can be more secure than self-hosting on-premises, security is a shared responsibility, and our cloud workload protection strategy does not fully fall into the hands of our cloud service provider. Surveys have discovered that many companies have at least one critical security flaw in their AWS configurations. We must take precautions and be proactive when it comes to protecting our cloud-native workloads, which could mean extending security policies and tools we have for our cloud-based systems.

    Security may be the greatest challenge for cloud-based workloads due to the myriad components that require a security analysis, along with the different attack plane combinations that these components produce together. We will dive deeper into what protecting our cloud workloads means in this new environment.

    Security Threats in Cloud-Native Environments

    We need to be aware of system vulnerabilities, data breaches, account hijackings and insecure APIs. We also need to identify any flaws in our identity and access management protocols and make sure two-factor authentication is enforced. Other considerations include running due diligence on third-party systems and understanding the implications of sharing cloud resources with other users on the same platform—we also may be compromised if they are.

    Finally, we need to understand containers and their implications in terms of security. Containers are particularly vulnerable because their ephemeral nature makes enforcing standard security difficult, and the variations in container images introduce more points of entry. This makes intrusion detection in cloud environments a tricky business.

    Discover

    Perform regular assessments that oversee and can uncover weaknesses in this dynamic environment. Make sure to routinely test code for any weaknesses, monitor new deployments for vulnerabilities and breaches, then keep a watchful eye on security logs and fire off appropriate alerts. Manage and monitor network security and ensure the visibility of all traffic.

    Visualize

    We can’t understand what we need to control if we don’t maintain an effective visibility strategy. This entails being able to visualize the workloads that we are running and being able to act swiftly if there are problems related to the workload. Ideally, some type of central interface can allow us to synthesize different workloads and events to help us manage our security solutions and tools.

    Protect

    Cloud platform providers are responsible for securing the physical hardware infrastructure and virtual machine instances, but we are responsible for securing access to and between our workloads that are running on this infrastructure. So we must maintain the operating systems that we have chosen to run on these virtual machines, and be diligent in applying security updates and patches and installing antivirus software.

    Implement some real-time policies and best practices that prevent the propagation of exploits, and restrict access to servers as much as possible and ensure the correct configuration of your firewall. Have a system for managing configurations, patching, logs and administration privileges. Regularly audit your system and procedures. Are you running any arbitrary code?

    Invest in a Cloud-Native Workload Protection Platform

    To optimize your cloud-native workload protection strategy, it may be beneficial to look into outsourcing the work to security platforms by trusted leaders in cybersecurity (e.g., Symantec Cloud Workload Protection Suite). These platforms can employ advanced and comprehensive protection for all types of cloud workloads against exploits that traditional methods cannot sustain. It can scale easily and automatically, be controlled from a single cloud-based dashboard and provide automatic discovery, visibility and protection of cloud-native workloads.

    Remediation

    Despite your best efforts, some security issues will arise. That’s why it’s important to have a remediation plan in place. Your remediation plan should include guidelines that define who will respond to security incidents, how information about incidents will be shared within your organization and how response actions will be recorded. For more serious incidents, it is also wise to have a plan in place for handling any legal- or PR-related concerns.

    Compliance

    Meeting compliance requirements is not the only reason to secure your cloud workloads. But it is one important reason. Non-compliance could mean fines, as well as a loss of reputation. Compliance needs vary from case to case, so you’ll need to determine which compliance frameworks apply to your business and workloads, then take steps to ensure that you are in compliance. You’ll also want a plan in place for revisiting your compliance policies periodically, to make sure that you remain compliant even as the rules and your workloads evolve.

    Data

    Data protection is a big topic, and even more so with the advent of the PII and GDPR data compliance regulations. As mentioned previously, the shared responsibility model between the cloud platform provider we choose and us means that we remain responsible for the protection and security of sensitive customer data. To avoid failing a data security compliance audit or suffering the consequences of a data breach, we must implement some protective policies and build governance and controls surrounding data stores into the business:

    • Enforce the appropriate levels of authentication according to the sensitivity and availability needs of the data. Use security groups to give authorized access and make sure that services and service accounts are securely implemented.
    • Refine your data retention policies. A breach of data is often unavoidable, so have a plan in place to periodically erase data that you no longer need to minimize the loss incurred during a breach.
    • Encrypt all data at rest and data in transit when dealing with your cloud-native workloads. Cloud platform providers cannot be relied upon to guarantee end-to-end encryption for data in transit.
    • Use additional external tools to safeguard data. Consider safeguarding to be the highest priority and leave nothing to chance.

    Learning More

    It is crucial for businesses adopting cloud platforms to have a cloud-native workload protection strategy, understand the threats that exist for this new landscape and ensure the security of their cloud implementation within the infrastructure from their cloud provider.

    This new complexity can be managed with a more sophisticated protection strategy. Symantec is a leader in cybersecurity, and its offerings are designed to meet the unique challenges that businesses confront in this cloud-native landscape. Find out how to use their tools to protect cloud-native workloads.

    Be sure also to check out Cloud Security Alliance, a non-profit organization that has performed comprehensive studies on best practices and new requirements for cloud computing.

    To see how Symantec’s CWP product can help you secure cloud-native workloads, sign up for a free trial.

    — Daisy Tsang

  • Data: the New Currency That Accelerates Business

    Data: the New Currency That Accelerates Business

    The amount of data created every day is staggering: 2.5 quintillion (that’s 18 zeros) bytes, to be exact. And that number is growing exponentially, thanks to mass deployments of internet of things (IoT) networks and devices. Cisco Systems estimates that by the end of 2019, IoT will generate more than 500 zettabytes of data per year. In the past two years alone, 90% of all the world’s collected data was generated.

    If it’s hard to imagine that such staggering volumes of data can be generated so quickly, just look around. There are IoT devices in our homes, vehicles, business offices, hospitals, shopping malls and parking lots and along highways, each generating and collecting data. Myriad sensors, cameras and listening devices collect human-generated data. Such devices are pervasive, and the number of data-capturing sources are making the information collected about us ubiquitous.

    Value vs. Privacy

    One might say we are human data machines. So, is data the measure of a person? While this is a relatively new phenomena, data has always been available about our activities, likes, dislikes, motivations and more—it just hasn’t been collected, aggregated and analyzed in the way it is today. It’s like we’ve discovered a vast vein of data gold that has always been hidden beneath the surface.

    IoT is becoming big business. Yet, the real value of IoT is in the data, as meaningful insights are derived and then leveraged for a business advantage. With artificial intelligence, automation and business intelligence, the more data collected, the smarter the applications become and companies gain more insights, increasing the value of data. However, there are legitimate security and privacy concerns as organizations collect our information, most often without our knowledge.

    Data is becoming the currency that drives our economies and a motivation manipulator that impacts our societies, with both good and bad results. If data represents currency, as it helps companies achieve greater revenue and efficiencies, how can consumers be in control of their own data and share in its rewards? How can they make sure their information is not put in the hands of those who want to use it for ill-gotten gains at their expense? These, and other important questions, need answers and solutions that protect consumers, as we look for innovative ways to put a boundary around our personal digital vaults before our data enters the marketplace.

    As consumers, do we care that this data is being collected? Is the benefit of receiving free services worth the price of our information? What should consumers do to make sure their data isn’t used in ways they don’t agree to? Perhaps consumers should expect companies to include them in their revenue-sharing. If they want to share in their data monetization, those collecting the data should ensure the data is protected and private, with full data-use transparency.

    Data Value Directly Related to Data Integrity

    When malicious actors gain access to data, not only does it put us at risk, but it also can lower the value of data. Consider this scenario of a hacker gaining access to data from the insurance company’s network-connected device in your car: The data on that device could have lowered your premium, but the hacker changed the information to show much higher speeds and extreme braking, then threatened you to pay a ransom or risk the hacker sending the altered data to the insurance company. In this case, because the IoT device was hacked, the financial impact of your data changed from a financial advantage to a financial disadvantage. Remember, data value is directly related to its integrity.

    There are two important things we must consider regarding IoT data. The first is to consider how the data is being secured. Is it encrypted while in transit? Are encryption and authentication deployed to prevent unauthorized users from accessing data when it is stored at rest? The second consideration is around privacy. How is the data being used and shared? Is sensitive data that we want protected being consumed and shared? Are device manufacturers providing consumers with transparency regarding whether and how their data is being used? Without establishing firm foundations in security, privacy and transparency, the protection and the monetization of our data will continue to be out of our control.

    — Mike Nelson