devopsy.kinsta.cloud

Tag: automation

Automated Certificate Management No Longer a Luxury

On December 6, 2018, approximately 32 million customers of the UK telecom provider O2 woke up to a frustrating reality: The most widely used mobile network in the country was experiencing a day-long outage of its 4G mobile network.

For some, not being able to check the weather forecast, traffic report or football scores was a minor inconvenience. For others, the impact was much more severe, as people throughout the country realized they couldn’t collect email, use their connected apps or perform any other task requiring data usage—no matter how critical to their business or personal lives. The world quickly came to realize that O2 subscribers in the UK were not alone: Other mobile providers around the globe, including Softbank in Japan, suffered similar outages in the same time frame.

What caused this global outage? Was it a software bug? A poorly tested update, shared too soon? Maybe even a cyberattack targeting mobile network infrastructure? In the end, the explanation was much simpler: A digital certificate expiration in a back-end mobile data service from Ericsson caused a cascading systems failure, ultimately resulting in data outage for these mobile services. This issue highlighted the potential impact of certificate errors in our modern digital enterprise and the value of automation in managing such a critical process.

Human Errors, Absences Lead to Outages

While the O2 and Softbank service outages had serious consequences in their home countries—the U.S. equivalent would be a provider like Verizon losing data service for an entire day—they are hardly the only ones to experience this issue. Earlier this year, during the shutdown of the U.S. Federal Government, a number of important sites went down or lost functionality because no one was there to manually renew expiring web certificates. Even popular agencies such as NASA and critical services such as those provided by the Department of Justice were not spared, and both agencies and the citizens they serve were forced to watch helplessly as their certificates expired and services went down.

Although it is of little comfort to the customers who woke up without mobile data or citizens who were unable to use government services, this scenario could easily have been prevented had automation been utilized to manage the certificates in question. When certificates are managed manually, it isn’t always easy to determine the cause of an outage like the one experienced by O2 and Softbank. An expiring certificate may not be the first answer that springs to mind, and may only be noticed after poring over code, looking at network monitoring and checking other potential culprits. Certificates are not always front of mind for IT teams, which underlines the value in simplifying their management.

The technology sector has made significant strides in this regard, and organizations have tools available to them that can help streamline certificate issuance, management, renewal and revocation. Gone are the days when certificates had to be manually issued one at a time, and organizations kept spreadsheets of certificate expiration dates hoping they remembered to renew them. While we’re not able to say specifically why or how the root-cause Ericsson certificate was allowed to lapse, unexpected certificate expirations can be eliminated through proper use of certificate automation—making such a disaster unlikely to repeat itself if the proper steps are taken.

Peace of Mind at Scale

Getting started is easier than you might expect. The rise of automation in managing digital certificates presents organizations with a simple and exciting solution that extends well beyond just outage protection. Perhaps best of all, many organizations operating in a Windows environment may already have access to auto-enrollment and auto-renewal tools using the Microsoft CA, which has been a pioneer in the field of automated certificate services. Chances are, you may already be familiar with using Microsoft CA as an easy way to take care of your internal (private) certificates.

But what about public certificates? True, Microsoft CA doesn’t handle those, so most organizations will need to find a third-party vendor to manage the SSL certificates for their web servers, load balancers, VPN devices and other networking gear. There are vendors capable of managing all certificate requirements under one roof, whether you operate in Windows, MacOS, Linux or other environments. The rise of standards like the Automation Certificate Management Environment (ACME) have made integrating different environments increasingly simple.

Designed by the Internet Security Research Group, ACME has become a popular certificate management tool. More than 150 million websites and 130 open-source client tools use ACME. In fact, ACME is supported by a number of tools familiar to those who use DevOps to manage deployments, including Kubernetes, Chef, Ansible, Salt Stack, Terraform, Istio, Docker and others.

As enterprises look to leverage DevOps with increasing regularity, effective certificate management can help facilitate container, micro-service and application security. Furthermore, orchestration tools such as Kubernetes support certificate management natively, leveraging ACME as the underlying protocol. The private keys may be stored in Kubernetes Secret or Hashicorp Vault so the certificate management system integrates seamlessly.

Digitally signing your DevOps containers with a private CA helps verify the identity of a given container as it communicates over TLS while also preventing unauthorized applications—and the tools available to ACME users add significant value. Of course, this is just the tip of the iceberg—using automation to manage these certificates means plugging into a vast ecosystem of resources, reducing deployment time and eliminating the possibility of human error. It really is that simple.

A New Automated Future

Certificate management is hardly the only area where human error can have an outsized impact, but the O2/Softbank outage clearly illustrates severe effects such errors can have. Service outages, damaged reputations and frustrated consumers lie in the wake of poorly managed certificates, and organizations are increasingly turning to automation to solve the problem. As protocols such as ACME become integrated into an increasing number of commonly used web tools and outside vendors expand their ability to shoulder the certificate management burden, it will only become easier for organizations to incorporate automated elements into their own environments.

— Abul Salek

July 26, 2019
HashiCorp Tools Bridge DevOps, Legacy IT Environments

At its HashiConf EU 2019 conference this week, HashiCorp extended the reach of its Terraform Enterprise tools for automating the management of IT infrastructure beyond the realm of DevOps into legacy IT environments.

Via integration with the Service Catalog from ServiceNow, IT administrators working with a traditional ITIL-based framework now can invoke Terraform Enterprise to provision IT services. HashiCorp CTO Armon Dadgar said rather than requiring organizations to hire an infrastructure engineer to automate the provisioning of IT infrastructure, IT administrators now can do the job.

At the same time this week, HashiCorp announced that policy sets may now be configured using the source policies available in multiple version control systems (VCSs), including GitHub, GitLab and Bitbucket. The goal is to make it simpler to enforce policies that would limit, for example, the number of virtual machines a developer might be allowed to spin up within the context of a set of GitOps processes, said Dadgar.

Finally, HashiCorp also announced that HashiCorp Consul service mesh networking software has been enhanced to include Layer 7 application traffic management capabilities alongside integration with the HashiCorp Nomad application and cluster scheduler software.

In general, Dadgar said HashiCorp is moving toward enabling its portfolio of offerings to span multiple types and classes of IT environments. HashiCorp Consul, for example, can be run across both legacy IT environments as well as Kubernetes. Most organizations will need to support both legacy monolithic and microservices applications using a combination of DevOps and legacy ITIL processes, noted Dagar. Rather than having to master new tools for each environment, HashiCorp is making a case for a common set of tooling that can be applied across multiple IT classes of processes.

While some organizations have fully transitioned to DevOps, many more organizations are still struggling to make the transition. Reasons for not making that transition faster range from simple inertia to a lack of skills. Rather than having to take an all-or-nothing approach to modernizing IT processes, Dadgar said HashiCorp is enabling each IT department to embrace IT automation wherever they happen to be on that journey, including being able to embrace new platforms as they become available.

Dadgar noted that level of extensibility is critical because most tools are still being brought into organizations from the bottom up by individual members of the IT staff, rather than being championed from the top down. In the absence of an executive mandate, Dagar said the path of least resistance is to make IT automation tools as accessible as possible to members of the IT team regardless of skill level.

There will, of course, come a day when IT leaders will try to rationalize processes around a common standard set of practices. The degree to which any IT organization has been able to achieve that goal, however, has been mixed at best. In the meantime, IT organizations will always need to keep their options as open and flexible as possible.

— Mike Vizard

July 10, 2019
How Systemic Approaches to ITOps Impact the Business Culture

We are at an inflection point in the rise of big data. Enterprise IT infrastructure is producing data at a scale like the world has never seen before. IT operations (ITOps) management has accelerated past the notion of data storage and processing (vis a vis the Hadoop model) and into the world of analytics-as-a-service. We now need systems to drive insights and manage change on-demand. In fact, we need a new operating model for IT in the face of these new effects.

Can a New System Save Us?

The key to transforming into a responsive, data-driven enterprise—while avoiding a flood of unnecessary processes and over-staffing—is to adapt automation to intelligently consolidate information flows. At the same time, the dynamic nature of today’s IT environment requires IT teams to become literate in a completely new approach to technology, one that spans all functions of the business. Consequently, risks that previously stayed inside organizational units now leak across domains, influencing decision-making for the entire company. These factors will try to pull the organization in multiple directions at once; only a unified data management system with the intelligence to extract operational insights can drive positive business change.

Culture Has Got to Change, Too

Agile processes such as DevOps and AIOps offer mechanisms to control the flow of data and turn it into actionable results. For example, continuous application development and deployment gathers data about functional use, user sentiment, efficiency and various other metrics to inform the overall process and improve the product. Such rapid dev cycles let organizations identify and fix issues more quickly and with fewer resources, ultimately delivering greater business value.

The resulting streamlined collaboration among IT, development and customer teams improves capabilities but also introduces dependencies that must be managed effectively. It changes the culture to adapt to the system, and thereby, the system of service delivery will fundamentally affect how organizations are structured, built and run. Eventually, there will be increased digitization of every human process, and that means human impact slowly will be replaced by AI impact. This doesn’t mean that machines are coming to take our jobs; instead, it means the modern enterprise IT organization must address a new future. How will it look?

Retraining and Reskilling Is a Fact of Life: In a world of dynamic IT, requirements are no longer static, and teams need to be prepared. Organizational units must be able to work in different ways with new technology, with new skills that include robotic emotional intelligence, cloud-native literacy and infinite adaptability.

System operators become stewards of the business because system operations directly impact business outcomes. It won’t be enough for workers to merely deploy, configure and manage IT while isolated from the goals of the company, because those goals are inextricably linked to how objectives are achieved through innovation. Employees will need to develop new skills for functioning in a dynamic system. Training staff in agile methodologies, service-oriented software design and six sigma-style approaches for process improvement will become critical for success.

Automation Is Everything: Automation by policy won’t be enough; even runbooks will become obsolete. Data and results are always changing, so simple automation will not drive meaningful change. AI can be used to drive automation policies and frameworks, with the potential to reduce errors and integrate disparate systems, which previously required multiple points of oversight.

The trend toward using AI for data understanding is driving intelligent services into other parts of the business, as well. Rather than just for correlating system and operational data, AI is being adapted to business process optimization. Moving forward, IT operations can begin to run themselves, led by deep data analytics. Alerting and response processes automatically will use their own feedback to update the system’s intelligence, identify emergent trends and take actions to deliver improved operational results.

Risk Becomes Universal: Increasing connectivity and generalized frameworks (such as DevOps and site reliability engineering) mean that risk is no longer siloed to one department or function. It’s spread everywhere, and cascading effects become commonplace.

In general, successful engineering outcomes rely on coordinated processes and policies that dictate operational needs, even after deployment. Introducing connectivity between formerly disparate systems can create instability that potentially affects operations in other unintended areas. Thus, broad risks arise if different systems can’t interoperate once unified into a single IT ecosystem. Addressing requirements under a generalized framework helps prevent gaps between application implementations.

The System Drags the Company Forward: Insights-driven action will transform digital business well in advance of executive priorities. Change will happen before anyone is ready. Indeed, with so much complexity, so many tools and processes and so many competing business demands, IT changes now impact culture in entirely new ways.

Business impacts will come from system intelligence rather than manual processes. Efficiency in operations management is derived from the entire system, including the people themselves. What was previously the IT culture is rapidly being replaced with a combination of self-adapting processes and workers who focus more on value than on plumbing.

Transforming the enterprise from silos into integrated platforms will propel the business forward. The resulting system will also drive cultural change at an accelerated rate, perhaps faster than the worker community is prepared to accept, but informed by real data instead of executive intuition.

— Bhanu Singh

June 24, 2019
Agile Stacks Extends Reach of DevOps Automation Platform

Agile Stacks this week announced it is extending the reach of its namesake DevOps automation platform to Kubernetes and additional cloud services.

Company CEO John Mathon said that by extending the reach of the company’s platform to include Google Cloud Platform (GCP), Microsoft Azure and Red Hat’s OpenShift distribution of Kubernetes alongside existing support for Amazon Web Services (AWS), Agile Stacks will make it easier for organizations to implement best DevOps practices across hybrid cloud computing environments.

Agile Stacks made the announcement at the HPE Discover 2019 conference. The company counts Hewlett-Packard Enterprise (HPE) among its investors, and at the conference this week, HPE this week outlined how Kubernetes is core to the company’s evolving hybrid cloud computing strategy.

Mathon said the Agile Stacks platform provides a framework for automating the creation of DevOps pipelines across multiple tools and continuous integration/continuous deployment (CI/CD) platforms. In all, there are more than 80 DevOps tools and platforms available in the Agile Stacks Service Catalog, including Kubeflow, Jenkins, GitLab, Spinnaker, Harbor, Minio, Ceph and Portworx.

At the core of the Agile Stacks platform is a SuperHub automation hub, which accelerates software delivery based using a set of custom composable stacks that can be configured by the DevOps team. Each component of a stack is auto-configured to deploy, upgrade, rollback, promote, clone, backup, generate test scripts, perform merges, conduct performance testing, scan for vulnerabilities and execute functional testing.

In effect, Mathon said Agile Stacks is essentially automating the DevOps automation. Organizations that have embraced best DevOps practices tend to rely on multiple tools from different vendors, which inevitably leads to silos of automation that often work at cross purposes. Agile Stacks provides an extensible GitOps framework for integrating all those tools that enable organization to evolve more easily as their DevOps processes mature, added Mathon.

In addition, he said Agile Stacks expects to play a role in making DevOps more generally accessible by reducing the complexity organizations initially encounter when trying to embrace DevOps practices.

As hybrid cloud computing starts to become more commonplace, thanks in part to the rise of Kubernetes, Mathon said DevOps complexity is only likely to increase. Organizations have made it clear they want to leverage multiple clouds alongside their existing on-premises IT environment. Right now, however, the DevOps processes associated with each of those platforms is, at best, fragmented. In addition to making it easier to unify DevOps processes across multiple platforms, Mathon noted Agile Stacks can also enable organizations to keep better track of cloud spending by applying automated cloud tagging via Superhub.

Of course, most organizations are challenged automating the development and delivery of software on a single platform. However, regardless of how far down the path any organization is toward DevOps, the business is moving forward. In many cases, the rise of hybrid cloud computing will finally force many organizations to implement a set of best DevOps process enterprisewide. As part of that process, organizations will look to standardize on a specific set of DevOps tools whenever possible, or at least until developers discover the latest DevOps tool to supplant all others.

— Mike Vizard

June 21, 2019
Building a Security Feedback Process for DevOps
The last few years have seen some major slip-ups in the security space among all major cloud providers, resulting in uncertainty and speculation. That’s understanding; cloud security is an extremely complicated subject as enterprises build and deploy applications faster than ever before to keep up with business requirements. Most of the security issues that occur in the cloud today are around compromised accounts, identity theft, malware apps and, of course, data breaches, which are business-critical and showstoppers, in many cases.

Given the dynamic environment of DevOps and the nature of the cloud, it’s a complex problem to address and fix. The most effective way is to embed security, or what is usually talked about as the shift-left of security, in the IT pipeline. If DevOps is the culture of collaboration to reduce risk by the size of change, DevSecOps is the automated security control built into each point to build a sort of self-defense mechanism.

DevOps is not about tools; it’s about philosophy and process. One of the most important processes to include to succeed at DevOps is a feedback loop that reinforces constant collaboration.

This is certainly true when it comes to security-related information. In a healthy DevOps organization, security problems that are detected at any point in the delivery pipeline are communicated quickly to all stakeholders, who can, in turn, respond accordingly.

In this post, we will look at the steps involved in building a security process that will ensure the right information reaches the right people quickly and that teams collaborate without duplicating efforts.

Automate Your Way to Better Feedback

Automation is the first step, and a no-brainer when it comes to DevOps. Without automated security tools for code analysis, configuration management, access control, vulnerability management and secrets management, it is very difficult to scale security in a dynamic DevOps environment. Manual efforts in the cloud are doomed to fail in many cases, as environments change rapidly. Hence, automate. The recommendation is to automate every security control and the entire feedback process as much as possible.

Manage Roles Right and Control Their Accesses

Identity and access management has become the linchpin in cloud security. Since it’s very easy to give people the wrong access, it becomes important to figure out who can access what resources. Access to information and resources must be secure, yet easy to obtain. All access should be given to a group rather than an individual. Mapping security access to the organization’s hierarchy is an effective way of establishing a communication loop. The right information reaches the right group of people. It is important to be able to answer questions such as the following to ensure the communication loop is efficient:
- Who can delete this VM?
- What are the resources a certain user has access to?
- Who has service accounts access?
Tools that show a relationship between users, groups, roles, permissions and resources help answer these questions.

Encrypt Secrets, Manage Their Keys

Another important step to building an effective communication cycle is controlling access to application-level secrets. Security can be improved by tightening information flow between relevant groups, which can be accomplished through creating and controlling access to secrets using principles of least privilege. Access keys to those secrets can be kept in a centralized secrets management tool or a key management tool. The rotation of these keys can be automated as often as required.

Designing for Failure and Proactively Managing Vulnerabilities

Though security personnel may cringe at the mention of the word “failure,” designing for failure means thinking ahead and having a plan B. The cloud is dynamic by design and things happen at an incredible pace, often resulting in issues of elasticity, configuration and cloud power.

Context becomes an important parameter for managing vulnerabilities because there are multiple vulnerabilities at any given time. Automated vulnerability management and configuration management solutions are available to take the complexity out of protecting cloud environments. Automation capabilities and continuous monitoring provide deep insight, intelligent recommendations and continuous protection of the cloud, thus ensuring effective management of information in the feedback cycle.

Log Everything — Everything

Just embedding security to shift left is not enough to ensure security. It is important for all relevant collaborating groups to know the status when some code is deployed. The primary source of visibility and feedback comes from logs. Effective logging strategy is the paramount attribute of security design. Logging should be enabled everywhere possible: data access logs, identity logs, admin activity logs, logs within the cloud environment, OS, network platforms, etc. Logs, though cumbersome and volume-heavy, are non-intrusive and a critical visibility tool. You need to know what’s going on before deciding to do something with the event itself, which brings us to the next step: monitoring.

Monitor Those Events

Testing departments often seem paralyzed with too many flags, which prevent them from prioritizing critical issues. Having log and event data is an important first step, but the absolutely necessary next step is to relay them into an automation strategy that will separate the unimportant, voluminous issues from business-critical priorities. Automated monitoring for events triggered makes the feedback cycle more effective. Otherwise, logs can become noise—a copious amount of noise.

Alerts and Notifications

We embed security in the code. We manage access and roles. We turn on the logs and monitor them. An event occurs. Now, who do we alert? An effective alert notification policy will ensure that notifications are sent to the right group of individuals, depending on the notification filters—the level of criticality and relevance of the issue type. Notification policies specify the conditions that trigger a notification that will be sent to the recipient. Notifications can be sent by email, SMS, voice, push messages, etc. A recipient can then acknowledge the alert, which indicates to other users that someone is working on the issue, thus avoiding duplication of efforts.

Conclusion

To conclude, automating security controls and feedback cycles enables a productive DevOps ecosystem by exposing vulnerabilities early in the product life cycle. Automation, streamlined communication and appropriate response to priorities ensure availability, network connectivity and resilience. At the same time, they reduce the possibility of a data breach and duplication of efforts. At the heart of everything is the ability to deliver code in a secure and reliable way to meet ever-evolving business needs with agility.

To see how Symantec can help maximize security in a DevOps environment, check out a free trial of Symantec Cloud Workload Assurance.

This sponsored article was written on behalf of Symantec.

— Twain Taylor
June 20, 2019
Unravel: A System Architect’s Guide to Data Operations
Big data is an older fashionable term that, when combined with agile methodology, AI and ML and analytics, has become a powerful trend once again in data operations (DataOps). If you’re a systems architect, it pays to understand the philosophies, benefits and tools to power DataOps and improve your data-driven applications. This article serves as a guide to define enterprise architectures that support the needs of an organization.

Inside Statistical Process Control (SPC)

The DataOps Manifesto defines a set of key principles to guide systems architects and implementation teams. In summary, they focus on the customer and analytics, driving positive change through a continuous improvement process that relies on both teamwork and automation to help drive efficiency and overall data and system quality. This is based on statistical process control (SPC), which is a quality-driven approach to using data and associated statistics to monitor, control and improve systems and processes. Quality, in this sense, translates to maximum application uptime, increased usefulness to the customer and reduced waste in terms of delivery time and resource usage.

Data is king in SPC, and most data-driven application architects go through a set of phases when they design systems. These phases include:
- Data Discovery: Knowing the sources of data used by applications, as well as identifying new sources of data about your applications and their environment. This includes performance statistics and other systems and process measurement data.
- Data Relationships: From the data and their sources, knowing the relationships and hidden value in your data is the next phase. This is the realm of big data and analytics where you search for business meaning buried in the data to be used to expand your offerings to customers while also enhancing and improving the customer experience.
- Data Monitoring: This means you want to ensure system uptime by monitoring your data-critical systems (e.g., data feeds, databases, analytics and so on), but it also includes automation of the first two phases above.
In line with SPC, performance data is used to ensure that your systems and their processes are stable, with the uptime and responsiveness needed to meet customer demand. To achieve this, the reach of DataOps extends from the data scientists and business analysts within your organization who solve business problems with data and algorithms, through to the database administrators and the developers who work with that data (e.g., ETL) and build applications from it.

From here, DataOps gives you insight into value discovery, turning these into new methods and tools, with an emphasis on continuous refinement. Tools and techniques will help you automate data mining, the discovery of potential performance improvements, identifying areas to create more efficiency and lower costs and prepare you for the future in terms of business and architectural needs.

Leveraging DataOps: Automation Is Key

Putting feedback loops in place helps to automate the entire data flow, from business intelligence, where tools are used to help data analytics processes, to data science, where new mathematical and architectural approaches are used to drive deeper understanding data, to developers and operations staff, where the data is aggregated into applications and further measured to drive new value back through this process.

Automation is used to build efficient feedback loops without overburdening your IT staff. In turn, automation tools are used not to replace your analysts, data scientists, developers or other IT personnel, but to enhance their jobs and make them more efficient at what they do. As data production and consumption continue to grow, your organization’s ability to discover hidden value in this data and drive it through to real system improvement will be a key competitive advantage.

AI-Driven DataOps

While data visualization is important to effective DataOps, displaying graphs and pictorials can go only so far. Gaining actionable insight into metrics is what drives improvements in performance and efficiency. To remove the potentially error-prone manual approach to achieve this, organizations are leveraging an AI-driven approach to DataOps. This helps discover hidden dependencies between applications and their resources (e.g., databases and other data sources) along with their users and their needs, providing unified visibility and operational intelligence to optimize your entire ecosystem.

AI can help you dive deeper into the effectiveness of your data, going beyond operational efficiencies to identify improvements to code, overall system architecture and the configuration tools and frameworks in place across the cloud or data center. Going further, AI-driven DataOps can help you optimize your consumption of cloud-based resources or identify key components to migrate (with recommended changes to deployment) when moving to the cloud.

The Big Picture: Cloud, Microservices, Serverless and DataOps

As a systems architect, you need to maintain a big-picture view of the data, systems and processes used within your organization to best serve your customers. This means you need to understand the entire application ecosystem to communicate and simplify your system and data operations. Tools help provide and maintain a 360-degree view into the performance, interactions and dependencies between components in your software systems and data workflows, along with all of the storage, compute and network infrastructure that supports them.

To enable the value in DataOps and drive visibility of the data and interactions within your systems, adopting newer architectural approaches can help. For instance, breaking apart monolithic applications using a microservices approach helps drive real value throughout your technology ecosystem; it helps improve the makeup and velocity of your development teams, improves the benefits of operational efficiency, drives a test-driven approach to development (since these services are more easily verifiable) and helps enable cloud deployment.

Other approaches such as the use of containers and serverless architecture help remove system dependencies from implementation, reducing the impact of deployment changes on your applications’ code. With all of these changes, the data orchestration and workflows without your applications (including your customers’ usage of this data, in some cases) become more apparent and measurable.

Call to Action: Leverage Tools and Frameworks

DataOps helps operations staff, app developers and enterprise architects reduce the complexity of delivering reliable application performance. Whether you’re new to DataOps or have been on board for some time, knowing the tools and frameworks to help drive both your architectures and operational control is key to success. Tools such as those from Unravel (see Figure 1) provide monitoring solutions that map dependencies between applications and their data sources, and all of the components in between.

Figure 1: DataOps tools such as the one from Unravel should support your existing implementation and technology choices, while supporting the three phases of DataOps work.

The tools you use should automate the discovery of value and optimizations to be made, where possible, from the viewpoint of business owners, IT operations staff and the data scientists and developers defining and building the systems. All of this should be achievable regardless of the platforms, frameworks, languages and cloud providers you use to enable your applications and users. Finally, make sure your DataOps approach and tools support your organization in the future, with support for continuous improvement of overall architecture as well as data processes.

This sponsored article was written on behalf of Unravel.

— Eric Bruno
June 18, 2019
How to Automate an API Security Program Without Adding Staff
In today’s information economy, data is a primary raw material and a source of value to both providers and consumers. For many companies, entire business models are built on the exchange of information.

Consider a ride-sharing business that owns no vehicles of its own. What the company does possess, however, is a database of private vehicle owners who are willing drivers, a roster of people who will gladly pay for a ride and an application that can instantly connect a rider with a driver via an application programming interface (API). The ride-sharing company makes its money by taking a cut of the fare.

Another example is a company that collects and refines financial data and makes it available to investors who might want to buy stock. Companies such as E*TRADE, Merrill Lynch and Fidelity buy that investment data to make it available to their clients. APIs are at the heart of these important data exchanges.

How Prominent are APIs?

The information economy is synonymous with the API economy. APIs are increasingly the way business partners—internal and external—exchange and monetize data and services. APIs allow for machine-to-machine data retrieval, essentially removing barriers and accelerating access to data. There is hardly a modern application today that doesn’t provide an API to integrate with other applications and data sources. The typical mobile app today uses an average of 10 to 15 APIs to move meaningful data into and out of the app. Modern web apps called SPAs (single-page apps) do the same. APIs have proven to be so efficient and useful that many companies have begun creating new business ventures by mobilizing their data to third-party developers and the public.

DevOps teams heartily embrace APIs to advance their ability to produce applications much more quickly. APIs, however, are largely a mystery to security teams that must ensure that this means of rapid data exchange does not create unnecessary risk and negative exposure for the company or its brand. Enterprise security teams are right to question the vulnerabilities and threats posed by APIs. Insecure APIs have led to some very serious data breaches, some of which are outlined in the Dark Reading article, “Expect API Breaches to Accelerate.”

For example, the mobile payment app Venmo exposed sensitive details about hundreds of millions of transactions through a poorly secured API. In another case, an insecure Panera Bread API used by a mobile app exposed users’ private details through a searchable API that required no authentication to access. As a result, as many as 37 million records, which included customer personal information identifiers (PII) such as names, email addresses, physical addresses, birthdays and the last four digits of credit cards—all in plain text—were exposed. T-Mobile, McDonald’s, Symantec, Instagram and numerous other companies have suffered through significant data breaches traced back to insecure APIs. The damage is high because most APIs are designed to serve data without security in mind.

DevOps and security teams can agree that a data breach coming from their applications is completely untenable. Thus, designing and applying a security program that spans across all APIs across all applications is an essential function in protecting data and brand. Just exactly how can this be achieved? Let’s look at the steps of automating an API security program without needing to add additional security staff.

The Challenge of Securing APIs

The very nature of identifying all APIs makes it a challenge to secure them. A developer can create an API in a matter of minutes and publish it on the internet with almost no resistance when using public cloud services such as Amazon, Microsoft and Google. What’s more, she can make changes to that API as often as she wants, and every change cycle presents an opportunity to introduce new risk. It’s not unusual for a developer to change her API weekly or daily.

Let’s go back to that example of the financial services company that sells data to stock brokers. Suppose the developer makes a change to her API which unintentionally enables the brokers to see the client lists of every other broker that consumes data via the API. That would be a catastrophic breach, yet this is precisely the type of API breach suffered by the U.S. Postal Service a year ago.

APIs today increasingly are built on serverless infrastructures such as Amazon Lambda, Azure Functions or Google Cloud Functions. Traditional firewalls, gateways and agents can’t protect this type of API built on ephemeral infrastructure. Now consider that a large enterprise might develop and/or use hundreds or even thousands of APIs. The magnitude of the challenge to secure every API becomes apparent. Manual security oversight and enforcement are simply out of the question. Automation of security assessments done on a continuous basis is the only scalable approach.

Start With an API Security Framework

Every organization that creates or uses APIs needs an API security framework that consists of three steps:
1. Continuous API discovery and specification creation.
2. Continuous API specification analysis and inspection.
3. API policy enablement and enforcement.
Let’s have a look at each step and its impact.

Know What APIs Are in Use and What They Are Intended to Do

Gathering API specifications is an important first step in defining what APIs are intended to do. However, many APIs exist without any specification. A service that creates API specs helps to deal with those APIs void of documentation. However, this spec creation service also needs to continuously monitor and discover new APIs to remedy the problem of not knowing what APIs exist and their intended function. It’s common for a developer to document what her API is designed to do initially, but updating that documentation every time she makes a change to her API is rare. To compensate, the organization needs to have an automated tool that gathers this information and notes each time an API changes, then accordingly updates the API specification.

For example, consider an API that exists to transfer customer information between a company’s back-end server and a mobile application. The developer adds a feature or two and the API changes to support these new features, resulting in the API doing things it wasn’t originally expected to do. How would the organization know about these changes? There is a need to continuously analyze the API for such changes to understand the current state of specifications for what the API is doing.

The organization needs to create specifications of all the APIs, especially publicly facing APIs hosted in the cloud. This service needs to operate continuously to look for changes in the APIs constantly. There are tools that can automate these various activities, including discovering new APIs and changes to existing APIs. These tools also can uncover other cloud resources and data services related to these APIs; enumerate API domains, functions and associated methods; and generate a standards-based (Swagger, OpenAPI v3) spec if one does not exist.

Analyze the APIs for Security Threats

The next step is to perform a security inspection on each API operation every time it changes. The security team needs to know: Does it have the right data encryption? Is proper authentication in place? What type of authorization policy is being applied? What is the API’s level of availability? What kind of data sources does it access? Knowing each API’s current security posture is critical if the organization wants to avoid a data breach.

This API analysis needs to take place continuously and at scale since manual processes are ineffective for organizations using numerous APIs. Fortunately, there are tools and technologies that automate this process and perform tasks such as finding potential vulnerabilities within the authentication and encryption layers of the API. The tools also can generate security tasks with recommended changes for developers to remedy their respective API problems and alert security personnel when there are discrepancies between the API specification and its functional operations.

Create and Enforce Security Policies

No two companies have the same appetite for risk. If there’s one aspect of creating an API security program that requires a manual investment of time, it’s policy creation. Automation then can be used to enforce the policy. A security policy for APIs should start with two primary questions: Who should be able to utilize the API? And what level of sensitivity, regulatory oversight and/or privacy concerns does the API have? Based on the answers to those first two questions, you can form the basis for the security policy.

For example, a bank has an API that provides transaction statements for a specified account. That API has access to extremely sensitive and private information for each of its customers. One customer should never be allowed to access another customer’s account statements. Therefore, the authentication, authorization, encryption and availability of that API would be set to the most restrictive security standards available to ensure data exposure is tightly controlled. Another API from this same bank may provide the current interest rates of savings and money market accounts. This is data the bank likely would share with anyone potentially interested in opening an account with them. In this case, the level of security restriction would be lowered to support a wide variety of systems, browsers, SDKs and apps to ensure this information is highly available.

Historically, API policy enforcement for authentication, encryption and availability was often done at the network gateway layer. As applications have migrated to modern architectures such as mobile and cloud, the gateway approach has increasingly become less practical, less scalable and more restrictive. Application developers are often leveraging authentication, encryption and availability zones provided to them through SDKs and/or cloud services. In these cases, an automated service that integrates with these security services is a better approach for API policy enforcement versus traditional network gateways.

Automation is Key

Security teams are notoriously overworked and short-staffed. Worldwide, there’s a dire shortage of skilled security professionals available for hire, so adding additional security staff to build and execute the API security program isn’t a practical option. Bringing in consultants is expensive and can be budget-busting. Thus, automation through tools and technology is a sensible approach.

Automation provides the benefits of saving time and money when it comes to executing the three steps outlined above. What’s more, an automated program provides consistency in the repetitive tasks necessary to discover, document and analyze the APIs in use, and enforce corporate policies to control risk.

APIs are critically important to modern applications. DevOps people heartily embrace them, and security teams can learn their value as well once they understand how a robust and continuous API security program will benefit their organization.

— Doug Dooley
June 13, 2019
The DevOps Security Stack
The whole idea behind DevOps is automation. DevOps automates the development process by eliminating or minimizing human error. If an error occurs, the applications fail quickly so the errors can be resolved promptly on a smaller scale. DevOps and cloud-native applications complement each other perfectly.

Even though DevOps offers a more efficient and faster way to develop and deploy applications, security remains an issue. On paper, DevOps and cloud-native applications might be more secure than their traditional counterparts, but the reality is not that simple. With the newer, more complex, highly distributed infrastructures common to DevOps come new challenges that need to be identified and addressed.

This article walks through those challenges to highlight what it takes to secure a DevOps workflow. I won’t focus on specific tools because DevOps is not about specific tools; rather, I’ll focus on the components of DevOps workflows that present special security challenges and explain how to address them.

The Vulnerabilities You Just Can’t Ignore
1. Careless use of recycled code
One way DevOps improves developer productivity is by letting developers recycle code from previous projects or open source repositories such as GitHub. But without proper screening and sanitization, these pieces of code can pose threats to an application.
1. Multiple platforms make the attack surface larger
When deploying code in the cloud, you can’t use traditional security services. You need to rethink your security strategy. DevOps teams leverage various platforms, from serverless computing to microservices to multi-cloud in their applications. There’s no single solution to secure cloud-native applications. These applications are highly distributed and have various components, so organizations need a multifaceted approach to better secure these applications.
1. The dynamic nature of cloud-native applications doesn’t help
Cloud-native applications are continuously integrated and deployed, which makes it difficult for security teams to identify and mitigate risks. Traditional security tools can’t match the sheer velocity, scale and dynamics of cloud-native applications, leaving them useless.
1. Secrets need to be secured
When following the DevOps approach, security professionals need to create privileged accounts and login details and share them over business networks automatically. With serverless applications and the cloud, security teams don’t have a traditional location such as the operating system to execute security policies. A host of useful data, such as keys, storage account credentials, secrets, database passwords and embedded passwords, are stored in repositories. It’s crucial this data is kept extremely secure, as attackers can use this information to cause some serious damage.
1. Containers can leave your applications more vulnerable
Since the microservice approach involves a distributed infrastructure, it exposes your system to the network, which makes it easy for attackers to sneak into the system. Containers can be spread across different systems, which makes the threat landscape even larger. Containers are highly replicable, so if there is a vulnerability in one microservice, it can get replicated every time the source code is used.

Using open source software in containers can make way for an increase in vulnerabilities. DevOps teams should make sure unauthorized containers are not used in production. These containers can be used by attackers to launch an attack on the entire application.
1. Hosts aren’t automatically safe
Even if containers are properly secured, the hosts they run on are vulnerable to new threats and day-zero attacks. Container runtime should be up-to-date to avoid risks that can lead to container escape. This can help attackers to take control not only of a container but also the host operating system. So, it is important that hosts are given appropriate protection.
1. Kubernetes clusters need additional security
Kubernetes is a widely accepted container orchestration platform. However, it’s difficult for security to be enabled at multiple layers. Kubernetes uses APIs to manage containers, and recently, it was discovered that K8’s API server is quite susceptible to attacks—attackers can make use of the API server to perform malicious activities or install malware in the application.

Running multiple workloads on one cluster is not a good idea. Sensitive workloads should be run on a dedicated set of machines to avoid attacks through less secure neighboring applications. There’s also a need to secure sensitive metadata, which can be stolen and used to change privileges in a cluster, thereby paving the way for unauthorized changes in an application.

DevSecOps: The Future of DevOps Security

In traditional applications, security holds less priority, and security policies are always employed after development is completed. But with DevOps, security can’t be an afterthought. Companies depending on the DevOps approach need to make security their top priority. One of the ways to make sure security is in the foundation of a DevOps implementation is a DevOps security stack. To employ the DevOps security stack, IT teams no longer should be walled off from the DevOps team.

Security experts’ involvement can help them gain insights into what the application is and how it’s supposed to function so they can identify risks and take steps to mitigate them. This DevSecOps approach has become popular. DevSecOps is a collaboration between the development, security and operations teams who work together to ensure security is at the root of an application, not just on the surface.

Symantec Cloud Workload Protection: A Modern Solution for Modern Problems

Most organizations use hybrid cloud storage comprised of workloads distributed among public clouds, private clouds and on-premises, which can lead to an increased attack surface and vulnerabilities.

Symantec Cloud Workload Protection (CWP) helps organizations monitor and protect their workloads, no matter where they reside. With CWP, organizations don’t have to look for multiple products to meet their many security needs. CWP offers a single console to monitor and manage security across various platforms. It offers automatic discovery of workloads across AWS, Azure and Google Cloud, and visibility into security postures and software, which enables automatic workload monitoring and protection. With continuous delivery workflows and malware prevention, CWP is essential for modern software development.

Sign up for a free trial of CWP.

— Twain Taylor
June 13, 2019
The Human Value in Automation-Driven DevOps Testing
Enterprises are embracing DevOps underpinned by automation to ensure the quality of software across all digital touchpoints. In fact, automation has become the crux of continuous integration and delivery, the vital DevOps testing goals. Without automation, DevOps would become a distant dream, and businesses would struggle to keep pace with seamless delivery and integration. However, DevOps entails more than the implementation of the latest automation tool and demands a more holistic approach. It does so by bringing together processes, people and technologies under a shared vision. As defined by Microsoft, DevOps is the coalition of products, people and processes to facilitate the seamless delivery of value to the end customers.

DevOps is considered to be the latest advancement of the agile movement. It includes a greater number of stakeholders in the process of delivering value by linking the silo-based teams. It may not be the best practice, but it’s a discipline of people-centric management. Continuous delivery is the technical foundation of DevOps automation, which extends to the agile foundation as well. The close collaboration between the operations and development teams is known as DevOps, which is used to automate the delivery pipeline.

Role of Automation in DevOps to Increase Productivity

Since digital transformation has taken over the current business model, more and more companies are embracing the DevOps practice to drive the automated deployment of software. We are shifting toward an organizational culture in which development and testing take place simultaneously, quickly and more often. This helps maintain consistently high ratings on the quality of software with faster turnaround time. The main agenda of DevOps testing is to derive quality outcomes all along the software development life cycle (SDLC) and beyond. So, with the help of DevOps culture, a new code of development, testing and delivery is introduced in the overall SDLC.

The automation in DevOps empowers companies to:
- Enhance the reliability, re-usability and quality of all components.
- Save costs of upgrades and maintenance.
- Improve the rate of success for standardized processes to ensure the rapid delivery of applications.
Automate Everything — Is It Even Possible?

With the help of the DevOps approach, developers automate and accommodate operational activities, from the source code commit to operational productivity. The primary goal of DevOps automation is to remove bottlenecks in the pipeline by implementing automation in a single level of development. However, DevOps is not a fully automated process and is not operational without human intervention. It is the DevOps testing specialists who are responsible for the process to run flawlessly. Governance at every stage of DevOps testing is still indispensable in various processes undergoing automation. Importantly, human administration is still crucial in handling exceptions to complete the tasks of automation.

The Strategic Role of Humans in DevOps to Ensure a Continuous Pipeline

Remove the Boundaries of Roles

The primary feature of DevOps automation culture is about an enhanced collaboration between the operations and development teams. It has led to a cultural transformation at the organization’s level and inside the teams as well, to encourage continuous delivery. Since shared collaboration is an important aspect of the DevOps culture, it strengthens the collaboration within teams. Before the DevOps approach, the perceived lack of interest by the development team regarding maintenance and operations used to affect DevOps testing and business goals. However, this scenario has since changed due to an organizational shift toward automation and DevOps testing.

The gap between the silos of operations and development no longer exists with the arrival of DevOps. It has removed the boundaries of several organizational roles. Teams are required to focus on the shared goals of DevOps to enhance the end user experience, so the team members need to rethink every investment and decision around this shared goal, which requires human intervention. Automation cannot substitute the decision-making process of humans, for the latter has become vital across the organizational ecosystem as, and when, needed.

Omit One-Size-Fits-All Approach in DevOps Testing

Automation cannot be viewed as the eliminator of roles. The organization still requires expert knowledge and skill set inside your team. The focus should be on developing agile and cross-functional teams that are recognized for removing long-established stumbling blocks between the operations and development teams, and for facilitating collaboration. Developing efficient and functional teams would ensure the delivery of superior quality products.

Continuous Monitoring, Testing and Improvement

One of the key agendas of DevOps testing is to carry out a deeper level of monitoring. Since rigorous monitoring is needed at every layer of the testing process, from integration to infrastructure and transactions to components, it has become a crucial part of the road map of the DevOps approach. In this digitally connected world, monitoring and testing are no longer distinguishable. Continuous monitoring by testers offers a positive response to the user experience while eradicating pre- and post-production obstacles.

Conclusion

With the impetus in the growth of DevOps and agile practices in the past few years, testing has taken a new structure and form. It has evolved into a crucial discipline, and it’s not restricted to a phase of the development cycle anymore. It has become an essential part of the operational activities of production. So, is the role of a tester transformed radically from only a checkpoint to a crucial active participant in the DevOps testing? Teams need to cooperate based on the shared vision of enhancing augmented user experience. We need to understand the fact that automation is not an elixir to deliver on DevOps goals, but merely a part of the DevOps strategy.

— Komal Lopez
June 12, 2019
Scalability in the Age of DevOps: A Must for Success

DevOps seems to be an emerging topic “du jour” in the world of software development. And like it or not, it is showing no signs of slowing down. If used correctly, DevOps can be helpful—from making teams (developers and testers) work seamlessly to automating routine tasks, DevOps has proved its awesomeness in just a few years. So what is helping make DevOps a huge success? Scalability!

Technically speaking, scalability is something that defines the exact capability of the system, and it has to increase frequently to accommodate the ever-changing needs of the world. As customers rely more heavily on your services, achieving hyper-scale is becoming more crucial. To ensure scalability, you may need to redesign or rethink, your different processes and systems.

Say, for instance, customer demand seems to have reached its apex. Preparing comprehensive strategies that focus on scalability has become prime importance.

To get the most out of your application, invest in DevOps-related time and dollars. With the introduction of the cloud, developers are focusing on building features and products while ops (operations) are making sure everything stays in sync. At the same time, many traditional ops functions have been outsourced to cloud providers.

Scaling DevOps

Here are some best practices that can assist you in creating scalable, secure and high-performance deployments.

Be Vigilant

With the increase in the adoption of visualization, networks are more complex and heterogeneous, and so are the security risks associated with them. The only way left to manage these networks is by automating key management functions. Here is when DevOps comes into play. But for that, the method must be included with security as a key component, or the volume and pace of network would eventually change—several technologies such as SDN and virtualization will skyrocket the level of IT risk like never before. Another challenge is that security is still considered as an afterthought. And so-called business inhibitors are seen advising an organization on what can’t be done instead of how to do things securely. This is an area where security, developers and operation teams must foster a level of trust, or collaboration won’t exist. The only way to do this is incrementally and with vigilance.

Watch Changes Carefully

Considering DevOps as a collaborative mindset enforces professionals to come up with faster and more efficient ways to deploy, operate and upgrade applications. When new releases are delivered, several tools can ensure consistent configuration of infrastructure and software components. These tools also can be used to remediate security concerns by constantly validating the proper application of security’s best practices and taking automatic countermeasures.

Ease of Use

Automation and repetitiveness are what DevOps is all about. Fortunately, there are a few tools that are capable of automating the building and configuration of VMs. However, make sure you have the adequate horsepower to run this capability, either in-house—which is trickier to scale dynamically—or in the cloud, if your product is amenable to that. At the end of the day, it’s all about how you make your product installation and configuration easy.

Managing Gateways

While building any new goal or coming up with the best culture between development and operation teams, keep certain gates between functions to ensure that the production environment remains stable. Try to structure it in such a way that operations include development discussions and daily scrums. By doing this, operations teams simultaneously will understand what changes will be made in future releases. By integrating automation in both testing and operations, you will be able to find issues before they reach production. In addition, you’ll realize cost efficiencies and better quality operations.

Conclusion

In today’s fast-paced development environment, a well-thought preparation strategy for mass scalability is a must to make the entire process smooth and easy. DevOps reduces lead-up anxiety by getting a head start on planning, so you have more time to enjoy the fruits of your labor. Adding scalability in your strategy enables you to experiment and try new things without fearing failure. Testing new approaches ahead of time to see what does or doesn’t work will only help you learn and make your scalability efforts that much more effective for the future, as your company continues down the path of hyper-growth and hyper-scale.

— Herman Morgan

June 12, 2019