devopsy.kinsta.cloud

Category: Enterprise DevOps

Open Mainframe Project: Zowe Ready for Prime Time

There is a lot of interest in updating mainframe technology/interfaces across traditional enterprises. As development environments and toolsets have evolved outside the mainframe, there is a struggle to keep up—partially because backward compatibility requirements make wild changes difficult and partly because the very architecture of mainframes is different.

There have been several fragmented efforts over the last few years—some of them quite good—to update how a mainframe is accessed from external source and the ways that developers can code for the mainframe. After all, the number of developers proficient in mainframe toolsets continues to decline while the number of workloads does not, so something has to change.

The Open Mainframe Project’s Zowe attempts to address these issues in ways that are both more widely applicable and more technologically advanced. It takes advantage of Open Source REST systems to front-end hardcore mainframe functionality, so developers who are accustomed to application server development can call things such as Spring Boot-based interfaces and use languages including Node.Js to access their mainframe systems.

This week, OMP announced that Zowe 1.0.0 is production-ready. That means serious experimentation can be started at your organization to see if it addresses some or all of your mainframe modernization needs.

There will likely always be some (at least operational) differences in mainframe systems usage because the software infrastructure is designed a certain way, but increasing modern tool usage makes learning those different parts easier across the board. By implementing a vendor-independent architecture that has guidelines for users and vendors to add functionality from anywhere, Zowe is a powerful and expandable tool for getting the job done.

I sound a bit like a fanboy, but for most verticals, this is an issue that will continue to be a pressure point. It’s been more than a decade since companies started bringing back retirees to work on mainframe projects because new developers did not have the skills or interest to complete those projects. As more and more skilled CICS/RPG/PLI/etc. developers and mainframe operations specialists retire, this pressure will increase until it is resolved or unbearable. So I am a fanboy of any company/solution that is addressing this space instead of flocking off to the coolest new DevOps tool du jour. I’ve not yet used Zowe, so can’t directly recommend it, but reading over their documentation, I believe it’s a good idea to use tools we’re familiar with. So I’m rooting for it.

If you test it out, drop a comment and let the rest of us know what you think.

And keep rocking it. Whether in VS-2019 or VT-70, you are making the company engines hum.

— Don Macvittie

February 14, 2019
Self-Service Integration: 3 Areas to Watch
Self-service integration is a priority—more enterprises are turning to it for help in retaining their customer base, delivering a delightful customer experience and making it easier to do business with. It has become a de facto standard for modern-day integration solutions. Enabling business users to build integrations strategically addresses a dire need to establish a smoother symbiosis between people, processes and technologies; marshal them in an order; and align them with business objectives.

This approach is taking root across digital economies, and more advanced self-service integration features have emerged over time, which means bare minimum features are no longer enough. Enterprises with basic integration capabilities face the same old IT issues they faced earlier with conventional development tools.

In this blog, we’ll take a look at three critical features of self-service integration capability that can help enterprises sharpen their edge:
- Business User Capability: Enables business users to develop integrations and helps IT to drive innovations. Business users get built-in constructs to build integrations with heavy coding and rollouts, while IT can monitor all processes and technologies in a B2B network and avoid potential IT issues. With this advantage, business teams can operationalize data without IT getting involved and reduce time to revenue.
- Any Data Conversion: Modern-day integration solutions offer AI-led data transformation capabilities to convert data between source and target elements. The solution uses past traffic for suggesting data mapping recommendations and further reducing the time that goes into mapping data. Business users get a single interface to map data between source and target elements. Moreover, they can convert multi-gigabyte data in different formats in few minutes, rather than months.
- IT Governance: Self-service integration provides a drift-to-drive approach for IT governance. IT teams can combine old and new enterprise information systems and set up a concurrent integration strategy to use them for best of advantage. Start-to-end integration helps IT teams to streamline governance and enables business users to expand B2B connectivity. Teams can connect with a universe of applications without heavy resources involved.
Factoring these three capabilities in the zero code integration strategy can help teams to improve efficiencies, reduce the IT cost and improve B2B data exchange. They can counteract IT challenges in the form of multiple sources of truth and IT silos in minimum steps.

These three capabilities can help enterprises in improving process, people and technologies across business. By using these capabilities, enterprises can acquire knowledge faster and interpret data faster. Physical and digital environments can be populated with heuristic methods. Enterprise networks can be connected for more openness, connectedness, intelligence and scope.

End-to-end data integration improves the analytics process and drives the decision-making process: Data can be prepared for complex operations without complex coding.

Additionally, with self-service integration, enterprises don’t have to train citizen data scientists for data integration. Data scientists can use sophisticated functionalities for processing data from multitude of sources, and can re-engineer every piece of integration for syncing data with extremely powerful edge devices. They can establish strong neural networks for setting up diverse and robust architectures. Such systems can be used for complex use cases such as biometrics optimization, personalized medicine, material sciences and more.

A self-service integration framework is more than an enterprise service bus (ESB) or SOAP web services integration model. it helps in driving IT innovation, keeping things flexible and reducing costs. Modern- day integration solutions deliver out-of-the-box flexibility and reusability for bringing together composite services and executing them in a particular order across enterprisewide deployments. They help organizations grow their B2B networks and add new partners, suppliers, services and sales and support channels.

Enterprise information systems today generate massive volumes of data. There is a tsunami of information distributed between different B2B networks. Today’s data is being generated in variety of forms and the workloads have become difficult to predict. This veracity—or the ability to handle uncertainty caused by incompleteness, inconsistencies and anomalies of data—is a major barrier to unlock audience intelligence and customer segmentation in high-density computing.

Coalescing around a self-service integration strategy around these features is important for enterprises. It can help them in decimating the constraints they face on the road to B2B data integration: IT complexity that steadily decreases the pace of business, and silos that make them difficult to work with and slow down time to revenue.

— Jitesh Banga
February 4, 2019
APIs and DevOps

As always, there are interesting trends in DevOps, and I wanted to point out one that is starting to bother me: the “API economy,” and how it impacts DevOps decision-makers.

Traditional infrastructure—SAN, LAN, physical systems, etc.—were generally controlled the best from the command line. While there are a million of them, Cisco IOS is probably the one that you’ll remember best (or you stopped using to read this—just depends upon the shop). When large-scale automation started, these command lines became a bit problematic. After all, scripting was possible, but painful, and often had non-portable hardcoded values.

Virtualization, and later cloud—and even later SDN and NFV—all made this better, but on a layer over the top of (and normally not well integrated with) the physical layer. Years ago I bemoaned that we had doubled the workload of operations by re-implementing what was already implemented at the physical layer. So we were configuring the physical—even for cloud you have to get to the cloud—and then configuring the virtual.

Then “Software Devs as Kingmakers” came along, followed by “the API Economy,” and finally we started to see some serious attention paid to APIs by infrastructure vendors that should have done so long ago (some former employers of mine might belong in this group).

Here is the thing: There is a difference between having an API and having an API that is useful. When shopping for solutions, be they physical or software, don’t allow APIs to be a check box because it’s “the API Economy.” And don’t assume that an API useful to DEVops is what is needed to automate infrastructure. Monitoring APIs—particularly ones that provide you detailed information about an app as it runs—are useful to DEVops, and assuming you have tools that are or can be integrated, might provide value to devOPS.

But monitoring is not management. devOPS is increasingly automated, and increasingly needs viable APIs that can help them deploy/configure infrastructure. Vendors are responding, but the quality of APIs at this point in time varies widely. So make certain that evaluation looks at APIs available with an eye to suitability to task. devOPS needs management and might be able to use monitoring, while DEVops generally needs monitoring and has little use for management.

This is not 100% true of course, there are a billion different shops, most doing things in slightly different ways, but think of storage (because it can be done in few enough words to fit in a blog)… An app needs access to storage, ways to be notified of storage space limitations, and possibly to metadata. Operations needs to provision, migrate, backup, replicate… The list goes on. These all contribute to the availability and quality of the application, and are all DevOps, but the focus is different. And having a simple “number of blocks free” API will not help with the DevOPS needs even if it helps developers manage a piece of the application.

But we have more options than we ever did before! So fear not, just watch what your prospective vendors are doing, and check in with long-term vendors on occasion, too—because things are constantly changing, nearly always improving.

And keep rocking it. Presumably you are working this out already, knowingly or not, or you wouldn’t have time to read this blog.

— Don Macvittie

January 24, 2019
Collaboration Between Competitors is the Future of Enterprise Technology

Mark my words: 2019 will be the year enterprise technology companies go back to kindergarten and realize that sharing really is caring—about their customers. Several key moves last year point toward a future full of collaboration between large and small technology vendors—even those that compete against one another.

Companies rarely use a single vendor for all their enterprise IT needs, and their shift toward increased agility and their need for specialized capabilities will only lead to greater fragmentation. The vendors that make it easiest for their customers to connect IT systems with one another will be the biggest winners in the current IT market, even if that means they must work with previously sworn enemies.

While integrating products with those of a competitor may seem like a risky move, the vendors that pull off those integrations have the ability to actually strengthen their defensibility through the application of network effects. Think of it this way: An ERP system becomes more valuable to the company that uses it with each successful connection the system makes to the company’s other systems of record. In turn, that makes it more difficult for the company to turn around and replace those systems.

Key Examples

Over the past five years, competing enterprise technology companies increasingly have partnered with one another to benefit their shared customers. Last year was a significant year for that sort of collaboration in the enterprise space.

Salesforce.com announced several key strategic partnerships last year, including work with AWS that will make it easier for customers to send data back and forth from Amazon’s cloud. Of particular interest is the work the two companies are doing on Amazon Connect, the Seattle-based company’s contact center software that competes with Salesforce’s own Lightning Dialer product. Google Cloud and Salesforce are also working together on IoT and analytics applications to serve their shared customers, even though Salesforce already offers its own analytics suite.

On the end user side, Google now allows Box users to store files from Google’s G Suite collaboration service in the content services company’s cloud, and is working with Dropbox to provide similar functionality. Box and Dropbox compete with Google Drive, but working with the other companies allows Google to broaden its appeal, especially to companies that would prefer their data not get stored on Google’s servers.

But there’s a new sort of collaboration on the horizon: Multi-company alliances that provide even broader benefits to customers. Microsoft, SAP and Adobe announced last fall that they’re working on a new project called the Open Data Initiative that’s supposed to bring customer data from the three companies’ disparate products together into a common data lake for processing and analysis. Under the new initiative, customers should be able to read and write data from all three companies’ business applications.

Microsoft CEO Satya Nadella said at the time that customer demand for collaboration and integration was the driving force behind this initiative. Pooling business application data is critical to enterprises’ futures. Companies that want to make the most of advanced analytics will need to access previously disparate data sources. Organizations must break down their internal silos between functions to increase enterprise agility—and technology must follow suit.

Partnership Evolution

Why now? It’s easier for different vendors to integrate with one another when business applications reside in the cloud. Different companies can build APIs to facilitate connections with one another. Vendors can also ensure their customers are running on consistent managed environments, which simplifies connections between different vendors. Customers also can more easily change vendors in the cloud era, which puts increased pressure on tech companies to provide products that meet enterprise needs, which means they must integrate more broadly.

These companies also can build upon the foundation of their individual partnerships to form broader alliances. Both SAP and Adobe have individual strategic partnerships with Microsoft, and those relationships likely make it easier for them to forge a broader-reaching agreement.

Finally, if tech companies don’t partner with one another, other firms will come along to do the job. A fleet of businesses including Informatica, Sapho and Mulesoft (now a part of Salesforce) exist to help enterprises create and operationalize connections between data sources. If vendors choose not to build these integrations, customers will find a way to create critical connections and may even abandon siloed products for more open alternatives.

How Far Does This Go?

There are still some fairly clear (if unstated) boundaries that haven’t yet been crossed. For example, Salesforce’s partnership work with Microsoft could be best described as minimal, ever since Microsoft’s attempted acquisition of Salesforce fell through and Microsoft beat out Salesforce to acquire LinkedIn. Similarly, AWS and its hyperscale cloud competitors aren’t doing much in the way of strategic partnerships with one another, and are instead content to keep sniping at each other through public announcements and new product launches.

As such, enterprises should expect there will still be barriers to full and true openness among their technology vendors, and those barriers won’t be overcome easily. Case in point: For all of Microsoft’s talk of openness with the Open Data Initiative, the data lake in which customers are sharing their information with SAP, Adobe and Microsoft is hosted in Azure, not another cloud. If Azure remains the sole storage option for this initiative, it seems logical that not every Microsoft competitor will play ball and participate, since doing so would drive use of one cloud platform.

However, there are some glimmers of hope on the horizon for broader collaboration between major cloud players AWS, Microsoft, Google and IBM. All of them are now members of the Cloud Native Computing Foundation and are working on the Kubernetes open source container orchestration software. As Kubernetes forms the foundation for an increasing number of modern applications, it’s possible we’ll see major cloud providers opt to work more closely with one another on building foundational components of application infrastructure.

While such collaboration is far from friction-free, it does provide opportunity and a foundation for these firms and others to increase their collaboration, which we expect will be more important in the future as enterprises continue to embrace hybrid- and multi-cloud deployments.

— Blair Hanley Frank

January 18, 2019
How AIOps Helps You Get More Out of DevOps
DevOps may seem like an overused term. It involves taking an application’s source code and running it in an environment. It can cover the processes, the technology or even the people that maintain the technology that is running those very processes.

At its heart, though, DevOps is about helping developers to be self-sufficient when it comes to the basic operations around getting their application to a real environment. Once things are running in a test or production environment, there are other IT functions that are required to keep it running and performing well. While there are teams that specialize in these functions, such as site reliability engineers (SREs) and system administration, they are usually overwhelmed by the sheer amount of data their areas of responsibility generate without considering the data that applications going through their DevOps processes are also creating.

This is what makes AIOps the perfect complement to the DevOps space.

What AIOps Does

What AIOps does best is break down silos within IT. Having data from all the functional areas processed and handled by modern machine learning (ML) in a single place allows correlation, which leads to faster problem identification and resolution.

Before machine learning was introduced, for humans to process data, more or less they operated in three ways:
- They knew exactly what data to look for, such as when a specific business function isn’t working. There is a typical message in the monolithic application that handles all the core business.
- They routinely sifted through data only from their area of expertise.
- When they went outside their area, any logs and other data they had access to often was summarized data, which has time gaps and can miss a lot of detail that proper correlation needs.
Now with ML, the AIOps data platform can perform trend analysis, look for patterns and exceptions across all the data that is being generated in the environment and highlight what it considers important through web interfaces or using an event management solution for after-hours support.

Over time, as the AIOps platform continues to learn the environment, it can be given the ability to automatically resolve certain error conditions, as it will be able to identify them faster than human operators and often resolve them before they become service-impacting.

How This Helps DevOps

Machine learning aids DevOps processes in three ways when working with logs and data:
- Knowing exactly what data to look for: In a true polyglot DevOps environment that is leveraging microservices, it is next to impossible for any one person to know the most common error messages that might occur. In this model where a new business function can be conceived, written in one of many languages, and deployed into production without Operations even being notified, ML allows the AIOps platform to quickly identify what “normal” looks like for a new microservice and start looking for anomalies.
- Only routinely sifting through data from their area of expertise: When working in DevOps, developers need to be able to quickly find data from multiple areas in IT that is related to whatever they are trying to diagnose. Using AIOps, they will be able to quickly pull data from systems directly related to their incident. Whether it is as simple as using timecodes or a correlation GUID (globally unique identifier) that is put in the logs purposefully to aide tracking, AIOps will guide the developer through the other systems’ data.On the flip side, if there is an infrastructure problem, the same AIOps platform can help correlate what downstream services may have been impacted, from failed builds to Kubernetes pods that were restarted.
- Summarized and incomplete data: The true value of machine learning is that it has the ability to process a higher volume and variety of data at a much higher speed and accuracy than a human, and sort and index it for later use. Not having to summarize the data to one data point every five minutes could result in a problem being discovered that has occurred often but not consistently, as it may have been dropped in the summarization.
Conclusion

AIOps is very young in the technology industry, but has great promise as it builds on previous generations of IT operational analytics products, combined with the latest in machine learning capabilities. DevOps is a key field in IT that has more logs than it knows what to do with. Combining the technologies and expertise in these two growth areas will have a huge positive impact on productivity across enterprises with improved service management.

— Vince Power
January 10, 2019
Blame Replaces Strategy: Enterprise IT War Rooms Causing Incalculable Losses
The concept of a “war room” is time-honored. One of the early examples is the Churchill War Rooms, an underground bunker that allowed Britain’s leaders to plot the allied route to victory during World War II. British Prime Minister Winston Churchill, as plain-speaking and self-assured as ever, said, “This is the room from which I will direct the war.”

Today, the war room concept also serves the world of business—specifically, organizations’ IT departments when troubleshooting a major issue impacting the business applications and their supporting IT infrastructure. The problem? So many parts of the infrastructure, whether hardware or software, can be implicated as the root cause of any problem. As a result, IT war rooms can encompass many stakeholders who are experts in those individual parts. And, while the goal is to collaborate to resolve the problem and minimize the damages from it, IT war rooms often result in a case of too many cooks in the kitchen, all speaking different “languages.”

War rooms are a symptom of a much bigger problem within the IT organization. At the highest level, it results when IT operations teams have insufficient visibility and have limited proactive control of the infrastructure. For one thing, in a siloed organization (which most enterprises are), discussion across silos is rare. Plus, siloed monitoring tools for resolving infrastructure issues lack a common context and have no inherent understanding of the business impact of the applications. By contrast, the hardware, software and data that span silos do “talk.” In a sense, they collaborate. That’s the nature of connected infrastructure. Data, for example, flows throughout the enterprise, altogether oblivious of silos.

The lofty view of war rooms, in the words of Dennis Drogseth, vice president at analyst firm Enterprise Management Associates, is this: “Our war rooms can be either physical or virtual. Highly automated or not. Made up of consistent, well-defined teams, or not. But what makes them war rooms is the need for collaborative decision-making across silos.”

How Low Can IT War Rooms Go?

The reality is that, despite their noble charters, IT war rooms have devolved into blamestorming—rancorous debate and finger-pointing to indict application development, database analysts, IT operations or DevOps for a failure. Drogseth described a bleary view of war rooms as “disastrous assemblages of finger-pointing adults caught up with siloed versions of the truth.” In an even more disillusioning example, one vendor stated that “mean time to innocence” becomes the IT war room’s chief metric”—rather than isolating and resolving the root causes of issues.

With so many factors in play, dollar costs of IT war rooms are difficult to calculate. Taking aim at the problem, Enterprise Management Associates said, “The average War Room comprises 15 people and takes 5-6 hours to respond to and resolve the incident that caused it to convene. 29% of War Rooms take more than 11 hours to respond and resolve.” And those figures are over and above the substantial losses that occur if a server or switch, for example, goes down and war room staffers bicker over who’s to blame. Revenue losses can easily reach into the millions of dollars per hour of outage.

Furthermore, the problems and costs are growing. Infrastructure complexity has ballooned with the growth of hybrid data centers. The hybrid model uses cost-effective, flexible compute and storage via the cloud, while retaining the control and security provided by on-premises infrastructure. However, the reality is that highly virtualized, multi-cloud environments are amazingly complex and problematical when it comes to ensuring application service delivery. Out-of-control infrastructure leads to outages and slowdowns—and finger-pointing.

Remedies

There are, however, several ways to mitigate the downsides of IT war rooms:
- Discover technology solutions that provide cross-domain insights. Stated more simply, that means finding tools that can trace infrastructure issues to their source—exactly where the issue originates—even if the symptoms appear elsewhere in the infrastructure. Achieving this level of sophistication effectively eliminates finger-pointing.
- Learn from history. Issues repeat themselves; if an issue occurs but is not traced to its source and remedied the first time, it will almost certainly recur. Often, it requires analytics-based tools to make the necessary correlations and remedy an issue.
- Create what EMA calls an “informed context.” An informed context can help resolve an issue, moving from triage and diagnostics, to remediation, to validation. Knowing where applications live on the infrastructure at any given time and the business priority or tier of each application is one example of “informed context.”
IT will likely always have its war rooms, and many organizations will claim that they’re necessary. At its best, a war room is a highly motivated, solutions-driven SWAT team. Of course, SWAT teams, like anything to be valued, will forever come at a cost to the enterprise. One way to view the IT war room is that it’s just one more cost center. Maybe the better way to view it is by quantifying its value: If a war room saves minutes in getting a server up and running after an outage—or prevents an outage altogether—the savings could run in the millions of dollars. But too often, the opposite is the case: IT war rooms further contribute to the lost revenue from a failure.

Let’s hope that enterprises remember the original intent of IT war rooms and reorient them around strategy rather than blame. If not, enterprises might want to consider closing the door of the IT war room for good.

— John Gentry
January 9, 2019
9 Pillars of Continuous Security Best Practices
Without proper consideration given to security best practices, the continuous delivery of software changes facilitated by DevOps is risky. On the other hand, DevOps provides an opportunity to reduce security risks if security is integrated into the continuous delivery pipeline according to best practices.

This blog enumerates best practices for security across nine pillars of DevOps: Leadership, Collaborative Culture, Design for DevOps, Continuous Integration, Continuous Testing, Continuous Monitoring, Elastic Infrastructure, Continuous Delivery/Deployment and Continuous Security. Examples of best practices for each pillar are listed. These practices can be used to assess an organization’s maturity within the journey to Continuous Security, often referred to as DevSecOps.

Leadership

Leaders need to understand and sponsor a clear vision for security.
- Leadership demonstrates a vision for organizational direction, team direction and three-year horizon including security practices.
- Leaders intellectually stimulate the team status quo by encouraging asking new questions and question the basic assumptions about the work including security practices.
- Leaders promote personal recognition by commending teams for better-than-average work, acknowledging improvements in the quality of work and personally compliment individuals’ outstanding work including security practices.
Collaborative Culture

Culture in organizations that work well with DevOps have a collaborative continuous security mindset.
- The culture encourages cross-functional collaboration, shared responsibilities and avoids silos between developers, operations, project management, quality assurance and security.
- The DevOps system (toolchain) is created by an expert team and reviewed by a coalition of stakeholders including security.
- Changes to end-to-end DevOps workflows are led by an expert team and reviewed by a coalition of stakeholders including security.
- DevOps culture empowers and trains team members to take personal responsibility for security, compliance and privacy obligations.
- Security engineers/architects are involved in the design for modular components, and also consulted when security patterns change within modules.
Design for DevOps

Designing software for DevOps at speed requires application designers to master the best practices for continuous security.
- Software source code changes are pre-checked with static analysis tools, prior to commit to the integration branch. This assures that the modified source code does not introduce critical software faults and security vulnerabilities such as memory leaks, uninitialized variables, array-boundary problems and SQL injection.
- Software component analysis scans third-party components for known security vulnerabilities and identifies risk during the build process.
- Security frameworks for technology stacks that are used (such as Apache Shiro or Spring Security) are documented and shared with developers for their respective technology stacks.
- Software code changes peer code reviews include checks for defensive coding and security vulnerabilities prior to committing code to the integration/trunk branch.
- Common security components such as identity, authorization, key management, audit/log, cryptography, protocols, etc. are maintained, published, readily available and used within module development.
Continuous Integration

Continuous integration (CI) within organizations that have multiple teams working concurrently on a project and different code bases is challenging. During the integration stage it is critical to assess the application and understand the impact of code changes from a security point of view.
- A software version management system is used to manage versions of all changes to source code, executable images and tools used to create and test the software.
- Incremental static analysis pre-commit and commit checks are wired into CI to catch common mistakes and anti-patterns quickly by only scanning the code that was changed. These checks identify security vulnerabilities through control flow and data flow analysis, pattern analysis and other techniques. These techniques find security-related issues such as mistakes in using crypto functions, configuration errors and potential injection vulnerabilities.
- Binary artifacts are digitally signed and stored in secure repositories.
- Changes to security patterns used within software source such as session management, authentication, authorization and encryption code trigger a notification or pull request to security engineers.
Continuous Testing

Continuous testing (CT) with DevOps has significant advantages for continuous security when best practices are followed.
- New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
- Security tests for each DevOps pipeline stage are automated and may be selected automatically according predefined criteria.
- Release regression tests include security tests. At least 85 percent of the security regression tests are fully automated and the remaining are auto-assisted if portions must be performed manually.
- Test results that indicate possible security concerns are tagged for security analysis.
- Dynamic or interactive application security tests exercise the application for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
- If containers are used, image repositories are scanned for images with known vulnerabilities, hash checks for image drifts and run-time checks for vulnerabilities during image deploys.
- Attack patterns, abuse cases and tests are built for application module profiles.
- Unit, functional and integration tests around—and especially outside—boundary conditions are run during CT. Tests include error handling, exception handling logic and negative tests.
- Automated security attack testing to include the OWASP Top 10 in integrated into automated testing.
Continuous Monitoring

Continuous monitoring (CM) of security considers the dynamic nature of artifacts and infrastructure and a proliferation of objects and services to secure.
- Metrics and thresholds are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example security metrics include:
  - Number of security defects identified pre-production
  - Percent of code coverage from security testing
  - Number of failed builds due to security checks
  - Mean time to detection
  - Mean time to resolution
- Vulnerability information in consolidated to provide a comprehensive view into vulnerability risks and remediation across tools, pipelines and apps—and over time.
- Metrics and events from production security controls such as WAF, RASP, etc., are used to improve security testing.
- Insight into security threats and events are shared and visible across DevOps teams to enable “attack-driven defense” methodologies.
Continuous Security

Continuous security (CS) is itself a distinct pillar with standalone best practices that cross all of the other pillars.
- All information security platforms that are in use expose full functionality via APIs.
- Immutable infrastructure mindsets are adopted to ensure production systems are locked down.
- Security controls are automated so as not to impede DevOps agility.
- Security tools are integrated into the CI/CD pipeline.
- Source code for key intellectual property on build or test machines are only accessible by trusted users with credentials. Build and test scripts do not contain credentials to any system that has intellectual property.
- External penetration tests (done out of band) are scheduled either periodically or on a regular cadence are used to perform deep-dive analysis.
- Telemetry from production security controls such as WAF and RASP are delivered back to development teams to inform application updates.
- Accurate inventory of all software packages and version information is documented via infrastructure as code. Automated detection is used to identify whether any of the packages have known CVEs associated and define specific remediation actions.
Elastic Infrastructure

Elastic infrastructure environments offer advantages compared to legacy or traditional static infrastructure. However, elastic infrastructures need to follow best practices for continuous security because flexible infrastructures offer a broader range of attack surfaces.
- Configuration code includes automated checks including:
  - Ensuring unnecessary services are disabled and only ports that need to be open are.
  - Permissions on files, audit and logging policies are enforced.
  - Development tools are not installed on production.
- Security-approved OS, software versions and frameworks are used to compose required infrastructure. Security-related controls such as ACLs and FIM are defined as a part of infrastructure where applicable.
- A least-privilege model is enforced for processes running on shared infrastructure.
- Smaller clusters are used to reduce complexity between teams.
- Service provider partners security controls are validated to ensure they meet business requirements in their domains of the shared security model.
- IaaS or PaaS service provider security controls are validated to ensure that they meet business requirements in their domains of the shared security model.
Continuous Delivery/Deployment

While DevOps enables new feature deliveries to users quickly, to minimize risk during deployment the following best practices are important:
- Release-to-production decisions are determined according to predetermined metrics, which include security metrics.
- A whitelist policy for application segmentation is enforced during deployment for each environment, especially production.
- Continuous deployment processes trigger run-time security and compliance checks including:
  - Ensuring unnecessary services are disabled and only ports that need to be open are.
  - Permissions on files, audit and logging policies are enforced.
  - Verify development tools are not installed on production.
- All secrets used for deployment are vaulted and retrieved programmatically during run-time or initialization of the continuous delivery process.
In this article we detailed best practices for nine pillars of continuous security, otherwise known as DevSecOps. While is it clear that DevOps offers immense value for software deployment, the adherence to best practices is essential to reduce risk and assure security. Each organization is different and has different security postures. The specific practices, security tools and metrics that are appropriate may vary according to a risk assessment for each organization.

Aggregation of end-to-end security metrics, from planning through to operations, is important but beyond the scope of this blog. End-to-end security monitoring tools, such as value stream management applied to DevSecOps and enterprise security dashboards, is a topic that we intend to cover in a subsequent blog.

This article was co-authored by Imran Chaudhari. Imran is a principal security consultant at Trace3, where he assists clients in assessing and designing cybersecurity initiatives from a strategic, organizational, and technical perspective. His experience covers expertise in secure software development, application and cloud security, as well as identity and access management and DevSecOps assessments.

— Marc Hornbeek
January 8, 2019
DevOps and Agile: Your New Year’s Resolution

Don’t apply Agile principles to DevOps. That’s not how it works.

We in the west often celebrate the new year with a promise to do more or less of something in the coming year. While these promises are often broken, they can occasionally be transformational.

And it’s time your organization had a transformational one: More DevOps, less Agile++.

It is still a pervasive issue that people look at the successes of Agile development and attempt to apply them to the operations side. That’s not how it works. Really. There is a fundamental difference between DEVops and devOPS that means what is perfect for one is terribly imperfect for the other.

While having a developer who specializes in the language du jour is important for a DevOps project—just like having a systems administrator that is proficient in given platform X—that really is where the similarity ends.

Operations is a daily thing. It takes specialized knowledge and uses it to keep systems purring along on the chosen platform with vendor X, Y and Z. For a lot of organizations, storage has moved to the cloud. For a lot more it has not, and there is a difference there. But in the end, management of that massive blob of storage falls on your storage specialists. If your environment is internal, they’re worried about hardware and software both, but even if you’ve taken a massive leap to the cloud they have access and backup and hierarchy concerns.

And given our worry about virtualization sprawl, a large number of organizations seem to have pitched that out the window when cloud and DevOps came along. Those instances still need monitoring and management, even if some of it is automated. In fact, the growth in numbers pretty much negates the automation goodness.

So take a close look at the devOPS side of your operation. Think about it. Include it in DevOps from the start, rather than applying Agile principles to a group that it just doesn’t fit well. We’ve (mostly) gotten past the “Developer is Kingmaker” mentality, and have started to reintroduce standards so that we don’t have 50 different database products to support. But far too many have yet to truly approach DevOps as an integrated whole that relies on the wisdom of operations—even though operations is going to be involved with this project far longer and more intimately than development.

And they are disparate functions. DevOps makes getting to market and maintaining the application far more streamlined, but when it comes down to it, 90-plus percent of you aren’t keeping a java developer watching an application for life. They’re moving on to the next project, and operations is left with more of development’s input and advice than previously, but all of the responsibility.

So step up, apply DevOps in the manner it makes sense for a given bit. If you are running DevOps on VMware, operations input will be different than if it is on physical servers or public cloud or private cloud. Take that input and develop around it. Personally, I’m a fan of application release automation (ARA) for achieving the melding, but however you do it, don’t say, “We’re DevOps,” and mean, “We’re going to wedge Agile into the entire org,” anymore.

And have a great year! Here’s to more apps, more responsiveness and more security!

— Don Macvittie

January 8, 2019
DevOps and Retail: Transforming Brick-and-Mortar to Brick-and-Click

Brick-and mortar-retail is rapidly transforming with the digital age. As physical storefronts turn digital, it’s safe to say that traditional retail is quickly becoming a mesh of “brick-and-click.”

So, how are retail giants surviving in this new landscape? Interestingly enough, the remaining brick-and-mortar retail are reinventing themselves by applying helpful DevOps principles.

In this post, we’ll peek into cases of innovation and horror stories from historically brick-and-mortar retail chains and brands. From Target, to Balian Group, Sears and other companies, we’ll discover the importance of DevOps in this new brick-and-click multi-channel retail environment.

Target

In an effort to increase agility, Target engineering teams embrace something they call “the Dojo”—immersive, six-week training sessions with hands-on DevOps and Agile coaches. As Target CIO Mike McNamara recently told the Enterpriser’s project: “The Dojo has been fantastic in getting teams engaged with agile and DevOps, removing the natural resistance and fear of change, and then supporting the team through the changes while maintaining productivity. It’s been a huge success for Target.”

The result of these sessions is increased comfort with methods including containerization and tools that enable continuous integration and continuous deployment. McNamara also noted their grassroots approach; they encourage open source activity and listen to their engineers when they petition the use of new tech.

“Our engineers have fully embraced DevOps,” he said.

Bailian Group

For another case study, take Bailian Group, a large Shanghai retailer that operates 6,000 grocery and department stores. By the time Bailian entered the online marketplace, it suffered from a severe lack of agility, non-native web apps and underutilized server assets.

“Our transition from traditional brick and mortar to omnichannel business presented a great opportunity but an equally large challenge,” noted Lu Qichuan, director of IaaS and Cloud Integration Architecture at the Bailian Group.

Qichuan and Bailian realized that to stay relevant, they had to cut long development cycles and embrace “new cloud application development tools and processes that foster a CI/CD and DevOps culture and increase innovation and time-to-market,” he said.

They ended up implementing a private cloud using OpenStack. Now, midway through their cloud transition, they estimate to have 10 million virtual machines by the end of 2020.

Sears

While the previous two examples had positive outcomes, digital transformation has not gone over well for other retailers. Sears declared bankruptcy in October 2018.

In its final years, Sears was in a sticky situation. To the chagrin of former CEO Eddie Lambert, surveys found their customers preferred to shop in stores as opposed to online. Disregarding customer desires, however, Sears attempted a big data play with its “Shop Your Way” program, under which Sears gathered user data to then sell. The program floundered, resulting in consumer irritation and loss of sales as reported by Business Insider.

Writing for InformationWeek, Emily Johnson reminded us that evaluating DevOps using the wrong metrics will ultimately lead to failure, citing customer satisfaction as the No. 1 goal.

In its final years, an abrasive Sears leadership halted any sense of DevOps to flourish. A disengaged Sears CEO Lambert created “a climate of fear,” according to Busines Insider reports. It appears the retailers that stay afloat encourage agility and intrapreneurship—this means encouraging innovation, not fear.

DevOps Aids the New Omnichannel Retail Benchmark

Brick-and-mortar are evolving into omnichannel retailers that bridge physical and virtual spaces. Take IKEA’s augmented reality app, which visualizes items in a house. From Disney, to Startbucks, Nordstroms and elsewhere, physical storefronts embrace digital offshoots of their core model to augment the traditional retail experience.

Surprisingly, physical storefronts aren’t completely dead. They still offer conveniences for urban environments with high walking scores. See the emergence of Amazon physical locations for ironic evidence of their benefit. Other storefronts and pop-up shops have taken on a museum-like quality. The flagship Funko store, for example, creates impressive displays and interactive exhibits, pulling in flocks of weekend shoppers that want more than a dull e-commerce shopping cart.

DevOpsGroup similarly describes a “reverse showrooming” movement; consumers research products online to then purchase in person. This enables brands such as ASOS, which is emphasizing social media and free worldwide deliveries, to stay competitive.

“Retailers have been forced to re-evaluate their business strategies and understand how new technologies will affect their offering,” according to the DevOpsGroup website.

DevOps Turns Bricks to Clicks

Online shopping has sufficiently disrupted retail, and brick-and-mortar must reinvent themselves to stay relevant. An open-minded leadership and a commitment to fast development, dynamic scaling, uncompromised availability and open source technology underpin a successful evolution.

Of course, this increased reliance on technology necessitates a hardened look at how development and operations communicate, and brick-and-mortar retailers must adopt the behaviors of tech companies to retain agility.

This may come in many forms: new user-facing digital experiences, microservices for composability in niche omnichannel environments, containerization to package code that scales well, distributed data stores, heightened user data security or smart incident management techniques, to name a few.

Traditional retailers, then, should dissuade their doubts against agile. DevOps can aid retail by reducing redundancy, cutting costs and streamlining production and distribution.

— Bill Doerrfeld

January 8, 2019
Don’t Let Your DevOps Transformation Be a Parade of Tools
Choosing the right DevOps tools should be a careful, thoughtful process.

On May 12, 1848, a store owner named Sam Brannan held a “one-man parade” to announce the start of the San Francisco Gold Rush. “Gold! Gold from the American River!” Brannan shouted up and down the market street in San Francisco. He held his hat in one hand and waved a bottle of gold dust in the other. San Franciscans had received false news of gold before. But by all accounts, Brannan’s performance sent residents running in search of riches.

Brannan had a good reason for spreading the news rather than panning for gold himself. The canny entrepreneur owned a general store that served the workers, and in the week between learning about the discovery and yelling about it in San Francisco, he’d bought all the picks and shovels in the city. Brannan’s announcement helped spur a seminal event in California’s history. As Brannan raked in money selling mining supplies, his actions also, years later, led to the coining of a famous maxim. (source)

During a Gold Rush, Sell Shovels

In the current age of DevOps and Agile rush of enterprises, there is a lesson to be learned from the gold rush: Don’t let your DevOps transformation become a parade of tools. While automation is critical for the success of modern ways of technology delivery and operations, the wide range of tooling choices can often lead to wrong decisions—especially well-intentioned, top down purchase of tools, duplication in toolsets and, most importantly, ignoring the voice of the teams on the ground.

With such variety of tools available in the market today, we all have opinions regarding the tools that we would like to use. Our preferences are based on either our history of using certain tools, because we have invested time and effort to learn a tool, or partnerships your service vendors might have with tooling vendors.

This leads to two general attitudes toward tooling adoption, with each having their own advantages and disadvantages:

Autonomy for teams: Let teams decide the tools they feel are suitable for their context.

The advantage is that teams can experiment and pick a tool that suits their context better. The disadvantages can, potentially, include security risks due to lack of patching and maintenance, and limited cross-team collaboration and sharing of tooling skills and know-how.

Standardized tool suite: A centralized way of dictating tooling choices.

The advantage is that it allows for capacity planning, integration between tools, larger pool of community for collaboration and create platforms-based solutions.The disadvantage is that this method at times prevents teams from using tools that best suit their unique needs and creation of avoidable dependencies on centralized tooling teams in the organization.

These contrasting views lead to debates at every level for the choice of tools and, in some cases, slowing down decision-making—or even leading to the purchase of tooling licenses without input from technical teams on the ground. The decision-making process must be improved, and the need for autonomy must be balanced with standardization and optimization. Also, the two approaches of “autonomy for teams” and “standardized tool suite” do not always have to be binary and will change depending on the context of the team needs and area.

By considering desirability, viability and feasibility of the DevOps tools, we can help achieve the balance between autonomy and alignment.

Here are some tips to help ask the right questions:

Desirability
- Do teams want this? or is this a solution chasing a problem?
- Do you anticipate that teams will need this in the near future?
- Does this help improve team productivity and effectiveness?
Viability
- Does this make commercial sense?
- Have we considered maintenance costs and total cost of ownership?
- Do we have existing license arrangements with the tooling vendors?
- Do we have comparable tools already in use and that can do the job?
- Do we need to purchase licenses throughout the enterprise?
- Are there any open source alternatives?
Feasibility
- Can we implement this in the enterprise IT landscape?
- Have we considered the ease of setup and configuration of the tool?
- Does it scale across the enterprise? Does it have to scale across the enterprise?
— Aditya Vadaganadam
January 7, 2019