devopsy.kinsta.cloud

Tag: data management

  • Becoming a Data Company: Back to the Future

    Becoming a Data Company: Back to the Future

    Transformations are complex, and it’s often difficult to see the big picture when you’re in the middle of one. 

    The elegance of the end-state of harmonious synergy between software and data belie the complexity and chaos of the transition we find ourselves in the midst of. Not that long ago, there was a general balance between the pace and sophistication of software delivery and the speed and maturity of data analysis. Markets and customer bases grew steadily allowing organizations to keep pace in their software and data competencies to handle increasing mutual scope and complexity.  

    However, the consumerization of IT—the ability of lay consumers to consume sophisticated software experiences from browsers and mobile devices—unleashed access to a massive, eager and demanding customer base measured in the billions. This gave birth to the current wave of digital transformation. Along with it came the race to get to market, i.e. build and deliver high-quality software exponentially faster, in order to capitalize on massive global opportunity.

    Innovation thus started in the software domain. With faster, better software came the need for purpose-built data stores (document databases, graph databases, etc.), the need to manage the velocity, volume and variety of data collected from millions (or billions) of users and the need to extract insights and value fast enough to further fuel the pace of software innovation. Thus, software innovation planted the seeds for the current focus on data innovation.

    While the innovators quickly traversed this software journey, most enterprises are only now fully in the midst of embracing new software methodologies and technologies to deliver software quickly and effectively. Those enterprises at the vanguard of the software journey are now embarking on the data journey—to find ways to safely and quickly harness the value locked away in enterprise data assets. This includes everything from analytics to rapid provisioning of copies of production data for software testing. In fact, 451 Research states 86% of companies that consider themselves mature in DevOps journey are planning on increasing their investment in DataOps. That is, companies that have traversed the software competency curve are now embarking on the data competency curve.

    The Dawn of the CDO

    The growing traction by data science, AI, analytics and DataOps technologies and vendors point to the rising importance of developing a data competency. The recent emergence of the Chief Data Officer (CDO) is the culmination of this trend. 

    Given the nascent focus on data at most enterprises, most CDOs are still grappling with defining their role, navigating the dynamics of the C-Suite, defending against regulatory and compliance risk and justifying their role by identifying growth opportunities. Despite these immediate challenges, they do have one thing going for them: they know they’re in the midst of mind-bending change, have the benefit of hindsight on the software journey and the many parallels it carries, and are mentally prepared for the organizational, technological and cultural changes that will need to be navigated. Most companies were stunned by the sheer suddenness and velocity of change that came with the DevOps and the software transformation journey. With DataOps and the data journey, they are now aware, prepared and committed to the associated journey and changes for the long haul. They can be proactive and avoid being reactive.

    Planning for the DataOps Era

    As cycle times for going to market and learning from the markets shorten, and innovation cycles accelerate, CDOs can expect that data-centric organizations will need to drive net-new top and bottom-line results without incurring undue risk. This will come through adoption of new technologies, organizational structures, cultures and workflows. New roles focused on data-centric activities, and new forms of collaboration between different personas will emerge. 

    In the modern enterprise, production will no longer be the most important data environment in enterprises. There is a growing need for fast access to secure, governed data outside of production—where the work of the rest (non-IT part) of the enterprise is centered. What used to be non-prod for IT will now be prod for the rest of the enterprise. To succeed, IT will need to deliver data quickly and securely to the rest of the organization or even across organizational boundaries to partners and other ecosystem players. Embracing collaborative processes and workflows that support this new pattern of enterprise operation will be critical. Enter DataOps—the unique mix of people, process and technology approaches required across teams in an organization to become agile with data.

    Given the similarities outlined earlier of the data journey to the software journey, organizations’ DataOps journey can benefit from the lessons learned in their DevOps journey. For instance,  developing a data competency will require new layers of organization and workflow patterns supported by corresponding technologies. Proactively planning and laying down infrastructure for DataOps will yield far better results rather than adopting technologies ad hoc until a data competency blueprint emerges through trial and error. 

    Building a Data Company Technology Stack

    As with any complex shift, it makes sense to deconstruct data competency into a series of layered capabilities, each building on the one that came before, that together help develop a sophisticated competency. For example, when building their software competency, in order to apply software in sophisticated ways, (e.g., cloud native architectures and microservices), organizations needed to manage distributed software elements (e.g., scale, heal, monitor) which in turn relied on the ability to test and deploy incrementally, consistently and rapidly. 

    A similar approach can be applied to data. Building collaborative processes around appropriate technological tooling can yield layered component capabilities that collectively define a sophisticated data competency. While the layering was emergent in the software journey, CDOs have the opportunity to be deliberate rather than reactive in developing a layered approach as the diagram below illustrates. 

    At the foundation of a successful data competency lies the capability to deliver data anywhere in the organization and secure it to meet regulatory and compliance mandates. Being able to quickly secure and move any type of enterprise data across a hybrid cloud context is hard work, but also table stakes for basic data management hygiene. These are critical prerequisites and this foundation needs to be solid in order to support the layers above. 

    Next, there are various data management activities to master, such as the self-service access, governance, data catalogs, data cleansing, collaboration and versioning, etc. These define the next level of data competency and must be mastered before data can be applied in different ways, e.g. in machine learning, test data management, analytics, data science, etc. 

    A thoughtful, layered approach to building a data company requires building the right abstractions between layers and will avoid tool sprawl consisting of a mesh of ad hoc point solutions that inevitably require human glue to drive data workflows.  

    CDOs should evaluate their organization’s needs through such a structured lens to evaluate the organization’s needs. Each of the activities will bring with them ecosystems of vendors, engendering further evaluation of vendor capabilities. Critical questions to ask include (among many): Is the solution enterprise ready? Does it work with all your data sources across hybrid cloud? Will it evolve with your changing needs? How will it interoperate with other solutions? Can it be automated?

    As CDOs think through these infrastructure investments to connect different parts of the enterprise, the similarity to laying down railroad tracks to connect a large country are unmistakable and perhaps instructive: the resilience and stability of lower layers enable flexibility, variety and innovation at the higher levels just like the simplicity and reliability of simple railroad tracks enables running electric, steam engines, luxury bogies and cargo bogies alike. 

    Toward a Data Company

    The refrain of digital transformation has been that every company is a software company. But to complete the transformation, every company must also become a data company. Software development practices and tools have evolved significantly in the last decade. We’re in the early stages of a similar evolution in data practices and tools that will take us through the next decade. The symmetry and coupling between software and data is unmistakable and there is much to be learned from the software journey.

    As CDOs, CIOs and CEOs plan the transformation of their enterprises in this new era, they will need to consider building a foundation for their data competency that is stable, resilient, compatible with their software development processes and adaptable to inevitable change. They should be prepared to deliver their data from where it is produced to where it is used throughout the enterprise.

    The data company is an organization where data is treated not just as a byproduct of operations, but as the lifeblood of the organization—that drives software development, improves quality assurance, informs decision making and potentially generates revenue in its own right.

    — Karun Bakshi

  • What Are AWS Application Patterns?

    What Are AWS Application Patterns?

    The cloud-native approach is not just about compute power and cloud services. A “set it and forget it” attitude won’t work with cloud native; there are far too many moving pieces and there is just way too much complexity. Instead, this approach requires constant testing, optimization and improvement. This is epitomized in the two key phrases of the DevOps methodology: continuous integration and continuous delivery.

    To make this a reality, however, it takes a strategic approach to defining application patterns that will govern the way in which cloud-native applications are built and shipped, rather than a one-off effort. Since AWS is the leading cloud provider, we should discuss which application patterns are relevant to the AWS platform. There are many aspects to consider when it comes to AWS application patterns, and we will cover them here.

    Operational Excellence

    The best starting point for application patterns is a commitment to operational excellence. Typically, the term “Ops” is used to identify the IT team, but in this case, operational excellence rests on the shoulders of Dev, QA and Ops teams. In cross-functional DevOps teams, every member should strive for operational excellence at every step–even in the smallest of tasks.

    In their white paper on operational excellence, AWS encourages its users to manage operations as code. This helps avoid human error, enables automation of complex processes and allows for quick and easy rollback when errors occur. Failure, according to AWS, is an opportunity for learning. They even advise performing “pre-mortems” that try to anticipate potential points of failure before a deployment.

    AWS Trusted Advisor is a useful tool that enables this type of accountability and shared responsibility within a team. It checks and reviews your new and existing AWS workloads looking for opportunities to save costs, improve performance, tighten security or follow best practices for scaling operations.

    Security

    Security is a pattern that is non-negotiable. The term “DevOps” has inspired another variation called DevSecOps that signifies this fact. Security operations take center stage when talking about DevOps.

    The interesting thing about DevSecOps is it actually results in more freedom, flexibility and trust across teams when done right. This is unlike traditional approaches to security which created silos, stifled creativity and slowed down innovation. Security is now a key piece of the puzzle, not an afterthought. AWS has many machine learning-based security solutions such as Macie which automatically identifies and secures personally identifiable information (PII).

    Along with this, configuring role-based access controls using AWS IAM is essential to have greater control over who or which applications access data and for how long. Access Control Lists govern access to data stored in AWS S3 buckets.

    The news is full of instances of data breaches due to weak security policies and access controls. Capital One is a recent example of a data breach that affected 100 million of its users. While security in the cloud is more complex than before, AWS has the necessary tools and systems in place. It’s up to organizations to implement a security strategy that prevents misuse of their data.

    Reliability

    Today, there is a low tolerance for downtime because of the potential loss of revenue. With the stakes so high, reliability is a pattern any application custodian would have high on their list of priorities. AWS has multiple reporting and health monitoring tools to keep you constantly on the pulse of your applications.

    Architectural patterns like microservices enable better resilience by decomposing monolithic applications into more manageable chunks called services. This “divide and conquer” approach leads to higher availability, which is critical for SaaS applications.

    Patterns like chaos engineering seek to intentionally kill random instances and services and force the DevOps team to figure out workarounds and backups for every scenario.

    Application Performance

    With the explosion of data, applications are always dealing with lots of data to transfer, store and process. An application’s performance depends on three factors: the backend data storage layer, the middleware components and the frontend.

    The backend data layer is the most critical component when it comes to performance. Choosing the right storage formats, database types, storage devices and organization patterns all contribute to the speed of data access. AWS has an amazing array of storage services, including S3, RedShift, Aurora and Glacier. AWS gives its users every type of storage solution, all of which should be considered to improve data management.

    AWS is devising new ways to query large datasets without bearing the load on your existing systems. RedShift Spectrum is one example of this. It is a great way to “minimize the need to scale Redshift with a new node, which can be expensive.” If you’d still rather not have to manage any RedShift clusters on your own AWS Athena is a great serverless alternative

    That gives you all the power you need to query large datasets in S3 without having to maintain any clusters of your own.

    This pay-per-use model is making its ways across the entire AWS stack with services such as ECS and Fargate offering completely serverless experiences when operating containers at scale. What started out with AWS Lambda is catching on fast, and many AWS services like the ones mentioned above enable serverless experiences with a focus on application performance.

    Cost Optimization

    When moving to the cloud, pricing is completely different. Rather than paying for perpetual licenses, you now pay a small fee for every small resource that you use. Though small on its own, you’ll soon realize that cloud costs can shoot up easily if you’re not paying close attention to your spending.

    Cost optimization is necessary as you scale operations in the cloud. Cost optimization includes looking for more efficient services, whether that’s within the same vendor platform or a different platform. At a minimum it may require a lift-and-shift approach to transition from expensive old infrastructure to inexpensive new alternatives. For example, shifting workloads from VMs to containers to serverless within the same platform can show great cost savings, since containers and serverless are generally more cost-effective than VMs.

    It pays to leverage the expertise of experts in this area. They should ideally be people who have spent years using AWS, understand Well-Architected best practices and have found ways to cut costs and derive more value from the same budget.

    Application Patterns Enable Extensibility

    Microservice applications adopt a pluggable architecture. This enables them to be easily integrated with external applications. Typically, API-based communication patterns are most common. AWS API Gateway is a service that manages API design for AWS applications. In some cases, an SDK is the better choice–for example, when you need more control over the developer experience and integrations. It’s important to know when to choose either option.

    Application Patterns Enable Portability

    Portability is the ability to move workloads across instances without interrupting their functioning. As instances change, the workloads which they support shouldn’t be affected. This is only possible if there is a clear separation layer between the workloads and the infrastructure that runs them.

    Application Patterns Enable Shift-Left

    The DevOps methodology is known to encourage a shift-left approach to software delivery. This means that not only Dev, but also QA and Ops contribute to the design and development stage of an application. This fosters deeper collaboration across teams and breaks down silos.

    A shift-right approach holds true as well. Shift-right means developers are responsible for the code they write. They are equal owners with operations teams for the resilience of the apps they build and also the bugs they introduce into the system. This practice of putting ownership and responsibility where it belongs is very healthy for a DevOps team.

    AWS facilitates pipeline management using tools such as CodePipeline, CodeCommit and CodeDeploy. They each perform similar but complementary roles and help build a CI/CD pipeline. They make it easy to get started with CI/CD within the AWS platform.

    For organizations further down the line in their cloud journey, AWS’ pipeline management tools may seem too simplistic for something like GitOps. Instead, they’d want to look outside AWS for tools such as Jenkins X or Weave Flux which are designed with GitOps automation in mind. Whether it’s the simple or advanced route, a pipeline brings great benefits over a manual development process.

    Conclusion

    Though tools enable the implementation of helpful application patterns, on their own, they are ineffective. Application patterns need a deliberate strategy and an approach for a start. Only after a strategy is in place, can you move to deciding which tools to use. With a strong strategy in place, any of the tools mentioned here can be swapped out for other better tools and the results will be the same. It’s important to not be distracted by the next shiny tool but keep a keen focus on the strategy for implementing AWS application patterns.

    As we’ve seen here, application patterns are diverse, but they all build upon each other. Without even one of these pillars, organizations would not be able to realize the goal of cloud-native applications. On the other hand, if they give careful thought to each of these application patterns, organizations would deliver cutting-edge applications that can only get better with the passing of time.

    This sponsored article was written on behalf of Eplexity.

    — Twain Taylor

  • Rethinking Data Management in a DevOps Environment

    Rethinking Data Management in a DevOps Environment

    You often hear data is the new oil. This valuable, ever-changing commodity has begun to play a starring role in many cloud-native applications. Yet, according to a number of DevOps teams, data issues continue to plague their efforts to continuously integrate, test and deploy frequent software releases. More specifically, issues with persistent data (and its underlying database engine) often appear to be the culprit.

    Your organization might be pursuing emerging IoT applications that incorporate and analyze sensor data from multiple sources. Or, your DevOps team may be trying to develop applications that extract further actionable insight about customers. Whatever the use case, there’s no doubt back-end database architectures have gained growing importance to the success of such projects. Yet, in many cases, such database systems appear to be having difficulty just keeping up with the pace of today’s DevOps pipelines.

    According to one report on the state of database deployments in 2019, 46% of DevOps teams on an accelerated release schedule (with weekly, or even daily, releases) found it “extremely or very difficult” to speed their database release cycles accordingly. In a related Redgate report on the 2019 State of Database DevOps, 20% of respondents cited slow development and release cycles as one of the biggest drawbacks to “traditional siloed database development practices.” Another 23% saw a higher risk of deployment failure and extra downtime when database changes were introduced in a traditional database environment.

    Getting to the Heart of the Data Problem

    Are such database issues the result of the wrong choice of underlying database technology–such as the use of an RDBMS, NoSQL or even a NewSQL system? Possibly.

    Maybe database issues are also caused by poor (or non-existent) cross-communication between database experts and their developer counterparts? Undoubtedly, this is true as well.

    Would you be surprised to learn both situations (wrong technology and poor communication) are often to blame? Also, to blame is the urge to solve new data problems in the same old way organizations approached their earlier, legacy use cases.

    Ultimately, as with many things DevOps-related, the answer to the data problem is likely to require change along three, separate fronts: People, process and technology. It will also require a fundamental shift in how such issues are approached at the start.

    Starting with People and Process

    Instead of worrying about database slowdowns, what if you began again with how DevOps teams first approach the many facets of the underlying data layer? This means:

    1. From the start of a project, include DBAs as part of the cross-functional DevOps team. This will help promote healthy cross-communication. They will also positively influence the development of appropriate underlying data layers and infrastructure that support your emerging use cases. In larger organizations that manage many data sets, these may be specialized database administrators (DBAs). In smaller companies, these may be more full-stack engineers who also have high-quality expertise in database operations.
    2. Identify early the different data types, domains, boundaries and optimal cloud-native patterns associated with your data. Once these are established, the team can gain a better understanding about the different ways to develop applications and data architectures for each new use case.

    In effect, organizations need to make a concerted effort to shift left with their overall data architecture discussions. This shift allows more of the right questions about data to be asked, answered and incorporated early on in the design of the overall application.

    Asking the Right Questions About Data

    Part of this early collaborative work may involve asking key questions about data. These include the best ways for an application and its users to interact with the underlying data, such as:

    • How will data be ingested and from where?
    • How will we access the data?
    • How and where will we store it?
    • How do we update and/or maintain legacy databases as we build for new use cases?
    • What types of data will we have? How will the types change or be increased over time?
    • How will we query from it? And what are the most common queries users will want to run?
    • How will we manage and scale the data in use?
    • How will we protect the data from data loss, disaster or corruption?
    • How will we secure the data?

    If these questions sound a lot like features in a data lifecycle, you’re right. But, such basic questions about data are often overlooked in the rush to deliver, integrate, test and deploy application code. So often, what happens instead is an application team creates a design, then hands it off saying, “Okay, now create a data architecture to support the application.”

    This fundamental miss with data and collaboration at the start often leads to poor database architecture decisions later. It can also cause DevOps teams to resort to workarounds to accommodate such wrong architecture choices made at the start–which may cause some development teams to choose to circumvent DBA involvement altogether. In this case, they might even find themselves opting to use public cloud services instead (or some other types of data management) to support their application.

    Instead, application teams should choose the path of early collaboration between developers and all functions of IT (including DBAs). This approach can offer better DevOps outcomes. It also helps move organizations toward a future where data architects provide an effective bridge between all parties critical to the application’s success: From the needs of developers to the platforms provided by DBAs and the underlying infrastructures needed to enable them.

    Changing Skillsets, Changing Mindsets

    As with most things DevOps-related, success at the data layer has as much to do with changing the education, mindset and culture toward database architectures and database deployment methods as it does with the specific steps in the DevOps pipeline.

    For example, if most DBAs in your organization are narrowly focused in one area (such as an Oracle DBA or SQL Server admins), your organization may benefit from investing in education to close skill gaps. Such investment can help expand the knowledge and exposure of that workforce to DevOps practices or emerging database approaches to cloud-native application patterns.

    It can also help some organizations to first identify those individuals in the organization who want to transform, innovate and learn about new tools, methods and practices.

    Hiring those with that expertise already in place is another option. When you hire new skillsets, however, the answer may not be just to hire several NoSQL administrators. This tactic will not necessarily solve your data problem, either. It may be just as beneficial (or more so) to hire someone with little specific NoSQL skills but who is good at thinking about data in a different way.

    Business First, Technology Second

    In our practice, we are often asked to weigh in on technology choices to support DevOps efforts in emerging areas such as IoT, Big Data and advanced analytics. Organizations ask us about using traditional relational databases (SQL Server, Oracle, etc.) versus NoSQL databases (MongoDB, Cassandra, etc.). They ask about the merits of one NoSQL iteration over another. They ask about how to manage persistent data in container environments. They even ask about workarounds when the data choices they’ve made cause other unexpected problems.

    We do our best to answer these questions. But, whenever possible, we also tell organizations to back up a few steps and start, instead, from their business objectives:

    • What are you trying to accomplish?
    • What are your business drivers?
    • How do you hope data will be used in these contexts?

    Answers to these questions can help you make better choices about the best data architecture to support your growing applications.

    Ultimately, new tools and technologies can enable a lot of impressive data architectures. But, just throwing tools at the data problem is not enough. Early effort and investment in people, process and cross-departmental communication is just as important to a successful data outcome for any project.

    Many of the discussions we have now are trying to take everyone back to the steps they skipped. This harks back to the fundamentals of software engineering: Addressing things early so you don’t have problems later.

    Unfortunately, data is the one component in the technology stack that you can’t easily undo once you’ve chosen a specific path with your application’s data architecture. When it comes to planning for data, it really is a pay now or pay later equation.

    Isn’t it best to take the time and choose your data path wisely rather than paying later for slowdowns or other issues with deployment and performance?

    — Jeff Bozic

  • How the World Measures DevOps Quality

    How the World Measures DevOps Quality

    With continuous everything, knowing whether each new release will ultimately enhance or undermine the overall user experience is essential. Yet, most of today’s go/no-go decision still hinge upon quality metrics designed for a different era.

    Every other aspect of application delivery has been scrutinized and optimized for DevOps. Why not re-examine quality metrics as well?

    Are classic metrics like number of automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure the steady stream of updates are truly fit for production?

    To provide the DevOps community an objective perspective on what quality metrics are most important for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in the 55-page report, Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts. The report takes a deep dive into the global findings, complete with heat maps, quadrant mappings and some fun lists such as “Most Overrated,” “Hidden Gems” and “Top DevOps Differentiators.”

    One of the most common questions we received after publishing the report was: “How do the results vary across regions?” In response, we performed some additional regional analysis—and I’d like to share those results here.

    To start, let’s take a look at the global top 20. The following metrics were ranked as the most valuable by the DevOps experts who measure them (across all regions).

    Europe DevOps Quality Metrics Trends

    Looking specifically at Europe, the top 20 changes as follows:

    Interesting trends in this region:

    • There is a greater commitment to measuring quality metrics. European respondents reported a higher level of DevOps quality metrics measurement across the board. For almost all metrics, the usage rate was at least 6% higher than the global average. For metrics related to time, coverage, risk, effectiveness and efficiency, the usage rate was over 14% higher. This speaks to European organizations’ commitment to scrutinizing and continuously optimizing their quality processes—especially in terms of time and resource utilization.
    • Risk and coverage metrics are valued more than in the global average. European respondents also ranked risk and coverage quality metrics a surprising 21% higher than the global average. This could be related to the fact that the respondents from this region came primarily from the financial services and insurance sector, with healthcare and government close behind. In such highly-regulated industries, measuring and mitigating risk is certainly a core concern. This finding could also indicate European organizations place a greater emphasis on protecting the corporate brand.
    • Test data preparation time seems to be a greater concern. European respondents were more likely to measure (+18%) and highly-value (+23) time spent preparing test data than their global peers. Given the restrictions GDPR placed on test data as of May 2018, it seems likely that European organizations have significantly changed their test data management processes (e.g., to masking and more synthetic test data generation), and are cautiously monitoring how the changes are impacting their overall efficiency.

    Asia Pacific DevOps Quality Metrics Trends

    Now, let’s shift focus to Asia Pacific. The following 20 metrics were ranked as the most valuable by the Asia Pacific DevOps experts who measure them:

    Notable trends in this region:

    • End-to-end testing metrics are valued—and measured—more than in the global average. Although Asia Pacific respondents measured fewer build and functional validation metrics than the global average, they measured (and valued) end-to-end testing metrics much more than their peers around the world. For example, percent of automated end-to-end tests was measured by 47% of the organizations (versus 36% globally) and highly-valued by 84% (versus 70% globally). Risk coverage measurement was significantly higher; it was measured by 49% (versus 34%) and highly-valued by 71% (versus 59%). This speaks to the region’s focus on digital transformation and commitment to delivering exceptional user experiences.
    • API testing metrics were also valued—and measured—more than in the global average. Asia Pacific respondents also measured and valued API testing quality metrics more than the global average. Overall, API quality metrics were measured by 16% more organizations in this region than globally. The highest valued API quality metrics were API test coverage (63% versus 39% globally) and API risk coverage (79% versus 62% globally). This prioritization of API testing is likely a side effect of the regional trend towards API-driven open banking (the majority of respondents indicated they were in the financial services and insurance sector).
    • There is a greater leader/laggard quality metrics measurement gap. Part of the study involved classifying the respondents as either DevOps leaders or DevOps laggards, based on their responses to various questions about the maturity of their processes. Although the percentage of DevOps leaders in the region was lower than the global average (18% versus 26%), the DevOps leaders from Asia Pacific generally measured quality metrics at a comparable rate to their global peers. However, the DevOps laggards in Asia Pacific generally measured quality metrics at a much lower rate than their global peers. This suggests the select set of firms that have truly prioritized DevOps initiatives have made great strides—and the laggards have a lot of catching up to do in order to remain competitive.

    Research Methodology

    Here’s a look at the process behind the research discussed above:

    1. Survey 603 global enterprise leaders responsible for their firms’ DevOps strategies.
    2. From that sample, identify the firms with mature and successful DevOps adoptions (157 met Forrester’s criteria for this distinction).
    3. Learn what quality metrics those experts actually measure, and how valuable they rate each metric that they regularly measure.
    4. Use those findings to rate and rank each metric’s usage (how often experts use the metric) and value (how highly experts value the metric).
    5. Compare the DevOps experts’ quality metric usage versus that of DevOps laggards. If there was a significant discrepancy, the metric is considered a DevOps differentiator.

    — Wayne Ariola

  • DevOps Chat: Data Protection in the Cloud With Druva

    DevOps Chat: Data Protection in the Cloud With Druva

    As any CIO knows, a data management architecture is essential to caring for one of the organization’s most valuable assets: data. Data management sounds simple but it’s not. Backup and recovery, disaster recovery, data retention, governance and e-discovery are all parts of any effective data protection strategy.

    As businesses move and grow their presence in the cloud, the same capabilities apply in this very dynamic and elastic environment.

    Recently, Druva received another investment round of $130 million led by Viking Global. Mike Palmer, CPO at Druva, joins us on this episode of DevOps Chat where we talk about Druva’s planned use of those funds and Druva’s approach to providing a SaaS-based data protection platform.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Mike Palmer, chief product officer at Druva. Our topic today is how data protection is in your future as you move to the cloud. Mike, welcome to DevOps Chat.

    Mike Palmer: Thank you, Mitch. I have been looking forward to this conversation with you.

    Ashley: Me, too. Glad we’re together online and talking about data protection. Would you first start, though, just let’s introduce yourself, tell us a little bit about what you do as chief product officer and tell us a little bit about Druva.

    Palmer: Well, as you say, running product is my primary job as chief product officer, as you would imagine. I run engineering and pricing and strategy for the company. And Druva is an amazing organization, one that I came to around about a year ago, coming from various positions in the industry, watching the evolution of both storage as well as data protection software and how it’s been evolving in a world of public cloud adoption.

    Druva is the first and only SaaS player in the data protection space, which made it super intriguing for me because, as we look at enterprises and how they adopt SaaS, whether that’s been from CRM to ERP to collaboration platforms, what stood out as the next big wave of SaaS adoption was, in fact, data protection. So, we’re focused on that space and growing rapidly.

    Ashley: You guys are doing real well. As a matter of fact, I know a little bit of old news here, about 30 days, but you had a $130 million investment. I think Viking Global led that for you—congratulations, by the way. That’s awesome.

    Palmer: Thank you very much. I don’t think $130 million ever feels old.

    Ashley: [Laughter]

    Palmer: But it was a great validation of the strategy and Viking is a tremendous investor. Folks in the industry know kinda their origins dating back a couple of decades and we were glad to have them aboard. They were a great partner in talking through the evolution both of our business as well as the industry over the next, what we consider to be, kind of 5 to 10 years and they saw what we saw, you know, which was kind of a relentless pace of adoption in cloud and cloud storage, the desire for enterprises to find ways to simplify their current environments and move technical skills from some of the legacy areas like data protection into some of the more advanced ones or next generation ones like machine learning and analytics, and we stood at the intersection of these trends, and they got on board with us and have been a great partner.

    Ashley: Clearly see a good future for you. Let’s talk about—so, this 130 million and the other investments that you’ve had, they didn’t give that to you to put it in the bank, they want you to put it to work. So, what does the future of Druva look like as you start to invest those kinds of funds?

    Palmer: Yeah, it’s a great discussion, because we are competing, frankly, on a few different _______ looking forward. You know, number one and the basic one is, how do we drive the best possible TCO for enterprises who have increasing amounts of data to manage, increasing amounts of risk and liability related to the regulatory environment surrounding that data. How do we expand the applications and the workload coverage for them as they support current apps, but also the ones that they’re adopting in the native cloud?

    And then, as we look a little bit forward, as I was told by an investor, you know, this is either a good investment or a great investment and it’s gonna hinge around how we can expand the data protection definition into one in which customers get more value from that copied and stored data. If we can find ways to help them use that data and extend it into the ecosystem, get better visibility into it, it will effectively not only change the architecture, but we will have changed the entire value proposition, and that’s what we’re shooting for.

    Ashley: Now, you’re operating in AWS in an elastic environment, you know, storage is right there, you want some more, you just grab it, use it. I would imagine that’s very easy to get out of control proliferation. You need some data management capabilities around what you have as well as the whole process of backing up data, restoring, recovering, making sure you’re not losing information—all of that.

    What are some of the key things that are different about operating in a cloud environment versus maybe a traditional on premise data center?

    Palmer: Great question, and let me start by giving us some background statistics to frame up how big this challenge is and the opportunity is. You know, the thing that we keep most in mind is, if you look at Amazon’s five year Kager of cloud storage pricing, it’s negative 60%.

    Ashley: Hmm. Going down.

    Palmer: In a world like that—it’s going down and if you talk to Amazon, they will say that that is the path that they will continue on over the next five years, enough to challenge customers with, “Tell me, when you buy an appliance today, how much cheaper is it gonna be next year or the year after or the after?” It’s likely it’s gonna be the same, so they’ll lose out on the innovation in the industry, even just around the ability to create better price and performance. ESC did a very nice survey and asked CIOs in their premises data centers what were the top five opportunities for cost efficiency? And number one on that list was storage management.

    Ashley: Mm-hmm.

    Palmer: You know, enterprises are dealing with this long history, and you had talked earlier about your history as a CIO. All CIOs deal with generations of proliferation of storage vendors as well as the hardware itself, the incompatibility, and just even the ability to track it, tough challenges, and the third stat, which is the one that I find most interesting because it indicates the tipping point is, in 2021, so just about a year and a half from now, we’ll see the crossover point where there are more dollars spent in cloud storage than premises storage. So, this will be the point at which companies can effectively be viewed as cloud first from a storage point of view to the point where they’ll have to defend [Audio skip]. And in that world, your first question about what do we have to help them with—we have to help them realize the promise of those savings.

    You made a good point—it can easily get out of control, especially where you have very distributed teams who are building applications and taking advantage of that infrastructure in a more decentralized way. We have to help them manage the tiering, we have to help them manage the data classification, we have to help them manage not just the cost side, but give them advice on performance. Much of this can be automated through machine learning. So, we’re gonna give them a new model for what old terminology like ILM or information life cycle management or HSM or hierarchical storage management, we’re gonna turn that, really, into a background automation process and let them focus on what matters, which is, do I have the right policy, can I leverage this data for other uses, and can I do that without special skills?

    Ashley: Well, you know, you bring up some really good points and that is, it’s one thing for a startup or someone who’s building a Cloud Native application as a young company or maybe a one-off kind of application. Enterprises look for environments that are gonna de-risk or things that are gonna de-risk their move to the cloud. And I would imagine having a structure that it’s at least familiar so you have not only data protection and data management, but you’ve got governance and things like that. You’re able to do some forensics, predictive analysis things like that. The kind of functionality that they’re used to having on prem, but maybe in a cloud environment or possibly even a hybrid model.

    Palmer: I think—yeah, you nailed it. And one of the words that I used frequently is convergence. You know, one of the benefits of cloud is that because you’re provided with all of the substrate, you’re provided with compute, you’re provided with storage and networking and policy and all kinds of things surrounding you that the job of a company like Druva and even the job of enterprise developers is just to focus on the application logic.

    And what that means in our space is, as you mentioned, you have a lot of surround in your data center. Customers that, in my former world, they would buy a product for backup, they would buy a product for disaster recovery, they’d have an e-discovery product, they’d have an archived product, they’d buy a data classification product.

    Ashley: Been there, done that. [Laughter] Been there on all those.

    Palmer: Painful, right? You know, and all you see is your costs going up and complexity going up and different skill sets and your training and conferences to go to and it’s slowing you down, right? And when you look at Druva in a single platform, we simply consider those ancillary value propositions like e-discovery or data classification as on demand, callable apps.

    Ashley: Mm-hmm.

    Palmer: So, when you back up with us, you simply tack on the DR service, you simply tack on the data classification. You have a work flow for legal hold. And by the way, you pay for this on demand, you don’t have to pre-pay capacity licenses, you don’t have to pre-pay storage.

    Ashley: Mm-hmm.

    Palmer: So, you have a world in which not only do we replace your current apps, but we have a world in which, as customers come to us and we release new capabilities every other week, that they can put something on a roadmap and see a functional working application in weeks and months over their existing data set, paid for only when they use it. You know, and this is revolutionary for most CIOs.

    Ashley: Usually you have to buy the amount of storage you think is sort of your top capacity and use it as you go and then potentially upgrade where, of course, in the cloud, it’s elastic, it’s fungible. We can change, add more as we need it.

    Palmer: And of course, you have to do the same with your software licenses, you have to then integrate those products, you have to find space in your collocation facility. You know, when you really analyze how much cost and effort it is to adopt a new piece of software or a piece of hardware, it’s far beyond the obvious.

    And not to mention, by the way—and this is a stat that I quote to a lot of people. It has been in my experience that, when a software vendor releases a new version of a product, it can be 12 to 18 months before even half the install base adopt that version. So, if agility and speed matter to you as a business in adopting capabilities, you have to take risk and wait a long time before you get a capability, whereas in SaaS, you get it on demand with no effort—again, completely changes the model.

    Ashley: I mentioned hybrid earlier, something we haven’t talked about yet. Most enterprises can’t lift and shift everything into the cloud. As a matter of fact, they may not want to. They may want to keep some hybrid either multi-cloud or on premise. Is that an area that you can readily address with them?

    Palmer: Yes, and thanks for asking that, because one of the difficulties in being unique in your space is just explaining your model in the first place. And from a SaaS point of view, while our application architecture is Cloud Native, we service customers in their premises data centers and into the public cloud, including their SaaS providers as well.

    So, what that means is, as you said, you know, if you have a customer that has—and most are within this category—have a data center where they have workloads where their recovery times are perfectly fine to take advantage of all of those benefits I mentioned earlier in the cloud, we help move them to the cloud and manage and restore and so forth for them to their local premises facility. However, if there is an application that has an RPO or a recovery point or a recovery time objective that requires it to be within the data center, we also kind of provide them with a virtual appliance that solves that problems as well.

    And then, as they build native cloud applications and, for example, Amazon, AWS, or they adopt Office 365 or Salesforce, we also help data protection and classification, both natively in Amazon as well as with SaaS providers as well. So, we really are in them with their current architecture. We’re also with them in their future architecture.

    Ashley: Yeah, that’s the key to transition, I think. Again, I mentioned earlier about de-risking and part of that is familiarity as well as giving me a path to get where I want to go, and I may not fully go to the cloud, right? I may always have to have something, as you mentioned, because of RTO or RPO requirements to be on prem.

    Palmer: That’s right.

    Ashley: Tell us a little bit about some of your, some of the capabilities that are just kinda native with the product that, you know, what does it take to get started using Druva? Can I just plunk down the credit card and, like Amazon, I’ve got all of it available to me, or is there a set of planning processes that are helpful to go through to properly architect a data protection architecture like this?

    Palmer: Oh, and we love this question, because again, coming from the industry where the idea of even doing a POC is a very complicated process of governance rules and software installation and finding hardware and disk capacity. With Druva, you literally sign up on the online platform. We give you access to the agent type that you’re looking for, so let’s use VMware as an example. You can download a proxy, provide it your credentials, put a policy in, do all of this within about 10 to 15 minutes and you can initiate your first backup.

    And you can do this and we have tested this over and over without having specialist backup administrator skills, which means you can also start delegating privileges to application teams and such where you as an administrator have the ability to enforce and create overall policy. So, now you have a world where your most mundane, everyday tasks are made so simple that you can delegate them to your customers, your customers from an IT provider point of view—which increases their satisfaction, because they can do these things on demand, and they free up your time to do more important things.

    Ashley: That’s awesome. I was breaking out in a cold sweat as I was asking that question. Having done this at multiple companies, it is not a fun process to stitch it all together with different products and really have to architect it properly because you have to worry about things like how are you gonna monitor compliance. You mentioned e-discovery. Now, we have things like GDPR compliance as well as our own internal processes about data retention and making sure we’re protecting ourselves against ransomware, things like that.

    Palmer: And we deal with some of the largest companies in the world, and one particular use case we saw super well is on the end point side where customers are dealing with either legal situations or they’re concerned about end user behaviors and what they’re doing with data. So, we not only provide them the ability to protect devices, but we also provide them with ideas about what’s going on with the data, and then certainly with workflows as you mentioned to go and pull that data aside or do a hold where they need to—again, this is all just workflow in an interface. It’s not trying to get one product to work with another product or making sure that versions are upgraded or compatible or patched sufficiently. We take care of all of that.

    Ashley: Mm-hmm. As I’m kind of peeling the onion here remembering things that you have to address with a data management approach, you know, you have to deal with things—you mentioned tiered storage earlier. Does de-duping apply in a cloud environment? Does caching apply in a cloud environment? Are those things that you also address?

    Palmer: All of the above. So—and you asked a very good question earlier. If I’m a customer adopting the cloud, where are some of my risk areas or things that aren’t provided to me? And one of those examples is de-duplication. So, if I’m going to adopt S3 or I’m going to move data into Glacier, how am I gonna do that efficiently in a world where I’m potentially coming from a de-duplication position on premises, I’m gonna lose that in the cloud.

    With Druva, we de-duplicate that data for you, so we are minimizing your cloud storage footprint. And once we’ve done that, we’re also taking care of the tiering into whether it’s S3IA or Glacier or Glacier Deep Archive, we’re not only replacing your HSM strategy, we’re doing that, as I said earlier, in an automated way.

    Ashley: Mm-hmm. What’s nice, too, is—again, I’m kinda having the negative flashbacks here. The old process of which are tier one, how much in memory storage are you gonna have at your first tier of storage, how much are you gonna have at this speed, how much are you gonna have at your third tier of speed. With the different services that Amazon provides, that’s what you’re bringing together in one collection in a full data management architecture so you can, of course, dynamically allocate any of those as you need them, but that’s what you’re tying into is those services as well as what you provide.

    Palmer: Capacity planning is a non-issue in a cloud and SaaS world. Of course, we will work with you where, as you mentioned earlier, where you have RPOs and RTOs on site where you know that the use case doesn’t apply to a remote data center. We’ll work with you with a virtual appliance. We also have customers that will always want a land speed restore, but they’re increasingly collocating in places like Equinix, for example, where they can cross connect into the cloud provider.

    So, even where you have requirements that look like data center requirements, it’s almost as if your data center is moving into clouds in certain collocation facilities where you can take advantage of SaaS even without the virtual appliance.

    Ashley: What do you feel like is the biggest pain point that you address for customers? Obviously, they have needs, functionality, capabilities—all of that kinda thing. But when someone comes knocking on your website and says, “I think I need this,” what is most commonly the biggest problem that they’re trying to address?

    Palmer: There are two problems, but the compelling events are different. So, the two problems we’re solving for them are cost. They look into the future and they realize their costs are going up, not down, and cloud can help solve that problem.

    And the second one is complexity. They don’t want to have to hire more storage administrators or backup administrators, track different vendors. They don’t have to do hardware refreshes. They don’t want to have siloed systems. These are problems that are all solved with SaaS provides like Druva and the public cloud. So, those are the core problems.

    The compelling events, on the other hand, can be very different. We have Fortune 100 companies that have data centers all over the world, and their cloud journey can include moving the satellite facilities to the cloud today and increasingly looking at cloud and the core data centers tomorrow. So, they have a more data center migration consolidation use case.

    We have others that have remote offices or field facilities where supporting an appliance is expensive if not impossible because they don’t have the staff. So, it’s refreshes and getting equipment out of facilities is the second one.

    Ashley: Mm-hmm, mm-hmm.

    Palmer: We have customers that come to us because of ransomware, where we offer an air gap solution where the only alternative on premises was tape. So, we give them the same security via encryption and have an air gap solution to their primary network that no one else can. So, they’re adopting the cloud with the safety of tape.

    And then, of course, your basic use cases of backup and disaster recovery, where customers want not just to be able to restore data but move an application, have the full orchestration capability, and instantiate that without the additional costs of owning their own architecture or frankly even managing their own cloud integrations.

    So, we integrate a DR capability in our platform. So, customers come, they get native backup, they get native disaster recovery all in one interface without other products or other training. So, we have compelling events and we’re saving our customers 50% based on their existing legacy architecture and we have proven that a few different times via external experts as well as customers themselves and complexity and really taking all of the legacy mundane tasks and skills and making them obsolete.

    Ashley: Well, Mike, I’d like to thank you for joining us. We’ve run out of time, here. I think we could talk another hour about data management [Laughter]—

    Palmer: [Laughter] For sure.

    Ashley: —share some war stories along the way, too. So, I’d like to thank you, Mike—Mike Palmer, chief product officer at Druva, for joining us.

    Palmer: Mitch, thank you very much. It was a great time talking to you. Maybe we’ll do it again.

    Ashley: That would be great. We’d love to hear an update as things kinda move forward and you spend some of that money. [Laughter] And I’d, of course, like to thank you—you, our listeners—for joining us today. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat podcast. Be careful out there.

    — Mitchell Ashley

  • Cloud Complexities Hinder Effective Enterprise Data Management

    Cloud Complexities Hinder Effective Enterprise Data Management

    Enterprises everywhere are rushing to embrace automation and AI to improve existing business processes and innovate. But they’re learning that if APIs are the glue they use to build their systems, data is the fuel that drives these efforts forward.

    According to data management provider Actian, however, enterprises are experiencing significant challenges when it comes to obtaining fresh data, effectively wrestling with the complexity of data and getting to the point of driving innovative initiatives.

    The Actian Datacast 2019: Hybrid Data Trends Snapshot surveyed 300 IT decision-makers and professionals in businesses with 259 or more employees. Ninety-four percent of respondents said it’s important for them to have a system in place that makes certain users keep getting data, yet 58% of respondents admitted it is only somewhat likely, or not likely, they are actually getting fresh data to their users. So much for smart, real-time decision making.

    That could explain why just 34% of respondents who said they are using data for decision-making are using it to push innovation forward, instead, the majority are using it for traditional business operations and reporting.

    “Business as usual operations keep enterprises running from day-to-day, but limiting data to operational reporting tasks means missing a key piece of the data puzzle–new insights that lead to awareness of products, markets, consumer trends, strategy and more,” said Actian in a statement.

    The costs of data management are also taking their toll. The survey found 84% of enterprises would deploy more data if they could do so more cost-effectively.

    Those costs could be driven, in part, by the complexity created with data being stored within application siloes and across multiple cloud environments.

    “Over 50% of these businesses say data complexity issues, due to siloed applications, are a top barrier to entry for accessing data and gaining effective real-time insights,” said Actian.

    Actian concludes, unsurprisingly, that enterprises must get to the point that they can exploit their data in real-time and better manage data across cloud platforms.

    “Slower decision-making is only one consequence of having to wait for data to become available for analysis,” said Jack Mardack, vice president at Actian. “There are exciting new analytics use cases, like customer-360 and hyper-personalized real-time offers, that simply don’t work with stale data”

    “Enterprises are struggling to keep up with the growing demand for business-as-usual reporting that it leaves no viable bandwidth for generating new insights. This requires that enterprises provide not only the necessary data management infrastructure to make data from all their sources available to analytics discovery, but also that they invest in and empower data teams that can pursue new insights from new data spaces,” stated Actian.

    — George V. Hulme

  • CAP Theorem No Match for Mainframes

    CAP Theorem No Match for Mainframes

    One of the biggest problems in modern distributed computing is data management. These days data are spread out all over the place. As a result, getting the data you need, exactly when you need it, becomes a problem. Wider distribution creates more network latency. If you have a system in which some machines are in California and others are in New York, logic dictates it’s going to take more time for data to travel coast to coast than from within the same geolocation. It’s a matter of common sense.

    The data management problems created by distributed systems became more pronounced with the growth of the internet, which is, by nature, highly distributed. Eric Brewer, a computer scientist at the University of California, Berkeley, published a paper in 1999 that describes the dynamics of data management in a distributed system as a set of immutable constraints. Brewer published the work as the CAP principal. Three years later Seth Gilbert and Nancy Lynch at MIT proved the principal to be a theorem. Today, the CAP theorem is a central consideration when designing and implementing applications at web-scale.

    Simply put, the CAP theorem asserts that when your data is distributed over a variety of machines and geolocations, if you want the latest, most accurate data, you’re going to have to wait. And, if you don’t want to wait, the data you get will probably be stale. However, if data resides on a single machine, then everything changes. In such a case, you can have the latest data on demand, at any time.

    Of course, having a single data source implies having a machine that supports the demands of web-scale. Sadly, the x86 machine has yet to be made that can fit the bill. But a mainframe can. And, not only can it handle an enormous operational load, under the right set of circumstances it also can be a cost-effective way to meet the need at hand.

    Allow me to elaborate.

    The CAP Theorem Explained

    Before we go into the details as to why a mainframe computer such as an IBM Z or LinuxONE can make the problems presented by the CAP theorem go away at a reasonable price, let’s look at the theorem itself.

    Essentially, the CAP theorem asserts that of the three aspects of data management in a distributed system—consistency (C), availability (A) and partition tolerance (P)—you can have only two at any point in time.

    Figure 1: The CAP theorem dictates that of the three aspects of data management in a distributed system, only two are available at any point in time.

    In other words, if you want the latest information (consistency) from your data source immediately (availability), that information needs to be stored in a single, fail-safe location (partition tolerance). If you want to spread the data out over many machines (partition tolerance), either the data will be accurate at some point in time but not immediately (consistency) or you can have data immediately but it might be stale, awaiting an update to the most recent state (availability).

    The CAP theorem is more than theoretical. It has an actual operational impact on companies. A telling example is a problem that Netflix experienced when implementing its Monet system.

    Netflix Gets CAP-Ped

    Back around 2018, Netflix decided to implement an internal ad management tool under design using GraphQL, an open source API specification originated at Facebook that was becoming popular in many companies that operate at web-scale. In fact, GraphQL is still growing in popularity. So Netflix decided to take the plunge toward adopting GraphQL with an internal project named Monet.

    The way Netflix decided to implement Monet was to have the GraphQL API server act as a facade for existing RESTful APIs. The thinking was to reap the benefits provided by GraphQL, yet reuse as much existing code as possible. The idea was good, the actual deployment fell short. Once Monet was put into production, it ran dog slow.

    It turns out the REST APIs were distributed among a variety of data centers (see Figure 2). Network latency was killing them.

    Figure 2: Initially Netflix’s Monet system queried data from a variety of data centers.

    The company’s solution was to bring all the REST APIs into a single data center (see Figure 3).

    Figure 3: Netflix improved Monet’s performance by moving all RESTful APIs into a single data center.

    The result? Once all data resided in the same data center, performance improved dramatically. It makes sense. Shorter distance to travel means less latency in the system overall.

    Now, how this relates to the CAP theorem is interesting. Remember, according to CAP, as partitioning goes down (P), consistency (C) and availability (A) go up. So, while it’s true that Netflix was not using a single machine as a data source, moving everything into a single data center created what one could consider a single virtual data source. Yes, there might be latency issues within the data center. But, rack-to-rack latency is nowhere near as high as the latency between geolocations. We’re talking the difference of 1 to 10 milliseconds as opposed to hundreds, if not thousands, of milliseconds.

    In Netflix’s case, the company addressed the constraints imposed by the CAP theorem by reducing the breadth of partition at the geolocation level. All the data lived in the same data center. Imagine if partition size could be reduced at the machine level. What if instead of having data distributed among many machines within the same data center, data actually resided within a single machine? Partition size would be reduced to one. Then, applications could have the latest data immediately. All that’s needed is a single machine that can handle the load. This is where using a mainframe comes into play.

    Mainframes Are Built for Big Data and Agile DevOps

    Mainframes since their inception were intended to be able to store and process enormous amounts of data. Today 80% of the world’s data still resides on mainframes. As mentioned in a previous article about this topic, one mainframe can process 2.5 billion transactions daily. That’s the equivalent of 100 Cyber Mondays in a single day. That’s a lot of computing power easily equivalent to more than a few racks of commodity hardware in a data center. In fact, in some situations, one mainframe can provide the computing power found in an entire data center.

    So, let’s go back to the Netflix problem. What if instead refactoring those distributed RESTful APIs into a single data center, the time and money spent was used to refactor those RESTful APIs into a single mainframe? Yes, there are some technical hurdle to overcome, but considering the running Linux on a mainframe is standard these days, it’s not as if we’re talking transforming a bunch of Python code into COBOL. Databases from standard RDBMS such as DB2, Oracle to the NoSQL MongoDB can run perfectly well on a mainframe. There’s work to be done, but reinventing the wheel is not one of the tasks. Using a mainframe is a viable solution except for one sticky little problem: The conventional thinking is that using a mainframe is prohibitively expensive. Thus, the bias toward racks of commodity hardware or simply going with a cloud service.

    Mainframes as a Cost-Effective Solution

    The general perception in most IT shops is only big banks, insurance companies, governments and large-scale manufacturers use mainframe computers. They’re just too expensive for small to medium-size business. While many mainframes come with a price tag that can exceed $1 million, there are still surprisingly affordable models. For example, an entry-level IBM z114 will run about $75,000.

    If you’re a company spending $50,000 a month on AWS or Google Cloud computing resources, then for the $600,000 a year cost you’re spending on a cloud provider and the latency that comes with it, your company can well afford not only an entry-level IBM Z system to handle a production load, but also a second one to serve as backup, should failure occur.

    In terms of system administration, just about all the tools commonly used to support modern CI/CD processes today in the DevOps world have corresponding equivalents for mainframe implementation. As with any adoption of unfamiliar technology, there is a learning curve. But, again, we’re not reinventing the wheel. The tools, resources and expertise exist. Adoption is just a matter of following standard change management practices typical in medium-to-large-size companies that are well-positioned and use a mainframe to take advantage of implementing a cost-effective, no-partition data source strategy.

    Integrating Mainframe Technology into a DevOps-Driven Ecosystem

    Mainframes are capable of doing DevOps in an open, agile and collaborative fashion. The tools, resources and expertise required to support a mainframe’s participation in CI/CD processes exist today. For example, Zowe is an open source software framework that provides solutions that allow development and operations teams to securely manage, control and develop on the mainframe like any other platform. DB2 DevOps Experience for z/OS is one of the first commercial products built on Zowe. It allows IT the control over the mainframe database, its business rules and standards, while allowing development teams the ability to work independently using self-services and integration with modern CI/CD tools to speed their application release time with better quality and development practices.

    When you think about it, in certain mission-critical situations, using a mainframe to have the latest data at any time not only makes sense from a technological point of view, but it’s also attractive in terms of return on investment. The trick is to be able to identify the right use case. Of course, if you have a situation in which 90% of your users are in California and you have a mainframe situated in Bangalore, the laws of physics will apply. Moving the bits from the mainframe to user will take time. But, if you have situations in which user and data are in continental proximity, using a mainframe as a standard data source begins to look very attractive.

    Again, having the right use case is key. But, everything else being equal, using a mainframe for sensitive, data-driven applications warrants significant consideration.

    Putting It All Together

    The constraints on distributed computing imposed by the CAP theorem remain a challenge to those who design and implement large applications at web-scale. The “pick two” limitation CAP imposes can be an annoyance for simpler websites that provide content intended for the general consumer. But, for large scale, data-intensive applications, having accurate data immediately is not optional, it’s essential. Thus, using mainframe technology to create a no-partition data infrastructure becomes a viable way to address the limitations imposed by the CAP theorem. Under the right conditions, making a mainframe the central repository for your application’s data provides a significant return on investment hard to match when compared to commodity-based cloud providers.

    The nice thing about using a mainframe is when it comes to the CAP theorem, you still only get to pick two, but the two you get to pick are consistency and availability. In the scheme of things, those are the two that count.

    — Bob Reselman

  • Tips for Migrating and Managing Data Workloads in the Cloud

    Tips for Migrating and Managing Data Workloads in the Cloud

    Unless you are an alien life form who just beamed down from space, you’ve heard all about the why of moving data workloads to the cloud. The cloud is a more scalable, more flexible, more available and possibly less expensive solution for storing and processing data.

    What is not always obvious is the how of moving data workloads to the cloud. In several key respects, cloud-based data workloads are fundamentally different than those hosted on-premises.

    Understanding those differences is critical for moving data workloads to the cloud effectively. While it’s true that the cloud offers a variety of benefits for hosting data workloads, actually realizing those benefits requires a plan for managing data effectively once it is in the cloud.

    Let’s take a look at what that entails.

    The Special Challenges of Cloud-Based Data Workloads

    Let’s start by identifying the main ways in which cloud-based data workloads differ from those hosted on-premises:

    We could/should add a section here about assessment/planning before the actual migration. We have a bunch of published material on this.

    • Migration: The ways in which you migrate data to the cloud (and migrate it from one location to another within the cloud) are different from those you use on-premises. Not only do the tools typically differ (in the cloud, you’ll usually need to use your cloud vendor’s tools and interfaces for uploading and moving cloud-based data), but the speed and process are usually different, too (in the cloud, you’ll typically have to transfer data over the network, which is slower than some on-premises data transfer techniques). In extreme cases, migrating data over the network into the cloud may simply be unfeasible, which is why AWS Snowmobile is a thing.
    • Availability: In some respects, cloud-based data workloads are more available; in general, the cloud is less likely to fail or become unresponsive than your own data center. That said, the considerations for data availability in the cloud are different than on-premises. For example, DDoS attacks are typically a greater threat to data availability in the cloud. As a result, the strategies (discussed below) for maximizing cloud data availability are also different.
    • Performance: The factors that impact data workload performance in the cloud differ in some respects from those on-premises. In the cloud, it’s not the number of CPU cores or memory in your servers that matters, since these resources easily can be increased. It’s the way you build your data workload architecture and how effectively you avoid bottlenecks that shapes the performance of your workloads.
    • Cost: In an on-premises data processing environment, the lion’s share of your costs are the upfront capital expenditures required to build your infrastructure. In the cloud, your costs are monthly bills based primarily on resource consumption and the types of services you use. This means cost structures and the strategies for optimizing costs are very different for cloud-based data workloads.
    • Security: Cloud security is a discipline unto itself. Depending on which types of services you use in the cloud and how much control you have over the physical infrastructure (typically, you have minimal control), your security strategy for cloud workloads may look very different from the one you use to secure on-premises infrastructure.

    Working Effectively With Cloud-Based Data Workloads

    How do you manage the special challenges that cloud-based data workloads impose? Following are some tips.

    Perform a Migration Assessment

    As noted above, data migration into the cloud is a common pain point. Given this, and the fact that your migration strategy lays the foundation for the ongoing success of your data workload, it’s important to perform a migration assessment before beginning your migration.

    A migration assessment involves determining not only how to get your data into the cloud, but also how you will adapt your data architectures and strategy to fit the cloud. Will you simply lift and shift data into the cloud, keeping the same general architecture in place? Or will you modernize your data workloads by taking advantage of new technologies you didn’t use in your on-premises environment?

    The major cloud vendors offer a suite of tools to help answer these questions:

    Use the Network Effectively

    The network can be the biggest bottleneck in a data pipeline. Given the centrality of networks to the cloud, it’s therefore important to ensure you leverage the network efficiently.

    One way to do that (especially during the data migration stage) is to take a provisioning approach, which involves moving data gradually into the cloud and provisioning cloud infrastructure to fit each specific data workload in a cost-efficient way. This can be more cost-effective than lifting and shifting all of your data into the cloud at once, which can leave you with hefty data migration fees.

    Replicate Cloud Data Intelligently

    One of the best ways to maximize data availability in the cloud is to replicate it across multiple cloud data centers and/or regions. Doing so maximizes the chances of keeping your data available in the event that one cloud data center fails, which is rare but does happen.

    Of course, the more data replication you use in the cloud, the more you will pay, so you need to balance your replication strategy with your budget.

    Used Tiered Storage Effectively

    Most cloud vendors offer different tiers of storage. The default tier is designed for data that needs to be accessed on a frequent basis, but you can save money by choosing tiers designed for infrequently accessed data or archival storage. The price you pay for using lower-cost storage tiers is slower data access time, but in cases where you don’t need to access data quickly or frequently, low-cost tiers can do a lot for your budget.

    Access Control, Access Control, Access Control

    In the cloud, you often have little control over underlying infrastructure or software environments. You can’t harden the host operating system for your cloud servers and the metrics you can feed from a cloud environment into your SIEM are often limited.

    What you can do to help secure most cloud environments, however, is to set strict access controls. Configuring IAM policies may not be most engineers’ idea of a good time, but it’s critical to perform this tedious work to help secure cloud-based data workloads.

    Backup

    When you move data workloads to the cloud, it can be tempting to assume you don’t need to back them up, because the cloud never disappears.

    It’s true, the cloud is very resilient, but that doesn’t eliminate the need for data backups. There is always a chance—however small—that cloud-based data could be lost permanently. You also face the (less uncommon) risk of having your cloud-based data become corrupted or infected with malware—in which case it’s useful to have clean backup copies from which you can restore.

    So, back up your cloud-based data, too. As a best practice, it’s wise to follow a 3-2-1 backup strategy, which entails keeping multiple backups of your data in isolated locations.

    Conclusion

    The cloud is a fundamentally different beast than your on-premises infrastructure. Making the most of cloud-based data requires you to rethink your strategies for data migration, availability, security and more in certain respects. You need not discard everything you know about on-premises data management, but you need to adjust your operations to address the special challenges of the cloud.

    This sponsored article was written on behalf of Unravel.

    — Chris Tozzi

  • How DevOps Teams Can Skill Up on DataOps

    How DevOps Teams Can Skill Up on DataOps

    If you’re a DevOps engineer, you probably don’t think of data operations as a key part of your job. You help to write code and deploy and manage applications, while someone else handles data management and analytics, right?

    Probably not. Although it might be comforting to imagine that every company has a dedicated team of data specialists on staff who oversee data operations, the reality is that the vast majority of companies employ no data scientists at all. Only around 6% of large enterprises have data scientists on staff, and virtually no small and medium businesses employ data scientists.

    What this means is that at most companies, effective data management is the responsibility of any and all IT employees that these companies have on staff, including (especially) the DevOps engineers who bridge together development and IT operations. If the DevOps team doesn’t take responsibility for data operations, it puts at risk the applications that depend on healthy data flows to do their jobs.

    That begs the question: How do DevOps engineers (most of whom weren’t trained in the intricacies of data management) learn to support data operations? Keep reading for tips.

    Data Operations and Software Delivery

    Data operations (DataOps) refers to all of the processes and workflows required to collect, store, manage and analyze data.

    Traditionally, DataOps were siloed away from software delivery and management. The people who wrote, deployed and monitored applications were different from those who dealt with an organization’s data. That approach worked well enough in previous decades, when the data that organizations had to manage was less massive and less valuable.

    Today, however, data has become too critical a component of effective software delivery to be left in a silo. Many modern applications depend centrally on data. If your app includes a predictions engine or a recommendations feature, it requires data to do its job. Similarly, the monitoring tools that IT teams use to assess application health are driven by data. Data collection and analysis are critical for security operations, too.

    Thus, modern application delivery pipelines depend not just on the smooth flow of code from development into production, but also on a healthy flow of data. You can no longer isolate DataOps from application delivery (and even if you could, you probably don’t have a dedicated team of data scientists to handle your data anyway, as noted above).

    Adding DataOps to DevOps Software Delivery

    How do you actually go about integrating DataOps into an application delivery pipeline in a DevOps organization? Consider the following pointers.

    Automate, Automate, Automate

    Automation is just as important for effective DataOps, as it is for DevOps software delivery. By embracing tools that automate data collection, management and analysis, IT teams ensure that DataOps flow as smoothly as the software delivery pipeline.

    What’s more, data automation tools can help reduce the amount of DataOps work that general IT staff need to perform. That’s a key benefit for DevOps engineers who don’t have extensive DataOps experience.

    Embrace AI

    Another way of reducing the amount of DataOps work imposed on IT staff who are not trained as data specialists is to leverage AI to help them make decisions within DataOps workflows.

    Until just a few years ago, only a few tools used AI in a practical way to power DataOps. That is now changing, and modern data management tools make effective use of AI to reduce the decision-making burden on human engineers.

    Focus on the Data that Matters

    Within the context of software delivery (and elsewhere), some data have more value than other data. Since you have finite staff and material resources to devote to DataOps, identifying which data is most important for your applications is critical for optimizing your DataOps workflow.

    In other words, just because you collect data doesn’t mean you necessarily need to store and/or analyze it.

    Be Infrastructure- and System-Agnostic

    It can be tempting to rely on the data management tools provided, along with the cloud infrastructure you use or the platform you run on. However, although these tools might be helpful to a limited extent, relying too heavily on them puts you at risk of being locked in and having to rebuild your DataOps from scratch if you migrate to a new infrastructure or platform.

    For that reason, choosing DataOps tools that work with any environment is a wise idea.

    Educate Your Team

    The final pointer for helping your DevOps team do DataOps is simple: You need to educate them about data and what makes it work well. Don’t expect to turn them into data scientists, but they should understand concepts such as data quality, and recognize the bottlenecks that can get in the way of healthy data performance. Although these concepts may seem basic to someone trained in data management, they are usually foreign to DevOps engineers.

    Conclusion

    To say that we live in a data-driven world is a cliché. But it’s a cliché that is eminently true if you work in software delivery. In many cases, you can’t do DevOps without having healthy DataOps, too. That fact, combined with the lack of dedicated data teams at most companies, means that DevOps engineers must learn to integrate DataOps into their workflows.

    This sponsored article was written on behalf of Unravel.

    — Chris Tozzi

  • Data Mapping and GDPR: How Are They Related?

    Data Mapping and GDPR: How Are They Related?

    Every business enterprise receives data from an array of diverse data points. The volume of data is growing, and businesses understand the importance of leveraging data and converting them into actionable insights. Owing to data diversity, it becomes important to integrate them in a standard format. This is where data mapping takes a precedence. It helps establish relationships between different data models. Although data mapping offers several business benefits, it is also an integral aspect of the General Data Protection Regulation (GDPR) compliance. This post discusses the concept of data mapping and how it helps you adhere to GDPR.

    What is Data Mapping?

    As the name suggests, data mapping is a term for mapping the journey of data in an organization. A data map answers questions ranging from the origin of data to its journey through the organization. Today, most business enterprises are using various types of data mapping software, including ETL data mapping tools, to achieve their data mapping targets. Depending on the capabilities of these software applications, data mapping is used to perform a wide range of data management tasks such as data integration, data migration, data warehousing and data transformation.

    How Does a Typical Data Map Look Like?

    Typically, a data map includes the following information:

    • Type of data collected
    • Basis of data processing (personal or sensitive)
    • Purpose of data collection
    • Storage period of data
    • Place of storage
    • Conditions of data storage
    • Various data transfer destinations
    • Location of third party vendors (especially, international third party vendors)
    • Protocols followed during the data transfer (to international third party vendors)
    • Legal basis for data processing

    GDPR Compliance and Data Mapping

    The GDPR mandates business enterprises to ensure fair handling and security of the information they own. To assure the security, it is important to track the data journey from its point of origin to a conclusion. Without a detailed insight into data lifecycle, it is difficult to implement any security. Thus, data mapping becomes an essential step to achieve GDPR compliance. The following GDPR articles and their requirements will help you understand the importance of data mapping.

    • Article 30: Maintaining a written record of data processing activities in an organization is mandatory. A data map serves as a best written record, which organizations can easily retrieve and present at the time of GDPR audit.
    • Article 25: Data mapping helps organizations showcase their commitment towards data privacy.
    • Article 6: This article governs personal data processing under a lawful basis. Data mapping helps organizations to show the basis of processing–personal or sensitive.
    • Article 28: Data maps enable GDPR auditors to understand the extent of third party access to organizational data. All organizations are required to maintain a list of third-parties with whom they might be sharing their business data for various reasons.
    • Article 35: This article discusses Data Protection Impact Assessment (DPIA), and data maps enable business enterprises to meet this requirement effectively. To analyze the risk of a particular process, an organization needs to understand the data flow across an organization–how the data is being collected, where it is stored, who uses the information, how long it will be retained and so on.

    How an Efficient Data Mapping Software Helps in Achieving GDPR Compliance

    Now, what is an efficient data mapping software? It can be any application that aids a business organization to exploit the benefits of data mapping while adhering to the GDPR regulations. The following are a few important features of such GDPR compliant data mapping tools, which helps distinguish them from others.

    • Ability to Identify and Map Personal Data: Advanced data mapping software applications, which may or may not involve ETL data mapping software, allows users to identify and map personal data flow in an organization. They are designed to handle a large influx of data. These solutions utilize metadata to identify various data elements. The metadata helps in creating data maps that answer questions ranging from the origin of data, type of data collected, data subjects involved in process, service level agreements, among others. Thus, advanced data mapping software applications enable users to maintain, manage and monitor data maps for business efficiency and GDPR compliance.
    • Helps Ensure Data Security: Professional-grade data mapping software solutions are equipped with various security features that enable users to secure their database while providing them access to data, whenever required. They also allow organizations to conduct risk analysis of their data and ensure adherence to DPIA.
    • Assures a Better Control: Earlier reference to GDPR articles suggest how GDPR mandates enterprises to ensure data protection while handling personal data. An efficient data mapping software will help them address this need. They enable users to modify their applications to establish control, as well as safeguard personal data effectively.

    Achieving GDPR compliance is not an easy task. However, you can easily address it at various stages and it will help your business adhere to privacy standards while effectively meeting customer demands. You can take advantage of various data mapping software applications including advanced ETL data mapping tools to create effective data maps, stay compliant and, above all, drive your business forward.

    — Chandra Shekhar