devopsy.kinsta.cloud

Tag: DataOps

  • Actifio Looks to Meld DataOps with DevOps

    Actifio Looks to Meld DataOps with DevOps

    Actifio and IBM have formed an alliance under which IBM will make available data management software developed by Actifio under its own name as part of an effort to close the gap between DevOps and DataOps.

    At the same time, Actifio will make available a software-as-a-service (SaaS) offering for the first time, dubbed Actifio Go, which promises to make it easier to manage and move data across multiple clouds.

    Actifio CEO Ash Ashutosh said with the rise of both DevOps and DataOps there is now a unique opportunity to unify the management of pipelines across the enterprise. While storage administrators have been creating data pipelines for databases and applications for decades, the pipelines developers create to consume that data have been developed and managed separately. Now there is a nascent DataOps effort based on a set of best practices for automating the management of data that borrows many of the same principles being advanced by DevOps proponents for the building and deploying of applications.

    Actifio’s core Virtual Data Pipeline technology advances DataOps, Ashutosh said, by making it easier to copy a virtual instance of a data set running in a production environment and update it with the most recent copy of that data at any point in time. That capability is not only being employed to, for example, copy data into the cloud, but also to make data readily accessible to DevOps teams as needed on a self-service basis, he said, adding the goal is to make it easier for developers to dynamically consume the latest version of any data set as needed, with a minimal amount of intervention required on the part of a storage administrator.

    IBM plans to make an instance of Virtual Data Pipeline available as part of the IBM InfoSphere portfolio of data management software it provides. That agreement extends Actfio’s existing reseller relationship with IBM.

    Ashutosh conceded there is much work to be done before DataOps becomes a natural extension of DevOps. Most storage administrators today are not all that familiar with how the data pipelines they create are consumed by applications. Most storage administrators continue to think in terms of volumes and files rather than in terms of managing data pipelines, he said.

    But as those application environments become more dynamic thanks to the rise of microservices, pressure to modernize data management will only increase, Ashutosh noted. For example, more DevOps teams will be integrating the ability to copy data directly into their continuous integration/continuous delivery (CI/CD) platforms. Achieving that goal will be easier if the software employed to manage data within the context of a DevOps processes is invoked as a SaaS application, he said.

    Most IT organizations are a very long way from melding DevOps and DataOps. Most are still struggling with mastering the fundamentals of DevOps. But the convergence of DevOps and DataOps is all but inevitable. The only thing that remains to be seen is the degree to which IT organizations will proactively seek to achieve that goal versus waiting for developers to force the issue.

    — Mike Vizard

  • Top Analyst Firm Releases Groundbreaking New Research Report Highlighting Strong Momentum of DataOps

    90% of respondents in new 451 Research report say they will significantly increase investment in DataOps technologies this year

    REDWOOD CITY, Calif. [February 4, 2019]: Leading research and advisory firm 451 Research today issued a new report on digital transformation strategies for 2019, which reveals enterprises around the globe are planning a significant investment in the ever-growing discipline of DataOps. The report, “DataOps Lays the Foundations for Agility, Security and Transformational Change” found that over the next 12 months, 86% of respondents plan to increase investment in DataOps strategies and platforms, and 92% expect this strategy to have a positive impact on their organization’s success.

    Matt Aslett, Research Vice President, Data, AI and Analytics, 451 Research, defines DataOps as, “The alignment of people, process, and technology to enable more agile and automated approaches to enterprise data management in service of business goals. It aims to provide easier access to enterprise data to meet the demands of various stakeholders who are part of the data supply chain (developers, data scientists, business analysts, DevOps professionals, etc.) in support of a broad range of use cases.”

    In interviews with executives and data experts across banking, financial services, healthcare, retail and other markets, the research firm found nearly all are grappling with the increasing tension between the need to innovate to keep pace with ever-changing consumer demands and global markets, versus the need to comply with new data privacy regulations that vary across geographies. The research indicated that these executives are looking to DataOps strategies as a way to resolve this tension and stay ahead of the competition.

    The report found that these global brands believe DataOps will help navigate compliance and regulatory concerns, accelerate important digital transformation initiatives, and serve as a lever for increasing their competitive advantage in today’s Digital Economy.

    Compliance, Speed Driving Adoption of DataOps

    The report illuminated top data-related challenges these companies are facing, which include:

    • Prohibitively long wait times for exponentially growing volumes of data;
    • Increasing complexities of managing disparate data sources;
    • Slow and risky cloud migrations;
    • Growing security and compliance concerns.

    Compliance and security were notable call-outs in the report, as nearly three-quarters of respondents cited security and compliance as a top perceived benefit of DataOps.  They expect new DataOps technologies to help lessen the growing friction between the need to innovate faster and faster versus the need to comply with GDPR, CCPA, and other data privacy regulations.

    Aslett and his team further conclude that there is a real place for DataOps as part of a modern data strategy, “It has become clear that if enterprises are to realize business value from the development and delivery of data-driven applications and data-driven decision making, then more agile and automated approaches to database provisioning and data management are required, approaches that are more responsive to changing business requirements.”

    “Today, every company is a data company and the ability to keep pace with ever-changing customer demands is the key to survival,” said Chris Cook, CEO of Delphix.  “We’re seeing increased demand for DataOps platforms by some of the largest brands in the world as a way to ensure the delivery of fast, secure data to drive continuous innovation and success in the digital economy.”

    “As DataOps continues to mature, enterprises will reap the additional benefits of empowering development teams to overcome data friction and achieve the velocity of innovation demanded by the digital economy. In leveraging the Delphix platform and its full suite of partners including, Amazon AWS, Microsoft Azure, Oracle, and many more, IT leaders can improve cloud migration, address security concerns, and boost data modernization,” adds Aslett.

    Click here to download the full report.

    About Delphix

    Delphix’s mission is to free companies from data friction and accelerate innovation. Fortune 100 companies use the Delphix Dynamic Data Platform to connect, virtualize, secure and manage data in the cloud and in on-premise environments. For more information, visit www.delphix.com. You can also follow us on Twitter, Facebook, and LinkedIn.

     

  • Trifacta Extends Data Preparation to DataOps with New Functionality for Data Engineers

    New features including RapidTarget and Automator allow data engineers to more efficiently operationalize data preparation workflows in production

    SAN FRANCISCO, 14th of November 2018 — Trifacta, the global leader in data preparation, today announced a new set of capabilities designed to enable data engineers within data operations (DataOps) practices to more efficiently develop, test, schedule and monitor data preparation pipelines in production. With RapidTarget, Trifacta customers can utilize an existing data model to intelligently guide transformations and accelerate the process of generating a new  output dataset that matches the predefined schema. Automator provides end-to-end management of scheduling, monitoring and refining preparation workflows. These new features, along with the new Deployment Manager framework, facilitate the work of data engineers in administering data pipelines that feed analytics, machine learning and data science initiatives.

    “Today’s organizations are increasingly adopting DataOps to more efficiently scale their analytics and data management practices ,” said Wei Zheng, VP of Products at Trifacta. “Data engineers are at the center of this shift, playing a critical role in the data preparation practices that form the backbone of their data efforts. Our team has extensively studied the emerging role of data engineers within our customers’ organizations and focused these latest enhancements on improving their ability to transition data prep workflows into scalable, repeatable data pipelines. Our introduction of these new capabilities demonstrates the expanding role Trifacta is playing within the data management practices of our customers.”

    DataOps is clearly on the rise, with 73 percent of organizations indicating they planned to invest in the methodology in 2018. In light of this growing adoption of DataOps practices, the role of the data engineer in configuring, managing and scaling data pipelines has risen as a critical position within data management. Trifacta, through its work with large-scale enterprises including New York Life, GlaxoSmithKline and Deutsche Boerse, has seen first-hand how data engineers are a driving force behind the maturation of data preparation from an ad-hoc activity to a standard enterprise data management process. These new features in the Trifacta platform further support the role that data engineers perform in promoting data preparation flows developed by analyst end users into scalable data pipelines that deliver value to the broader organization.

    “Data engineering is a critical function at Malwarebytes—ultimately, we’re the ones responsible for the secure adoption of new analytics use cases across the organization,” said Manjunath Vasishta, Director of Data Science and Engineering, Malwarebytes. “Trifacta’s data preparation platform has been instrumental to that objective, allowing us to provide our business users with increased accessibility to data, while also maintaining governance. As data preparation usage continues to grow, the platform’s functionality provides data engineers with even further assurance to scale and operationalize data prep workflows in order to keep pace with the demands of the business.”

    The new features from Trifacta to further support data engineers and DataOps-focused organizations include:

    • RapidTarget: Allows data engineers to set a predefined schema target that provides automated guidance for how diverse data sources must be prepared and joined together in order to map to that target. Aligning the transformation process to an existing data model is critical when integrating unfamiliar or external data into an analysis. RapidTarget offers intelligent suggestions to accelerate how users prep and blend data sources in order to align their output to fit a desired data model.
    • Automator: Trifacta’s system to intelligently manage the scaling, scheduling and monitoring of data prep workflows in production. Users can set the schedule for flows to automatically run in production at a given time, when data updates or programmatically using APIs. Upon scheduling, users can define parameters or variables to customize what data is input into a workflow and how outputs are published.  Finally, they can monitor the status and performance of jobs, and set alerts when input data changes or anomalies occur.
    • Deployment Manager: Provides a framework for testing, versioning and managing data preparation workflows as they transition into enterprise-wide data pipelines. Before a new workflow can bring value to the broader organization, the flow must be tested at scale. With Trifacta’s Deployment Manager, data engineers can seamlessly test new workflows across development, testing and staging environments prior to being transitioned into production pipelines. The framework also allows data engineers to manage the versioning of flows and even rollback to prior versions of flows if needed.

    According to a recent Gartner report, “Providing the right data to the right people at the right time is a constant challenge. Data engineers are emerging as a crucial role in addressing this challenge. Data and analytics leaders must therefore develop a data engineering discipline as part of their data management strategy.”*

    Learn more about how data engineers are using Trifacta as part of broader adoption of DataOps practices on Trifacta’s blog here.

    *Gartner, Data Engineering Is Critical to Driving Data and Analytics Success, Roxane Edjlali, Nick Heudecker, Ehtisham Zaidi, 4 October 2018

    Gartner Disclaimer

    Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

    About Trifacta

    Trifacta is the global leader in data preparation. Trifacta leverages decades of innovative research in human-computer interaction, scalable data management and machine learning to make the process of preparing data faster and more intuitive. Around the globe, tens of thousands of users at more than 8,000 companies, including leading brands like Deutsche Boerse, Google, Kaiser Permanente, New York Life and PepsiCo, are unlocking the potential of their data with Trifacta’s market-leading data preparation solutions. Learn more at trifacta.com.

  • DataOps: DevOps Plus Big Data

    DataOps: DevOps Plus Big Data

    In traditional DevOps, there are the complimentary forms of development operations (that I call DEVops), and development operations (that I like to call devOPS). Between them they automate the toolchain and bring the people working on getting an application out to users onto the same team—not necessarily the same organizational team, but using their strengths in tandem to meet the needs of the business.

    DataOps is an interesting extension of DevOps. You still need all of the coding side to get data into the system and queries consistently maintained. You still need all of the operations side to get your database/noSQL/whatever up and running. In fact, while the development side in a DataOps environment might be (but isn’t necessarily) lighter, the operations side is almost always more complex. No matter the big data engine in use, it is a complex system in addition to other supported systems in a normal environment. My first installation of a big data environment (Cloudera, as it happens) was a weeks-long learning voyage. Only after I’d completed it did I use an automation tool (which is no longer available) to make it easy. My second round was hours … But it assumed the knowledge I had gained in the first attempt.

    The other bit of DevOps—ongoing monitoring and management—is also more complex in a big data environment, but we’ll come back to that in a moment.

    Once the normal DevOps systems are in place, ETL/data import tools will need to be supported also. The volume that these tools crunch through in a day makes their inclusion in DevOps in a data-heavy environment critical. If the data uptake is slow or the data itself inaccurate, there is an impact on the organization. This step also requires inclusion of data scientists, people who traditionally have not been pulled into the DevOps model. But they are the ones that can gauge the accuracy of data, and normally are responsible for data acquisition anyway.

    Which brings us to the monitoring and management stage of DataOps. Normally when we talk monitoring and management in DevOps, we are talking about tools that ultimately help with things like availability, responsiveness, auto-scaling and recovery. DataOps needs all of these things, like any other application does. And we’ve already mentioned that it has a complex environment embedded in your complex environment. But it also (some would argue more importantly) needs data monitoring. A knowledge of the state of the data is imperative. Whether from a piece of code gone wrong or data poisoning, monitoring the health of big data archives is essential to keep results from getting out of whack. This monitoring will again involve the people who know the data—data scientists by whatever title. This iterative process will need to run relatively constantly, checking on whatever makes the data valuable, normally using cross-field checks to verify that data hasn’t somehow been corrupted, or comparing trending averages to current average to monitor for sudden swings in results.

    Of course, for a system (and in research there are many) that spins up, finds results and spins down, the monitoring part is less essential, but for anything that keeps accessing and adding to the data pool, it is the key to a healthy application or application portfolio based upon the dataset.

    If all that sounds a bit intimidating, you’re not alone. The phrase “DataOps” has been around for a few years, and while data scientists use it, it hasn’t yet gained serious traction in the enterprise. Here’s how I would approach it as CIO: Given that DevOps is running on other projects, that is the low-hanging fruit. Given the rate of change for servers (as queries—particularly long-running ones—increase and decrease, systems needs change is one example), traditional DevOps is a greater benefit to big data than many other corporate systems. So roll out DevOps first without serious concern for DataOps beyond how DevOps impacts ETL—not ETL data quality, but ETL data availability/performance. Only when the system and the team are in place and the software/hardware/code part of the overall system is fully in a DevOps environment, start looking at ways to move data into it. Data monitoring will be the big one, but ETL under the initial round of DevOps implementation will have shown places it could be improved and ETL data integrity can be added, so improving those areas also. A measured, step-by-step approach that builds off of organizational experience is key here. Data monitoring and management will require different tools than normal DevOps monitoring and management, so having the data scientists look into available toolsets and how they mesh with existing tools early on will help with eventual implementation.

    Make no mistake, it is a big job, moving large data driven environments to DevOps. But the benefits are increased awareness of data quality and data quality issues, along with more responsive systems and teams that DevOps offers every business application. The amount of data available to any give organization is currently on a massive increase as IoT, open datasets, partnerships and ML drive greater pools. It is worth getting the automation and cooperation of DataOps into the mix as early as possible.

    — Don Macvittie

  • Delphix and Datical Extend Global Partnership, Bring New DataOps Solutions to Modern Enterprises

     One-half of all Fortune 50 companies work with Delphix, Datical to deliver high quality software – faster and more securely – to close Data Velocity Gap

     REDWOOD CITY, Calif., October 23, 2018 – Delphix, the company accelerating innovation through DataOps, today announced it has extended its partnership with Datical, the leading provider of database release automation solutions, to bring new DataOps solutions to market that unlock the potential of data to drive business transformation. Already in use by  one-half of the Fortune 50, Delphix and Datical are driving the emerging DataOps ecosystem that will enable more Global 2000 companies to open up the flow of fast, secure data to move fast and compete in today’s Digital Economy.

    With worldwide enterprise data expected to grow to 163 zettabytes (ZB) by 2025 (Data Age 2025, IDC), software development and DevOps teams are drowning in data; unable to use it to deliver innovation that keeps pace with ever-changing consumer demands. Delphix and Datical offer proven solutions that eliminate delays through automation to achieve 100x faster high-quality software releases, 10x less cost and no production outages.

    “As large enterprises look to accelerate their digital businesses, two of the biggest challenges they face are the incredible volumes of data being generated and the related lack of velocity from their database layer,” said Stephen O’Grady, Principal Analyst with RedMonk. “While application development has been revolutionized by the DevOps phenomenon, database management has lagged behind. This issue is what’s created the demand for an equivalent ‘DataOps’ style movement, and is why Delphix and Datical are focusing on these needs so intently.”

    “Businesses are feeling the debilitating effects of data friction, challenging their ability to move fast and confidently compete in today’s digital economy,” said Tony Orlando, SVP of Global Field Operations, Delphix. “Through our partnership with Datical and the DataOps ecosystem at large, we are enabling the secure, continuous flow of data within the world’s most complex enterprises, empowering them to accelerate innovation, reduce costs, increase quality and lead in their markets for years to come.”  

    Modern Enterprises Rely on DataOps to Unlock the Potential of Data

     “Data remains locked in legacy systems, and is holding [CEOs] back,” said Salesforce CEO Marc Benioff in a recent earnings call. Over the last three years, Delphix and Datical have worked with some of the most important brands in the world to help them unlock data to accelerate software development and close the costly data velocity gap.

     Together, Delphix and Datical enable enterprises to dramatically accelerate software development and testing. The unique combination of these solutions reduces the amount of time it takes to provision production-quality data from days or weeks to just minutes, while also eliminating manual tasks, such as reviewing and validating database schema and logic changes. Integrating these solutions into a software development toolchain is at the heart of the DataOps approach that transforms how businesses deliver new releases.

    A Top 5 investment management firm based in the US (with over $2 trillion in assets under management) uses Delphix and Datical to increase the speed and quality of application releases that are the backbone of its digital transformation efforts. In a short period of time, the global brand achieved:

    • A 100% increase in rate of application deployments;
    • A 90% reduction in data provisioning times;
    • Dramatic improvement in quality from eliminating data-related defects.

    Pioneers in the DataOps movement, Delphix and Datical are at the forefront of building the platforms and technologies that support this emerging discipline. As Gartner recently recognized in its Hype Cycle for Data Management for 2018, DataOps is the next chapter in the DevOps story that will accelerate business and help create the modern data enterprise.

    “Software is reshaping all industries and the pace of change is accelerating. Companies must speed up innovation in order to survive in the digital era,” said Derek Hutson, CEO and President of Datical. “To achieve their goals of faster time-to-market, companies continue adopting faster, more iterative development methodologies to be able to change and adapt more quickly – but data has become the new constraint slowing them down. Datical and Delphix are in the perfect position to help enterprise leaders remove that constraint and modernize the way data is managed throughout the software delivery lifecycle.”

    About Delphix

    Delphix’s mission is to free companies from data friction and accelerate innovation. Fortune 100 companies use the Delphix Dynamic Data Platform to connect, virtualize, secure, and manage data in hybrid cloud environments. For more information visit www.delphix.com.

    Follow Delphix on Twitter, Facebook and LinkedIn.

    About Datical, Inc.

    Datical’s mission is to transform the way businesses build software so they can deliver better customer experiences faster. Datical solutions deliver the database release automation capabilities technology executives need to get the most out of their Agile, Cloud, DevOps and Continuous Delivery investments. With Datical database release automation organizations can shorten the time it takes to bring applications to market while eliminating the security vulnerabilities, costly errors, data loss and downtime often associated with current database deployment methods. For more information, visit www.datical.com, call 737-402-7187 or connect via @datical.

    Follow Datical on Twitter, Facebook and LinkedIn

  • DevOptimism: The Future of DevOps is Becoming Clearer

    DevOptimism: The Future of DevOps is Becoming Clearer

    DevOps has quickly matured from a fairy tale to a bestseller at most large enterprises. However, we are still at the beginning of the DevOps story, with many exciting chapters still to be written. This article looks at several emerging technologies shaping DevOps today and a handful of developing trends that will shape the ecosystem in the years to come.

    Today’s IT organizations increasingly are moving away from monolithic applications deployed through grand-slam releases with security slapped on after the fact. We are seeing many enterprises transitioning to more atomic applications released in continuous deployments with application security baked in from the beginning. Fueling today’s emerging DevOps is a bevy of new products focused on architectures including microservices, service mesh and function-as-a-service (FaaS); advancements in pipeline automation; and more adaptable security paradigms such as interactive application security testing (IAST) and runtime application self-protection (RASP). While these solutions herald a great step forward, they do demand that legacy applications be refactored into smaller granularity services, layered onto newer architectures and migrated to processes that allow greater automation.

    OK, that’s where we sit today: Great progress creating great opportunities and great challenges. But let’s head a little further down the road to understand where today’s advancements are driving DevOps tomorrow.

    There are five trends consistently appearing on the whiteboards of venture firms and forward-thinking startups that will shape DevOps in the upcoming years. At a macro-level, DevOps must become wider and smarter. As we dig layer deeper, we see five distinct patterns appear in which emerging technologies will make the release pipeline more insightful and proactive while allowing it to handle a wider spectrum of pipeline artifacts beyond just applications:

    • AI for DevOps: Artificial intelligence (AI) is making inroads into all facets of IT and DevOps is no exception. AI techniques soon will make the DevOps pipeline smarter, able to predict the impact and risk of deployments, spot procedural bottlenecks and identify automation shortcuts. Advancements in robotic process automation (RPA) lend themselves directly to optimizing the various handoff and automation points of the release pipeline. AI-based predictive analytics also will allow for higher fidelity operational capacity planning and pre-deployment fault prediction.
    • DevOps for AI: Today we are seeing a handful of new tools, commonly called MLOps, emerging on the market that allow developers and modelers to deploy new AI models rapidly and repeatably. However, these products have yet to collaborate closely with the Ops side of the enterprise. They are generally a development aid, not an operational aid—or said another way, they are all “ML” and very little “Ops.” This space will mature rapidly, however, allowing release of AI models via the DevOps ecosystem so as to include operational aspects. Conversely, tomorrow’s DevOps pipeline will also allow AI models that have been learning from real-world data to be extracted from production and moved back into the development environment.
    • DBOps: Today, new databases, schemas, indexes, triggers and stored procedures typically are deployed manually by DBAs outside the DevOps pipeline, with their operational impact being provided by best-guess capacity planning. This glaring gap soon will be closed by products that allow databases and their associated artifacts to be released through the deployment pipeline along with the applications with which they are coupled. This will not just encompass traditional relational databases, of course, but also support non-structured databases and object stores. The prime objective being that applications and their supporting data repositories will be developed, tested and deployed via the same automated pipeline as applications without the need for specialized database handholding.
    • DataOps: When new apps are deployed, the associated analytics and visualizations are typically brought online by data scientists after the fact through manual deployments. Looking forward, expect data analytics to be introduced into the DevOps pipeline soon after the requirements for the release are defined. Much like DBOps above, the goal is to eliminate specialized manual promotion. But beyond just synchronous deployment, DataOps will allow data governance, data loss prevention (DLP), data lifecycle management and copy data management features to be integrated during each new deployment.
    • Blockchain for DevOps: Today’s and tomorrow’s DevOps evolutions will increase the velocity, variety and volume of pipeline traffic dramatically. While this sounds promising, it undoubtedly will be a nightmare to audit, verify compliance and ultimately control and manage. One technology being eyed to untangle this spaghetti is a distributed consensus mechanism such as blockchain. Stop rolling your eyes—we’re not talking about Bitcoin or ICOs. We are describing the need for a centralized, permissioned, secure, distributed ledger to record, track, authenticate and audit the torrent of pipeline transactions. Blockchain seems to fit the bill nicely and also has the added benefit of enabling digital contracts at each pipeline stage that can be verified at runtime to trigger artifact promotion and also audited long after the fact to ensure that all rules were followed.

    DevOps has already become the standard deployment paradigm in many enterprise IT shops and its adoption will continue to become more pervasive. But this growth requires DevOps products and processes to become smarter, more secure and more controllable as the release pipeline incorporates a wider set of deployment artifacts beyond just applications. This is a daunting challenge to be sure, but there are exciting new products that will emerge to help write the next chapters in the DevOps story.

    — Mark Campbell

  • DevOps & DataOps: Catalysts for Organizational Transformation

    DevOps & DataOps: Catalysts for Organizational Transformation

    Over the past few decades, use of the internet has increased exponentially. Nowadays, the need to build quality web applications and managing its huge data effectively is a huge concern of any organization. Thus, there has been continuous exploration for finding better methods of software development and data management.

    To meet the need of today’s internet users, many technology companies have stepped up and are continuously utilizing their resources to find more efficient ways of developing web applications in less time and with better quality, and are finding more effective ways to manage and analyze its data.

    To achieve the same in application development and data pipelining management, many big organizations have started following a set of practices—DevOps and DataOps respectively.

    In this article, we’ll take a look at what DevOps and DataOps are, and then we will focus on the ultimate question: Are they really worth to implement in an organization, and will following the same set of practices really improve the application development and data management—and consequently the growth—of an organization?

    DevOps and DataOps: An Introduction

    In simple words, DevOps and DataOps are a set of practices for development of software and management of data respectively, for a particular project or to follow at an organization level.

    DevOps emphasizes on automation of processes of various teams—software development team, quality assurance team and IT operational team—so as they can complete the relevant tasks (i.e. plan, code, build, test, release, deploy, operate and monitor) of application development quickly. Also, the these teams will be able to maintain the higher quality of software application by delivering/publishing software in smaller parts with higher deployment frequency while keeping the cost low, thus following the agile methodology more seamlessly.

    DataOps, in comparison, can be defined as DevOps with data analytics.

    As mentioned earlier, we know that DevOps emphasizes the collaboration of software development team (including testers) with system administrators (operational team). In a similar manner, DataOps weighs on collaborating the teams of data engineers, data scientists and other data professionals who are handling the data-related processes (i.e. store data, analyze data, deriving decisions from data, archive data or deliver data).

    The concept emphasizes both communication and collaboration between various teams of the same organization. The combined team can have more visibility of the project and come up with faster results. It also will improve the morale of the organization by proving the fact that people will be always above processes and tools when it comes to management of projects.

    DevOps DataOps
    Automation and Monitoring of Software Development Automation and Monitoring of Data Pipelining/Management
    Combination of

    –       Software Development

    –       Quality Assurance

    –       Technology Operations

    		 Combination of

    –       Data Engineering

    –       Data Integration

    –       Data Quality

    –       Data Security/Privacy
    Collaboration of

    –       Software Engineers (Developers)

    –       System Administrators (Operational Team)

    –       Testers

    		 Collaboration of

    –       Data Engineers

    –       Data Scientists

    –       Data Analysts

    –       Data Professionals
    Typically aims at

    –       Shorter development cycles

    –       Increased deployment frequency

    –       Quickly moving from Dev –  to/from Test – to Production

    –       More dependable releases

    		 Typically aims at

    –       Automation of Storage Workflows

    –       Monitoring the Operational Flow

    –       Optimization of Data Analytics

    –       Creating Infrastructure that will enforce data quality rules

    Now that we’ve gone through the basic details regarding DevOps and DataOps and what they are doing for software development and data management respectively, let’s take a look at some of their potential benefits to see if they can really act as a stimulator for an organizational transformation.

    Benefits of DevOps and DataOps to adapting Organization

    There are numerous benefits of adapting the defined processes of DevOps and DataOps in project execution and running the organization. Here are some of the key benefits for applying the practices of DevOps for applications development in an organization.

    Benefits of DevOps

    • Transparent Processes and Culture across Organization – By continuous communication and collaboration between the development team, QA team and IT operations team for every deployment cycle, each employee will be updated regarding latest information of project and will be more aware regarding the processes of other departments.
    • Quicker and Higher Quality Deployment Cycles – By applying the set of practices of DevOps, especially the shorter development cycle, will boost the frequency of deployment. As the focus of teams will be limited—i.e. on small task/module—for a deployment cycle, quality of each deployment will improve.
    • Faster Resolution of Bugs – As tasks are smaller and continuously deployed on relevant servers, the development team will be given feedback very quickly from the QA team, thus enabling the development team to provide much quicker resolution of any non-working functionality. Application stability will be a byproduct of this process, as the reason behind any error can be identified very easily (because particular deployment cycle of an error is known).
    • Flexibility to Handle Unplanned Work – As agile methodology insists on the handling of the raised change request, teams streamlined with DevOps will be able to handle such unplanned work more efficiently.

    Benefits of DataOps

    • Faster Processing of Data – As various teams are getting collaborated, communicating continuously and working on micro databases rather than all at a time, processing of data at various process stages are done much quicker.

    • Quicker Solution of Problems – Similarly to DevOps, in DataOps data are being processed in bunches rather than at once. This approach can help to identify issues pertaining to a bunch of data more quickly, and their resolution can be derived more quickly, as the focus is limited.
    • Better Data Security – As the data security professionals are in continuous touch with data integrators and data analysts, securing data at each process level becomes easier.
    • Improved Resource Output – When the team adheres to the fundamentals of DataOps, seamless communication and transparency between the various teams will aid in the performance of each team member and will lead to better resource output.

    Implementation Procedure of DevOps and DataOps

    Defining the processes for implementing DevOps and DataOps is simple, but implementing same processes do require tremendous dedication of teams. Let’s focus first on the general steps for implementing DevOps processes:

    • Collaboration, communication/seamless integration between formerly siloed teams, i.e. software development team, quality assurance team and IT operations team
    • Breaking down the major modules/big development tasks in particular smaller tasks
    • Processes for DevOps should be followed for each deployment cycle/each smaller tasks
    • Tools – There are various tools available in the market to accomplish processes of DevOps, and there are some techniques for continuous development and performing agile operations‚ such as Kanban, Scrum, etc., which should be followed for seamless application deployment.

    With DataOps, there are no specific software tools available in the market to implement DevOps. But there are some frameworks and its supporting tools that can assist the DataOps approach for agile-based project management.

    • Data storage and analytics infrastructure should be redefined, i.e. cluster-based or redundant data storage approach is recommended. Also some latest data analytics techniques should be adopted.
    • Use of microservices-based architecture and software that can allow blending of various type of data, i.e. structured and unstructured, are also recommended
    • Implement ‘built for purpose’ database engines – By doing such, accessing and analyzing large quantities of data become much easier (these are the same type of database engines that drive Google Search, YouTube etc.)
    • Supporting tools – Tools that aid to agility of project and collaboration between teams i.e. ETL/ELT tools, data curation and cataloging tools and log analysers should be used. For data blending, software tools such as Map Reduce, HDFS, Kafka, Hive and Spark can be used

    Conclusion

    All IT organizations have been managing software application development and its relevant data for a long time. But right now, we are in an era where managing the same with velocity and quality can make your organization stand above the rest of peer companies. Adopting DevOps and DataOps demand sheer dedication and focus of the teams, but the output after implementation will be very positive and such followed standards will definitely transform the organization constructively.

    — Vikash Kumar

  • Everything Ops: Are DevOps Principles Being Applied Too Broadly?

    Everything Ops: Are DevOps Principles Being Applied Too Broadly?

    From big data to storage to security, the DevOps concept is being applied to every field of IT operations under the sun today. This “Everything Ops” movement reflects the popularity of DevOps—but is it also diluting DevOps’s significance?

    DevOps itself focuses on software delivery. It’s about designing, building, testing, deploying and managing software applications.

    If you follow the DevOps conversation, however, you know that DevOps principles are now being extended to lots of niches that don’t have much or anything to do with software delivery chains. Consider these examples:

    • DataOps: DevOps principles for data analysts.
    • ChatOps: The idea that chat tools should be integrated into workflows.
    • SecOps (or rugged DevOps, or DevSecOps, or whatever else you want to call it): DevOps practices for security teams.
    • Storage Ops: The insertion of DevOps practices into storage.

    The list could go on. Taken together, these many flavors of DevOps reflect what I like to a call an “Everything Ops” tendency. Fans of DevOps want DevOps to take over the entire world of IT.

    Benefits and Drawbacks of ‘Everything Ops’

    If you believe that DevOps represents a better way to deliver software, then the Everything Ops movement is a positive thing. It’s a way to help people who work in areas of IT other than software delivery to innovate as well by taking cues from their DevOps colleagues.

    But there is also a danger that Everything Ops can be applied too broadly. You can’t fit every single thing in the IT universe into the DevOps mold. For example, CloudOps (the DevOps-related way of managing cloud infrastructure, not the company of the same name) just doesn’t make a lot of sense. It’s basically just a reiteration of the ideas that are already at the core of cloud computing: Scalability, agility and location-agnosticism.

    Concepts like ChatOps can also be problematic because they refer to a specific way of using a particular type of technology, whereas DevOps is about cultural practices. ChatOps is a fundamentally different sort of thing from DevOps.

    Conclusion

    In short, by overextending the DevOps conceptual framework, you run the risk of diluting the importance of DevOps itself—and of losing your focus on software delivery, which is the area that DevOps benefits most.

    This is not to say that we shouldn’t extend DevOps to other fields. We certainly should. But we should be careful when doing so. Don’t try to make DevOps eat the entire IT world, because that just doesn’t make sense.

    — Chris Tozzi

  • Are We Leaving Developers Out of DevOps Spinoffs?

    Are We Leaving Developers Out of DevOps Spinoffs?

    SecOps. DataOps. NetOps. Reading these terms, you get the sense that the key to IT efficiency is to make IT Ops work with everyone else. But that’s a mistake, because it leaves developers out of the picture.

    It’s no secret that the DevOps movement has generated a number of other *Ops initiatives, including those listed above. The driving idea behind all of them is that IT Ops teams should work more closely with security teams (in the case of SecOps), data analytics teams (in the case of DataOps) or networking teams (in the case of NetOps). These are only a sampling of *Ops movements; the list could go on.

    To be sure, there are advantages to gain by having IT Ops collaborate more closely with other parts of the organization. But there are an equal number of advantages in doing the same thing with developers.

    To optimize security operations, you need developers, IT Ops and the security team to work together. If Dev is left out of the picture, it’s much harder for the other two parts of this equation to make sure that the code the organization deploys and manages is secure.

    For similar reasons, DataOps, NetOps and almost all the other *Ops you can come up with work well only when Dev also participates. The people who actually design and write the applications are important.

    Rethinking Ops

    Why does this matter? My point is not merely that terms like SecOps and DataOps are misleading because they don’t mention developers.

    On the contrary, giving equal weight to developers is important because if you don’t, you risk isolating developers. Some are already complaining that DevOps is “killing” them. The trend of defining new *Ops practices that don’t mention developers is not helping.

    So, maybe we should start mentioning and thinking about developers more prominently when we are talking about ways to extend the DevOps philosophy to other parts of the IT organization. This is already being done somewhat in the case of SecOps, which you sometimes hear called SecDevOps. But Google lists nearly 13.5 times as many results for “SecOps” as for “SecDevOps.” The numbers are similar if you search for “NetOps” as compared to “NetDevOps.” And I’ve never heard anyone say “DataDevOps.”

    The takeaway is this: Developers are as important as IT Ops to DevOps. Don’t leave them out.

    — Chris Tozzi

  • DataOps: Time to Bring DevOps Ideas to Big Data

    DataOps: Time to Bring DevOps Ideas to Big Data

    What can data scientists learn from DevOps? A lot, actually. By applying DevOps principles to the world of big data, you get DataOps, which streamlines data analytics.

    If you’re reading this site, you’re probably already familiar with DevOps. The DevOps philosophy emphasizes continuous collaboration between software developers (the “dev” people) and system admins (the operations, or “ops,” folks)—as well as everyone in between, such as software testers.

    It’s possible to apply similar ideas to the world of big data and data analytics. There, as in the world of software development and delivery, different teams traditionally have worked in isolation from one another. Those teams include storage admins, who keep data storage systems running; data analysts, who write and execute programs for interpreting large volumes of data; data quality specialists, who ensure data consistency; and data security teams, which protect data from attackers.

    As DevOps professionals know, keeping different teams isolated from one another is not efficient. It limits collaboration and maximizes confusion when problems occur.

    The data world, therefore, has much to gain by ending this isolation between teams and adopting a DataOps workflow. This will keep data operations flowing smoothly and provide continuous insight into the data that organizations collect.

    DataOps is not a brand new concept. People like Andy Palmer have been writing about it for a little while now. But the term and the idea have yet to gain nearly as wide a following as DevOps. (There is not yet even a Wikipedia entry for DataOps—and you know an idea is still in its infancy when no one has taken the time to write about it on Wikipedia.)

    Making DevOps Bigger

    Meanwhile, from the DevOps perspective, the growth of interest in DataOps is significant because it exemplifies the broad applicability of DevOps principles. So far, the DevOps conversation has focused on the relatively narrow context of software development and delivery. But it can extend much further, too. DataOps is a start.

    So, if you’re a data analyst, storage admin, data security professional or someone who works with data in any other capacity, it will be worth your while to borrow some ideas from your DevOps brethren. And if you work in software development, testing or delivery, it may be worth your time to educate your data friends on DevOps principles. It will make their workflows more efficient, while introducing DevOps to a wider audience.

    — Chris Tozzi