Tag: DataOps

  • 3 Steps to Turn a Data Deluge Into Actionable Intelligence

    3 Steps to Turn a Data Deluge Into Actionable Intelligence

    Digital transformation is creating even more data for organizations to manage effectively. Here’s how to find the intelligence needle in the data haystack

    Data and analytics transform how companies across all industries identify and act on opportunities to gain a competitive advantage. But many organizations are discovering that ever-growing volumes of the former make the latter increasingly difficult. They struggle to turn raw data into a strategic asset that informs decision-making and accelerates growth. Overcoming that struggle requires conceiving and implementing an effective data transformation initiative. Fortunately, there are three steps to implementing sustainable initiatives that any company can follow, no matter its maturity or size.

    But before taking the first step, it is vital to secure the commitment at the board and management levels that the initiatives will be a companywide priority. That is critical to ensuring the allocation of adequate resources and budget dollars at the outset. Failure to secure senior leadership’s enthusiastic support will impede progress if not completely derail the entire project.

    You might not think that would be a difficult sell. But research firm KPMG found that the opposite is true when it examined the fundamental role data analytics will play in the financial services sector over the next 10 years:

    “(Many CEOs are) responding incrementally and, at best, tactically with better front-end digital services for customers or process improvements in the back-office. But they’re not addressing the more fundamental issue – that their very business model will need to change. And fast.”

    If you encounter resistance, here’s the crux of your argument: Your company is a data company. It doesn’t matter if you sell software, cars, clothing, food or any other product or service; the ever-growing volumes of data you’re collecting inform all business decisions. Or at least, they should.

    Curt Hopkins, editor in chief for Hewlett Packard Labs, summed up this point perfectly: “Data has become the new gold that backs the value of companies, though it is a gold that companies must locate, mine and refine before spending. Business has shifted from making things to knowing things, so realizing usable data is the order of the day if a company wants to remain profitable.”

    Step 1: What Data Do You Have?

    Once you have secured the full support of management, you’re ready to take the first step to launch a data transformation initiative: gain an understanding of precisely what you have and where it “lives.”

    Take the time to conduct a thorough exploration of all your raw data sources and where those volumes are stored—likely some combination of individual users’ laptops and other devices, on-premises systems and in the cloud (public, private or hybrid).

    Next, determine who has access to your data and how. If you decide to work with an analytics services provider, do you have the necessary extraction processes in place to deliver raw data to your partner? If not, you will need to build that pipeline or move the data into a cloud-based analytics database.

    Step 2: What Do You Want to Accomplish?

    Once you know what you have and where it is across your organization, the next step is to formulate your initiatives based on your objectives. Classifying all datasets under two broad categories—internal use or external use (i.e. commercialization)—will help narrow down the relevant data for a specific campaign.

    Embed responsibilities for secure data handling and usage across all parts of the organization—don’t put them only on the technology teams’ collective shoulders. Assume you will need to organize, transform and enrich data to realize its full value. Remove any barriers to sharing data under NDA and considering a nuanced build-versus-buy approach. Finally, ensure the initiatives map to management by objectives (MBO) to maintain a focus on using data to help the company achieve its business goals. At this point, you’ll need to answer the build-or-buy question to determine if you have these transformation capabilities internally or need to find them externally.

    Step 3: How Will You Track Progress and Evaluate Outcomes?

    After executing the data initiative, you should regularly measure its outcomes. Hold your teams accountable by setting time-based goals and tracking progress versus plans. Anticipate the need to iterate because harnessing data is a fluid process. Create and maintain a dictionary to keep employees and teams aligned as they progress on their tasks and projects.

    One additional and no less critical consideration is to ensure you strike a balance between gaining the ability to use data assets with protecting them from security threats, both cyber and physical. According to  “The Cost of Insider Threats Global Report 2020” by IBM and the Ponemon Institute, the average cost to recover from a breach is between $3 million and $4 million, and that does not account for the loss of customer trust and brand loyalty. Protecting customer and employee personally identifiable information (PII) and all other sensitive data types and complying with all data rights agreements is paramount.

    Fifteen years ago, British mathematician Clive Humby famously proclaimed, “data is the new oil,” and the analogy holds up today. Like the raw crude that comes out of the ground, raw data only represents potential value. Just as oil has to be refined into fuel to become useful, data must be analyzed and refined to become functional.

    By understanding your supply of raw data, formulating your initiatives and executing and measuring those initiatives, you will be better able to identify opportunities and make more informed decisions that give your organization a competitive advantage.

  • DataOps: The Key for Real-Time Data Application Development

    DataOps: The Key for Real-Time Data Application Development

    Data engineers and developers face challenges every day to help their organizations digitally transform. To do this, they must deliver real-time data applications faster, better and cheaper. With businesses in every industry now data-driven, data professionals must work more efficiently and accelerate time to market for their products. That’s where DataOps comes in.

    Development of real-time data applications slows down when strategic business project teams are dependent on small, centralized engineering or IT teams for basic data operations such as exploring data or even building most data processing applications. Needlessly, engineering teams are notorious for being a bottleneck for strategic digital transformation initiatives. Rather, they can become known as strategic project enablers that provide the best tooling to their customers—instead of building it and being known as gatekeepers.

    Business users, outside of IT and engineering, need access to data but lack the necessary deep technical knowledge to work with the latest data technologies. DataOps brings people closer to data, helping organizations accelerate the delivery of their strategic projects, reduce costs and minimize the need for hard-to-source engineering skills.

    To accelerate digital transformation initiatives, teams and organizations should onboard new users directly onto a data platform quickly with the right tooling and resources required to get to work on their projects. DataOps empowers these business-focused users to work with data tools on their own to self-serve. A DataOps approach makes it easier for more people across an organization to work with its data, resulting in better alignment for data engineers and developers with their business and technical partners and improved data access.

    Serious new security, compliance and governance risks are introduced when less-technical users increase their interaction with technology, such as onboarding to a data platform. More people now understand that personal data has tremendous value. Initiatives such as the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) have brought the need to respect data privacy to the forefront. The stakes for safeguarding data are increasingly high and regulators are unforgiving—costing some organizations several billion dollars in fines, lawsuits and loss of market cap for some data breaches.

    When providing access to data, stringent management, control and real-time monitoring must be applied from a governance perspective. A surprise compliance auditor visit can result in questions such as which applications are reading credit card data, how an organization is keeping customer data safe, who has access to data and what are they doing with it. While locking down for compliance, you must help your business stay agile and avoid any slow down in data initiatives.

    Build vs. Buy to Streamline Operations

    In addition to adopting a DataOps approach, a key step is to onboard users to a data platform. This will optimize your data strategy and will help take the worry out of managing data projects, putting you on a path to faster execution and accelerated time to market.

    When onboarding users to a data platform, you need to choose between building your own or buying. It takes time and resources to develop your own system to onboard users and clients and can take years to build your own technology in-house. If your engineers are spending all their time working with open source technology to build tooling for onboarding, they’re not focusing on better ways to use data and get solutions to market fast—which is an inefficient approach.

    When your users are onboarded, the cycle begins again, as users request access to more data sets. More time at the command-line console means less time for your engineering teams to focus on higher-value work and strategic initiatives.

    Get a Leg Up on Competition With a DataOps Approach

    Buying a solution requires an upfront investment to support onboarding and, when combined with DataOps principles, is formidable to remain competitive. Together they enable support and manage the onboarding of clients to a data platform at scale and to maintain security and compliance. A DataOps approach fosters improved communication between data and business professionals to share data more effectively and to build a data mesh architecture.

    A Swedish digital bank, Avanza, needed to ensure the continuous improvement of the customer experience of its digital services to remain the leader in the Swedish market. Doing so required giving developers direct and constant access to real-time business data in Apache Kafka.

    The organization applied a DataOps approach and intelligent data masking rules to meet GDPR and protect customer confidentiality, featuring a large single data platform to address how it delivered data access and applied data governance to support more than 20 strategic projects. Using the platform, tenants who build streaming applications gained visibility into the health of their Kafka clusters and improved insight into data flows. This self-service approach means fewer support calls to data engineers, minimizing bottlenecks and freeing them up to work on more strategic initiatives.

    Data is the fuel that propels insights by tying back to analytics, decision-making and business systems. The better the shared data, the more it can be applied to supporting better business decisions, better customer experiences and execution of strategic business objectives and goals.

    DataOps also builds compliance and governance into your processes. It provides the visibility needed to understand who is accessing data and how to maintain governance and ensure people are acting ethically with the data they are enabled to access.

    The daily challenges faced in data-driven organizations only continue to grow. A DataOps approach helps data engineers and developers meet these challenges to digitally transform and build real-time data applications that help organizations gain a competitive business edge in their respective marketplaces.

  • Consider DataOps for a Competitive Edge

    Consider DataOps for a Competitive Edge

    DataOps seeks to eliminate existing barriers between people, technology, tools and data

    It’s no secret that COVID-19 has put the economy under enormous strain and future economic prospects are uncertain. A smart way to give your organization a leg up against competitors is DataOps. Adopting a DataOps culture makes the best use of your data—it helps to optimize the data powering your business because it successfully monetizes and systematizes your approach to working with data. It will help to get your new products to market faster and also accelerates business agility.

    For example, a DataOps approach can help a data scientist who has spent weeks developing a new machine learning model that is complete but not ready to go into production. She will need to set up data stream monitoring and other basic requirements before she can put it in place.

    This type of challenge is routinely faced by DevOps teams and the best practices they implement help avoid potential setbacks to quickly put their hard work and innovation to use. Applying the same development practices to the deployment of data-intensive services and applications significantly speeds up project completion times.

    Why DataOps?

    DataOps puts the principles and powerful DevOps practices such as automation, continuous delivery and quick feedback cycles to work in a data environment. Its strategic focus is to bring people closer to data, by delivering proven tools that deliver data to people with big-picture business knowledge. DataOps also removes complicated engineering and IT processes that stand between data and those with business knowledge to give projects an important and meaningful business context that greatly improves outcomes.

    The goals of a DataOps approach are to eliminate the friction that’s holding business stakeholders, developers and data scientists back and to increase project productivity and business outcome success. Automating and streamlining processes help people to quickly and easily move from ideas to development to production without lag time.

    DataOps also helps take concepts into production by fitting data projects into familiar, standard CI/CD development workflows and plugging them into any existing framework to deploy applications.

    How to Put DataOps to Work With GitOps

    GitOps is becoming a core component of DataOps practices and puts them into action. It allows data engineers to define data processing applications and pipelines as configuration and manage them through Git and mature CI/CD pipelines. This brings with it higher levels of standardization, security, governance and, as a result, an accelerated rate of delivery. Adopting GitOps in the data management world can pose problems for some organizations working with powerful and fast-changing technologies such as Apache Kafka and Kubernetes, but the automated processes it brings turbo-charge delivery time.

    Cloud migration is another way in which a GitOps approach makes a difference for organizations that want to take advantage of managed services in Microsoft Azure, AWS or Managed Streaming for Apache Kafka (MSK). GitOps helps simplify migration of Kafka clusters used for streaming to the cloud service of an organization’s choice—simply define the desired state, step back and GitOps takes over.

    GitOps’ pull-based approach to deployment, in which an operator watches Git and applies the desired state, creates several benefits. One benefit is not needing to poke holes in production environments to provide push-based actions from software such as Jenkins, which has numerous common vulnerabilities and exposures (CVEs). With a GitOps pull-based approach, attack surfaces are reduced, enabling solution migration across technology and clouds.

    A Better Process Results in Better and Faster Outcome

    Real-time business is here to stay and there is huge pressure on businesses to change fast. As the saying goes, “You snooze, you lose.” With the right tools and strategy backed by a good communication plan, DataOps will work to your advantage, bringing the benefits of a DevOps culture.

    Faster time to market via increased productivity is one advantage of a DataOps approach. Putting people closer to data, together with the right tools, helps you empower business users and developers so they can add new features and capabilities to any deliverable, and deploy and monitor it quickly.

    When working at a faster clip, don’t overlook the importance of governance and security. Build governance into the DataOps process to ensure deployment of the correct applications in the right way. And, maintain business continuity without disruption to existing infrastructure and processes.

    DataOps seeks to eliminate existing barriers between people, technology, tools and data. By bringing business and technology imperatives closer together, you gain invaluable context and a better ability to focus on driving strategic business outcomes that matter.

  • Hasura Launches Beta of GraphQL-Based Remote Joins Tool

    Hasura Launches Beta of GraphQL-Based Remote Joins Tool

    Hasura today announced the public beta of Remote Joins, which makes use of an open source GraphQL query language for application programming interfaces (API) to join disparate data sources residing in different repositories.

    Company CEO Tanmai Gopal said Remote Joins extends a GraphQL engine the company has developed around a project that is now hosted by the GraphQL Foundation, an arm of the Linux Foundation. The GraphQL Foundation counts among its members Airbnb, Amazon Web Services (AWS), Apollo, Coursera, Elementl, Facebook, GitHub, IBM, Intuit Hasura, Paypal, Prisma, Shopify and Twitter.

    Originally developed by Facebook, GraphQL has emerged as the basis for a more streamlined alternative to REST APIs. GraphQL provides a complete description of the data made available via an API and is organized in terms of types and fields rather than traditional endpoints. In addition to making it more clear what data is available via that API, developers can employ types to avoid writing manual parsing code.

    Hasura is now taking that concept a step further by enabling IT organizations to join data residing in different databases and microservices to enable that data to be queried as if it were a single data source. The core Hasura engine automatically generates the GraphQL engine, which enables those remote joins to be created in a few minutes.

    Gopal said joining data across a complex portfolio of modern applications based on microservices and legacy monolithic applications is becoming increasingly difficult as the number of microservices in an IT environment continues to proliferate. The Hasura engine addresses that issue by providing data access and flow tool base on a GraphQL type system for managing APIs that can be applied to GraphQL APIs as well as REST APIs or other protocols, he said.

    The join, authorization logic and consistency checks are all managed by the Hasura engine. Hasura also provides data change management, observability, auditing, security and performance management capabilities. That approach makes it possible for data sources and APIs to evolve independently while still allowing organizations to query data flexibly, noted Gopal.

    As DevOps and DataOps processes continue to converge, IT organizations will need to find a way to advance application and API development in a way that doesn’t result in data access becoming a major bottleneck. Too many DevOps teams are rapidly building applications and APIs only to find themselves waiting to be given access to data. GraphQL could provide a foundation on which IT teams can manage DevOps and DataOps processes more collaboratively.

    It’s not likely IT teams will be replacing existing REST APIs wholesale overnight. However, it’s also apparent that REST APIs may not enable IT environments to scale as efficiently as GraphQL, which is the primary reason why Facebook developed it. Of course, it’s still early days as far adoption of GraphQL is concerned. However, with more IT teams under pressure to innovate faster, interest in a more streamlined approach to building and managing APIs is likely to gain additional momentum.

  • DataOps and DevOps: A Match Made in Heaven

    DataOps and DevOps: A Match Made in Heaven

    The adoption of IT technologies has increased exponentially over the last few decades. Nowadays, the need to create quality web applications while simultaneously managing their extensive data effectively is a major concern for any organization. As such, there have been continuous efforts to come up with better approaches to software development, data analysis and data management.

    In order to meet the needs of today’s internet users, technology companies have intensified their efforts and are increasingly making use of all their available resources to find progressively efficient ways to create web applications within a shorter time-space and with superior quality, and to find more effective ways to integrate data-driven decision-making.

    In order to accomplish these feats when it comes to application development as well as data pipelining management, most of the larger organizations are now following a set of practices, DevOps and DataOps, respectively. Within this environment, developers must create applications that can function across various hosted providers, each with its own specific administrators and requirements. 

    This can only be accomplished by looking past the traditional storage capabilities. Instead, developers and organizations they work for are creating an underlying data fabric that facilitates what they’re trying to do, and any cloud provider they want to do it with.

    An Introduction to DevOps and DataOps

    DevOps and DataOps are a set of practices that relates to software development and data management, respectively. 

    These practices can be applied to a single project or can be followed at an organizational level. DevOps emphasizes the automation of the processes of the different teams (software development team, quality assurance team and IT operational team) so that they can easily complete their relative tasks (i.e., design, code, create, check, deliver, deploy, operate and monitor).

    DataOps, on the other hand, can be described as DevOps that includes data analysis, which empowers businesses to create processes to meet the needs of users throughout any data usage life cycle. 

    DevOps emphasizes the cooperation of the software development team (including testers) with system administrators (operational teams). In a similar way, DataOps relies on the cooperation of teams of data engineers, data scientists and other data practitioners who manage data-related processes (i.e., data storage, data processing, data decision-making, data archiving or data delivery).

    The concept underscores both communication and collaboration between the different teams within the same organization, ultimately removing all obstacles for seamless integration between the practices and the end product, including cross-device compatibility. 

    The combined team can make the project more visible and deliver faster results. It also increases the morale of the company by showing that, when it comes to project management, people will still be above processes and resources.

    Benefits of DevOps

    The benefits of DevOps include:

    • Transparency — By regular communication and coordination with the development team, the QA team and the IT operations team will be updated with the latest project details and will be more aware of the processes of other departments.
    • Quicker and high-quality deployment cycles — On any great DevOps platform, the pace of deployment will be improved by applying DevOps practices. The scope of the teams will be limited (i.e., for a small task/module) for a deployment period, as such their efficiencies will increase.
    • Faster bug fixes — Because tasks are smaller and continually distributed on the appropriate servers, the development team can receive input very easily from the QA team, allowing the development team to provide much faster resolution of any non-working components. Application stability would be a by-product of this method, because the cause behind any bug can be established very easily.
    • Flexibility — Because agile methodology insists on managing the raised change request, teams aligned with DevOps will be able to manage any unplanned work more effectively.

    Benefits of DataOps

    Now, let’s take a look at the benefits of DataOps:

    • Faster data processing — When different teams cooperate and interact on a daily basis and work on micro databases rather than on all at once, they not only avoid data overwhelm, they also accomplish much faster processing of data at all the different stages. 
    • Quicker problem solution — Similarly to DevOps, businesses can benefit from DataOps in multiple ways. DataOps data is processed in batches as this helps teams to recognize issues related to a batch of data faster as the emphasis is limited.
    • Better data security — We can protect our data in the cloud by using a range of safety features such as two-factor authentication and virtual private networks, the best of which provide a greater level of security than is available merely through your ISP. As data security professionals are in constant communication with data integrators and data analysts in a DataOps scenario, data security at each process level is made easier.
    • Improved resource output — When the team conforms to the basic principles of DataOps, seamless communication and transparency between the different teams will help the performance of each team member and lead to better resource output.

    The Lessons We Can Learn from eharmony

    The eharmony membership pool consists of singles in the U.S. as well as more than 200 countries around the world, making up about 5 million visitors per month.

    The company is continuously bringing out new features such as their most recent one-on-one video communication feature aimed at helping their members to maintain meaningful connections in the current era of social distancing. But the company has come a long way since its debut in 2000, learning very valuable lessons along the way.

    Overcoming Several Challenges

    eharmony is leading the online dating industry through innovation by continuously improving its two key applications — Singles and Matching. But they’ve had to make significant changes on their journey to remain at the top. Their challenges included:

    • The IT team needed a considerable amount of time to refresh the database, requiring the involvement of DBAs, storage specialists and network administrators.
    • The labor-intensive process of creating, scanning and storing database snapshots made it difficult to create test datasets for new releases. As a result, QA often carried out tests using stale data. 
    • Database improvements triggered development freezes, which hindered the team’s ability to deliver new features and functionality.
    • Their data volumes were increasing constantly, with several Oracle databases requiring between 20 to 70 TB of storage.

    Combining DevOps and DataOps

    eharmony has also implemented a self-service data management solution called Delphix, which enables the individual teams to develop and update environments as needed.

    Today, their refreshes can be completed within an hour, providing accurate and relevant information to development teams on demand. 

    Other solutions included:

    • They have saved over five thousand person-hours in just three years by automating database provisioning, refresh and rollback tasks. In the process, this has also saved the company over $3 million in operational costs.
    • Through compression and virtualization, eharmony has reduced their database storage footprint to a fraction of the original size, reducing their storage requirements by almost 80%.
    • As development teams adopt agile methods and more frequent releases, demand for non-production environments is expected to increase, further increasing storage cost savings. eharmony has also eliminated data related defects by providing accurate, up-to-date test data. Unrestricted access to test environments significantly improves the ability of QA teams to identify issues before they reach production.

    Why DataOps and DevOps Are a Match Made in Heaven

    It is easy to see why DataOps and DevOps go together so well. DevOps calls for continuous software monitoring, testing and deployment. DataOps are inherently modular because they are designed to execute a single function. Software that is scalable and easily fits into the DevOps framework.

    Using DataOps in the DevOps process will help to fully recognize the benefits of continuous integration by validating the implementation promised by DevOps. These tools can be combined to optimize computing in a way that should be used during development whenever possible.

  • How to Get Your Digital Data Game on with DataOps

    How to Get Your Digital Data Game on with DataOps

    As a new mindset, methodology and practice within data management, DataOps focuses on improving the communication, integration and automation of data flows to simply and successfully help developers, IT and the business create real-time data experiences and reduce the risk of project failure. DataOps enables developers and organizations to better adapt to fast changing business requirements to become data driven. 

    DataOps, like DevOps, aims to provide continuous delivery of new capabilities with higher quality, but it’s focus is on data rather than putting technology and application development first. A DataOps approach ensures data pipelines are deployed, SLAs are met and that data versioning, compliance and auditing are in place. It also helps improve visibility into the quality of data. 

    How Can Enterprises Maximize the Value of the Digital Data They’re Collecting? 

    Organizations need to move quickly, and the faster knowledge is in the hands of those with domain expertise, the higher the value.

    Enterprises need to react quickly to events, both digital and physical, because the rate at which data value perishes increases with time. The key to understanding and interpreting this stream of data is to be able to share and disseminate the information to those with the business and domain expertise, without the typical time-consuming back and forths. If we take the current coronavirus outbreak, we can use technology to track and correlate but if we can’t share it with the medical professionals in a usable real-time form, we lose.

    The way Babylon Health does this is a great example of bringing in DataOps practices that reduce the turnaround time in delivering healthcare data to medical professionals and patients.  

    They have adopted DataOps. All their data personas (engineers, analysts, etc.) have different needs but use the data integration platform in the same way. For them, DataOps is a big reason why they have been able to expand and offer so many great AI healthcare data products in many countries so quickly.

    What’s the Best Way to Build an Effective Data Optimization Strategy? 

    Organizations need to provide developers with tools that bind data technologies, increase data literacy, promote and incentivize data sharing, while ensuring it comes with transparency to protect data ethics.

    Data optimization comes from knowledge and adoption of DataOps practices. When companies deliver data to those with domain knowledge and cut out the complex IT and engineering processes—currently in between the data and that knowledge—data projects succeed. 

    This practice ensures that focus remains on the business and its needs, with technology as an enabler. 

    Organizations must also stop thinking that one data solution fits all. Instead, they should use a data mesh leveraging best-of-breed data technologies and look for a solution that makes those technologies accessible to anyone, not an elite few.

    Increasing data literacy and working with a common language, such as SQL, is a key best practice. For example, Airbnb has been very successful increasing its line of business employee data literacy through its Data University courses. When business partners and developers leverage SQL to solve their problems it reduces barriers to optimizing data. 

    Organizations must also drive a mindset of data sharing by including measurement of teams and individual performance on how much data they share across their organization. When teams and people know data sharing is part of their performance evaluation they are more likely to take steps and adopt practices needed to be effective in their data sharing.

    However, with increased democratization of data, and a higher data-sharing policy, comes the need for greater transparency to ensure data ethics are upheld. Look for tools that increase observability to uphold data ethics. Improved transparency into who is doing what with the data also enhances accountability.

    What Are the Leading Data Optimization Challenges? 

    Organizations struggle to provide timely access to data with governance, with developer productivity and getting applications that process data to production.

    A big contributor to this is that many data platforms have a number of open source technologies. The industry has not accounted for how these technologies can be easily integrated into an organization. Platform teams spend too much effort integrating these individual technologies—when solutions exist that allow them to instead focus on delivering products to market.

    This wasted time and inefficient use of talent exacerbates the existing skills shortage.

    A further challenge is changing mindsets. How can organizations ensure and encourage developers and engineers to select technologies that best support business initiatives and not technology for technology’s sake? 

    How Can IT Leaders Bring Their Staff Up to Speed on the Latest Data Optimization Technologies and Practices? 

    We are now in an era of commoditized technology. Previously complex and hard to manage open source projects are now offered as managed services by all major clouds, along with every incarnation of data store possible. Importantly, they also provide features an organization needs in order to be successful. For example, encryption and secret management, not to mention the compute resources to deal with the ever increasing data workloads.

    Tech intensity is upon us. Let’s not recreate technology that has been commoditized, and focus instead on building data intensity with a real-time DataOps approach. 

    The question is not about upskilling developers and IT, but about enabling business partners to be more data literate so they can self serve.

    Developers can help by working to build a culture focused on what the business needs and switching the mindset from the line of code to the daily, weekly, monthly outcome to deliver bottom line improvements to the business.

    People perform better from a business perspective when they care about the product the business delivers versus what framework they use with solutions that are ill suited to business needs.

    Developers should also measure their success. Having objectives and key results (OKRs) is a great framework to establish goals and a roadmap to achieve them, but they must be aligned with the OKRs of the business. Organizations should move away from 360 reviews because they are fraught with irrelevance to business goals and political intent.

    A better approach is to educate developers on the goals of the business, then the department and the team. Make them feel part of the machinery, give them the chance to have a say and get involved to yield better outcomes.

  • 5 Ways Your Business Can Benefit From DataOps

    5 Ways Your Business Can Benefit From DataOps

    The amount of data that businesses generate and collect continues to soar, and enterprises that want to accelerate their end-to-end processes and obtain business insight can’t keep using the manual data management processes they have relied on for decades. And the situation will only get worse in the coming years. IDC expects the amount of data to grow at a 32% CAGR to 180 zettabytes by the year 2025. Fortunately, DataOps can help.

    DataOps is a relatively new and still-maturing discipline that arose about five years ago. It’s built on the foundations created by agile development and DevOps, and the goal is to bring similar principles to data analytics and data science to improve data quality and reduce the time required for obtaining actionable business intelligence.

    Over the next few years, enterprises that resist adopting DataOps will spend even more time reacting to data errors and broken manual processes, and fall even further behind in their ability to provide timely, accurate information to business leaders. Meanwhile, those that embrace DataOps will create streamlined, automated data pipelines that enable data administrators and data scientists to optimize business processes, focus on higher-value tasks and support decision makers with the best possible intelligence.

    In this article, we’ll review the principles and power of DataOps and five specific ways every organization can benefit from it.

    What Is DataOps?

    Because of the massive size of datasets, performing data analytics today requires automation to be able to do validity testing, analyze the activity and behavior of the data pipeline itself, and detect anomalies and outliers that can indicate a quality issue with the data or the pipeline. Based on the efficacy of agile development and DevOps principles to deliver high-quality results orders of magnitude faster than previous approaches, relevant principles and best practices were borrowed and codified in the 18 principles of the DataOps Manifesto to enable the same types of automation and quality-detection capabilities within the data pipeline.

    A foundational concept of DataOps is that “analytics is code,” which means that everything–collection routines, ETL routines, even the analysis routines that drive metadata and business intelligence-level summaries–needs to be modular, automated and easily and instantly repeatable. As businesses work toward this goal, operational benefits will accrue, such as the ability to receive real-time alerts on streaming data, so quality issues can be resolved before bad data gets propagated or has a chance to impact decision making.

    The Power of DataOps

    DataOps is already enabling businesses to transform their data management and data analytics processes. For example, like DevOps, DataOps lets teams easily spin up isolated, safe and disposable testing environments that allow them to experiment and innovate (Principle 12 of the Manifesto). However, while developers typically focus on applications with small test databases, data analysts and scientists may need to spin up a sandbox environment that includes applications along with terabytes or even hundreds of terabytes of data. By easily implementing intelligent DataOps strategies such as automation, cloning, predictive analytics and more, spinning up massive disposable data environments becomes possible.

    DataOps principles are also enabling businesses to act on their massive production datasets in ways that were unimaginable just a few years ago. For example, DreamWorks can now easily share the datasets of its films in development with teams of creative artists around the world, enabling rapid collaboration and dramatically shortening production times. Another example is the genomics company, WuXi NextCODE, which has developed a genome platform that can compare human DNA–millions of bits of data–and integrate the data on the fly to explore the differences or mutations that may cause cancer or rare diseases.

    There are also many financial services companies, in America, EMEA and APAC, all leveraging NetApp for moving to a hybrid cloud data pipeline model. The hybrid model allows them to maintain compliance for protected and sensitive data, while taking advantage of cloud cost savings for non-sensitive data and applicaation components. This allows for a strategy that delivers both regulatory compliance and the flexibility of a hybrid data pipeline architecture.

    However, you don’t have to be a film studio, genomics company or financial services firm to benefit from DataOps. Every company that needs to obtain timely actionable business intelligence will benefit in five critical ways.

    Top 5 Benefits of DataOps for Any Business

    1. Reducing toil: Similar to DevOps, DataOps is fundamentally about process-oriented methodologies and automation, which dramatically increases the efficiency of workers. By baking intelligent testing and observation mechanisms into the analytics pipeline, teams can stay focused on strategic tasks instead of poring over spreadsheets looking for anomalies.
    2. Better-quality data: Creating automated, repeatable processes, along with automatic code checks and controlled rollouts, reduces the chance that any type of human error will get distributed to multiple servers and take down the network or produce erroneous results.
    3. Faster access to actionable intelligence: Reducing toil and improving data quality leads directly to faster access to actionable business intelligence. Automated ingestion, processing and summary analytics on incoming data streams, combined with the elimination of errors, can deliver insights into customer behavior patterns, market shifts, price fluctuations, etc.–instantly instead of hours, days or even weeks later.
    4. Seeing a bigger picture of dataflow: Beyond the business-critical day-to-day insights, DataOps can provide an aggregated view over time of the entire dataflow, across the organization and out to end users. This can reveal macro trends, such as adoption rates of features or services, or search pattern deltas over time. Even behavioral or geographic patterns for focused or global data sets. Creating such a view would never be possible for teams that are constantly reacting to anomalies and errors with manual processes.
    5. Career enhancement: Data analytics and operations professionals who learn how to implement and manage DataOps processes will enjoy career benefits as they become the leaders of the next generation of data teams and set the standard for data practices for at least the next 10 years. The business also benefits from increased employee satisfaction and retention as repetitive and monotonous processes are replaced by an exciting and fast-moving organization focused on innovation.

    Over the next five years, DataOps will become mainstream in the same way that DevOps did. The benefits are too compelling, and the consequences of ignoring it are too dire. However, as companies progress on the DataOps journey and find success in leveraging its principles to drive business intelligence and streamline processes involving large datasets, they will eventually confront the limitation of their infrastructures. As a result, they will need trusted technology partners that can help them ensure data replication, data distribution and data availability at a scale that dwarf’s today’s requirements.

  • GigaSpaces Embraces Kubernetes Alternative for DataOps

    GigaSpaces Embraces Kubernetes Alternative for DataOps

    GigaSpaces has added a DataOps framework to its in-memory computing platform that provides an alternative to Kubernetes by employing Nomad software from HashiCorp as a scheduler.

    Yoav Einav, vice president of product for GigaSpaces, said many enterprise IT organizations aren’t in a position to adopt Kubernetes, yet they still need a means to embrace best DataOps practices to process massive amounts of data in real-time.

    Version 15.2 of the company’s namesake platform adds a GigaOps Stack module based on ElasticGrid, an alternative to Kubernetes; and Ops Manager; gsctl; and a monitoring tool. GigaSpaces already natively supports Kubernetes, but Einav said there is a significant number of IT teams that don’t have the skills required to deploy and manage Kubernetes clusters.

    DataOps has emerged as a corollary to DevOps in that it defines a set of practices for automating the provisioning of storage systems that makes data readily accessible to applications. The need to automate those processes has become more acute as organizations build and deploy applications infused with machine learning algorithms, noted Einav.

    ElasticGrid is designed to provide a more declarative alternative to achieve that goal without having to learn how to programmatically spin up Kubernetes clusters, Einav said. That approach makes ElasticGrid more accessible to the average IT administrator, he added.

    In addition to being able to support both virtual machines and containers, Einav said ElasticGrid is approximately 20% faster than Kubernetes.

    Other capabilities added to the latest release of GigaSpaces include a stream iterator optimization tool that increases throughput for read-intensive workloads by a factor of two and support for memory-level encryption and compression per property to better adhere to data privacy regulations.

    GigaSpaces is making the case for an in-memory computing platform that is optimized to run applications in near real-time. As organizations embrace a wide range of digital business applications, legacy IT infrastructure that was optimized to process applications in batch mode needs to be updated to support applications that need to process data in memory to support applications that are processing data in near real-time.

    It’s not clear to what degree organizations will be investing in new platforms during the economic downturn brought on by the COVID-19 pandemic. There is no doubt some organizations will pull back on those investments. However, a significant number of organizations are also accelerating the rate at which they are building digital business applications to drive a newly minted business continuity strategy. Recognizing the need for more resilient applications that can be accessed by customers and suppliers from anywhere, many organizations are accelerating the deadlines for delivering many of these applications in the wake of the pandemic.

    Of course, there are multiple options available when it comes to deploying an in-memory computing platform to support these applications. The challenge many organizations will encounter, however, is finding an in-memory computing platform that also addresses the inherent challenges associated with moving large amounts of data in and out of the platform on-demand.

  • DataOps and Beyond: How DevOps Methodology Transformed Our Approach to Data Science

    DataOps and Beyond: How DevOps Methodology Transformed Our Approach to Data Science

    The DevOps methodology has become synonymous with forward technical thinking–a workplace culture that reinforces best cultural practice and promotes more, better quality output by synchronizing the functions of development and operations teams. As a testament to its popularity, the term has expanded in reach to refer to a broader set of transformational software-related practices. The list continues to grow: DevTestOps, DevSecOps, GitOps, DataOps. 

    In 2020, organizations are drawing upon thousands—if not tens of thousands—of data sources, which exist in hundreds of different formats and fall under unique sets of restrictions and compliance regulations. Figuring out where all this data comes from, what it means and how to ensure that the right teams can access it when they need to, are common data management challenges. To address these challenges, some solutions were borrowed from effective practices in software development; automating analytics for improved quality, and allowing data scientists to shift away from time-consuming tasks to focus on value-adding projects instead. 

    What happens when software development and deployment methods are applied to processing and integrating data? Does that mean that DataOps is exactly like DevOps, but for data? And what benefits do organizations expect from adopting this approach to data management?

    The xOps Mindset

    At their core, the ops practices are linked by fundamental characteristics that shape their goals and practices. Let’s explore some of these commonalities:

    • xOps ties together workflows from create to run: In an ideal environment, production and distribution don’t function as separate entities, but are synchronized to meet target metrics that can include hundreds, or thousands, of daily deployments. Culture changes have rewritten rules around internal communication and feedback loops in a development environment where the conditions or requirements for success are constantly changing.
    • xOps automates the mundane (non-creative) tasks: Automation not only increases an organization’s agility, but when executed correctly can eliminate communication gaps between teams—for instance, automating traceability and issue tracking within code changes. Within larger, more complex projects, automation can save on time and cost by executing and managing CI/CD—ensuring that software builds and changes reach their designated target environment with little to no human involvement.
    • xOps builds in best practice: Equally as important to the functions they perform, xOps tooling plays a key role in building in best practice from the top down and bottom up. Best practice might not look identical between organizations, but the common goal is delivering value for a range of stakeholders through continuous production, in an environment where it’s encouraged to learn through experimentation and to respond to problems as they occur.

    In short, DataOps loosely applies the DevOps process to data pipelines—using automation and Agile methodology to reduce the amount of time spent fixing issues in pipelines, and to get data science models into production quicker. There are also some obvious differences between DataOps and DevOps. While DevOps refers to the collaborative process between two technical teams, DataOps facilitates cooperation between data analysts, engineers, scientists and any members of an organization who use data. This makes DataOps a much more multifaceted process than its DevOps cousin. 

    xOps for the Big Data Age: Benefits of Running DataOps 

    We know that for many organizations, converting raw data into usable, accessible and relevant (that is, profitable) business intelligence has given rise to a plethora of data management issues. The sheer volume of most enterprise level datasets has simply outgrown many of the inadequate older systems and processes built to manage it, and we continue to produce new data at stunning rates. There is a general consensus among data scientists today that the amount of data in the world doubles every 12 months.  

    • Automation and Continuity: Within an organization, business data moves through a set process—akin to the software lifecycle for the DevOps analogy—where data is ingested in one form and exits in another. Engineers must build data pipelines, test them and change them before they are deployed. By adopting the standards for automation and best practices outlined above, you’ll ideally have a constant stream of data flowing through a pipeline. This unlocks the potential to obtain real-time insights from data, shortening the length of time it takes to turn raw data into valuable business information.
    • Cultural Shift: When completed correctly, DataOps changes who within an organization becomes a data user. A higher level of transparency and an assurance of quality means that data is more openly available to decision makers within non-technical teams. Consider the business value-add for a sales team who wants to expand into new markets and are able to access a wealth of customer data.
    • Machine Learning: We talked about the role of feedback loops in the xOps mindset. When machine learning modelling meets this mindset, you improve the quality of the model being produced because you’re unlocking a higher quality of data through version control, continuous integration and continuous deployment. Improved insights in machine learning offer nearly unlimited potential for extracting value from DataOps.

    — Hannah Leslie

  • Becoming a Data Company: Back to the Future

    Becoming a Data Company: Back to the Future

    Transformations are complex, and it’s often difficult to see the big picture when you’re in the middle of one. 

    The elegance of the end-state of harmonious synergy between software and data belie the complexity and chaos of the transition we find ourselves in the midst of. Not that long ago, there was a general balance between the pace and sophistication of software delivery and the speed and maturity of data analysis. Markets and customer bases grew steadily allowing organizations to keep pace in their software and data competencies to handle increasing mutual scope and complexity.  

    However, the consumerization of IT—the ability of lay consumers to consume sophisticated software experiences from browsers and mobile devices—unleashed access to a massive, eager and demanding customer base measured in the billions. This gave birth to the current wave of digital transformation. Along with it came the race to get to market, i.e. build and deliver high-quality software exponentially faster, in order to capitalize on massive global opportunity.

    Innovation thus started in the software domain. With faster, better software came the need for purpose-built data stores (document databases, graph databases, etc.), the need to manage the velocity, volume and variety of data collected from millions (or billions) of users and the need to extract insights and value fast enough to further fuel the pace of software innovation. Thus, software innovation planted the seeds for the current focus on data innovation.

    While the innovators quickly traversed this software journey, most enterprises are only now fully in the midst of embracing new software methodologies and technologies to deliver software quickly and effectively. Those enterprises at the vanguard of the software journey are now embarking on the data journey—to find ways to safely and quickly harness the value locked away in enterprise data assets. This includes everything from analytics to rapid provisioning of copies of production data for software testing. In fact, 451 Research states 86% of companies that consider themselves mature in DevOps journey are planning on increasing their investment in DataOps. That is, companies that have traversed the software competency curve are now embarking on the data competency curve.

    The Dawn of the CDO

    The growing traction by data science, AI, analytics and DataOps technologies and vendors point to the rising importance of developing a data competency. The recent emergence of the Chief Data Officer (CDO) is the culmination of this trend. 

    Given the nascent focus on data at most enterprises, most CDOs are still grappling with defining their role, navigating the dynamics of the C-Suite, defending against regulatory and compliance risk and justifying their role by identifying growth opportunities. Despite these immediate challenges, they do have one thing going for them: they know they’re in the midst of mind-bending change, have the benefit of hindsight on the software journey and the many parallels it carries, and are mentally prepared for the organizational, technological and cultural changes that will need to be navigated. Most companies were stunned by the sheer suddenness and velocity of change that came with the DevOps and the software transformation journey. With DataOps and the data journey, they are now aware, prepared and committed to the associated journey and changes for the long haul. They can be proactive and avoid being reactive.

    Planning for the DataOps Era

    As cycle times for going to market and learning from the markets shorten, and innovation cycles accelerate, CDOs can expect that data-centric organizations will need to drive net-new top and bottom-line results without incurring undue risk. This will come through adoption of new technologies, organizational structures, cultures and workflows. New roles focused on data-centric activities, and new forms of collaboration between different personas will emerge. 

    In the modern enterprise, production will no longer be the most important data environment in enterprises. There is a growing need for fast access to secure, governed data outside of production—where the work of the rest (non-IT part) of the enterprise is centered. What used to be non-prod for IT will now be prod for the rest of the enterprise. To succeed, IT will need to deliver data quickly and securely to the rest of the organization or even across organizational boundaries to partners and other ecosystem players. Embracing collaborative processes and workflows that support this new pattern of enterprise operation will be critical. Enter DataOps—the unique mix of people, process and technology approaches required across teams in an organization to become agile with data.

    Given the similarities outlined earlier of the data journey to the software journey, organizations’ DataOps journey can benefit from the lessons learned in their DevOps journey. For instance,  developing a data competency will require new layers of organization and workflow patterns supported by corresponding technologies. Proactively planning and laying down infrastructure for DataOps will yield far better results rather than adopting technologies ad hoc until a data competency blueprint emerges through trial and error. 

    Building a Data Company Technology Stack

    As with any complex shift, it makes sense to deconstruct data competency into a series of layered capabilities, each building on the one that came before, that together help develop a sophisticated competency. For example, when building their software competency, in order to apply software in sophisticated ways, (e.g., cloud native architectures and microservices), organizations needed to manage distributed software elements (e.g., scale, heal, monitor) which in turn relied on the ability to test and deploy incrementally, consistently and rapidly. 

    A similar approach can be applied to data. Building collaborative processes around appropriate technological tooling can yield layered component capabilities that collectively define a sophisticated data competency. While the layering was emergent in the software journey, CDOs have the opportunity to be deliberate rather than reactive in developing a layered approach as the diagram below illustrates. 

    At the foundation of a successful data competency lies the capability to deliver data anywhere in the organization and secure it to meet regulatory and compliance mandates. Being able to quickly secure and move any type of enterprise data across a hybrid cloud context is hard work, but also table stakes for basic data management hygiene. These are critical prerequisites and this foundation needs to be solid in order to support the layers above. 

    Next, there are various data management activities to master, such as the self-service access, governance, data catalogs, data cleansing, collaboration and versioning, etc. These define the next level of data competency and must be mastered before data can be applied in different ways, e.g. in machine learning, test data management, analytics, data science, etc. 

    A thoughtful, layered approach to building a data company requires building the right abstractions between layers and will avoid tool sprawl consisting of a mesh of ad hoc point solutions that inevitably require human glue to drive data workflows.  

    CDOs should evaluate their organization’s needs through such a structured lens to evaluate the organization’s needs. Each of the activities will bring with them ecosystems of vendors, engendering further evaluation of vendor capabilities. Critical questions to ask include (among many): Is the solution enterprise ready? Does it work with all your data sources across hybrid cloud? Will it evolve with your changing needs? How will it interoperate with other solutions? Can it be automated?

    As CDOs think through these infrastructure investments to connect different parts of the enterprise, the similarity to laying down railroad tracks to connect a large country are unmistakable and perhaps instructive: the resilience and stability of lower layers enable flexibility, variety and innovation at the higher levels just like the simplicity and reliability of simple railroad tracks enables running electric, steam engines, luxury bogies and cargo bogies alike. 

    Toward a Data Company

    The refrain of digital transformation has been that every company is a software company. But to complete the transformation, every company must also become a data company. Software development practices and tools have evolved significantly in the last decade. We’re in the early stages of a similar evolution in data practices and tools that will take us through the next decade. The symmetry and coupling between software and data is unmistakable and there is much to be learned from the software journey.

    As CDOs, CIOs and CEOs plan the transformation of their enterprises in this new era, they will need to consider building a foundation for their data competency that is stable, resilient, compatible with their software development processes and adaptable to inevitable change. They should be prepared to deliver their data from where it is produced to where it is used throughout the enterprise.

    The data company is an organization where data is treated not just as a byproduct of operations, but as the lifeblood of the organization—that drives software development, improves quality assurance, informs decision making and potentially generates revenue in its own right.

    — Karun Bakshi