devopsy.kinsta.cloud

Category: DataOps

  • Best of 2022: Agile/Scrum is a Failure – Here’s Why

    Best of 2022: Agile/Scrum is a Failure – Here’s Why

    As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    Welcome to The Long View—where we peruse the news and strip it to the essentials. Let’s work out what really matters.

    This week, something a bit different: I’m devoting my entire column to Agile and Scrum. They’re increasingly getting a bad reputation, being associated with the worst aspects of toxic workplace culture: Sexism, racism, burnout, micromanagement. (more…)

  • Cisco Unveils 800G Networking Platform to Advance DataOps

    Cisco Unveils 800G Networking Platform to Advance DataOps

    At the Open Compute Project (OCP) Global Summit conference, Cisco announced it developed an 800-gigabit switch that consumes significantly less power than the previous generation of its networking equipment.

    Thomas Scheibe, vice president of product management for cloud networking for Cisco’s Nexus and ACI product line, said the throughput provided by the latest 7-nanometer iteration of Cisco One ASIC processors is primarily needed by organizations processing massive amounts of data to train artificial intelligence (AI) models either in a local data center or in the cloud. The challenge is that many of the organizations are looking to build AI models that process massive amounts of data while simultaneously reducing their IT infrastructure’s carbon footprint, he noted.

    Cisco has been able to achieve that goal by continuing to invest in proprietary ASIC processors; they are at the core of its networking portfolio to improve data operations (DataOps), said Scheibe. Many IT teams are also looking to replace legacy network routers and switches that consume more power and reduce energy costs by as much as 77%, added Scheibe. In terms of climate impact, Cisco claimed its 8111-32EH switch can now provide 25.6Tbps at 15% of the power requirement. Cisco projected that switch will save about 10,000 kg CO2e per year compared to a 12.8Tbps switch. That equates to the greenhouse gas emissions generated by an average gasoline-powered passenger vehicle driving 26,155 miles, according to Cisco.

    IT teams also have the option of deploying either Cisco’s network operating system or SONiC—Software for Open Networking in the Cloud—network operating system (NOS).

    Cisco is also providing IT organizations with the option to configure its latest routers and switches to run at either 800, 400 or 100 Gigabits with an eye toward upgrading throughput sometime in the future.

    In general, DataOps as an IT discipline is maturing as organizations realize they need to implement best practices to optimize data flows across their organization. Cisco is giving IT teams the option of employing a standard Ethernet fabric or an alternative fabric that increases throughput by predicting which packets need to be delivered to a specific location based on their attributes and the behavior of previous network traffic. As the volume of data that needs to be accessed by low-latency applications continues to expand, DataOps will become a more critical IT discipline. Historically, storage administrators tended to be responsible for data management, but as these applications continue to proliferate, a new class of DataOps engineers is emerging to optimize the flow of data across distributed computing environments.

    In the longer term, it’s not clear how much carbon dioxide emissions are factoring into IT decisions these days, but there are organizations that have begun to track it as part of an effort to lower their carbon emissions. Most cloud service providers have also committed to being carbon neutral. Cisco is betting achieving that goal will require IT infrastructure upgrades at a time when the amount of data that needs to be processed will only continue to exponentially increase.

  • 5 Ways to Transform DataOps With Human-in-the-Loop Automation

    5 Ways to Transform DataOps With Human-in-the-Loop Automation

    We are in the middle of a data renaissance. Today, it’s not just about data instrumentation but also learning how to make DataOps a real business advantage for the entire organization. Data has become an inextricable part of products, used to enhance the quality of user experiences. Reliable access to data and the integrity of that data is imperative to drive innovation and business success.

    But just like any process, problems will arise. There will be pipelines that break or data that isn’t instrumented properly. The question then becomes, how can teams more quickly identify problems and continually improve? How can the journey toward data integrity become one of iteration and improvement?

    DataOps teams are by no means strangers to automation—automated CI/CD pipelines and infrastructure-as-code (IaC) are a huge part of their skill set. But there are some components of automation that are less pervasive throughout DataOps, ones that can help promote data integrity and encourage a culture of improvement. 

    It all starts with a different approach to automation—a human-in-the-loop (HITL) approach. Human-in-the-loop automation enables parts of a process to be fully automated while enabling humans to step in at critical points to take action, make decisions and decide the path forward.

    Here are five ways to use human-in-the-loop automation to drive continuous improvement and achieve optimal data integrity.

    Create Better, Earlier Interruption of Problems

    As is life, inevitably things break, issues arise and teams need to step in to remediate. The goal should be to enable better interruption of these problems when they do occur—identifying issues earlier and bringing the right stakeholders together more easily and swiftly.

    HITL automation can help teams not just identify issues earlier but kickstart the incident response and coordinate across teams and stakeholders. Ideally, automated workflows can be triggered from an incoming signal; for instance, from an observability tool like DataDog or BigPanda. An incident response could be started automatically, triggering further automation—like creating tickets, starting a Zoom meeting, creating a Slack channel and bringing together the right team members. 

    By instantly bringing the right people together, the team can investigate collaboratively and more quickly understand the state of the issue. HITL automation should then enable teams to run further scripts to take action and remediate as needed.

    Bake Improvement Into the Process Through Self-Documenting Workflow

    Many companies today are asking, “How do we make sure that we have that full integrated loop, down to the collection point?” Communication between teams is critical to this endeavor. But even before teams can collaborate on improvements, they need to understand the current state of their data so that if, for instance, a team finds that a report isn’t providing what’s needed, there’s a really clean way to get that communication back to the DataOps team to improve the instrumentation in the first place.

    HITL automation can bake improvement into DataOps workflows by automatically documenting every human and machine action throughout a process. Because it’s connected through APIs to all a team’s tools and services, a self-documenting workflow slurps up every action and creates an audit trail. This audit trail is the basis by which teams can more accurately understand not just the integrity of their data, but also how teams are implementing and running processes.

    Bring Human Data to the Forefront of Improvement

    DataOps teams play a big role in making sure the right data is being collected. However, a lot of people are still using systems that are very manual. They may have a good understanding of their log and monitoring data but still don’t have a good sense of the human data around their organization—what processes and workflows people are running.

    Understanding how humans run processes is just as important as understanding technology problems. The self-documenting workflow described above should include documentation of human data, from manually-run scripts to Slack or Teams conversations and, ideally, even auto-update tickets so even maintaining a system of record becomes less manual. Only by looking at the full picture—of human and machine data together—can teams find areas of improvement in their automation, communication and workflows.

    Take an Incremental Approach

    Don’t let “perfect” be the enemy of “done”. Every great transformation I’ve seen started with a small team, a great mandate, and strong support from the executive team. Adding tons of new technology and hiring more people may not be the silver bullet teams need.

    Instead, teams should look at the expertise they have in-house first—the people, technology and processes. Then, begin by taking an incremental approach. HITL promotes incremental, approachable automation by enabling teams to first codify institutional knowledge, analyze, learn and then begin to automate pieces in small batches.

    Diving head-first into a fully automated approach can leave teams in a worse state than nothing at all. Automation is a journey—we learn by doing, by recording human and machine data and then, using insights as our guide, begin to find real value in small doses of automation. These small steps can make a huge impact and, over time, add up to substantial business value.

    Buy for Industry Standards and Build for the Gaps

    As I mentioned above, teams should first start by evaluating their current resources. What skill sets and technology do they already have and how can they use those resources to achieve their data goals? At the same time, many teams starting their automation journey face challenges—lack of development resources, scripts that live on one person’s machine, difficulty bringing all the human and machine data together into a recorded timeline.

    To help data teams effectively and reliably implement HITL automation, investment into an off-the-shelf platform can help bridge the gaps teams and organizations have. Platforms should enable automation through low-code interfaces while providing code-level customization. This combination enables teams to buy for industry standards while building for the gaps—not managing the platform means development resources can be focused on expanding and improving automation.

    Platforms should enable teams to easily connect to APIs and adapt to custom APIs. The continuous change of DevOps processes and tooling coexist with rapidly changing APIs. Therefore, the most promising automation platforms must manage the complexity of APIs, enabling users to focus on the intent, not the mechanics of custom APIs. Automation technologies should seamlessly handle identity, authentication, pagination and caching. 

    Find Leverage With HITL Automation and Drive Business Impact

    As data becomes increasingly ingrained in product life cycles and user experience, automation holds the key to faster innovation, product quality and enhanced reliability. Creating a culture of continuous improvement and implementing technology like HITL automation that promotes data integrity throughout the product life cycle is where I’ve seen the greatest leverage for organizations. 

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2022 – October 24-28, 2022.

  • Where DataOps and Opportunities Converge

    Where DataOps and Opportunities Converge

    In today’s data age, getting data analytics right is more essential than ever. A robust data analytics implementation enables businesses to hit key performance metrics, build data and AI-driven customer experiences (think ‘personalize my feed’) and capture operational issues before they spiral out of control. The list of competitive advantages goes on, but the bottom line is that many organizations successfully compete based on how effectively their data-driven insights inform their decision-making.

    Unfortunately, implementing an effective data analytics platform is challenging due to orchestration (DAG alert!), modeling (more DAGs!), cost control (Who left this instance running all weekend?!) and fast-moving data landscapes (data mesh, data fabric, data lakehouses …). Enter DataOps. Recognizing modern data challenges, organizations are adopting DataOps to help them handle enterprise-level datasets, improve data quality, build more trust in their data and exercise greater control over their data storage processes.

    What is DataOps?

    DataOps is an integrated and agile process-oriented methodology that helps businesses develop and deliver effective analytics deployments. It aims to improve the management of data throughout the organization. 

    While there are multiple definitions of DataOps, below are common attributes that encompass the concept while going beyond data engineering. Here’s how we define it:

    We broadly define DataOps as a culmination of processes (e.g., data ingestion), practices (e.g., automation of data processes), frameworks (e.g., enabling technologies like AI) and technologies (e.g., a data pipeline tool) that help organizations to plan, build and manage distributed and complex data architectures. DataOps includes management, communication, integration and development of data analytics solutions, such as dashboards, reports, machine learning models and self-service analytics.

    Why DataOps?

    DataOps is attractive because it eliminates the silos between data, software development and DevOps teams. The very promise of DataOps encourages line-of-business stakeholders to coordinate with data analysts, data scientists and data engineers. Via traditional agile and DevOps methodologies, DataOps ensures that data management aligns with business goals. Consider an organization endeavoring to increase the conversion rate of their sales leads. In this example, DataOps can make a difference by creating an infrastructure that provides real-time insights to the marketing team, which can help the team to convert more leads. Additionally, an Agile methodology can be employed for data governance, where you can use iterative development to develop a data warehouse. Lastly, it can help data science teams use continuous integration and continuous delivery (CI/CD) to build environments for the analysis and deployment of models.

    DataOps can Handle High Data Volume and Flexibility

    The amount of data created today is mind-boggling and will only increase. It is reported that 79 zettabytes of data were generated in 2021 and that number is estimated to reach 180 zettabytes by 2025. In addition to the increasing volume of data, organizations today need to be able to process it in a wide range of formats (e.g., graphs, tables, images) and with varying frequencies. For example, some reports might be required daily, while others are needed weekly, monthly or on demand. DataOps can handle these different types of data and tackle varying big data challenges. Add in the internet of things (IoT), such as wearable health monitors, connected appliances and smart home security systems, and that introduces another variable for organizations that also have to tackle the complexities of heterogeneous data as well. 

    OK, so, how can we make this a reality? First, to manage the incoming data from different sources, DataOps can use data analytics pipelines to consolidate data into a data warehouse or any other storage medium and perform complex data transformations to provide analytics via graphs and charts.

    Second, DataOps can use statistical process control (SPC)—a lean manufacturing method—to improve data quality. This includes testing data coming from data pipelines, verifying its status as valid and complete, and meeting the defined statistical limits. This enforces the continuous testing of data from sources to users by running tests to monitor inputs and outputs and ensure business logic remains consistent. In case something goes wrong, SPC notifies data teams with automated alerts. This saves them time as they don’t have to manually check data throughout the data life cycle.

    DataOps can Automate Repetitive and Menial Tasks

    Around 18% of a data engineer’s time is spent on troubleshooting. DataOps enables automation to help data professionals save time and focus on more valuable high-priority tasks.

    Consider one of the most common tasks in the data management life cycle: Data cleaning. Some data professionals have to manually modify and remove data that is incomplete, duplicate, incorrect or flawed in any number of ways. This process is repetitive and doesn’t require any critical thinking. You can automate it by either setting customized scripts or installing a built-in data cleaning software tool.

    Additional processes that can be automated via DataOps include:

    • Simplifying data maintenance tasks like tuning a data warehouse
    • Streamlining data preparation tasks with a tool like KNIME
    • Improving data validation to identify flags and typos, such as types and range

    Building Your Own DataOps Architecture 

    To develop your own DataOps architecture, you need a reliable set of tools that can help you improve your data flows, especially when it comes to crucial aspects of DataOps, like data ingestion, data pipelines, data integration and the use of AI in analytics. There are a number of companies that provide a DataOps platform for real-time data integration and streaming that ensures the continuous flow of data with intelligent data pipelines that span public and private clouds. Looking to increase the likelihood of success of data and analytics initiatives? Take a closer look at DataOps and harness the power of your data.

  • What Should Elon Musk Do? | Passwordless Future: Tense | WebKit iOS Monopoly Ends?

    What Should Elon Musk Do? | Passwordless Future: Tense | WebKit iOS Monopoly Ends?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Everyone has Twitter advice for Elon, passwordless vision is vacuous, and Apple prevented from forcing Safari. (more…)

  • DataOps Vs. DevOps: What’s the Difference?

    DataOps Vs. DevOps: What’s the Difference?

    There is a mindboggling amount of data today; to even measure it requires using a byte measurement called a zettabyte, which is one sextillion bytes (that’s 21 zeros). Currently, because such a ridiculous amount of data exists, there is a growing urgency to end wasteful data processes. From this environment, DataOps was born. Similar to the way that enterprises adopted DevOps to formalize and streamline wasteful development practices in the past, today, many large organizations turn to DataOps to formalize modern data management practices.

    What, Exactly, is DataOps?

    Primarily, many companies are adopting these principles because they are either trying to avoid or rectify data debt, which is the amount of money required to fix data problems due to the mismanagement of data processes. Data debt is a strong motivator for revamping outdated processes and policies, particularly when decision-makers and stakeholders require metrics before implementing change. Unpaid data debt can be detrimental to a business; the longer it remains unpaid, the more it costs to maintain a data landscape.

    By implementing DataOps principles and data governance, an organization can effectively reduce its data debt and prevent it from growing any larger. Moreover, DataOps practices and software engineering can be used to detect inefficiencies, minimize knowledge loss and capitalize on missed opportunities related to data usage.

    Similarities Between DataOps and DevOps

    Much of the processes that enable DataOps were borrowed initially from the same foundations that built DevOps. Likewise, just as companies need DevOps to provide a high-quality, consistent framework for software and feature development, data enterprises also rely on these same features to realize rapid data engineering and analytics development. For organizations that already have a DevOps framework in place, leveraging DataOps is relativity straightforward. Several important DevOps concepts adopted by DataOps include:

    • Agile development
    • Focus on delivering business value
    • Continuous integration and continuous delivery (CI/CD)
    • Automated testing and code promotion
    • Reuse and automation

    The Differences

    Despite the similarities between the underpinnings of DevOps and DataOps, there are several major differences.

    The human factor: The people using DataOps and DevOps have divergent personalities and skillsets. DataOps participants may be tech-savvy, but often their knowledge is theoretical. DataOps professionals can include data engineers, data scientists and analysts who focus on creating models and visual aids. DevOps, however, was made for software developers and engineers–coding is in their DNA.

    The process: The life cycles of DataOps and DevOps do share similar interactive properties. But, the former deviates in that it consists of a data pipeline and an analytics development process, both active and intersecting. While conceptually, the pipelines of DataOps resemble the development processes of DevOps, typically, experts note that the DataOps process is more challenging.

    Orchestration: In the DevOps process, application code does not require complex orchestration. However, for DataOps, both the data pipeline and analytics development orchestration is an essential component. Although orchestration in the DataOps pipelines occurs frequently and drives data flows, there is usually no such coordination of pipelines in application development and DevOps processes.

    Testing: Again, the two pipelines of DataOps create a significant difference from DevOps; testing in DataOps occurs during both the data pipeline and the analytic development process. These tests attempt to catch anomalies, flag abnormal data values and–unlike DevOps–validate new analytics before deployment. Likewise, these tests get embedded into a data quality framework for continual monitoring.

    Test data management: In most DevOps environments, test data management hardly takes priority; with DataOps, it’s vital to accelerate analytics development so that innovation keeps pace with agile iterations.

    Tools: DevOps is the ‘father’ of DataOps, and as such, the tools needed to support the latter are still in their infancy. While testing in DevOps is primarily automated, DataOps doesn’t have the same luxury – most users modify testing automation tools or build their own from scratch.

    Exploratory environment management: Generally, data teams use more tools than software development teams. Moreover, exploratory environments in data analytics are more challenging from a tools and data perspective; data teams also naturally depart from data islands across the enterprise.

    While foundationally, the concepts of DevOps serve as a starting point for DataOps, the latter involves additional considerations to maximize efficiency when operating data and analytical products. Nevertheless, both serve their intended audiences, reducing data debt and evolving data products or shortening systems development life cycles or providing continuous delivery. For businesses looking to make internal data-related processes more efficient, they should start by examining best practices associated with DevOps.

  • DataOps: DevOps Plus Big Data

    DataOps: DevOps Plus Big Data

    In traditional DevOps, there are the complimentary forms of development operations (that I call DEVops), and development operations (that I like to call devOPS). Between them they automate the toolchain and bring the people working on getting an application out to users onto the same team—not necessarily the same organizational team, but using their strengths in tandem to meet the needs of the business.

    DataOps is an interesting extension of DevOps. You still need all of the coding side to get data into the system and queries consistently maintained. You still need all of the operations side to get your database/noSQL/whatever up and running. In fact, while the development side in a DataOps environment might be (but isn’t necessarily) lighter, the operations side is almost always more complex. No matter the big data engine in use, it is a complex system in addition to other supported systems in a normal environment. My first installation of a big data environment (Cloudera, as it happens) was a weeks-long learning voyage. Only after I’d completed it did I use an automation tool (which is no longer available) to make it easy. My second round was hours … But it assumed the knowledge I had gained in the first attempt.

    The other bit of DevOps—ongoing monitoring and management—is also more complex in a big data environment, but we’ll come back to that in a moment.

    Once the normal DevOps systems are in place, ETL/data import tools will need to be supported also. The volume that these tools crunch through in a day makes their inclusion in DevOps in a data-heavy environment critical. If the data uptake is slow or the data itself inaccurate, there is an impact on the organization. This step also requires inclusion of data scientists, people who traditionally have not been pulled into the DevOps model. But they are the ones that can gauge the accuracy of data, and normally are responsible for data acquisition anyway.

    Which brings us to the monitoring and management stage of DataOps. Normally when we talk monitoring and management in DevOps, we are talking about tools that ultimately help with things like availability, responsiveness, auto-scaling and recovery. DataOps needs all of these things, like any other application does. And we’ve already mentioned that it has a complex environment embedded in your complex environment. But it also (some would argue more importantly) needs data monitoring. A knowledge of the state of the data is imperative. Whether from a piece of code gone wrong or data poisoning, monitoring the health of big data archives is essential to keep results from getting out of whack. This monitoring will again involve the people who know the data—data scientists by whatever title. This iterative process will need to run relatively constantly, checking on whatever makes the data valuable, normally using cross-field checks to verify that data hasn’t somehow been corrupted, or comparing trending averages to current average to monitor for sudden swings in results.

    Of course, for a system (and in research there are many) that spins up, finds results and spins down, the monitoring part is less essential, but for anything that keeps accessing and adding to the data pool, it is the key to a healthy application or application portfolio based upon the dataset.

    If all that sounds a bit intimidating, you’re not alone. The phrase “DataOps” has been around for a few years, and while data scientists use it, it hasn’t yet gained serious traction in the enterprise. Here’s how I would approach it as CIO: Given that DevOps is running on other projects, that is the low-hanging fruit. Given the rate of change for servers (as queries—particularly long-running ones—increase and decrease, systems needs change is one example), traditional DevOps is a greater benefit to big data than many other corporate systems. So roll out DevOps first without serious concern for DataOps beyond how DevOps impacts ETL—not ETL data quality, but ETL data availability/performance. Only when the system and the team are in place and the software/hardware/code part of the overall system is fully in a DevOps environment, start looking at ways to move data into it. Data monitoring will be the big one, but ETL under the initial round of DevOps implementation will have shown places it could be improved and ETL data integrity can be added, so improving those areas also. A measured, step-by-step approach that builds off of organizational experience is key here. Data monitoring and management will require different tools than normal DevOps monitoring and management, so having the data scientists look into available toolsets and how they mesh with existing tools early on will help with eventual implementation.

    Make no mistake, it is a big job, moving large data driven environments to DevOps. But the benefits are increased awareness of data quality and data quality issues, along with more responsive systems and teams that DevOps offers every business application. The amount of data available to any give organization is currently on a massive increase as IoT, open datasets, partnerships and ML drive greater pools. It is worth getting the automation and cooperation of DataOps into the mix as early as possible.

    — Don Macvittie