devopsy.kinsta.cloud

Author: Andrew Stevenson

  • Open Source: What IT Pros Wish the CIO Knew

    Open Source: What IT Pros Wish the CIO Knew

    Using DataOps and open source together, organizations can reimagine the way they deliver data and software-driven projects

    Real-time data is the fuel that powers today’s most critical systems and applications, as well as the digital transformation initiatives organizations need to compete effectively. For the people on the front lines building them, having the right technology alone is not always enough to succeed. To make the most of these game-changing technologies, you also need the right tools and the right strategy in place.

    If you’re not maximizing use of real-time data, you’re at risk of being disrupted by a competitor. For this reason, C-level leaders often adopt cutting-edge open source technologies that come with risks, making real-time data applications complex to build, deploy, operate and govern, and require a major investment in time and resources upfront to unlock their potential. You won’t get the most out of open source unless they can be operated and data can be accessed by a wider range of people outside engineering.

    For example, open source data technology is often lacking, and documentation is sometimes sparse at best. You may need to ask your data and platform engineers to spend weeks or even months building custom management tools before you can get to work on your first streaming application. That kind of delay can seriously impact your time to market.

    How Secure is Your Solution?

    Streaming data projects with multiple open source components can also introduce security and compliance issues into the picture. Even if you’re working with the very latest technology, open source often lacks built-in capabilities for security and compliance—especially if they are industry-specific for use cases such as health care or financial services. In fact, a recent survey by Veracode found that 70% of applications have open source security flaws, and 47% of those flawed libraries end up in code indirectly. If your project suffers from increased risk due to security and compliance issues, it can also lead to additional delays in projects, further slowing your time to market.

    In a world where the threat landscape is constantly evolving, your organization may wish to opt for a commercial offering for some types of use cases, especially if it includes the features you need to align with very specific compliance and governance needs.

    Creating a Culture of Engagement

    Arming IT professionals with the right tools and technologies they need is essential, but organizations also need to be sure they have a strategy in place to retain the best people—and keep them productive and satisfied. Fostering a workplace with the right level of engagement can provide a strong start. A recent Forbes article cited studies finding that highly engaged employees result in 21% greater profitability. On the other hand, the same article reported that 67% of employees reported feeling burned out at work—and were 2.6 times as likely to be looking for a different job.

    Open source tools are a great way to attract talented engineers because people love working with the latest technology. But as you give your IT professionals access to bleeding-edge capabilities, the complexities and time spent working with them increase. The right tooling can help manage the shortcomings of these technologies and help retain the talent you need by encouraging a positive work-life balance and lead to more successful implementations. If your engineering team isn’t spending time fighting fires and consumed by open source technology, there’s more time to focus on bigger picture issues such as a well-defined career path that keeps team members engaged and more valuable to the business in achieving its goals. With the right level of work-life balance and keeping employees involved and satisfied on the job, you can be more confident that your best engineers will become more than technologists and are strategic business partners who won’t be spending their spare time dusting off their resumes.

    Getting the Best of Commercial and Open Source

    How can you enable IT professionals with the tools they need to deliver their projects on time and on budget while empowering them with a more strategic role? Employing a DataOps approach to your initiatives can provide a strong start. Spanning many disciplines and skills across the organization, DataOps encourages an environment where people can collaborate, reduce process complexity and fully optimize the use of their data. DataOps allows teams outside of IT and engineering to operate their data directly, without being dependent on a small engineering team. It enables business stakeholders in your workforce to get closer to data and bring their own knowledge and context to data initiatives and improve business outcomes.

    For example, leading social and mobile gaming company Playtika needed to provide better visibility data streaming across Apache Kafka, its primary platform for the exchange of information between its microservices. Using a DataOps approach and platform, the company provided its analysts, product owners, operations and developers access to a fully audited GUI and CLI to explore and operate data in a compliant way without relying on data or infrastructure engineering. Providing direct access to data enables 600 developers to save 30 minutes a day as they work on delivering new features—which led to faster and more innovation when it came to developing new games.

    When utilized together with a commercial solution based on open source roots, DataOps helps you get your business and technical people on the same page, so your organization can achieve closer alignment to strategic business objectives, with data serving the business. At the same time, you can free up technical staff and developers from tasks such as tooling, and enable them to utilize open source for R&D and other innovative priorities.

    Using a collaborative DataOps strategy, your organization can reimagine the way it delivers data and software-driven projects, while also embracing new open source technologies.

  • DataOps: The Key for Real-Time Data Application Development

    DataOps: The Key for Real-Time Data Application Development

    Data engineers and developers face challenges every day to help their organizations digitally transform. To do this, they must deliver real-time data applications faster, better and cheaper. With businesses in every industry now data-driven, data professionals must work more efficiently and accelerate time to market for their products. That’s where DataOps comes in.

    Development of real-time data applications slows down when strategic business project teams are dependent on small, centralized engineering or IT teams for basic data operations such as exploring data or even building most data processing applications. Needlessly, engineering teams are notorious for being a bottleneck for strategic digital transformation initiatives. Rather, they can become known as strategic project enablers that provide the best tooling to their customers—instead of building it and being known as gatekeepers.

    Business users, outside of IT and engineering, need access to data but lack the necessary deep technical knowledge to work with the latest data technologies. DataOps brings people closer to data, helping organizations accelerate the delivery of their strategic projects, reduce costs and minimize the need for hard-to-source engineering skills.

    To accelerate digital transformation initiatives, teams and organizations should onboard new users directly onto a data platform quickly with the right tooling and resources required to get to work on their projects. DataOps empowers these business-focused users to work with data tools on their own to self-serve. A DataOps approach makes it easier for more people across an organization to work with its data, resulting in better alignment for data engineers and developers with their business and technical partners and improved data access.

    Serious new security, compliance and governance risks are introduced when less-technical users increase their interaction with technology, such as onboarding to a data platform. More people now understand that personal data has tremendous value. Initiatives such as the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) have brought the need to respect data privacy to the forefront. The stakes for safeguarding data are increasingly high and regulators are unforgiving—costing some organizations several billion dollars in fines, lawsuits and loss of market cap for some data breaches.

    When providing access to data, stringent management, control and real-time monitoring must be applied from a governance perspective. A surprise compliance auditor visit can result in questions such as which applications are reading credit card data, how an organization is keeping customer data safe, who has access to data and what are they doing with it. While locking down for compliance, you must help your business stay agile and avoid any slow down in data initiatives.

    Build vs. Buy to Streamline Operations

    In addition to adopting a DataOps approach, a key step is to onboard users to a data platform. This will optimize your data strategy and will help take the worry out of managing data projects, putting you on a path to faster execution and accelerated time to market.

    When onboarding users to a data platform, you need to choose between building your own or buying. It takes time and resources to develop your own system to onboard users and clients and can take years to build your own technology in-house. If your engineers are spending all their time working with open source technology to build tooling for onboarding, they’re not focusing on better ways to use data and get solutions to market fast—which is an inefficient approach.

    When your users are onboarded, the cycle begins again, as users request access to more data sets. More time at the command-line console means less time for your engineering teams to focus on higher-value work and strategic initiatives.

    Get a Leg Up on Competition With a DataOps Approach

    Buying a solution requires an upfront investment to support onboarding and, when combined with DataOps principles, is formidable to remain competitive. Together they enable support and manage the onboarding of clients to a data platform at scale and to maintain security and compliance. A DataOps approach fosters improved communication between data and business professionals to share data more effectively and to build a data mesh architecture.

    A Swedish digital bank, Avanza, needed to ensure the continuous improvement of the customer experience of its digital services to remain the leader in the Swedish market. Doing so required giving developers direct and constant access to real-time business data in Apache Kafka.

    The organization applied a DataOps approach and intelligent data masking rules to meet GDPR and protect customer confidentiality, featuring a large single data platform to address how it delivered data access and applied data governance to support more than 20 strategic projects. Using the platform, tenants who build streaming applications gained visibility into the health of their Kafka clusters and improved insight into data flows. This self-service approach means fewer support calls to data engineers, minimizing bottlenecks and freeing them up to work on more strategic initiatives.

    Data is the fuel that propels insights by tying back to analytics, decision-making and business systems. The better the shared data, the more it can be applied to supporting better business decisions, better customer experiences and execution of strategic business objectives and goals.

    DataOps also builds compliance and governance into your processes. It provides the visibility needed to understand who is accessing data and how to maintain governance and ensure people are acting ethically with the data they are enabled to access.

    The daily challenges faced in data-driven organizations only continue to grow. A DataOps approach helps data engineers and developers meet these challenges to digitally transform and build real-time data applications that help organizations gain a competitive business edge in their respective marketplaces.

  • Consider DataOps for a Competitive Edge

    Consider DataOps for a Competitive Edge

    DataOps seeks to eliminate existing barriers between people, technology, tools and data

    It’s no secret that COVID-19 has put the economy under enormous strain and future economic prospects are uncertain. A smart way to give your organization a leg up against competitors is DataOps. Adopting a DataOps culture makes the best use of your data—it helps to optimize the data powering your business because it successfully monetizes and systematizes your approach to working with data. It will help to get your new products to market faster and also accelerates business agility.

    For example, a DataOps approach can help a data scientist who has spent weeks developing a new machine learning model that is complete but not ready to go into production. She will need to set up data stream monitoring and other basic requirements before she can put it in place.

    This type of challenge is routinely faced by DevOps teams and the best practices they implement help avoid potential setbacks to quickly put their hard work and innovation to use. Applying the same development practices to the deployment of data-intensive services and applications significantly speeds up project completion times.

    Why DataOps?

    DataOps puts the principles and powerful DevOps practices such as automation, continuous delivery and quick feedback cycles to work in a data environment. Its strategic focus is to bring people closer to data, by delivering proven tools that deliver data to people with big-picture business knowledge. DataOps also removes complicated engineering and IT processes that stand between data and those with business knowledge to give projects an important and meaningful business context that greatly improves outcomes.

    The goals of a DataOps approach are to eliminate the friction that’s holding business stakeholders, developers and data scientists back and to increase project productivity and business outcome success. Automating and streamlining processes help people to quickly and easily move from ideas to development to production without lag time.

    DataOps also helps take concepts into production by fitting data projects into familiar, standard CI/CD development workflows and plugging them into any existing framework to deploy applications.

    How to Put DataOps to Work With GitOps

    GitOps is becoming a core component of DataOps practices and puts them into action. It allows data engineers to define data processing applications and pipelines as configuration and manage them through Git and mature CI/CD pipelines. This brings with it higher levels of standardization, security, governance and, as a result, an accelerated rate of delivery. Adopting GitOps in the data management world can pose problems for some organizations working with powerful and fast-changing technologies such as Apache Kafka and Kubernetes, but the automated processes it brings turbo-charge delivery time.

    Cloud migration is another way in which a GitOps approach makes a difference for organizations that want to take advantage of managed services in Microsoft Azure, AWS or Managed Streaming for Apache Kafka (MSK). GitOps helps simplify migration of Kafka clusters used for streaming to the cloud service of an organization’s choice—simply define the desired state, step back and GitOps takes over.

    GitOps’ pull-based approach to deployment, in which an operator watches Git and applies the desired state, creates several benefits. One benefit is not needing to poke holes in production environments to provide push-based actions from software such as Jenkins, which has numerous common vulnerabilities and exposures (CVEs). With a GitOps pull-based approach, attack surfaces are reduced, enabling solution migration across technology and clouds.

    A Better Process Results in Better and Faster Outcome

    Real-time business is here to stay and there is huge pressure on businesses to change fast. As the saying goes, “You snooze, you lose.” With the right tools and strategy backed by a good communication plan, DataOps will work to your advantage, bringing the benefits of a DevOps culture.

    Faster time to market via increased productivity is one advantage of a DataOps approach. Putting people closer to data, together with the right tools, helps you empower business users and developers so they can add new features and capabilities to any deliverable, and deploy and monitor it quickly.

    When working at a faster clip, don’t overlook the importance of governance and security. Build governance into the DataOps process to ensure deployment of the correct applications in the right way. And, maintain business continuity without disruption to existing infrastructure and processes.

    DataOps seeks to eliminate existing barriers between people, technology, tools and data. By bringing business and technology imperatives closer together, you gain invaluable context and a better ability to focus on driving strategic business outcomes that matter.