devopsy.kinsta.cloud

Category: DataOps

  • A DevOps Guide to IoT Technology

    A DevOps Guide to IoT Technology

    Internet-of-things (IoT) is a term used to describe the network of physical devices—from everyday household items to sophisticated industrial tools—connected to the internet, sharing and collecting data. With the advent of cheap computer chips and the ubiquity of wireless networks, it’s possible to make anything a part of the IoT. Add to that the exponential growth in artificial intelligence and machine learning, and the IoT is a truly significant part of our lives.

    The IoT allows devices to communicate and interact over the internet, and they can be remotely monitored and controlled using technologies many of us have never heard of, such as MQTT, QUIC and AMQP. The proliferation of these devices has led to an exponential increase in the amount of data we can gather, analyze and use to make more informed decisions. We’ve already seen the transformative potential of IoT in our homes, workplaces and cities, and it’s only set to increase as more and more devices come online.

    At the heart of this technology is the concept of seamless connectivity and communication. However, the rapid development and deployment of IoT devices pose significant challenges. This is where the DevOps approach comes in, offering a solution to these challenges by improving efficiency and effectiveness in the development and operation of IoT systems.

    Importance of DevOps in IoT Development and Deployment

    In the context of IoT, DevOps is crucial. IoT devices are often deployed in the field, in homes or in other environments where physical access is difficult or impossible. As such, the ability to update and manage these devices remotely is a critical necessity. DevOps methodologies enable this remote management, providing mechanisms for continuous integration and continuous delivery (CI/CD) that make it possible to push updates to devices regularly and reliably.

    Moreover, given the pace at which IoT technology is advancing, it’s crucial to be able to adapt quickly to changes and new developments. In this regard, DevOps is invaluable. By bringing together development and operations, it allows for more agile response to changes, faster issue resolution and overall better performance.

    The Role of DevOps in IoT

    Speeding Up IoT Development Cycles

    One of the key benefits of using a DevOps approach in IoT development is the ability to speed up development cycles. With traditional development practices, the time between identifying a requirement or an issue and implementing a solution can be lengthy. In contrast, DevOps practices, such as continuous integration and continuous delivery, enable rapid prototyping, development, testing and deployment of new features and updates.

    With the DevOps approach, developers and operations teams work together from the beginning of a project, collaborating closely to ensure that the software being developed is robust, reliable and meets the needs of the business. This cooperative approach, combined with automated testing and deployment processes, significantly reduces the time to market for new features and updates.

    Ensuring High-Quality, Reliable IoT Applications

    Another significant advantage of utilizing DevOps in IoT development is the increased quality and reliability of the applications being developed. With traditional software development methods, testing and quality assurance often occur after the development process. This can lead to issues being identified late in the cycle, which can be costly and time-consuming to fix.

    On the other hand, DevOps practices emphasize continuous testing throughout the development process. This means that issues can be identified and addressed much earlier, leading to higher-quality, more reliable software. In the context of IoT, where devices may be deployed in critical applications such as health care or transportation, the importance of this cannot be overstated.

    Enabling Scalability and Flexibility in IoT Systems

    Finally, DevOps plays a crucial role in enabling the scalability and flexibility of IoT systems. As the number of IoT devices continues to grow exponentially, the ability to manage and monitor these devices efficiently is becoming increasingly important. DevOps practices, such as infrastructure as code (IaC) and the use of containers and microservices, provide the tools necessary to scale and adapt IoT systems as needed.

    With IaC, the infrastructure required to run an IoT application is defined and managed as code, making it easy to replicate and scale. Containers and microservices, on the other hand, allow applications to be broken down into smaller, independent components that can be scaled individually, providing a high level of flexibility and efficiency.

    Implementing DevOps in IoT Development

    Integrating CI/CD in IoT Application Development

    Continuous integration/continuous delivery (CI/CD) is a fundamental DevOps practice that plays a vital role in IoT development. It involves automating the processes of software delivery and infrastructure changes. With CI, developers frequently merge their code changes into a central repository where automated builds and tests are run. CD takes this a step further by automating the release of these changes to the production environment.

    For IoT, integrating CI/CD can help maintain the quality of software, increase efficiency, and speed up the process of delivering software updates or new features to devices. It also ensures that any changes made can be easily rolled back, reducing the impact of any potential issues.

    Applying IaC for IoT Infrastructure Management

    Infrastructure-as-code (IaC) is another key DevOps practice that is highly relevant to IoT. IaC is the process of managing and provisioning computing infrastructure through machine-readable definition files rather than physical hardware configuration or interactive configuration tools.

    In the context of IoT, IaC can simplify the process of setting up and managing the complex infrastructure required for IoT applications. With IaC, developers can quickly and reliably replicate infrastructure setups, making it easier to scale IoT applications as the number of connected devices grows.

    Using Containers and Microservices in IoT Application Architecture

    Containers and microservices are two more DevOps practices that can significantly benefit IoT development. A container is a standalone executable package that includes everything needed to run an application, including the code, runtime, system tools, system libraries and settings. Microservices is an architectural style that structures an application as a collection of small, loosely coupled services.

    In the context of IoT, using containers and microservices can greatly simplify the process of developing, testing and deploying IoT applications. Moreover, they can enhance the scalability and flexibility of IoT systems, making it easier to adapt to changing requirements and scale as the number of connected devices grows.

    Challenges of Implementing DevOps for IoT and How to Overcome Them

    Dealing with Device and Platform Diversity

    One of the main challenges in implementing DevOps for IoT is dealing with the diversity of devices and platforms. IoT devices can range from simple sensors to complex machines, each with its own capabilities, requirements and operating systems. Moreover, these devices may communicate using different protocols and have different connectivity requirements.

    The key to addressing this diversity is to adopt a platform-agnostic approach. This involves developing software that can run on any device, regardless of its hardware or operating system. Containerization can help achieve this by providing a consistent and reproducible environment for running applications.

    Ensuring IoT Security and Privacy

    Security and privacy are significant concerns in IoT, and they can be particularly challenging in a DevOps context. The increased frequency of updates and changes that come with the DevOps approach can potentially introduce new vulnerabilities. Moreover, the interconnected nature of IoT devices means that a security breach can have far-reaching consequences.

    To address these challenges, it’s essential to integrate security practices into the DevOps process. This is often referred to as DevSecOps. This approach involves continuously monitoring and testing for security vulnerabilities throughout the development process rather than treating security as an afterthought.

    Managing IoT Data Volume and Complexity

    The sheer volume and complexity of data generated by IoT devices can also pose a challenge in a DevOps context. Traditional data management and analysis tools may not be able to handle this volume and complexity.

    To tackle this, it’s important to utilize modern data management solutions that can handle large volumes of data. Additionally, employing advanced analytics and machine learning techniques can help make sense of complex data and extract valuable insights.

    Coordinating IoT System Updates and Maintenance

    Finally, coordinating updates and maintenance for a large number of IoT devices can be a significant challenge. This is particularly true in a DevOps context, where updates and changes are frequent.

    To address this, it’s crucial to have a robust system for managing and tracking updates. Automated deployment tools can be invaluable in this regard, allowing updates to be pushed out to devices in a controlled and coordinated manner.

    Conclusion

    In conclusion, implementing DevOps in IoT development can bring significant benefits, including faster development cycles, higher-quality, reliable applications and improved scalability and flexibility. However, it also presents unique challenges that need to be effectively addressed. By understanding and properly implementing the principles and practices of DevOps, we can harness the full potential of IoT technology and drive forward innovation.

  • Adopting a Low-Code GraphQL Approach to Build a Core Data Platform

    Adopting a Low-Code GraphQL Approach to Build a Core Data Platform

    Our education platform has relied on an outdated monolithic legacy application for years. It’s an outdated system built of a single service layer on top of a database designed for use cases that have evolved over the last decade. But we’re using the wealth of data in the monolith as a foundation for our current projects, thanks to our newly adopted tools.

    With our product line continually expanding, our need for a centralized API that can sit over commonly used data has become even more critical. This includes data related to our customers’ organizational units, purchased products, licensing information, and student, teacher and class rosters, among other things.

    The stakes are high since we have multiple downstream clients depending on us. Our goal is to move some functionality and data out of the main system while making sure our design is flexible enough to adapt to future product changes. We envision a hub-and-spoke model where all our products interact through a central hub to access shared data. Efficiency, security and reliability are all top priorities. Because we’re a small team, we need to move fast and be open to new ideas. That’s why we’re using a low-code backend solution to minimize manual coding and maximize our productivity.

    A few months ago, we adopted a tool called Hasura. Its core value proposition is that it places a production-grade GraphQL API over any database, saving teams months of recurring effort in building, shipping and maintaining their APIs. We put the open source Hasura community edition to the test and found that it lived up to its claims. In this article, we’ll show you how it worked for us and how it can work for you too.

    Converting GraphQL to SQL

    Before a team can convert GraphQL to SQL, they must undergo a data modeling phase. After this data modeling phase, backend teams must complete some unavoidable tasks. One of these tasks is to build an API that allows clients to read and mutate data. Hasura provides this instantly, freeing engineers from the toil of writing basic CRUD APIs. Hasura compiles a GraphQL string into a GraphQL abstract syntax tree (AST), which is then compiled into a SQL AST. This abstraction lets us do powerful things, like using a bulk GraphQL mutation to transactionally and atomically insert or update data. GraphQL provides a flexible way for us to query our data in the form of a graph. This means that, as engineers, we get to focus more on business logic.

    API Federation, Aggregation and Composition

    Hasura’s support for remote GraphQL schemas enables us to design a service-oriented architecture in which Hasura functions as an API federation/composition layer and delegates to other Go microservices that provide specialized business logic for a particular domain. This ability to federate doesn’t just end at remote GraphQL schemas—Hasura also supports remote data sources, opening multiple possibilities for how we traverse through the graph and explore relationships in our data, including how we might join remote databases to each other or to remote APIs (REST or GraphQL).

    In greenfield projects, teams typically spend months crafting a fully-featured API, a common language that all clients can share. With Hasura, we get this in seconds and can spend that saved time writing automated tests or implementing business logic.

    Developer Dashboard

    Hasura’s Dashboard/Console improves developers’ experience. Using the dashboard, developers can craft experimental GraphQL queries and mutations against our local instance of Hasura. The embedded GraphQL playground makes it a breeze for new developers joining the team to introspect and discover our API. We can also visualize our database schema in the UI directly and make changes to it with a few clicks of a mouse (if needed), generating source-controlled SQL migration scripts like you’d find in other web frameworks.

    Observability

    No production-ready system would be complete without observability, or the ability to gauge a system’s internal state based on external output, usually in the form of logs, traces and metrics. Hasura ships logs and traces to our New Relic telemetry platform for each database operation, allowing our team to identify bottlenecks and receive early alerts of unexpected or degraded system performance.

    Authorization

    Hasura’s role-based authentication support allows us to check an additional box on our production-readiness checklist. Clients with certain roles may have permission to execute specific GraphQL API operations or access particular database tables. Hasura can derive a client’s role in one of two ways: Either the client will send Hasura a JWT auth header (which includes a signature containing claims providing the client’s role), or Hasura will refer to an external webhook which, in turn, makes the auth decision. Without Hasura, we would need to write and maintain the code to perform all of this work.

    Change Data Capture

    The last way Hasura enables us to build our core platform is through change data capture (CDC), or the process of tracking changes to a database. Hasura provides this in the form of event triggers, which function like a transactional outbox system. When a record is inserted, updated or deleted, a bona fide database trigger will record this event in a table. Then, a reliable process in Hasura will guarantee that the event is delivered at least once to its intended webhook (one of our microservices). This is tremendously useful for our team since we need to sync data from our new database to the older legacy database.

    There are many tools we could use for CDC, such as AWS SQS or Kafka (some of which we use downstream to integrate our platform with individual products), but having this feature baked into Hasura makes the process simpler, as it frees us from having to provision any new infrastructure or deal with the complexities of bringing on additional tools.

    Conclusion

    For the Teaching Strategies platform modernization team, we’re just getting started. As we continue to evolve and improve our product, we’re considering Hasura for finer-grained authorization, such as attribute-based access control (ABAC) over coarser role-based access control (RBAC). We will also investigate real-time streams so other systems, such as users’ browsers, can see data changes as fast as possible.

    Using the functionality and features of Hasura community edition outlined above, our team is well on the way to building a strong core platform. By leveraging this tool, we can meet our milestones faster and provide even more support for educators and caregivers, making their lives easier.

    Interestingly, Hasura is a combination of the programing language Haskell and the Sanskrit word asura—or demon.  As far as tech choices go, this one feels much more like a guardian angel.

  • Forking RHEL! Oracle and SUSE Join the Fight ¦ Silverman Sues AI Firms

    Forking RHEL! Oracle and SUSE Join the Fight ¦ Silverman Sues AI Firms

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: SUSE announces a Red Hat Enterprise Linux fork after Oracle said a similar thing, and Sarah Silverman says OpenAI and Meta have stolen her words. (more…)

  • Are LLMs Leading DevOps Into a Tech Debt Trap?

    Are LLMs Leading DevOps Into a Tech Debt Trap?

    Everyone thought AI would free us from toil. Unfortunately, that’s not true. What it will do, however, is lead to an explosion of technical debt. Some of the most alluring uses of LLMs have the potential to lead to some of the worst outcomes for enterprises and their DevOps and platform engineering teams.

    SK Ventures and Adrian Cockcroft raised the alarm bells in their warning posts about the tipping points LLM introduced into work and society. And in my own Gluecon 2023 presentation, I challenged just how powerful and disruptive the new generation of AI/ML will be for ITOps and DevOps efforts.

    Working Faster is not Working Smarter

    There is no doubt that LLMs are powerful and will change how we think about expertise and knowledge work.  While they are not truly innovating, their ability to draw on such a vast pool of human expertise makes it difficult to distinguish from true creative work. Since they are unbounded, they will cheerfully use the knowledge to churn out terabytes of functionally correct but bespoke code.

    It’s easy to imagine a future where LLMs crank out DevOps scripts 10x faster. We will be supercharging our ability to produce complex, untested automation at a pace never seen before! On the surface, this seems like a huge productivity boost because we (mistakenly) see our job as focused on producing scripts instead of working systems.

    But we already have an overabundance of duplicated and difficult-to-support automation. This ever-expanding surface of technical debt is one of the major reasons that ITOps teams are mired in complexity and are forever underwater. Generating even more scripts, even if they were better written, does not reverse the trend.

    Can AI/ML Avoid This Technical Debt Trap?

    It depends on how we use the expertise in the models. Instead of asking it to generate new code, we could ask it to interpret and modify existing code. For the first time, we have tools to take down the “not invented here” barriers we’ve created because of the high cognitive load of understanding code. If we can help people work more effectively with existing code, then we can actually converge and reuse our systems. By helping us expand and operate within our working systems base, LLMs could actually help us maintain less code.

    Imagine if the teams in your organization were invested in collaborating around shared systems! We haven’t done this well today because it takes significantly more time and effort. Today, LLMs have thrown out those calculations.

    Taking this just one more step, we can see how improved reuse paves the way for reduction of the number of architectural patterns. If we improve our collaboration and investment in sharing code, then there is increased ROI in making shared patterns and platforms work. I see that as a tremendous opportunity for LLMs to improve operations in a meaningful way.

    It comes down to whether we ignore complexity by replacing it or embrace complexity by understanding it. I see this as a choice between independence or collaboration, as illustrated by the table below.

    LLM Driving Independence

    LLM Driving Collaboration

    Create new code

    Interpret existing code

    Replace existing code

    Maintain existing code

    Design new architectures

    Be a guide and mentor

    Build new stacks

    Evaluate alternatives

    Bypass documentation

    Improve documentation

    Can AI Help us Collaborate?

    We keep taking the same hardware and software components and then building unique systems. That variation makes it impossible to leverage each other’s work. At its heart, the reusability problem lies in the difficulty of understanding systems, and that makes it nearly impossible for us to work together, collaborate, reuse and accelerate.

    Breaking this cycle has been our mission at RackN for nearly a decade. We’ve seen tremendous ROI from availability, reliability and speed when customers are able to leverage standardized infrastructure pipelines. We’ve seen firsthand that it is possible to guide teams into collaboration with sufficient guidance, mentoring and training. Collaboration creates 10x outcomes even without AI, but it takes building expertise, deliberate effort and leadership discipline to implement.

    Why does collaboration create 10x ROI?  It works because true innovation actually comes from understanding and building on top of systems rather than rebuilding them from scratch over and over again. The key is being able to leverage LLMs to better understand existing automation and platforms instead of creating yet another bespoke stack.

    Enterprises Should use LLMs to Drive Collaboration

    I believe that the most effective enterprises will use these tools to elevate their IT leaders and architects to focus on collaboration and reuse. They must actually direct them to produce less bespoke automation; instead, they can use expertise in these tools to find and maintain existing systems. LLM presents options, but collaboration requires new thinking and strong leadership.

  • Threads: Twitter Killer or Ad-Infested Hellscape?

    Threads: Twitter Killer or Ad-Infested Hellscape?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Meta launches its much-leaked Threads microblogging app. A spinoff from Instagram, it’s already reached 30 million signups at the time of writing—even before it’s available in the EU. But how is Meta’s DevOps team making it scale? (more…)

  • Hasura Unfurls Data Delivery Network Based on GraphQL

    Hasura Unfurls Data Delivery Network Based on GraphQL

    At its HasuraCon 2023 conference today, Hasura launched a Data Delivery Network (DDN) based on a set of GraphQL application programming interfaces (APIs) that makes it simpler to integrate distributed sources of data.

    In addition, Hasura is adding Hasura Schema Registry to streamline API governance using graph technology, a MongoDB connector, a Native Data Collector (NDC) for REST APIs and support for the Open Domain Data Specification (OpenDDS), an open framework for modeling APIs originally developed for the travel industry.

    Finally, Hasura is adding support for Native Queries and Logical Models capabilities that convert SQL requests into API calls.

    The Hasura DDN leverages the GraphQL API to enable sub-second integrations across a network that, in concept, is similar to how a content delivery network (CDN) is constructed.

    Hasura CEO Tanmai Gopal said the goal is to enable sub-second integration of data at a global scale across a network with more than 100 regions. Developers can integrate their APIs in less than a second regardless of the number of data models connected. Metadata changes can also be used to create builds that can be independently tested and applied to projects to enable API version control and collaboration among development teams.

    The Hasura DDN is accessed via Hasura Cloud, an instance of a data integration platform that can be deployed on multiple clouds, including Microsoft Azure, and will soon be made available for private clouds managed by internal IT teams.

    GraphQL is gaining traction as an alternative to REST APIs for integrating data, but it’s more complex to implement and manage. The Hasura DDN extends the reach of a Hasura Cloud platform that provides a simpler method for adding GraphQL capabilities to an IT environment in a way the also conserves network bandwidth, noted Gopal.

    GraphQL is a query language for APIs and a runtime for fulfilling queries that provides a complete and digestible description of the data in an API. It can also be used to pull data from multiple sources because it uses a schema that defines the different types of data available. REST APIs, in contrast, can only fetch data from a single source. As such, many organizations that are trying to drive adoption of reusable digital services are relying on GraphQL.

    It’s not always clear who within an organization is assuming responsibility for managing the APIs that developers create, but as the number of APIs increases, DevOps teams, in many cases, are moving to centralize API management. In fact, the more data an organization creates, the greater the corresponding use of APIs to programmatically access that data becomes.

    Hasura is betting that as organizations reach a critical mass of API integration requirements, it will make more economic sense to rely on a purpose-built data integration platform accessed as a cloud service versus relying on a set of tools and platforms they need to configure and manage. Regardless of approach, the one thing that is certain is the number and types of APIs that need to be managed are rapidly exceeding the ability of IT teams to keep pace.

  • Developers’ Role in Protecting Privacy

    Developers’ Role in Protecting Privacy

    We live in a world where we always expect things to happen seamlessly. Every time somebody walks into a café and signs in to the free WiFi and opens their Instagram or TikTok to stay up-to-date with trends and friends or retrieves a discount from a retailer by joining their newsletter circulation, rarely do they stop to think about how that convenience and value is derived.

    Technology is the key enabler working away in the background, and developers and DevOps teams are making that happen. Underneath the slick interfaces of the apps and websites that provide these highly efficient and personalized experiences is a complex machine that relies on data to fuel it, built by developers and delivered by DevOps teams.

    In many ways, data is a two-way exchange, and those end-users that are aware of this give and take appear to be happy with how the relationship works.

    Indeed, several surveys in recent years suggested that the majority of consumers are happy to share their data with companies if it improves their experience. Furthermore, a significant minority are still comfortable if they have no control over who sees their data, there is no clarity around how data is stored and used, even if the organization in question has a poor reputation for data privacy.

    The Privacy Balancing Act

    So, if most people already accept data trade-offs in some shape or form, should developers still care about building applications and solutions that guarantee data privacy?

    The short answer is yes, but as developers, that involves a tricky balancing act.

    Although sharing data has become commonplace in exchange for benefits and value, consumers are becoming more aware of privacy issues. Take the EU’s General Data Protection Regulations (GDPR) as an example. Over the past five years, awareness has more than doubled in notable European markets such as the UK, Spain, Germany, the Netherlands and France.

    Meanwhile, there is also commercial pressure, as employers rely on developers to innovate to remain profitable. At the same time, customers expect brands to be responsible with their data, and failure to do so at the expense of trying to commercialize a new application could be detrimental. Indeed, while the pandemic may have ushered in significant changes and altered consumers’ attitudes toward data privacy, end users remain unwavering about the importance of security.

    Maintaining this balancing act is becoming increasingly complex to achieve. However, the question of data privacy is becoming a key business priority, and that means developers have a big opportunity to show their commercial value to their organizations. Indeed, companies cannot afford to get the balance wrong, and developers are central to creating the perfect formula.

    In fact, the privacy conundrum has been around for a long time.

    When the internet was created, privacy was not considered an issue because all content accessed online was deemed to be public. Nothing was encrypted; the introduction of email complicated things, not least because anybody on the internet could, theoretically, intercept personal data and read private messages.

    This prompted the emergence of the cypherpunks, a group of cryptographers who began to contemplate what the future of digital privacy might and should look like. They started to create the tools they believed the world needed to safeguard data privacy–today, their mission is carried on by developers who value user privacy.

    Why Developers Need to Be Familiar With FHE

    However, securing the transmission of data is not enough, as many breaches still occur because data isn’t encrypted during processing.

    So, in a world where more and more data needs to be shared to deliver the sort of experiences consumers expect, is it inevitable that privacy and security will be compromised as the volume of data exchanges grows? Not necessarily.

    Fully homomorphic encryption (FHE) offers a solution, allowing developers to build applications and run services without needing to see or secure the underlying data. This is because FHE enables data to be processed blindly without having to decrypt it at any stage. FHE offers a way for developers to maintain that all-important balancing act between maintaining privacy and offering value to both business and customer.

    In FHE, to guarantee a user’s data privacy, all data is encrypted by the user with their own secret key. Users of a particular service, such as an online medical portal, will use a secret key to send their encrypted data to a server where blind processing takes place. The result is also encrypted and sent back, the user again using their secret key to decrypt the information.

    What this means is that, because data is encrypted end-to-end, the company providing the service can do so in a way that keeps the same customer experience, even though they cannot see the data that is being shared and sent back. From a security perspective, this is also critical. Governments, attackers and service providers cannot see the exchanged information because they do not possess the key–they also cannot break it, as the encryption used in FHE can resist attempts made even by powerful quantum computers.

    Privacy can also be protected during larger-scale data exchanges that underpin collaboration between organizations and nation-states on numerous strategic programs, with each contributor able to add their value knowing that their intellectual property and sensitive information is not compromised.

    Looking ahead, if developers can embed FHE into their practices on a broad scale, then a key milestone for privacy is within reach. Rather than being viewed as an objective relating to digital interactions, privacy can be treated as a naturally occurring by-product of every application, piece of software and service that the developer community creates.

  • The Tragedy of the Data Commons: Balancing Innovation With Risk

    The Tragedy of the Data Commons: Balancing Innovation With Risk

    Clean drinking water and fisheries are classic examples of the “tragedy of the commons,” the problem in economics and ecology where an individual is incentivized to consume as much of a shared resource as possible at the expense of the larger society. With fisheries, for example, a logical fisherman would like to make as many catches as possible. However, this can ruin the environment for everyone, as overfishing can endanger an entire species and even threatens whole ecosystems. The modern IT environment has given us a new case to consider related to the tragedy of the commons: Data.

    Data, much like drinking water, is a critical resource. Developers need to use this data and, logically, they want to use it as much as possible to build innovative applications and products. However, too much data flowing around a modern distributed enterprise environment carries risks to the larger company.

    The data developers use often includes customers’ personal data like names and birth dates. The risks of this information being damaged or stolen can be catastrophic. This is where database administrators (DBAs), who are tasked with managing and securing this data, come in. There is an inherent tension between developers, who want as much access as possible, and DBAs, who want to limit that access for security and privacy reasons. How data is used within an enterprise is critical, because it must be used to create new solutions and be kept secure. An overly open data policy could lead to breaches and leaks, while an overly restrictive policy limits innovation.

    Thankfully, there is a solution that is helping ease this tension. Observability can automatically analyze massive amounts of information across a business’s entire IT environment. Users can use it to pinpoint the causes of outages or performance issues and receive actionable insights to resolve these problems quickly. Modern observability solutions even use AI to predict and proactively prevent problems before they occur. DBAs and developers can use observability to strike a balance while boosting innovation and improving productivity. Here are the three biggest ways that observability can solve the tragedy of the data commons and ease the tension between DBAs and developers.

    Ensuring Data is Secure

    One of the biggest concerns that DBAs have regarding data access is security. Sensitive production data are no longer stored exclusively on a company’s on-premise servers, where DBAs have full visibility into how it is being used and who has access. Instead, it is now stored across various private and public clouds as well as on-premises, which has reduced visibility for DBAs.

    This sensitive data is the same information that developers often need to build and launch applications in a continuous delivery model, where developers build software in short cycles so it can be released at any time. This involves working across different clouds, including AWS and Azure, while leveraging complicated cloud-enabled technologies like microservices and Docker containers. When data is deployed across multiple clouds, the attack surface increases while visibility decreases, leading to new security concerns.

    Observability provides DBAs with a better understanding of how data is being used by developer teams so they can more easily ensure that the data remains secure and protected. Additionally, many modern IT environments deploy threat intelligence solutions that observe actions and detect threat signals to help predict security incidents and respond before they happen. Some threat intelligence applications are powered by AI, which can automatically gather and organize complex log data from the entire network. This data is then compared with an out-of-the-box threat database, helping to detect suspicious activity early. Observability and threat intelligence working in tandem provide a comprehensive overview to help relieve some of the security concerns for DBAs and create a more secure environment for data to be shared between teams.

    Remaining Data Compliant

    Ensuring that data is compliant is top of mind for every organization. Compliance regulations such as GDPR, CCPA and HIPAA carry severe penalties for data leaks, meaning that companies must maintain the integrity and availability of sensitive and regulated data.

    Ensuring compliance while managing requests for information from developers working across geographies and clouds can be a daunting task for DBAs. Not only must they remain compliant, but many businesses must demonstrate continuous compliance, which is only possible using complex, dynamic and automated systems.

    Observability allows teams to clearly see and follow the complex flow of data as it moves between DBAs, developers and other teams. Additionally, observability can be deployed to help ensure that any sensitive data that is required to be anonymized is done so before it is shared. DBAs can further use observability to track the lineage of data as it moves between teams to monitor that it is not changed or transformed in any way that risks a compliance violation. Making compliance a regular part of the IT environment should not be viewed as a burden. In fact, it provides many benefits, including improving data quality, which can be used to further innovate and build better products.

    Making Teams More Productive

    Observability has shown it can create harmony between DBAs and developers by helping mitigate security and compliance concerns. Not only is it addressing these issues, but it is being used to make each team more productive on their own.

    For DBAs, this means gaining additional insights that help ensure applications and services are running properly. Observability collects information from across hybrid and multi-cloud environments and presents it in a single view to allow DBAs to respond quickly and resolve issues. When DBAs gain the insights that observability provides, they can better manage critical data and anticipate future problems before they arise.

    Observability is key to helping developers build applications as productively as possible. The ongoing shift left has necessitated the use of observability as a way to gain single-pane-of-glass visibility into the build process. Developers are also provided insights into the performance of web applications and can move quickly to address any problems. Finally, observability is improving the speed at which teams can innovate, as it provides live code profiling that can be used to predict and prevent future problems.

    Observability provides a real solution to ensure that data is not a resource that falls victim to the tragedy of commons, where its use can harm an entire organization. Instead, it is powering teams to use data securely to work more productively and build innovative applications.

  • Is Your Monitoring Strategy Scalable?

    Is Your Monitoring Strategy Scalable?

    Every company is trying to get better insights into its operational effectiveness, and they are running into the same problem: Scale. So what does a scalable monitoring strategy look like and how can you safeguard against the most significant issue in observability?

    What is a Scalable Monitoring Strategy?

    We’ll begin by identifying the two things most impacted by scale: Cost and performance. Cost can be broken down into storage and computation. It is obvious that to hold more data, more storage is needed, but what about compute power? This is important because queries need to search through more data—and thus, use more compute power—to return a result.

    This creates a trade-off between performance and cost-effectiveness. It becomes more and more challenging to have queries that resolve instantly as the dataset increases, so engineers will often simply tolerate a performance decline. This further impacts usability and the general usefulness of the system. If it takes 10 seconds to resolve every query, you’ve just cut down your data discovery efficiency by a factor of 10.

    Scaling Means Performance and Cost-Effectiveness

    To scale a monitoring strategy, a wise architect needs to break this standoff between performance and cost and approach the problem like a data scientist. A scalable monitoring strategy starts with a simple question: What is the use case for this data?

    Some data is only ever ingested because it might be needed. It spends its entire lifetime undisturbed and is eventually compressed or deleted. On the other hand, some data is queried every minute of every day and is integral. Knowing how data is used means an engineer can decide the value of that data.

    In the battle to create a scalable monitoring strategy, there’s a three-step approach.

    Step One: Track Data Usage

    Which data is ingested and never queried? Which data never spends any time at rest? Build a map of how often data is consumed so that any strategic choices are informed by the use case.

    It is most common to group data into three different use cases:

    Frequent Access – Data that is constantly queried and needs to be available at a moment’s notice.

    Monitoring – Data that drives dashboards or trains machine learning models but which is largely useless after it has been processed.

    Compliance – Data that is held in case it is needed, but it is only queried sometimes. For example, audit logs.

    Step Two: Optimize Storage and Consumption

    Once the use case of data—whether it’s logs, metrics or traces—is understood, the next stage is to optimize. This means creating different storage and query solutions for the different usages we’ve seen above.

    Frequent Access – Rapid queries that can be optimized and tuned. OpenSearch is a good option, although management overhead can be painful, especially at scale.

    Monitoring – This is largely about transforming data. For example, ingesting logs, converting them into metrics and deleting the original log. This is very powerful because metrics take up far less space than logs and can be stored much more cost-effectively.

    Compliance – Low-cost storage like Amazon S3 is a good option, but this data must still be accessible, even if simply by reindexing.

    How Easy is it to Build These Capabilities?

    It is trivial to create an OpenSearch cluster, but it’s challenging to manage it at scale. Likewise, it’s simple to convert logs to metrics, but doing so in a performant way at scale is complex. Holding onto compliance logs as they scale near infinitely can be difficult, and reindexing this data is a non-trivial operation when dealing with large data volumes.

    However, the key takeaway from this analysis should be that these are capabilities that any organization should have if they intend to scale their monitoring solutions. If they can be attained through a SaaS vendor, then this is a serious option to consider because it enables companies to immediately take advantage of this capability without the upfront, unpredictable and often ongoing cost of in-house engineering.

  • US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: PyPI complies with a “string of subpoenas,” and LTO continues to grow, despite predictions of its demise. (more…)