devopsy.kinsta.cloud

Tag: data integrity

  • Best Practices for Configuring MySQL Replication

    Best Practices for Configuring MySQL Replication

    MySQL provides several replication configuration options. However, ensuring it is done correctly may take time and effort, with considerable choices. Replication is a crucial initial step to enhance availability in MySQL databases. A properly designed replication architecture can significantly impact the accessibility of your data and prevent potential management complications. This article will delve into best practices for replication, covering local and long-distance scenarios.

    Active and Passive Setups

    When using active/passive mode for replication in MySQL, one server acts as the source while others serve as read-only replicas. The replicas can handle read-only queries, but all writes must be directed to the source. This configuration distributes the load across all replicas. Still, in the default asynchronous replication mode, there might be a delay between when data is written to the source and when it’s available on the replica. Keep this in mind while designing your application.

    Alternatively, the active/active configuration allows multiple servers to handle both read and write operations. While it may seem beneficial to have two servers for write requests, each server still needs to process the query workload of the other, making write distribution less effective. Additionally, MySQL lacks native conflict resolution logic, so conflicts can quickly arise, making it challenging to determine the true source of data without significant data loss when a rebuild is necessary.

    I recommend using the active/passive configuration for replication and sharding if you require higher throughput from your database.

    Enable GTIDs

    Global transaction identifiers (GTIDs) are unique IDs attached to transactions in a replicated environment. By default, replicas track processed records in the binary log file of a source database. However, this method is fragile if the source crashes and logs need to be restored.

    With GTIDs enabled, each transaction receives a UUID-based ID, allowing replicas to determine if a transaction has been processed. The GTID is a UUID representing the source server, followed by an auto-incrementing ID (e.g., 14a54b2f-2ad0-43b6-b803-72b5d7151d3b:1). As transactions occur on the source, the UUID remains the same while the ID increases. When a replica processes a transaction, the GTID or GTID range (e.g., 14a54b2f-2ad0-43b6-b803-72b5d7151d3b:1-10) is stored in the gtid_executed table.

    This process significantly reduces the risk of data becoming unsynchronized due to unprocessed or excessively processed transactions.

    Replication Modes for Optimal Configuration

    MySQL offers two replication modes that behave differently and should be understood to ensure optimal configuration for your environment. By default, MySQL employs asynchronous replication, where transactions are sent to the source, read by replicas and processed independently without source validation.

    Alternatively, MySQL supports semi-synchronous replication, enabled as a plugin. In this mode, transactions are received by the source and processed by replicas, but the source waits for at least one replica to accept the transaction before responding. Semi-synchronous replication provides greater data consistency, though it introduces some overhead in response time.

    With semi-sync mode, there are additional options that can be adjusted. By default, the primary server waits 10 seconds for a replica with semi-sync mode enabled to acknowledge the transaction. If data consistency relies on semi-sync, consider increasing this timeout value to ensure guaranteed consistency.

    It’s important to note that these modes can be mixed and matched. For instance, you can configure one replica with semi-sync and another without, ensuring that one specific server always has an up-to-date copy of your database while other replicas catch up at their own pace. This approach aids disaster recovery by easily identifying the best candidate for recovery.

    Understanding the distinctions between these modes will help you make informed decisions that align with your business needs.

    Optimize Replica Instances With Separate Log Storage

    Replica instances in your environment rely on the binary logs of the source for accurate information, as mentioned earlier. These logs are stored on the same disk as the database by default. However, this setup can overwhelm busy databases due to high throughput, including database manipulation and reading binary logs for replication. To address this, storing the binary logs on a separate disk from the database is recommended.

    This approach offers performance benefits and cost savings in cloud environments where free volumes have strict IOPS limits.

    Monitor Replication and Metrics in Managed Clusters

    Monitoring is crucial for all infrastructure, including replication. With monitoring, it is possible to know if data is being replicated after configuration. SolarWinds database performance (formerly VividCortex) and Prometheus are popular solutions for monitoring replication and other metrics in managed clusters. Regardless of the chosen solution, it is essential to promptly notify the appropriate individuals when issues arise and address them before they escalate.

    Create a Failover Plan for Minimizing Disruptions

    Inevitable software or hardware failures necessitate failover planning to minimize potential disruptions. Utilizing replication offers resilience by ensuring multiple servers house your online data simultaneously. Therefore, your team must possess a well-prepared strategy should the primary data source fail. The following example outlines an unplanned failover process:

    ● Ensure the downed source remains offline to avoid unexpected replication issues.
    ● Identify and remove the read_only option from the chosen replica that will serve as the new data source. If using semi-sync, select the replica previously configured with the plugin and source.
    ● Update your application to direct queries to the newly promoted source.
    ● Modify the remaining replicas to start replicating from the new source.

    Optimize Cross-Region Replication

    Cloud providers typically provide services across multiple geographic regions comprising several data centers. These data centers, known as availability zones (AZs), are strategically located within the same region but far enough to withstand disasters. Replicating databases to different physical locations is recommended but requires careful consideration. Replication introduces additional latency due to the distance the data needs to travel. Fortunately, latency between AZs is generally low, with AWS claiming single-digit millisecond latency within the same region.

    In contrast, there is significant latency between regions. For example, at the time of writing, the latency between us-east-1 and us-west-1 was reported to be over 60ms by Cloudping.co. Replication across regions results in replication lag, which is the time difference between data being written to the source and being written to a replica. This lag is even more pronounced when replicating across longer distances.

    Asynchronous mode should be used for replication across regions to avoid unnecessary delays for applications making requests.

    A comprehensive understanding of replication configuration options is crucial. However, it is equally important to determine the optimal configuration for these options. This can significantly impact your MySQL cluster’s operational efficiency and data-serving capabilities.

  • Dependability is Crucial for Real-Time Experiences

    Dependability is Crucial for Real-Time Experiences

    Millions of people around the world experienced a major disruption after an October 2021 outage saw Facebook, WhatsApp and Instagram go offline. A few days prior, the popular workplace collaboration platform, Slack, also went down for some users. And in the last few weeks of 2021, Amazon’s AWS caused considerable global chaos after significant networking issues in its busiest data center, US East 1, disrupting streaming services such as Disney+ and Netflix.

    Downtime can be costly, causing organizations to lose $5,600 per minute and up to $300,000 per hour, on average, according to Gartner. But it’s not only financial and reputational loss that businesses need to worry about. Outages also seriously downgrade—and in the case of real-time experiences, completely disrupt—the customer experience.

    At a time when consumers are demanding more live and interactive digital experiences, organizations can ill afford disruption or downtime, as customers quickly move to other services that ‘just work’.

    It’s not just the capacity within a real-time system that matters, but also globally distributed failover capabilities when something does go wrong. While organizations could try to build their own failover mechanisms, it’s extremely difficult to failover so services continue seamlessly, regardless of the status of its underlying infrastructure. Simple failover systems not only take longer to resume but typically suffer data loss during the time that a region is disrupted. Businesses need to know that even if, say, AWS suffers an outage, they won’t be affected.

    This is why it’s vital that organizations get it right when looking to develop their own real-time infrastructure. Managing tens of thousands if not hundreds of thousands of concurrent connections while synchronizing data in real-time means that if anything does go wrong it’s immediately obvious to users. If tech behemoths like Amazon and Facebook can’t get it right, what chance do organizations with far fewer resources have when building and managing their own real-time infrastructure?

    The Steps to Success With Real-Time Infrastructure

    Regardless of whether it’s for synchronized multi-user collaboration, audience engagement or instant updates, developers have a hard enough time developing even the most basic of real-time infrastructure, let alone making sure it’s dependable. There are four key elements to achieve dependability: Predictable performance, data integrity, reliability of service and high scalability.

    • Predictable performance: When we think of performance we usually imagine more computing power or speed. But when it comes to real-time, it’s not just about minimizing latency and bandwidth, it’s about having a predictable system. If developers have global latencies and bandwidth within certain operating boundaries, it provides certainty in uncertain operating conditions. This makes it easier to design, build and scale features around this certainty.
    • Data integrity: As we all know, the internet can be an unreliable network. Users abruptly disconnect and reconnect all the time, resulting in data that gets lost, is delivered in the wrong sequence or that is sent multiple times. As a result, organizations must ensure their infrastructure can guarantee high integrity without sacrificing anything else.
    • Reliability of service: The live nature of delivering updates in real-time requires total reliability. Outages can cause distrust and annoyance from users. Organizations could find their reputation taking a hit or customers quickly jumping ship if reliability proves to be a recurring issue. To prevent this, organizations should develop infrastructure that’s fault-tolerant and redundant at both a global and regional level so they can deliver a great experience even if there are multiple failures.
    • High scalability: More often than not, delivering an experience in real-time can result in unpredictable traffic; for example, a huge number of people wanting sports updates for a World Cup game. If thousands of devices connect over a short period of time but capacity can’t scale up quickly enough, then an outage will occur. To avoid this, developers must use infrastructure that can be elastic and meet changing levels of demand. Otherwise, the user experience is seriously affected and real-time capabilities crash when the network is swamped with connections it can’t cope with.

    While organizations often build their own infrastructure with the best of intentions, it’s easy for them to run into trouble. Most lack the internal expertise, resources and time; plus, going it alone often results in a huge technical debt. It’s also a massive undertaking, due to the amount of ongoing complexity and cost involved.

    Third-Party Partnerships Enable Dependability

    Unless organizations have the resources, finances and expertise of the likes of Amazon or Google, then building your own real-time infrastructure is just setting yourself up for failure. Instead, they should save on development cycles and focus on generating revenue more quickly by offloading their real-time infrastructure needs to a third party. This will mean organizations can design without limitations and focus on solving the challenges that really matter to them while putting in place real-time infrastructure that is as dependable and future-proof as possible.

  • Why Trust and Integrity Are Critical to IoT

    Why Trust and Integrity Are Critical to IoT

    We hear a lot about how the IoT is transforming people’s work and daily lives, but for me, its impact goes deeper. As a Type I diabetic, I depend on data from an IoT device to stay healthy. If I can’t trust my latest blood sugar reading, the results can be catastrophic. Health care use cases like mine represent some of the many examples that underscore the importance of data integrity. Stacey Higgenbotham expressed it perfectly in a recent post: “If a business or consumers can’t trust the data, the IoT fails.”

    The IoT is all about making better, more informed decisions, and data is the foundation that makes those decisions possible. As the IoT gains maturity, we’re seeing its impact across more and more industries and use cases. 

    In agriculture, smart sensors are in widespread use for real-time crop monitoring to help farmers track the state of their products. Tools like motion detectors, light detectors and moisture sensors can be brought together to support predictive analysis and enable smarter, more automated decisions. IoT sensors are also tracking the location and health of livestock to help farmers detect disease outbreaks and isolate animals if needed. 

    Most of us have seen how the IoT has revolutionized the safety and efficiency of the home, providing unprecedented control over appliances, heating and cooling systems, smart metering and lighting. Security cameras, alarm systems and smart locks are strengthening physical security and bringing residents peace of mind. As these smart home systems mature, they are also becoming increasingly integrated, personalized and automated.

    The global pandemic is also helping shape IoT adoption. Forrester predicts that manufacturers, pharmaceutical firms and utilities are increasingly switching to remote experts and technicians for support. To manage and repair their assets without extended downtime and costly travel, these industries have begun connecting previously disconnected machines.

    What do these diverse use cases have in common? Without trusted data from connected sensors and devices, all of them become vulnerable and the decisions they support become questionable. Inaccurate moisture data from a sensor could cause a farmer’s crops to fail. A compromised smart lock could leave a home vulnerable to theft. In a manufacturing environment, inaccurate telemetry from a connected machine could lead to factory floor shutdowns or damaged assets. The positive business outcomes enabled by the IoT all hinge on working with accurate, timely and trusted data.

    Signing Is Key to Strengthening IoT Data Integrity

    Establishing data trust should be top of mind for IoT device manufacturers, and it is imperative that they take steps to ensure that data is not being manipulated in transit or at rest. This protection applies not only to data packets exchanged by the devices and applications but firmware updates and supply chain processes.

    The recent executive order signed by president Biden underscores the need for measures like code signing. Focused on improving the nation’s cybersecurity, the EO presented guidelines and general best practices for federal departments, agencies and contractors. However, its impact is also likely to extend to technology and infrastructure companies. 

    Some of the strongest direction in the EO is intended to improve the security of software as well as supply chains for technology manufacturers. The order establishes baseline security standards for the development of software sold to the government, including requirements for improved visibility into that software and making security data publicly available. The order acknowledges that too many devices and software releases contain vulnerabilities that bad actors could exploit. 

    Effective Code Signing Requires Strong Management 

    Fortunately, a proven solution is available today to help manufacturers ensure data authenticity, confidentiality and trust for their IoT solutions. Best practices like code signing can help manufacturers bake security into each and every stage of their development processes. By applying digital signatures, organizations can assume full control of their development. They can confirm the integrity of their code before it moves further along in the development cycle and is put out in the field for devices and their customers. 

    Code signing is only as effective as the processes and best practices that support its implementation. To make the most of digital signatures, developers and engineers need a holistic approach to management. The signing process should be seamlessly built into their development processes to simplify and ease adoption. 

    A common management platform should enable manufacturers to fully protect private key usage and signatures, as well as monitor and manage access to digital keys. The process should also enable organizations to support granular monitoring of their code signing processes and include historical tools like reporting and auditing for full accountability and compliance. With a process in place to continually sign firmware, software updates and other components that can ensure data integrity in device interactions, manufacturers can be assured that their IoT solutions will work as intended, using data their end customers can trust. 

    There’s no question that as the IoT continues to extend deeper into our business and personal lives, the importance of data integrity is also rising. The stakes will only increase as more systems, processes and life-or-death decisions rely on trusted IoT data. To sustain the soaring growth of IoT use cases and solutions, it’s up to manufacturers to build security into the supply chain. With a strong data foundation underpinning today’s rapid innovation, the potential of the IoT revolution is limitless.

  • The Benefits of Serialization

    The Benefits of Serialization

  • Data: the New Currency That Accelerates Business

    Data: the New Currency That Accelerates Business

    The amount of data created every day is staggering: 2.5 quintillion (that’s 18 zeros) bytes, to be exact. And that number is growing exponentially, thanks to mass deployments of internet of things (IoT) networks and devices. Cisco Systems estimates that by the end of 2019, IoT will generate more than 500 zettabytes of data per year. In the past two years alone, 90% of all the world’s collected data was generated.

    If it’s hard to imagine that such staggering volumes of data can be generated so quickly, just look around. There are IoT devices in our homes, vehicles, business offices, hospitals, shopping malls and parking lots and along highways, each generating and collecting data. Myriad sensors, cameras and listening devices collect human-generated data. Such devices are pervasive, and the number of data-capturing sources are making the information collected about us ubiquitous.

    Value vs. Privacy

    One might say we are human data machines. So, is data the measure of a person? While this is a relatively new phenomena, data has always been available about our activities, likes, dislikes, motivations and more—it just hasn’t been collected, aggregated and analyzed in the way it is today. It’s like we’ve discovered a vast vein of data gold that has always been hidden beneath the surface.

    IoT is becoming big business. Yet, the real value of IoT is in the data, as meaningful insights are derived and then leveraged for a business advantage. With artificial intelligence, automation and business intelligence, the more data collected, the smarter the applications become and companies gain more insights, increasing the value of data. However, there are legitimate security and privacy concerns as organizations collect our information, most often without our knowledge.

    Data is becoming the currency that drives our economies and a motivation manipulator that impacts our societies, with both good and bad results. If data represents currency, as it helps companies achieve greater revenue and efficiencies, how can consumers be in control of their own data and share in its rewards? How can they make sure their information is not put in the hands of those who want to use it for ill-gotten gains at their expense? These, and other important questions, need answers and solutions that protect consumers, as we look for innovative ways to put a boundary around our personal digital vaults before our data enters the marketplace.

    As consumers, do we care that this data is being collected? Is the benefit of receiving free services worth the price of our information? What should consumers do to make sure their data isn’t used in ways they don’t agree to? Perhaps consumers should expect companies to include them in their revenue-sharing. If they want to share in their data monetization, those collecting the data should ensure the data is protected and private, with full data-use transparency.

    Data Value Directly Related to Data Integrity

    When malicious actors gain access to data, not only does it put us at risk, but it also can lower the value of data. Consider this scenario of a hacker gaining access to data from the insurance company’s network-connected device in your car: The data on that device could have lowered your premium, but the hacker changed the information to show much higher speeds and extreme braking, then threatened you to pay a ransom or risk the hacker sending the altered data to the insurance company. In this case, because the IoT device was hacked, the financial impact of your data changed from a financial advantage to a financial disadvantage. Remember, data value is directly related to its integrity.

    There are two important things we must consider regarding IoT data. The first is to consider how the data is being secured. Is it encrypted while in transit? Are encryption and authentication deployed to prevent unauthorized users from accessing data when it is stored at rest? The second consideration is around privacy. How is the data being used and shared? Is sensitive data that we want protected being consumed and shared? Are device manufacturers providing consumers with transparency regarding whether and how their data is being used? Without establishing firm foundations in security, privacy and transparency, the protection and the monetization of our data will continue to be out of our control.

    — Mike Nelson

  • IoT, Not People, Now the Weakest Link in Security

    IoT, Not People, Now the Weakest Link in Security

    Do a quick search on “people weakest link” and you’ll see a raft of articles in which cybersecurity experts and computer scientists point to employees or end users as the biggest vulnerability. No doubt, people do silly (or outright abusive) stuff and open the door for a variety of enterprise security problems. Spoiler alert: There’s this technology called the Internet of Things (IoT) and it’s a bigger security risk than people ever were. The new theme of this rapidly changing IoT security story is data integrity.

    Increased cybersecurity risk is due in part to the variety of environments IoT applications touch. Most at-scale IoT deployments use some form of a three-tier architecture, with edge, gateway and cloud components.

    • Edge devices are those connected sensors and actuators touching the real world. They are usually small, low-power devices often running a wireless protocol such as Bluetooth, Thread, ZigBee, Z-Wave, sub-GHz, low power wide area (LPWA) networks, cellular IoT technology, Wi-Fi or other protocol.
    • Gateway devices are hubs that groups of sensors connect with. One big role of the gateway is to translate the various protocols from edge devices into Internet Protocol (IP) for transmission into an enterprise network. Gateways also perform a key role in provisioning edge devices securely. Smartphones running Android or iOS often serve as a gateway for consumer IoT devices. For industrial IoT devices, multiprotocol gateways typically are built on a small form factor computer running a real-time operating system or Linux. Edge computing or fog computing architectures use enhanced gateways for more algorithm processing.
    • Cloud platforms are very familiar to most enterprise developers. In IoT apps, data from the edge arrives for processing, storage and presentation. Algorithms can run on a server cluster or in a distributed computing framework such as Hadoop or Apache Spark. Business intelligence and predictive and prescriptive analytics also reside in the cloud.

    In our practice, we see many IoT cybersecurity issues that can be mitigated with proper attention to two areas. The first is shutting down simplistic endpoint attacks by using basic connected device security measures. Avoid edge devices that ship with a well-known default password or PIN—compromising those defaults can be as easy as a battery pull and a factory reset. If a device uses encryption, make sure it doesn’t use a hardcoded key that, if cracked, compromises an entire family. Look for edge devices using over-the-air (OTA) programming allowing for security updates over a long life cycle. As more device developers gain IoT security experience and provide the right features, closing these basic weakest links is easy.

    Now, for the more startling weakest link. When the IoT application runs, is the data any good? According to a just-released survey of 950 IT and business decision-makers, only about 48 percent of companies are confident they can detect an IoT security breach. (BTW, don’t get me started on blockchain; it’s one tool, not a panacea.) A lot can happen before your people get involved. By then, it may be too late.

    Data integrity needs constant monitoring during at-scale IoT deployment, not just during pilot. An easy way into an unprotected IoT network is to add an unauthorized edge device at a gateway, then use it to access other devices. Gaps in end-to-end encryption are also easy exploits. Untrustworthy public cloud elements are also bad news—a big reason we recommend hybrid cloud architecture. A private cloud layer gives you more control over algorithm performance and data stream security. Presentation and archiving can be left to a public cloud layer.

    Bad data can come from either a compromised or faulty device. It propagates quickly across a system, and when an unwitting user at the presentation layer runs a business intelligence tool on it, the bad data could cause issues. How do teams hunt down vulnerabilities and faults that lead to corrupted data and results?

    • Hire a penetration tester, preferably a contractor removed from your day-to-day development who can objectively assess your architecture during pilot and reassess it during deployment.
    • Ensure your architecture can tolerate edge device or gateway failures without ripple effects in algorithms; in other words, protect the application from prolonged operation on bad data streams.
    • Before turning AI loose on automated decision-making, consider an AI-based IoT security analytics tool to spot anomalies and quickly bring people into the loop.
    • Retarget at least part of your cybersecurity experts to ongoing data integrity efforts and help them stay abreast of IoT technology and trends.

    This all sounds like a good job for DevOps, doesn’t it? You know what to do about people; now you’ll need to work magic with devices. The long-held security axiom that people are the weakest link is no longer true when IoT applications run unattended all the way to presentation. There are a few exceptions when people have more interaction at the edge, particularly in the medical device arena. Generally, we’ll see bigger and bigger IoT applications with more data, more processing and less interaction with people.

    If someone on the IoT says there’s a security hole you can drive a truck through, they may mean it literally. Data integrity measures will be critical for DevOps teams moving forward. Do you agree? What types of data integrity issues are you seeing in IoT applications?

    — Don Dingee