devopsy.kinsta.cloud

Category: DevOps in the Cloud

Building Scalable Cloud Applications With MySQL

Relational database systems such as MySQL have long been the go-to choice for handling large and complex data handling tasks. But, with the introduction of new technologies like NoSQL, many developers started to question whether traditional relational databases still have the upper hand. However, when building high-scalability cloud applications, relational databases like MySQL still have the edge over NoSQL. Here are five reasons MySQL remains superior to NoSQL in supporting developers of high-scale cloud applications.

1. Enablement of Complex Queries

Structured query language (SQL) is the most widely adopted language for relational databases, and MySQL is no different. SQL gives developers a powerful platform for creating complex queries and ad hoc reports, efficiently managing large datasets and creating tables and indexes for data optimization. By contrast, NoSQL usually offers only rudimentary query and data manipulation capabilities, often proving inefficient when handling complex data sets. Most data analysts are also familiar with SQL, so it’s a drop-in tool they can use to help gain insights.

2. Scalability and Reliability

MySQL is noted for its scalability when handling massive data loads and is particularly well suited to cloud-based applications that require consistently high performance. MySQL ensures that the consistency of data is maintained. MySQL is the go-to option for scalability and reliability for larger cloud applications. Companies such as Facebook, GitHub and WordPress have built billion-dollar businesses on MySQL, and it’s been tested and validated in some of the harshest and most demanding workloads of any database.

3. High-Performance Replication at Scale

MySQL replication offers a reliable solution for cloud applications that need to scale. Unlike other databases, MySQL’s replication capabilities provide effective data safety, making it a robust choice for applications that require high availability and durability. Additionally, MySQL enables seamless version upgrades without disrupting operations, ensuring the system always has the latest features and security patches. The read scaling feature of MySQL improves performance by distributing the read load across multiple instances, making it ideal for large-scale cloud applications. With a proven track record at scale, MySQL replication is the optimal choice for applications that want to grow and scale efficiently.

4. High Availability and Durability

MySQL provides enhanced high availability and durability, ensuring that critical data is always available even during maintenance and disaster recovery. This is because MySQL is fully ACID-compliant (atomicity, consistency, isolation, durability) and provides full support for transactions. Even when the system fails, the transaction can be rolled back and the data is not lost. With NoSQL databases, data is primarily written in memory and doesn’t provide the same degree of durability or consistency.

5. Ease of Implementation and Debugging

MySQL is easy to install, implement and debug compared to NoSQL. MySQL is supported by a large community of developers and database administrators who use it for a diverse range of applications. MySQL is also open-source and free to use, making it a great choice for businesses of all sizes. NoSQL systems require the installation of additional tools, libraries, and other components for implementation, making it more complex and time-consuming.

MySQL provides many benefits for developers of high-scale cloud applications, ranging from its powerful SQL language for complex querying to its high availability and flexibility for managing multi-table joins. NoSQL remains a viable option in some cases, but developers who are building high-scale cloud applications are mainly sticking with MySQL due to its scalable and reliable nature, easy implementation and flexible query-handling capabilities. Choosing the right database engine is critical for any organization that relies heavily on large and data-heavy applications. MySQL continues to stand the test of time when it comes to ensuring performance and getting consistent results.

October 6, 2023
Latest DORA Report Highlights Importance of DevOps Culture

A 2023 Accelerate State of DevOps Report published today by the DevOps Research and Assessment (DORA) team at Google Cloud found teams with generative cultures composed of people who felt included on their team have 30% higher organizational performance than organizations without a generative culture.

The report was based on a global survey of 36,000 IT professionals and tracked deployment frequency, change lead time, culture, change failure rates and failed deployment recovery times.

Overall, the survey found that while just under a third of respondents were successful in terms of their ability to deploy on demand (31%) and keep remediation efforts after deployments to 5% (33%), only 18% had lead times for changes of less than one day, while only 17% had a failed deployment recovery time of less than an hour.

This is an area where DevOps teams appear to need the most amount of help; nearly two-thirds (64%) of software releases or updates required some level of remediation after deployment, according to the report.

Nathen Harvey, developer advocate for DORA and Google Cloud, said the one area most underappreciated when it comes to improving performance is documentation. High-quality documentation leads to 25% higher team performance, the survey found.

DevOps teams that rely on the public cloud also benefited from a 22% increase in infrastructure flexibility, resulting in 30% higher organizational performance, noted Harvey.

The survey also noted that, as far as reliance on artificial intelligence (AI) to automate DevOps workflows is concerned, it’s still early. But most respondents are, at the very least, experimenting with one or more use cases.

In general, the management of DevOps will soon be very different as more AI capabilities are leveraged to eliminate bottlenecks and reduce toil. The immediate challenge is developers are benefiting more from AI than software engineers, which is resulting in more code starting to simultaneously move through DevOps pipelines. There is little doubt that in the months ahead there will be increased reliance on AI to manage pipelines at greater scale. In the meantime, it will be increasingly difficult to manage codebases that are steadily getting larger as developers become more productive.

At the same time, more organizations are embracing platform engineering as a methodology for centralizing the management of DevOps workflows at levels of scale that previously were difficult to achieve.

Hopefully, the combination of platform engineering and AI will result in less overall burnout among DevOps teams. In the short term, the burnout issue may become more challenging to address as the amount of code being created increases.

In the longer term, the results of the DORA survey may be substantially different next year as AI and platform engineering are employed more widely. The current survey results are not substantially different than in previous years, in the sense that DevOps teams are still challenged by many of the same issues.

Of course, as important as it might be to improve those metrics, the most important thing any DevOps team should focus on is developer productivity which, despite the existence of DevOps pipelines, remains relatively low.

October 5, 2023
Something is Amiss in the Open Source World
You don’t have to be a psychic to grok that the winds of change are blowing hard in the open source world. For the last few months, I’ve seen hints and heard whispers about a new direction for what I believe is the most passionate and enthusiastic community in tech. I think we are about to sail off into unchartered territory—and it will be interesting to see what the repercussions are.

Of course, this will all play out against the backdrop of continued layoffs and belt-tightening in the broader tech sector. Before I detail what I think are the telltale signals of change, let me state that I am and have always been a big supporter of open source software. I firmly believe we live in a golden age for open source software, which I call the “Foundation era of open source.” At a time when 90% plus of organizations use open source software, many of the leading open source projects are under the domain of not-for-profit foundations (NFPs) such as the Linux Foundation, the Apache Foundation, etc. This allows multiple commercial companies who might otherwise compete with each other to work in coop-etition. This has, in many ways, enabled this golden age of open source. Prior to the Foundation era of open source, there were two prior dominant models. Initially, there was what I call the “Cathedral and Bazaar era,” named after the great book by Eric Raymond. This was a time of idealistic notions of “pure” open source, unsullied by commercial aspirations. To me, Dr. Richard Stallman embodied that time; if you were to make an analogy to history, it was akin to the 1960s era in the U.S.

From that era, let’s zoom to sometime in the early 2000s when we moved to what I call the “Big Brother era” of open source. In this era, open source projects were owned and managed by one vendor, for all intents and purposes. The project could have a vibrant community, but ultimately, it was the managing vendor or “Big Brother” who reaped most of the rewards. This made it difficult for multiple tool vendors to support the same project and stifled progress.

The dawn of the foundation era of open source changed all that, and we have seen transformational growth as a result. But nothing lasts forever—and I think the king of the hill may be on the way down. Why do I say this? Here are several events/trends I’ve noticed recently that lead me to believe this. Some of these things I was told in confidence, so I cannot reveal names to protect the innocent.
1. The move away from “pure” open source licenses. Over the last few months, several pillars of the open source world have moved away from accepted open source licensing. Red Hat has done so with Red Hat Enterprise Linux (RHEL) and now Hashi has, as well. Now, to some, this change is subtle and it’s not a big deal. Only paying customers get access to source code. The overwhelming majority of users never look at the source code anyway. Nevertheless, this is a huge red flag for the community—the accepted licenses and a core tenant of open source, that source code is available to everyone, are no longer being used.
2. Diminishing corporate financial support for open source communities and foundations. I guess this was to be expected in hard economic times. The path to profits from supporting open source foundations, projects and communities can sometimes be more of a dotted line than direct. And a dotted line doesn’t usually work when the bean counters are involved. In looking to cut expenses, these are low-hanging fruit.
3. Closing foundations. This is a natural result of the economic strife. Foundations may operate as not-for-profits, but without operating funds, foundations can’t operate. Generally, foundations get a lot of mileage from volunteers. However, volunteers can only go so far before hard costs are involved. Also, volunteers are able to volunteer because their corporate employers empower them to do so. When their employers pull back the funding for those activities, the volunteers have to do what they can to earn a living. I have heard of at least one NFP foundation forced to cease or curtail operations due to corporate sponsor funds drying up.
4. Orphaned projects. Again, this is a direct result of number two, above. With the foundations closing and supporting funds drying up, I know of at least two bellwether open source projects that are in danger of becoming orphaned. Both of these projects boast huge community numbers and dominate their markets. The fact is though, that supporting a major project costs hundreds of thousands of dollars. While millions of community members may use a project and support it, they don’t contribute dollars. Dollars that pay for hosting, infrastructure and other hard costs. Coughing up a few hundred grand to support a project is a bridge too far for some companies in the current economic climate.
What does this mean for open source software and you? Well, in the near term, if you are just a consumer of open source software, probably not much will change day-to-day. You may have to cough up some money if you want access to source code or support from the community. Bigger picture, though, is the winds of change are blowing. I think we are going to see more companies taking a Hashi-type approach to protecting their source code. This will result in more forks of open source projects. However, my experience is that most open source forks don’t succeed (with some notable exceptions—like Jenkins forked from Hudson). Companies who are building businesses around open source models will be the most at risk. I think an open core business model may be the best bet for an open source vendor. I think the future of NFPs as the stewards of open source may become a bit cloudy (no pun intended). What will take its place or fill the void? That remains to be seen.

For the record I was never a fan of that old Irish proverb, “May you live in interesting times.” This kind of upheaval is a little too interesting for me. But one thing I have learned in 30 years in tech (and more than that in life) is that change is constant. You can embrace it and thrive or shrivel and die.
October 5, 2023
Meta Mayhem: Hybrid Work FAIL ¦ Yet More Layoffs

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Meta’s enforced hybrid work plan is failing badly, and Meta makes more layoffs. (more…)

October 4, 2023
Tricentis Launches Mobile Application Testing Grid

Tricentis today added a Virtual Mobile Grid service to its portfolio to make it simpler to test mobile applications at scale.

The service is based on the platform Tricentis gained via its acquisition of Waldo and is now integrated within the Tricentis Mobile application testing framework.

Tricentis CTO Mav Turner said the Virtual Mobile Grid extends an existing Tricentis Device Cloud service for accessing mobile devices to include the ability to author, execute and analyze tests that can be run across thousands of real and virtual devices.

In effect, Tricentis Mobile brings together Tricentis Testim Mobile, Tricentis Tosca Mobile and the Tricentis Device Cloud into a single platform, he noted.

DevOps teams can now test end-to-end at every layer of a mobile application using a platform that leverages neural network technology to test user interfaces. They can also build and run no-code tests to identify and resolve critical user interface and performance issues through real-time tracking of more than 130 key performance indicators (KPIs).

Organizations building mobile applications that must be deployed at scale and regularly updated are integrating testing platforms such as Tricentis within their continuous integration/continuous deployment (CI/CD) platforms. Many of those efforts are tied to digital business transformation initiatives that rely on mobile computing platforms to access a wide range of backend services.

There is a lot of debate over how far left application testing should be shifted. In theory, the more developers test applications themselves, the fewer issues there should be once an application is deployed in a production environment. However, organizations need to be wary of increasing the cognitive load on developers, especially to the point where they spend significantly less time writing code, noted Turner. There will still be a need for dedicated testing teams to conduct tests across multiple stages of the application development process to ensure quality, he added.

Testing mobile applications is more critical than ever because the cost of failure—in terms of impact on the business—is much higher as more organizations depend on software to drive business processes. The challenge is that many of these app developers don’t have a software engineering mindset, so they need testing platforms that automatically create and run tests on their behalf in a way that can be customized for specific use cases.

Over time, the quality of applications should steadily improve as many of the low-level mistakes developers make are eliminated. That should improve everything from application security to the user experience as applications are continuously updated.

Theoretically, as more artificial intelligence (AI) is applied to the testing process, the number of reasons tests are not run as frequently as they should be will steadily decline. Today, it is still too difficult for many developers to author tests, so they naturally don’t conduct as many as they should. The simpler it becomes to author and execute those tests, the more likely it becomes a test will actually be run before an application gets deployed in a production environment.

October 4, 2023
Avoiding Cloud Lock-In Amidst Potential Regulatory Change

At the end of my street, there’s a beautiful and much-admired garden with vibrant flowers and a well-manicured lawn. Next to this splendid garden is an abandoned vacant lot, long-neglected and overgrown with weeds. Every time I walk by, I think about how this neglected space could be transformed into something magnificent (like this garden), but no one wants to take responsibility for it. Similarly, many CIOs and CTOs have “discarded vacant lots” to deal with in their own work lives: The problem of cloud lock-in.

Cloud lock-in can be a significant headache for business leaders. Even the UK’s communications sector regulator, Ofcom, has expressed concerns about the issue. Earlier this year, Ofcom announced that it would refer the UK cloud market for investigation to the Competition and Markets Authority, highlighting barriers to interoperability and concerns about high egress fees charged by the cloud hyperscalers.

But organizations shouldn’t just wait (and hope) for regulators to solve this problem. As a true believer in harnessing the full benefits of the cloud, I propose two practical ways to combat cloud lock-in.

The Upside of a Technical Approach

The first option is straightforward: Have an honest conversation with your cloud service account leader. Express your concerns about lock-in and negotiate more favorable terms and conditions. This approach might yield some results, but it’s not a guaranteed success. So, there’s a more effective strategy: Adopt a technical approach that empowers you to avoid vendor lock-in altogether, without costing you an arm and a leg.

Imagine that, instead of trying to turn your neglected space into a garden alone, you could access a ready-made gardening community, where multiple people work together to create something amazing. The same principle applies to cloud databases. Rather than relying exclusively on one hyperscaler’s technology stack, you can opt for platforms that offer portability and flexibility across different cloud providers.

Let’s consider databases, a critical underlying component of many mission-critical business applications. Each cloud hyperscaler will offer a range of SQL, NoSQL, and graph database options, which cater to various types and scales of technical problems. The challenge is to avoid getting locked into a single cloud stack, which hinders your ability to switch between clouds if and when needed.

One viable solution is to choose “vanilla Postgres” – the basic open source version of PostgreSQL available from postgres.org or any cloud provider. This will certainly provide initial flexibility, but you are likely to encounter limitations as your application grows and requires enterprise-level features.

The real benefit of the cloud lies in modernizing your data layer to fully exploit cloud-native features like scalability and resilience.

To achieve this, you need databases that run across different public clouds and share common APIs. Born-in-the-cloud technologies, designed from the ground up for cloud environments, offer the most flexibility and benefits.

Examining your current cloud vendor’s offerings is natural, as you’re already on their platform. However, keep in mind that these services are customized to their specific cloud environment only, so this could limit your future growth. If you decide to change cloud providers or adopt a multi-cloud strategy, moving across the cloud database “matrix” can be cumbersome due to the inconsistent APIs.

If you were restricted to a single set of gardening tools when trying to transform that discarded space into a beautiful community garden, it would be nearly impossible and would not be fit for purpose.

You need to realize that these services and their functionality will always need to be customized in some way to the specific cloud environment. AWS’s fully managed relational database engine, Aurora, is compatible with MySQL and PostgreSQL, but it was specifically designed to run on Amazon. This means it can’t run on-prem, and it certainly won’t work in GCP (Google’s cloud).

As a consequence, you will have to make time-consuming and costly changes to your application to accommodate a bespoke version of PostgreSQL that will run efficiently in the cloud environment of your hyperscaler partner. You’re going to be locked into that version of the application because it can’t really work anywhere else. The same applies to the Google equivalent of vanilla Postgres (CloudSQL for Postgres), or the Microsoft version (Azure PostgreSQL).

If you truly want a multi-cloud strategy, you need to consider how you go, for example, from mid-range Google to low-end Amazon. Scaling is much discussed, but surely one of the main selling points of cloud is that you can go down as easily as you go up.

The Business Advantages of Multi-Cloud are in Reach

To achieve the advantages of multi-cloud, you need to consider databases that offer common APIs and portability across clouds. There are several open source PostgreSQL-based database options available, which enable you to keep your cloud database options open and avoid the trap of cloud vendor lock-in.

The wait for regulatory intervention might be long and uncertain. Taking proactive steps now to mitigate cloud lock-in is crucial.

Adopt a scalable open source PostgreSQL option to immediately enjoy many of the benefits of multi-cloud. Embracing this strategy now can spare you potential panic moves when and if regulatory changes come into play.

By freeing your business from the shackles of cloud lock-in, you can unleash the true potential of multi-cloud environments. Just like that “discarded vacant lot” waiting to be transformed into a beautiful garden, your applications can blossom in a multi-cloud ecosystem, which allows you to easily scale up or down and enjoy the best that each cloud provider has to offer.

So, don’t wait for someone else to fix the problem. Take action today and start cultivating your multi-cloud future.

October 3, 2023
Best Practices for DevOps in the Hybrid Cloud

In rеcеnt years, various tools, technologies and frameworks have еmеrgеd with thе goal of enhancing flеxibility, performance and scalability. Traditional monolithic approaches have been replaced by more modular methods like microservices and nano-services. Additionally, the rise of cloud computing has led to on-prеmisеs software being replaced by cloud еnvironmеnts that offer a wide array of advantages and capabilities that were previously unavailable.

Bеing able to optimize spееd and efficiency through the adaptation of cloud settings is a vital capability. To leverage cloud-bursting capabilities and also safeguard their applications and data, businеssеs can embrace a hybrid cloud strategy combined with DеvOps. In this article, we will dеlvе into bеst practices for implеmеnting DеvOps in a hybrid cloud sеtting.

What is a Hybrid Cloud Environment?

A hybrid cloud, which combines on-premises and cloud computing modеls, allows you to harness the benefits of both approaches. Hybrid cloud computing allows businеssеs to blеnd their own on-prеmisеs infrastructurе with thе powеr and flеxibility of cloud computing. This approach allows them to optimize their current infrastructure while also benefiting from the scalability and cost efficiency offered by public cloud environments.

By lеvеraging thе strеngths of both, companies can optimizе their opеrations for scalability and flеxibility and еnhancе sеcurity for thеir applications and data. One notablе advantage of hybrid cloud еnvironmеnts is their ability to adapt rеsourcе capacity to mееt еvolving workload dеmands rapidly.

A hybrid cloud offers flexibility and cost optimization to businesses. With a hybrid cloud, enterprises can scale up their resources and adapt to changing needs, much like a public cloud. At the same time, it allows them to retain control over sensitive data and critical applications stored on-site.

By adopting a hybrid cloud environment, organizations gain enhanced data control and the ability to flexibly allocate workloads based on evolving business needs. Organizations can achieve improved flexibility, enhanced security and regulatory compliance by adopting a hybrid cloud model.

What are the Benefits?

Here are the benefits of a hybrid cloud:

Cost-effectiveness: The cloud offers a cost-effective solution for deploying workloads. When using the cloud, an organization does not have to spend much money to purchasе additional infrastructure. Moreover, most cloud environments provide support for a pay-as-you-go pricing model, which implies you only have to pay for what you consume.

Avoids Vendor Lock-in: Companies often face the challenge of becoming too dependent on one cloud provider during the transition to the cloud, making it difficult to switch to another provider if it becomes necessary to do so. However, by adopting a hybrid model and specifically migrating to a platform that supports multiple providers, companies can mitigate this risk and avoid being locked into one vendor.

Improved Developer Productivity: By deploying a hybrid cloud platform, developers will be able to significantly enhance their productivity. With this approach, developers can leverage Agile and DevOps methodologies more efficiently and deploy their applications across multiple cloud infrastructures simultaneously.

Infrastructure Efficiency: A hybrid cloud offers development and IT operations teams enhanced and smooth control over resources. Organizations can also leverage cloud services and combine them with data stored in the cloud or on-premises infrastructure to create more value through the hybrid cloud.

Better compliance and security: Hybrid clouds offer significant benefits for business growth and acceleration. They enable faster product development cycles, promote innovation and expedite time to market. Additionally, they facilitate prompt response to customer feedback and allow for the delivery of applications in closer proximity to customers. Hybrid clouds also facilitate the integration of new services from third-party providers.

Cloud-bursting: One valuable feature of hybrid cloud environments is called cloud bursting, which enables organizations to easily increase their resource capacity to meet varying work demands.

An Overview of Hybrid Cloud Architecture

A typical hybrid cloud architecture comprises of the following components:

On-Premises Infrastructure: This refers to your privately owned physical infrastructure that can be used for storage, network, host servers, etc.

Private Cloud: Generally speaking, a private cloud is an environment that is dedicated solely to the use of a single organization. It can be hosted either internally within the organization’s own premises or externally, but it is securely protected and not accessible to the general public.

Public Cloud: The term public cloud refers to a computing model in which services such as compute, storage, deployment, etc. is provided by third-party providers as a fully managed service over the internet.

Cloud Management Platform: A cloud management platform is responsible for managing the operation and allocation of computing resources across various cloud platforms. It also helps businesses effectively use their hybrid cloud environments.

Orchestration and Automation Tools: Streamlining tasks such as resource provisioning and data synchronization across different environments requires the use of orchestration and automation tools. These tools are crucial in ensuring efficiency and seamless operations.

Best Practices for DevOps in the Hybrid Cloud

Albeit the benefits it provides, successfully implementing DevOps in a hybrid cloud requires you to adhere to certain best practices:

Monitoring: Ensure your on-premises and cloud environments are monitored and managed effectively by using the right tools. These tools offer a comprehensive view of your resources, regardless of their hosting location.

Infrastructure-as-Code (IaC): Use tools like CloudFormation, Terraform or ARM to automate infrastructure provisioning and management. IaC ensures that infrastructure is built consistently and reliably.

Containerization: Container technologies such as Docker and container orchestration platforms like Kubernetes abstract the underlying infrastructure and make moving applications between on-premises and the cloud easier.

Centralized logging: Make sure you have a centralized solution for metrics, logs and traces (like ELK Stack, Prometheus and/or Grafana) that can collect data from both the platform and the cloud environment.

Automation: From CI/CD pipelines to infrastructure deployment, it’s critical to ensure that processes are automated by leveraging tools like GitLab CI, GitHub Actions and/or Jenkins.

Security: You should consistently implement a robust security strategy. Consider the unique security requirements of a hybrid system, such as security of data transfer between disk and the cloud, encryption at rest and in transit and consistent identity and access management.

Performance: Understand latency and bandwidth issues when transferring data between disk and the cloud. Choose the right architecture (such as edge computing) and optimize your applications to minimize these issues.

Data management and integration: You should understand where your data comes from and pay attention to data gravity. You should also link applications and data sources on the server and in the cloud to ensure smooth data flow.

Application architecture: You should design cloud-agnostic, modular and flexible applications. A microservices architecture often works well for hybrid cloud environments.

Disaster Recovery and Backup: Take advantage of both on-premises and cloud resources to establish a robust disaster recovery and backup strategy.

Administration and policy management: You should enforce consistent policies across both environments using policy-as-code by leveraging tools such as Open Policy Agent (OPA).

Training and skills development: Note that hybrid clouds require knowledge across multiple platforms. Ensure your team is trained and has the skills to manage and operate a hybrid environment.

Budget and cost management: You should monitor costs on the server and in the cloud. Tools such as AWS Cost Explorer, Azurе Cost Management and third-party solutions can provide insight into where your money is going.

Communication and collaboration: Although there are challenges and obstacles, you should encourage open communication and team collaboration.

Regular reviews and audits: Review your architecture, tools and procedures to ensure they align with business goals and are as efficient as possible.

What are the Challenges?

Despite its many advantages, hybrid cloud also comes with some challenges. These challenges arise from the complexities of managing multiple environments and integrating them together. Here are the key challenges of hybrid cloud at a glance:

Security Concerns: Hybrid cloud models can pose security vulnerabilities, particularly if there is inconsistent implementation of security protocols across different environments. It is important to properly secure data that moves between private and public environments to mitigate potential risks.

Data Transfer Costs: When moving data between private and public clouds, it’s important to consider the potential costs involved, particularly when dealing with large volumes of data. Being mindful of these expenses is essential when designing hybrid solutions.

Data Sovereignty and Compliance: You must ensure that data complies with regional and industry-specific regulations when it is stored or processed in multiple locations, including public clouds in different regions.

Latency Issues: One of the challenges that may arise when implementing cloud-based applications is the integration with existing legacy systems. This can be a problem, especially for applications that require real-time or high-performance capabilities.

Vendor lock-in: Vendor lock-in can still be a challenge even with a hybrid approach, especially if organizations heavily depend on proprietary tools or services from a specific cloud provider, making it difficult to migrate or integrate with other platforms.

Skills gap: One challenge in managing and operating a hybrid cloud is the potential for a skills gap, since you’ll need a diverse range of skills, such as knowledge of both on-premises and cloud technologies. To address this, organizations may need to provide training or hire new talent with the necessary expertise in both areas.

Monitoring: Achieving visibility and effective monitoring can be challenging when resources are spread across multiple environments. It often requires specialized tools or platforms to achieve end-to-end visibility and monitor effectively.

Governance and Policy Management: Although it is crucial for maintaining security and compliance, it can be challenging to implement consistent governance and policy management across both private and public clouds.

Cost Management and Optimization: While a hybrid cloud can provide cost benefits, it can also be difficult to track and manage costs across multiple platforms. This requires ongoing monitoring and the implementation of optimization strategies.

Network Connectivity: To achieve secure, reliable and fast connectivity between on-premises and cloud environments, a well-designed network infrastructure is essential. This may involve additional investments in network services or infrastructure to ensure seamless connectivity.

How Hybrid Cloud Supports DevOps Culture

Here’s how hybrid cloud supports DеvOps culture:

Automation: Automation plays a crucial role in both hybrid cloud and DеvOps. By utilizing tools and services provided by cloud providers, automation еnablеs the provisioning, scaling, and management of infrastructure. This aligns with the core principles of DеvOps, which include automated installation, integration, and utilization.

Flexibility: Unlike with a single platform or tool, DevOps teams have the freedom to select the most suitable services and platforms from various cloud providers and incorporate them into their own solutions.

Cost Efficiency: DevOps teams can achieve cost efficiency by using public cloud resources for temporary workloads like test environments or short-term projects. Your teams can optimize their costs by only paying for the resources they actually use, thereby optimizing the amount of money they spend on those resources.

Fast deployment of resources: The fast provisioning and de-provisioning of resources are crucial in a DevOps culture. Hybrid cloud allows for rapid deployment of environments, supporting the iterative and dynamic nature of DevOps practices.

Collaboration: Hybrid cloud solutions often include collaboration tools and platforms that promote effective communication and teamwork, aligning with the collaborative culture of DevOps.

Continuous Integration and Continuous Delivery (CI/CD): Hybrid cloud provides tools and platforms that support CI/CD pipelines and enable consistent and continuous integration, testing and transition to production (delivery/deployment).

Service Integration: Many cloud providers provide a wide range of services, including machine learning and databases, as part of their cloud offerings. Using these services will allow you to seamlessly integrate them into your applications, fostering a culture of continuous improvement and innovation.

Monitoring and feedback: Hybrid clouds provide robust monitoring, logging and analytics tools that offer valuable insights into application performance and user experiences.

Regulatory compliance: With a hybrid cloud approach, IT teams can maintain the security and compliance of sensitive data and applications by housing them in a private cloud or controlled environment. This way, organizations can meet their compliance and security standards effectively.

Conclusion

A hybrid cloud combines on-premises infrastructure with cloud computing, offering scalability, flexibility and improved security for applications and data. A hybrid cloud infrastructure should be more flexible than a traditional cloud to meet changing requirements. Therefore, businesses with hybrid cloud capabilities can adapt more quickly to business and technological changes.

Businesses can leverage several benefits by implementing DevOps practices in a hybrid cloud environment, such as improved agility, faster release cycles, enhanced scalability and better collaboration. With a hybrid cloud setup, companies can leverage the strengths of both on-premises infrastructure and cloud computing models to achieve scalability and flexibility.

October 3, 2023
The Role of Observability Engineers in Managing Complex IT Systems

The rise of microservices, the adoption of cloud-native architectures and the increasing shift to digital have led to the growing volume of telemetry data (logs, metrics, traces, events) contributing to the complexity of observing modern IT systems effectively and at the right cost. These challenges with processing the telemetry data on time and at the right cost make it difficult for organizations to maximize their observable surface area and ensure they perform at their best.

Enter observability engineers. These professionals are tasked with collecting, processing, analyzing and visualizing data from complex IT systems. They use this data to identify patterns, detect anomalies and understand system behavior. This awareness allows observability engineers to discover and address potential issues before they impact users.

Observability engineers play a critical role in ensuring the reliability, performance and security of complex IT systems, and are essential for organizations that want to stay ahead of the curve in today’s rapidly evolving technology landscape.

Let’s examine the essential characteristics and responsibilities of observability engineers to understand their role.

Characteristics of Observability Engineers

Observability engineers are problem solvers who specialize in optimizing system performance, ensuring reliability and driving actionable insights from telemetry data. They are the driving force behind data-driven solutions and operations. Their specialized skillset and unique focus on observability set them apart from other roles within IT operations. They may have previous backgrounds as system administrators, site reliability engineers (SREs) and IT operations members, who often react to incidents, but observability engineers are proactive and have adopted a shift-left approach to observability. They are adept at understanding the telemetry data and its formats, creating telemetry strategies, implementing comprehensive monitoring systems, and using advanced observability platforms to gain real-time insights.

Another important aspect of observability engineers’ expertise is their in-depth knowledge of telemetry data. They have the necessary skills to collect, analyze and process data such as metrics, events, logs and traces from various sources. They understand the transformations that the data needs for easy consumption by downstream SIEM, analytics, or visualization systems. They are well versed in the challenges posed by microservices architectures, cloud-native environments, and hybrid infrastructure setups and how to extract meaningful information from the system exhaust. This understanding enables them to design telemetry pipelines to move the right data, in the right format, to the right system; build monitoring systems; and implement observability practices that effectively capture, transform and analyze these complex systems.

Collaboration and cross-functional skills are other essential aspects of the observability engineers’ role. They break down the barriers between observability domains, including infrastructure, applications and networking. By coordinating and aligning the efforts of these domains, observability engineers create a more comprehensive understanding of the IT system’s behavior and performance. This approach allows for a holistic analysis beyond the limitations of individual domains and identifies cross-domain optimization and improvement opportunities.

Responsibilities

Observability engineers have many responsibilities focused on addressing problems within IT operations. Some of their key responsibilities are detecting anomalies, troubleshooting incidents, monitoring system health, optimizing resource allocation, enhancing user experiences, enabling data-driven decision-making and supporting compliance and security efforts. Through their skills and expertise, observability engineers empower organizations to maintain highly performant, reliable and secure IT systems.

They are needed throughout the lifecycle of IT systems. They play a vital role in system design and implementation, ongoing maintenance and monitoring, incident response and troubleshooting, optimization, performance enhancement and new feature development and releases. Their expertise ensures that observability is built into the system from the ground up, maintains optimal system performance, minimizes downtime and enhances the user experience.

How Observability Engineers Perform Their Tasks

Observability engineers use telemetry data to learn trends and patterns. Analyzing this data allows them to plan capacity, predict potential issues, and implement preventive measures. This approach enables organizations to continuously improve their IT systems, enhancing performance, reliability and user satisfaction.

To achieve their objectives, observability engineers use various tools for collecting and analyzing data. Telemetry pipelines are emerging as their primary tool for collecting data from various sources within an IT system, transforming it, and routing it to various destinations. These pipelines enable observability engineers to gather data from metrics, logs, events and traces. They process and transform this data to extract meaningful insights and send it to downstream analytics or visualization platforms or for long-term storage.

Various analytics and visualization APM and observability platforms allow observability engineers to analyze and visualize the data. Custom dashboards and visual representations offer them a comprehensive view of system behavior, performance trends and user experiences. With this information, observability engineers can troubleshoot and optimize system performance, finding root causes of issues and implementing effective solutions.

They also contribute to compliance and security efforts. By monitoring and analyzing telemetry data, observability engineers identify possible vulnerabilities and risks within the system. They help organizations maintain robust security practices and ensure compliance with regulations.

Investing in the Promise of Observability Engineering

Observability engineers are increasingly becoming essential for managing the complexity and unpredictability of modern IT systems. Their specialized skills and expertise enable them to address potential issues, optimize system performance and ensure reliability. As technology advances, the demand for observability engineers will only increase. Organizations striving to succeed in their IT operations must invest in these professionals.

October 2, 2023
FCC: Net Neutrality is a ‘Thing’ Again ¦ Meta Shutters Big UK Site

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Network neutrality is back on the FCC’s agenda, and unused London office gets Zucked. (more…)

September 27, 2023
Dell Taps Microsoft to Simplify Hybrid Cloud Computing

Dell Technologies, in collaboration with Microsoft, today unveiled a platform that tightly integrates an on-premises IT environment with the Azure cloud.

Caitlin Gordon, vice president of multi-cloud product management for Dell Technologies, said the Dell APEX Cloud Platform for Microsoft Azure is the first instance of a hybrid cloud computing platform designed specifically to address that requirement using a Premier Solution of the Microsoft Azure Stack hyperconverged infrastructure (HCI) platform that Microsoft added to its portfolio today.

The Premier Edition of Microsoft Azure Stack HCI makes available an instance of that platform that provides a more turnkey management experience than the existing editions, said Gordon.

In the face of increased economic headwinds, more organizations are looking to lower the total cost of managing IT by reducing the number of control planes that need to be managed. Each platform that an IT team typically supports today has its own control plane that requires IT staff to manage it. Reducing the number of control planes creates an opportunity to reduce the total headcount currently required to manage multiple IT platforms.

It also serves to streamline workflows in a way that enables DevOps teams to reduce the current level of friction encountered when managing multiple IT platforms, noted Gordon.

At the core of Microsoft’s approach to hybrid cloud computing that Dell is embracing is Arc, a management framework that makes it possible to manage Azure cloud instances alongside on-premises IT environments. In terms of the ability to manage hybrid cloud computing environments, Microsoft is significantly more advanced than any other cloud service provider, said Gordon.

It’s not clear whether IT organizations are embracing HCI platforms running an instance of the same core stack of software that Microsoft relies on to provide Azure services. If IT organizations expect to lower the cost of managing IT, there is no doubt they will need to modernize on-premises IT infrastructure. The older the platform, the more expensive it becomes to automate using various third-party frameworks.

Regardless of the approach, IT teams need to reconcile the need to invest in automating workflows that span multiple IT platforms. Over time, those investments will reduce the total cost of IT, but getting the required funding to start that process will require IT teams to make a strong case for increasing the return on investment (ROI).

In the meantime, the number of workloads running both in the cloud and at the network edge is only going to increase. At the same time, the bulk of existing workloads continue to run in local data center environments. Most IT teams will be required to find additional ways to automate the management of all those environments simply because the cost of adding additional IT staff to manage highly distributed computing environments as scale will otherwise be prohibitive.

As such, the issues that IT teams are facing today may be a matter of not if the management of IT will become more automated but how soon.

September 26, 2023