devopsy.kinsta.cloud

Category: DevOps Practice

  • Learning ‘The Second Way’ of DevOps – Continuous Feedback

    Learning ‘The Second Way’ of DevOps – Continuous Feedback

    As indicated in my prior blog Learning the ‘First Way’ of DevOps–Continuous Flow, many organizations are struggling to realize well-engineered DevOps, even the ‘first way’ of DevOps – continuous flow.

    The first way of DevOps, continuous flow, is the foundation for DevOps. Once mastered, an organization will naturally want to master the second way of DevOps to achieve greater benefits that continuous feedback offers when engineered well. DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Feedback

    There are no training resources dedicated solely to the second way of DevOps—continuous feedback. Many of the same resources used to learn the first way of DevOps have some relevant information, though. DevOps monitoring, release automation and security tools publications, articles and tool vendors are a good source. A DevOps expert who is experienced with mature DevOps can select the training information that is relevant to the specific continuous feedback tools and feedback metrics used by the enterprise.

    Why This is Important to Engineering DevOps

    Continuous feedback requires a mature understanding of DevOps concepts and practices. Specific skills are needed for implementation and operation of advanced monitoring tools and practices for application performance, database performance, advanced continuous testing, advanced release management, infrastructure performance and continuous security.

    How Is Learning Continuous Feedback Accomplished?

    The following are recommended engineering practices for learning Continuous Feedback.

    Roles and responsibilities are assigned to experts for defining a training program for DevOps continuous feedback (e.g., DevOps sensei, training PM, evangelist). Continuous feedback DevOps skills gap assessments are conducted periodically. Dev and Ops training strategies are designed to address gaps and priorities of the organization for DevOps continuous feedback. A training budget that includes allocations of expenses and time allocations per role exists to support continuous feedback. Continuous Feedback goals and SLAs are defined for each role.

    Recruiting job profiles match skill requirements for DevOps skills relevant for continuous feedback. Training resources are available and visible to those that need continuous feedback training. Training goals set for teams and individuals include goals for continuous feedback. Incentives and recognition programs support Continuous Feedback training. Continuous feedback training progress is tracked and visible.

    What Is Needed to Implement Learning?

    The following are recommended engineering practices for implementing a learning program for continuous feedback.

    Assign roles and responsibilities to experts for defining a training program for DevOps continuous feedback (e.g., e.g., DevOps sensei, training PM, evangelist). Conduct a continuous feedback DevOps skills gap assessment. Define a training strategy that best addresses the gaps and fits the needs and priorities of the organization. Define a training budget that includes expenses and time allocations per role and skill.

    Determine which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly commercial training Identify training goals for each role. Ensure recruiting job profiles match skill requirements for DevOps skills that are most relevant to the organization’s strategic goals.

    Put in place the means to make training resources visible and available. Set training goals for teams and individuals. Put in place incentives and recognition programs for training. Track training progress.

    Some of the best sources of DevOps training resources are listed below:

    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro
    3. Udemy.com
    4. Recorded talks and publications of IT Revolution.
    5. Books such as my own “Engineering DevOps,” by Marc Hornbeek.

    What This Means

    To accomplish the benefits of the second way of DevOps—continuous feedback requires a commitment to continuous learning and mastering skills beyond what is needed for the first way of DevOps–continuous flow. Specific skills are needed for implementation and operation of advanced monitoring tools and practices for application performance, database performance, advanced continuous testing, advanced release management, infrastructure performance and continuous security. Organizations need a trained workforce to succeed with DevOps. Organizations must approach DevOps training strategically and continuously to leverage training resources to their best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • Get the Benefits of Low-Code Without the Risks

    Get the Benefits of Low-Code Without the Risks

    Today, more than 300 vendors and various platforms offer different flavors of low-code solutions. However, the majority of these low-code tools are really no-code tools that benefit individuals or groups trying to solve a specific program. The list of benefits of low-code tools in your environment is long and compelling; however, there are security risks that businesses should be aware of when working with these powerful tools.

    When evaluating or using a low-code tool, it’s important to remember that while low-code tools abstract the process of writing code, code is still being written. Applications created with low-code tools are susceptible to the same attacks as any other web application. However, companies don’t have control over the code generated by a low-code tool, so the burden of developing secure applications is largely shifted to the low-code tool vendor. Therefore, the best way to minimize risk when using a low-code vendor is to properly evaluate that vendor’s security practices and test applications created using these tools as vigorously as you would any application developed using a traditional/pro-code approach.

    Risks to consider include access control and authentication (permissions, multifactor authentication), data security (encryption in transit and at rest) and application security from vulnerabilities and attacks (network security and intrusion detection). Other risks to consider are risk mitigation covering areas such as DDOS (distributed denial of service) attacks, backup and disaster recovery, and security incident handling. Responsibility for management of these risks belongs to the low-code vendor, the hosting provider and the company using the low-code tool.

    The following tips should be considered when bringing new low-code tools into your environment to minimize security risk. Remember, it’s critical to perform due diligence of the vendor. Items to consider include:

    Hosting Platform: Be sure the vendor is using a reputable hosting provider, ideally Amazon Web Services (AWS), Google Cloud Platform, Salesforce or Microsoft Azure. Leading players will offer secure, highly available platforms that address the aforementioned risks and will have completed third-party audits and certifications demonstrating their commitment to security.

    Vendor Infrastructure Management: Low-code vendors typically have some level of responsibility for management of the upper tiers of the overall stack; this is generally referred to as the shared security model. Review the vendor to ensure they have solid controls and processes in place to properly manage and secure the infrastructure. Ideally, the vendor regularly submits to some form of recognized audit, such as a SOC 2 Type 2 or ISO-27001. This helps ensure controls and processes are in place in areas such as security, availability, confidentiality, privacy and processing integrity to address the aforementioned risks in the area of the hosting platform the vendor is responsible for.

    Formal SDLC: The vendor should follow a documented software development life cycle (SDLC) and change management process used for the development of their low-code tool. A good SDLC process includes using a source code management system, dependency checking process, static code analysis, automated unit and regression tests, peer code review, secure coding practices training, application vulnerability scanning, quality assurance testing process and third-party penetration testing.

    Policies and Processes: Documented policies and processes covering information security, business continuity, disaster recovery, security incident handling, infrastructure hardening and more should be in place at the vendor. This helps demonstrate that the vendor is committed to providing a secure, highly available and high-quality offering.

    The company adopting the low-code tool itself should also consider internal processes and how it’s using the low-code tools. Items to consider include:

    Formal SDLC: Even though the company uses a low-code tool, there’s still the need to follow a formalized SDLC. It’s important for the company to emphasize and embrace secure programming practices, even for low-code developers. Low-code tools have the ability to implement customer-developed code snippets or components and it is recommended companies follow programming standards and secure programming practices as outlined by the Open Web Application Security Project (OWASP) foundation. Another area of emphasis that can sometimes be overlooked is the need for testing, including functional and security-oriented testing.

    Security Testing: Consider performing a penetration test of the completed low-code application. While ideally, the vendor has performed these tests on the tools itself, it’s a good idea to also perform one on the completed low-code application.

    Access Control and Authentication: Configure access and permissions according to the least privilege model. Leverage advanced authentication mechanisms like multifactor authentication and single sign-on where possible. Ensure an access removal process is in place to remove access for terminated employees.

    Data Security: Verify the application is configured to use HTTPS and that all databases are encrypted. Low-code vendors may or may not provide a database, and either way, the company likely has some level of responsibility for securing the data, regardless of who is hosting it.

    Policies and Processes: Documented policies, processes and related training for team members helps mitigate risk. Some level of formality is recommended for documentation and training for the SDLC, business continuity planning, disaster recovery and security incident handling. For example, the team responsible for the low-code application needs to know how to handle a security incident if one occurs, whether it’s due to an issue in the low-code application, the low-code tool or the hosting infrastructure.

    The benefits of low-code tools far outweigh the risks of introducing these tools into your environment, but with a little effort up front, you can save a lot of headaches down the road.

  • Leveraging AI in DevOps for Non-Linear Scaleup

    Leveraging AI in DevOps for Non-Linear Scaleup

    With technology evolving in leaps and bounds, AI is shaping the future of digital transformation for every business seeking speed, scalability, quality and flexibility.

    Since DevOps is on the verge of delivering exceptional software development environments, DevOps in AI is undeniably the next breakthrough in technology. We’ve witnessed every industry racing toward the adoption of AI to transform their business. To support this transformation, smart automation solutions coupled with AI implementations is becoming the new normal.

    So if you’re thinking about reinforcing your enterprise’s DNA by leveraging AI, DevOps principles for artificial intelligence are vital to these efforts. Here’s how businesses can unlock the true potential of AI and stay ahead of the curve.

    DevOps and AI

    DevOps has been s game-changer for numerous businesses. Applying AI can helps DevOps teams achieve the operational efficiency of machine learning (ML) models all the way through the journey from design to production.

    Artificial intelligence undeniably enhances automation and fast-tracks the performance of DevOps. Since the digital transformation process is quite challenging for diverse businesses, DevOps and AI ensures the right processes are in place that can further bring flexibility throughout the journey.

    DevOps enables continuous deployment and quick delivery of AI models via:

    • Speed: Improved overall development and delivery time by reducing unnecessary activities
    • Scalability: On demand and auto-scaling AI models
    • Quality: Boosting cleaning datasets and eventually promotes consistent learning that further enhances AI
    • Data monitoring: Decision making and computing helps handle huge amount of data with precise data monitoring

    Benefits of Integrating AI in DevOps

    There are endless benefits to integrating AI in DevOps.

    Analysis

    Data analysis is the key to business success, especially in times when there’s a huge list of competitors rushing to gain consumer trust. Since DevOps produces an enormous amount of data, it isn’t practical for humans to analyze the data without technological intervention.

    Artificial intelligence’s smart analytics eases the process by identifying and solving problems in a
    more systematic way, resulting in efficiency and increased consumer satisfaction.

    Data Correlation

    In the wider technology environment, development teams have their own set of issues and errors, which arrive while monitoring the processes. Since the communication scope is minimal and there’s a lack of mutual learning across teams, artificial intelligence can help to accelerate the learning cycle.

    Artificial intelligence helps to improve data insights from multiple platforms, which, in turn promotes better communication and establishes better data correlation.

    Failure Handling

    Machine learning is paving the way to better error resolution based on data, and eventually artificial intelligence will be able to predict signs of failure by examining patterns. Since humans cannot match the observation capabilities of AI, certain indicators of failure can only be analyzed through AI. This further enhances the overall development processes, as quick identification helps identify issues before they can impact the software development life cycle.

    Breaks Down Silos

    DevOps with AI has the ability to streamline business processes and establish proper communication among diverse systems, removing barriers that slow down the development and deployment processes. Furthermore, the ability to automate, continuously integrate and enhance communication helps business leaders to focus more on other aspects that can further accelerate business growth.

    Economic Growth

    Since the basic function of an AI-based model is automation, it can help businesses save money and enhance human resources. For any business, eliminating repetitive human work is a tough nut to crack. but a necessity when businesses are demanding continuous innovations. Artificial intelligence could be the best way for enterprises to cut costs and eventually helps the organization to use human resources more creatively and productively.

    Makes Software Intuitive

    Besides endless business benefits, AI-based software is more intuitive and user-friendly, since it prioritizes the reduction of human error. While DevOps is a well-known, reliable, business-driven approach for delivering software, intermingling artificial intelligence could enhance the user experience by integrating technology more deeply into the system.

    The perfect harmony of simplified interactions between humans and AI helps businesses achieve their desired goals of speed, efficiency, scalability and security.

    AI-driven systems are providing promising and scalable solutions for enterprises that are on a hunt
    for innovative and rapid ways to scale business growth. AI in DevOps is undoubtedly the best solution for accelerating growth through precise monitoring, automation and process improvement. This is the right time for businesses to ensure they operationalize artificial intelligence as a core business objective to stay ahead of their competitors.

  • Digital Customer Experiences: The Future is Modular

    Digital Customer Experiences: The Future is Modular

    With the proliferation of microservices and APIs, will UIs go modular too? Some technologists note a rising trend around smaller, intent-based customer experiences that more directly mirror API endpoints.

    I recently met with Adam Evans, co-founder and CTO of Airkit, to better understand what’s broken with today’s digital customer journeys and how they need to evolve. He predicted that single-purpose experiences will become more pervasive in the future. In the process, we’ll likely encounter fewer native apps and fewer downloads. To enable all this, he’s putting faith in low-code to bring easier integration and automation patterns to more developers.

    The State of Digital Experiences

    Digital customer experiences are becoming increasingly connected and web-based, and at times, disconnected from the core service. In headless environments, end users are engaging with companies through more and more abstraction layers, such as chat, SMS, voice or email notifications. These interactions are increasingly modular and event-driven, noted Evans.

    Increasingly, user encounters are the result of trigger-based patterns activated when user action is required. Not only is the communication style changing, but new devices and platforms have emerged, further making a case for a decoupled development approach. B2C apps become more modular in this transition, presenting experiences built around specific tasks and closely reflecting the specialized microservices or APIs that drive them. Consider the rise of the single page application (SPA) style or AWS Lambda as evidence.

    As more automation and subscription models enable a “set and forget” attitude to commercial software, Evans predicted we’ll likely see more intent- and event-driven workflows emerge. He said these reimagined UIs would shroud platform complexity and, instead, focus on single, specialized actions, like submitting an insurance claim, checking the status of a package or updating your coffee subscription.

    A Broken Customer Journey

    So why would end user digital experiences evolve into more intent-driven applications? Well, complex digital platforms can get feature-bloated very quickly. Bulky applications with many menus are difficult to navigate, thus impacting user experience.

    As Evans describes, generic interfaces often display tons of information irrelevant to the task at hand. “Don’t make me go through generic interfaces to help me do something specific,” he said.

    Clunky interfaces are at odds with small digital tasks that involve an attention span of 20 seconds or less. Evans describes these as “micro touchpoints in our daily lives where you don’t need traditional, large applications.” These actions could be, for instance, looking up COVID-19 infection rates, checking your credit card balance or updating a streaming subscription, he said.

    Organizations have other imperatives to improve user experience, too. For example, if a platform isn’t self-service, it can increase support calls, which can cost a company significant resources. Plus, if the service isn’t event-driven, users must routinely check in of their own accord, which could waste time and encourage tech anxiety.

    Single Page Applications

    In the age of modular microservices, the days of bulky apps and “manual polling” may be limited. To Evans, the solution lies in expressing the “minimum way to represent the transaction of that interface.”

    “How do we keep moving smaller divisions from the back end to the way interfaces are built?” Evans asked. Similar to the way microservices architecture separates domains, Evans noticed UIs segmenting attention along API endpoints. He envisioned more single-purpose experiences being constructed that extend APIs and microservices with a user interface.

    Technically, this may result in more single page applications being developed. Niche UI layers that map APIs would likely carry a data model, Evans described, so that micro apps are delivered with data for entering fields into API calls. As they are web-based, asynchronous single-purpose experiences could also transcend the barriers of app stores and third-party marketplaces.

    Potential Benefits of Modularity

    More intent-based workflows mean less overhead for the user, bringing quicker, more convenient access to the problems they want solved. Sure, users still register with a core service, such as their bank, coffee subscription service or insurance agency for centralized account management. But, users frequent account centers less often, as the periphery of interactions around them continues to grow.

    “The future of apps will look more like individual, intent-based micro apps,” Evans reiterated. These would reflect a metronome of touchpoints Evans called “little moments of delight.”

    Pinging users only when action is required could reduce check-ins, thus reducing digital anxiety. Aside from the user benefits, deployment and maintenance for small applications may be easier to support, said Evans. Lastly, more isolated intentional customer experiences increase engagement, response rates and sales, he added.

    Using Low-Code For Customer Experiences

    Connected digital experiences are relying more and more on APIs. Yet, API integration can be difficult, even for experienced programmers. So, how do we make it easier for people to build interfaces to hit these APIs? Building user-facing layers on top of APIs will require great developer experience, says Evans.

    Lately, low-code has been dominating these discussions. With greater emphasis on visual-driven development, low-code is empowering non-technical employees. To Evans, low-code capabilities will be essential to influence how future customer journeys are constructed.

    Evans noted how low-code “changes the whole game around how apps are built.” For example, branded experiences often require reusable code and common libraries, all of which low-code platforms excel at. In addition, when working with APIs, a “layer over REST design is essential to maintain state and enable long-term processing,” he said.

    Will The New Digital Journey Go Micro?

    Complex interfaces seem to work for conglomerate giants like Amazon. But other scenarios may benefit from more single-purpose workflows. As platforms become more extensive, the user interactions need to become leaner to avoid added stress. As a result, Evans predicts that micro, trigger-based B2C touchpoints will become more common in the coming years. In this paradigm, better UI extensions of common APIs could help bring people into the equation.

    “We’ll see a lot of cool stuff in the next decade,” he said. “Future apps won’t look like what past ones looked like.” As IT moves to support new, multiple channels, such as chat, voice, web and VR/AR, our perception of what an application is also will inevitably alter.

    In tomorrow’s “great unbundling of apps,” as Evans called it, “if you don’t use it every day, there’s a pretty good chance it doesn’t need to be there.”

    There are many schools of thought on the best methods to retain quality user experiences. Do you think modularity is the answer? What do you think tomorrow’s digital experiences will look like? Feel free to let us know in the comments below.

  • Modernizing Continuous Integration

    Modernizing Continuous Integration

    In the duality of continuous integration/continuous delivery (CI/CD), CI focuses on build automation. Leveraging CI practices for software delivery is normal for most organizations and can be seen as a solved problem. However, the demand for increasingly distributed applications has risen with the expansion of microservices, and development teams have the expectation that every commit should be a build.

    Continuous integration has, surprisingly, become an unexpected bottleneck with these expectations. To begin modernizing CI, it’s important to first understand some of the challenges. Let’s dig in.

    Challenges of Continuous Integration

    Given that build and release candidates follow advancements in development technology closely — such as new languages, packaging and paradigms in testing the artifact — expanding the capabilities in continuous integration implementations can be challenging. With the introduction of containerized technology, the firepower and velocity required to build greatly increased.

    Scaling Continuous Integration Platforms

    The infrastructure required to run a distributed CI platform can be as complex as the applications they are building because of the heavy compute requirements. Consider how much a local machine’s resources are tied up during a local build and test cycle. Now, multiply that by the number of folks on a team or in an organization and you start to understand just how much compute power is being used. Distributed build runners are just one area that can be complex, and managing when new build nodes are spun up and spun down can depend on the platform/end user.

    Keeping Up With the Technology Velocity

    The adage of “the only constant in technology is change” is true. New languages, platforms and paradigms are to be expected as technology pushes forward. The ease of including new technologies in a heterogeneous build or accepting new testing paradigms, however, can be difficult for more rigid CI platforms. These legacy platforms were designed for only a small subset of technologies. With legacy or rigid approaches, the dependency management required to maintain technical velocity becomes a significant burden.

    What it Takes to Modernize Continuous Integration

    The simplest way to change continuous integration practices and platforms is to take what I call the four-pillar approach: platform infrastructure modernization, engineering efficiency strategy, test optimization and rapid pipeline development. Making strides in any of these pillars will put you on the path toward modernizing your continuous integration platforms and practices. Making strides in all of the pillars will set up your DevOps teams for significant success.

    Platform Infrastructure Modernization

    The location where the actual builds and packaging take place (e.g. build nodes/runners) do most of the heavy lifting and have a fairly elastic workload nature. Builds (e.g. a JAVA JAR build) and packaging (e.g. a Docker Image Compose) are compute-heavy tasks. Once the build and packaging are complete, the runners can sit idle. This shows the importance of having an ephemeral build node. The build node spins up during the task, then spins down or is destroyed after the build task is done, so as to not drain resources.

    Engineering Efficiency Strategy

    A core tenet of engineering efficiency is meeting your internal customers where they are. For software engineers, this means being as close to their tools and projects as possible. Like many modern pieces of application infrastructure, shifting left to the developer means being included in the project structure in source code management (SCM).

    A common disconnect in CI platforms is dependency management. Over the past decade, for software engineers, this problem has been solved with package or build tools such as Maven, Gradle and NPM. Engineers need to simply define implicitly or explicitly what they need, and the dependencies will be resolved. Continuous integration tools can suffer from a disconnect, since we are leveraging several tools that potentially don’t have a common syntax. So, again, it’s important developers have close oversight on all the tools they need to complete their projects to reduce this friction.

    Test Optimization

    While initial confidence in getting tests into the CI pipeline expands, more tests — and sometimes inappropriate test coverage — is introduced by the ease of integration with the pipeline. An even harder problem to identify and rectify is flaky tests. A flaky test is one that both passes and fails periodically without any code changes. This becomes a twofold problem: increasing execution time and lack of confidence with flakiness. The most logical solution to avoid these issues is adding test optimization. A modern continuous integration solution should be able to visualize pipeline order, timings and overall execution to help identity (and eventually rectify) excessive coverage and flakiness.

    Rapid Pipeline Development

    Software is an exercise in iteration. The lower the barriers to entry for iteration to occur, the more gains in engineering efficiency and agility are achieved. Local builds happen dozens of times before reaching a committable stage, moving them forward to a dev-integration environment. It’s here where having a local environment is key. Oddly, most continuous integration pipelines are designed to run externally from a local machine. So, the ability to move to a local CI pipeline helps build confidence that the externally run pipeline will be successful.

    Continue on Your Continuous Integration Journey

    Continuous integration might seem like a solved problem for many organizations, but as with any technology, there is always room for improvement and modernization. Like any engineering efficiency paradigm, CI is a journey and the destination ever-evolving.

  • No-Nonsense Guide to Distributed Databases

    No-Nonsense Guide to Distributed Databases

    I got my first DBA job 15 years ago, and quite a few things have changed since then. The amount of data exploded, we saw the rise of NoSQL, and business logic moved away from being stored in relational databases to applications.

    To this day, the three pillars of reliability, scalability, and maintainability are the banes of a DBA’s existence. To spice things up, your monthly cloud bill for managed databases just made your CTO lose the will to live.

    So, now you have two extra things to worry about in addition to these three pillars: cost efficiency and vendor-agnosticism.

    In this article, I’ll (try to) run an open source, new-generation database, benchmark it and touch on some of the challenges of being cloud-neutral.

    TL;DR

    • Solving the SQL vs. NoSQL dilemma in the Kubernetes world
    • How to leverage a distributed SQL database in a multi-cloud world
    • Taming performance – single cloud versus multi-cloud
    • Running TPC-C as a transactional benchmark
    • Taming the costs – the question of egress costs

    New Packaging For a New Age

    Just five years ago, people were like, “Containerizing stateless microservices? Sure! No problem.”

    But when you actually containerize a database, your persistence layer is asking for trouble.

    Over time, containers grew on us. Now, it’s the default way of packaging and shipping software. Heck, even AWS Lambda caved and started running container images. Databases just couldn’t resist.

    Once a database is containerized – say, a PostgreSQL database – Kubernetes takes over dealing with some of the more painful aspects of reliability and maintainability (with the help of middleware-specific operators).

    Containers decouple data from the runtime, so upgrades and configuration are easier through infrastructure-as-code (YAML). Hardware failure and maintenance windows are covered by Kubernetes’ built-in capability to reschedule pending pods on surviving nodes. Full scheduled backups or point-in-time recovery are in the hands of the operator’s automation.

    But what about scalability?

    Kubernetes’ vertical pod autoscaler can adjust container resources, deleting and recreating pods with new resource settings. As a traditional relational database, PostgreSQL can have only one primary replica that writes data. This means that you’ll be facing a  short downtime while the new pod comes online. The majority of workloads are about five times oversized, so while sizing is painful, it’s better than no sizing at all (please, have mercy on your CTO’s budget).

    NoSQL has none of these limitations. Databases like Cassandra were built for scaling horizontally without downtime.

    Enter Distributed SQL

    What if we could blend Google Spanner’s ability to operate at distances with strong ACID consistency, Cassandra’s scaling and PostgreSQL compatibility?

    This is where YugabyteDB helps.

    YugabyteDB is an open source, relational, distributed SQL database management system. It was designed to handle large amounts of data across multiple availability zones and geographic regions, all the while delivering single-digit latency, high availability and no single point of failure.

    I hate when the new kid on the block brings their own syntax language because, you know, their case is so very special – just like everybody else’s.

    Yugabyte doesn’t do that. They have a wicked-smart storage layer (inspired by Google Spanner) that reuses the PostgreSQL query engine. This means no mental strain on the development team – you get to use the time-tested, marketable and incredibly powerful PostgreSQL language-level features that many are already comfortable with.

    If Google Spanner is so inspiring, why is it not taking over the world of distributed SQL? Well, it’s available only as SaaS, and heavily depends on the specifics of Google Cloud hardware like atomic clocks (trueTime) and Google’s backbone network. If your private data center or favorite cloud provider is not GCP – well, no Spanner for you, my friend. Personally, I’d be hesitant to marry into the ultimate vendor lock-in. Yugabyte doesn’t have any dependencies and would happily run on any cloud, private data center, VM or bare metal – and if your coffee machine runs Docker containers, you can run Yugabyte there, as well.

    If Yugabyte is robust, scalable and easily maintainable, you have the top three pillars covered. But what about the other two pillars your CTO is nagging you about (cost efficiency and a vendor-agnostic approach)?

    How to Make Your Database Vendor-Agnostic

    Being vendor-agnostic means being able to switch between major cloud providers at any point without depending on prohibitively expensive services like DynamoDB. This approach also allows you to leverage inefficiencies in cloud pricing so you can look for new opportunities to reduce cost and dynamically shift resources between cloud providers online.

    This is something CAST AI can do.

    The CAST AI platform acts as a cloud abstraction and makes the whole multi-cloud, vendor-agnostic requirement quite trivial.

    Developers just interact with standard Kubernetes (no CRDs), and the complexity of several clouds is abstracted. A Kubernetes cluster in CAST AI can span across several cloud providers at the same time, bringing some unique cost optimization opportunities.

    What if spot/preemptible instance interruption wasn’t totally random anymore, but something you could actually predict? You could be proactive and—by operating in several compute bidding markets at the same time—reduce the risk of running out of capacity.

    Why not slash your monthly cloud bill radically by running some of the database containers on spot/preemptible nodes?

    Highly Available Stateful Application on Multi-Cloud

    Yugabyte on a multi-cloud CAST AI Kubernetes cluster is a robust design without a single point of failure.

    Active/Passive solutions are so ’90s—failovers cause brief downtime, and DBAs tend to leave the failover to be manually initiated due to occasional short-lived network blips. Active/Passive solutions constantly require testing and validation. Will your disaster recovery work on darker days? Because these tests cause downtime, nobody does them regularly.

    The best highly available solutions are active-active, don’t require user failover, and don’t rot if not tested regularly.

    CAST AI has a tutorial in place that takes you through every step of building the above solution in about 10 minutes, from scratch.

    Go ahead, open your cloud provider’s console and release your inner chaos monkey by killing Yugabyte, business app pods or whole Kubernetes nodes.

    CAST AI, Yugabyte, and Kubernetes were built with a core tenet that nothing is guaranteed; nothing should be shared. But keep in mind, consensus algorithms work when there’s still a majority left to vote N/2 + 1.

    You may be asking, what’s the performance impact of increasing resiliency when going multi-cloud with a distributed database?

    Let’s find out.

    Let’s benchmark a distributed Yugabyte database on top of CAST AI Kubernetes spanning AWS and GCP. These cloud providers are arbitrary choices here; Azure or DigitalOcean can be used in the mix, as well. We picked GCP and AWS because their VMs boot up a bit faster.

    Mono-Cloud Vs. Dual-Cloud Network Benchmark Results

    Let’s compare a Yugabyte containerized database on Kubernetes in a single-cloud scenario versus a distributed database working in two different clouds.

    Setup

    We took the OSS Yugabyte HELM chart with the image version “2.5.1.0” with the replication factor of 3.

    Role

    YB-Master

    YB-Tserver

    Description

    Control plane, consensus

    Serves data

    CPU

    3 CPUs

    7 CPUs

    RAM

    3 GiB

    14 GiB

    Storage

    50 GiB

    300 GiB

    Remote Network Block storage:

    • EBS gp2 (300 IOPs)
    • Standard persistent disk (R225/W450 IOps)

    Nine pods, each on a separate VM to ensure plenty of resources:

    Mono Cloud

    Regional GKE k8s cluster with 9x GCE N2d-standard-8 distributed in three availability zones. Casual VPC network in Google’s US-East4 region.

    Dual Cloud

    AWS EC2 3x c5a.2xlarge

    GCP GCE 6x n2d-standard-8

    A network between cloud providers set up with a Wireguard-based full mesh VPN that encrypts traffic end-to-end (also inside the same VPC).

    To minimize noise from compute and local storage efficiency, we didn’t optimize for cost but wanted to isolate and better understand how distributed databases would work in situations where database nodes are distributed single-digit milliseconds away (in this example, less than 9 ms).

    Nine milliseconds of network round trip latency means a lot of flexibility in choosing between data center providers distributed within a few hundred miles.

    The majority of cloud provider regions lack imagination and are clustered in the geographic vicinity of a densely populated metropolis. California, northern Virginia, Montreal, São Paulo, London, Frankfurt, Amsterdam, Mumbai, Tokyo, Singapore, Sydney, etc.

    So, what are your options in, say, Frankfurt? AWS, Azure, GCP, DigitalOcean, Oracle, Alibaba, IBM Cloud, Hetzner, Linode—not to mention the white-label providers.

    We carried out tests on U.S. East (Ashburn) and Europe Central (Frankfurt) with similar results.

    How We Measured It

    If we want to test relational database performance – the gold standard ruler to date is a synthetic TPC-C benchmark.

    Why did we choose that? Because it’s the most mature OLTP benchmarking test suite that has stood the test of time. As stated here:

    “While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment, but, rather represents any industry that must manage, sell, or distribute a product or service.”

    We loaded an identical Yugabyte configuration in both scenarios with 300 warehouses worth of data and then ran benchmarks several times.

    Each test took half an hour, and the output was the single most important metric: transactions per minute. The test also provided different outputs per type of transaction, average and 99th percentile duration in milliseconds.

    Results

    This table shows a comparison of TPC-C results with the average duration for a single cloud on Google and AWS + Google in the same region, US-East.

    300 warehouses

    Mono Cloud

    Dual Cloud

    Num New Order transactions per 1800 s

    113,071

    113,247

    TPM-C

    3,769

    3,774

    Efficiency

    97.6%

    97.8%

    NewOrder Latency avg msec

    580.2

    498.8

    Payment Latency avg msec

    30.8

    46.7

    OrderStatus Latency avg msec

    33.8

    32.4

    Delivery Latency avg msec

    123.1

    174.9

    StockLevel Latency avg msec

    239.4

    465

    And comparing the 99th percentile:

    300 warehouses

    Mono Cloud

    Dual Cloud

    NewOrder Latency  p99 msec

    6985.6

    5498.7

    Payment Latency p99 msec

    110.5

    157.5

    OrderStatus Latency p99 msec

    146.3

    185.1

    Delivery Latency p99 msec

    331.5

    696.8

    StockLevel Latency p99 msec

    712.3

    1083.3

    As you can see, the difference between the mono cloud and dual cloud scenarios is quite small.

    Some operations, like OrderStatus, are plain read operations. The nearest server can quickly respond with data to the client—usually, a microservice running on the same K8s cluster.

    Yugabyte’s lowest isolation level is a snapshot. This means that those reads are protected from the undesirable effect of the “dirty read.”

    What about writes?

    Writes are a bit more expensive. Raft consensus, hybrid logical clocks and lockless multi-version concurrency control (MVCC) algorithm magic—all of this means a lot more chatter and replication between nodes to get write committed.

    We see that, in payment and delivery operations duration (30ms vs. 46ms and 123ms vs. 174ms, respectively) the geographic distances start to be noticeable.

    The actual network latency between nodes in different availability zones in the mono cloud scenario without network overlay and encryption is, on average, about 0.6ms. Network latency between two clouds with overlay (VXLAN) and encryption (Wireguard) is about 5ms.

    The laws of physics aren’t going away anytime soon. But as we can see, the results are much better than expected.

    YMMV, but industry accepted norm is that the majority of IO operations will be reads, at 70%; only remaining 30% writes, reducing the impact of speed of light even further.

    Having said that, the gold standard for measuring OLTP database performance for the last two decades says that performance results are identical (TPM-C 3’769 vs. 3’774).

    Data Gravity and Egress Costs

    Data gravity refers to a scenario where compute gravitates closer to data.

    For example, if you have a hybrid cloud or multi-cloud strategy but your data stays in a single location like an on-premises data center. Most of the compute with stateless services will tend to stay in the same data center, closer to data.

    The whole point of multi-cloud and a high-performing distributed database was to get rid of data gravity. If we have the same data on two or more clouds, then the data gravity effect disappears.

    If all your data is replicated to several locations, what about network traffic costs?

    All cloud providers allow unlimited incoming (ingress) data. It doesn’t matter where data is coming from—if it’s your on-premises data center, a different cloud provider, the dark web or most cloud-native services. All the incoming data is free.

    “You can bring as much data as you want to us. But if you want to take your data back – sorry, you need to pay.” This is universal between all cloud providers. All the outgoing (egress) traffic from your availability zone is paid! Some cloud providers, like Oracle Cloud, are very generous with egress costs, and companies like Zoom saved a ton of money when they moved some of Zoom’s infrastructure to Oracle Cloud.

    DigitalOcean gives a generous egress monthly allowance per Droplet (VM), depending on size and how long you run it. But, at a minimum, it starts at Terabyte per smallest Droplet. The monthly egress allowance is aggregated in your account, and you pay only for the combined extra.

    All clouds have some minimal free tier—but in general, egress costs start at around 10 cents per GB of data sent outside VPC, and, on average, 2 cents for data between availability zones in the same VPC (I bet you didn’t know that you pay 1cnt/GB in AWS and Azure for incoming traffic coming from same VPC/Vnet from another AZ).

    If you send more traffic, your cost per GB falls. The first drop is at 10TB and generally flattens at around 5 cents at ~150TB.

    How does it make you feel when cloud providers charge an eye-watering 97% margin on that egress traffic? It’s like we’re back in 1995 with dial-up modems, right?

    There is a more cost-effective alternative. But first, let’s answer the ‘Will it Blend?’ question. Let’s add Direct Connect, Express Route, Cloud Interconnect and Layer 2 or Layer 3 providers like Equinix or Megaport. Yes, you get a Dedicated Fiber Link smoothie out of the blender.

    Dedicated fiber link slashes costs to a mere 2 cents per GB. And if you have more egress traffic than 7TB, it’s more cost-effective than relying on the weird discount models from cloud providers.

    You might be thinking, “2 cents for every GB?! That’s way too much! I’m using RDS—yes, I have vendor lock-in and data gravity—but at least I have none of the problems with egress cost, resilience, scalability and overall affordability of the service, right?”

    That’s just the result of an opaque billing design.

    https://twitter.com/QuinnyPig/status/1372598432095891458

    All the traffic between Availability Zones in all clouds is paid at 2 cents per GB of egress+ingress. The only place where you pay for ingress is between Availability Zones in the same region (AWS/Azure). Some managed services offer free replication, but they charge double on compute costs, etc.

    Through the ServiceTopology feature, Kubernetes can keep your application pod’s traffic inside the same cloud service provider Availability Zone and cloud, to avoid round robin, random calls between clouds. This means that you get the best latency and pay no traffic premium.

    Most of the database activity is reads (70/30, YMMV) and are effectively free, egress-wise (coming from the local AZ due to the Kubernetes topology awareness). So, the only thing you replicate and subject to egress costs is your database change delta rate per month.

    Now you can see that the issue of egress costs is an incredibly overstated problem.

    The overwhelming push to the public cloud over the last decade brought on new challenges like vendor lock-in, data gravity and eye-watering monthly infrastructure bills.

    During that time, innovation in non-traditional RDBMS, containerization and workload scheduling removed some of the old barriers.

    Running a hands-free, strongly consistent, highly available and scalable database in heterogeneous data centers on a budget was a monumental task a decade ago. Today, it has become an affordable best practice for running our business applications.

  • How Low-Code/No-Code Can Transform Your Organization

    How Low-Code/No-Code Can Transform Your Organization

    For many companies, IT transformation is no small feat. Between the technology, process and organizational changes needed, it can take time to see a return on your IT transformation efforts and investments. So it’s no surprise that low-code/no-code is starting to be seen as a compelling solution for some.

    With low-code/no-code, you can empower teams of ‘citizen developers’ and enable experienced developers to accelerate deployment and reduce IT backlogs. To gain insights on the ways that low-code/no-code can transform your organization, I asked several DevOps Institute Ambassadors for their thoughts. Here’s what they said:

    Brendan O’Reilly, DevOps Specialist, Daysha DevOps

    Low-code/no-code introduces urgency into an organization. By removing the bottleneck that is scarce development resources, you can reduce the lead time to value.

    Mark Peters, Technical Lead, Novetta

    Low-code situations are likely best for highly tailored situations, with clearly defined purposes and minimal changes. Low-code can work when your business depends on scaling rather than creating new code. If the business is good enough, and one is adding microservices or more instances of the same expanded for a larger market, low-code can be perfect. Implementing low-code then allows the next opportunity to move from some code to no-code, providing a graphical interface to change architecture, implement policy or change recurring controls through mouse clicks rather than code. This also provides an excellent chance for improved ops control; if one is scaling the number of users, controls can be implemented to ensure the graphical delivery does not hard-code a password into the system, instead of relying on approval checks within an alternative interface.

    Vishnu Vasudevan, Head of Product Engineering and Development, Opsera

    When an organization practices DevOps, they continuously expand by team, product, platform and application because of different solutions coming into the market. Incorporating a low-code/no-code approach gives enterprises the ability to speed time-to-market and expand engineering capabilities beyond IT.

    If you have a low-code platform that can help manage and provide visibility into technology and security compliance, that’s a gold mine for any developers or any organization. For example, anytime a new application comes in, teams can use a template from the previous application and put it over the new application. The same security quality of the audit governance is quickly established as a baseline, and all the developers have to do is focus on coding.

    Shriniwas Sathe, DevOps Chief Architect and CoE Leader at Capgemini

    Low-code/no-code tools can help change the way that regular employees interact with technology. With no-code tools, every employee, regardless of their background, can create custom applications, websites, forms, etc. With low-code tools, this can be taken a step further by building upon the offerings of a no-code tool by coding in the unique features needed by the organization. This is much less time-consuming compared to traditional development processes, even though it requires a general know-how of coding.

    With every employee able to contribute to new applications and tools, there’s a better chance of obtaining unique insights and unforeseen capabilities. You can also simplify processes to an extent where regular employees can benefit from the freedom from tedious and repetitive tasks. There are low-code/no-code tools for most industries, which gives you a choice of either a ready-made solution or one that can be customized to your particular requirements.

    Peter Maddison, Founder, Xodiac, Inc.

    Low-code development platforms are great at automating important procedures using drag-and-drop editors that allow citizen developers to create virtual models of an app’s business logic and user interface.

    A major advantage of having an enterprise no-code platform is the presence of built-in governance capabilities. No matter what an IT team’s size, a central console offers visibility for all applications and environments so that they can keep track of any issues or vulnerabilities.

    Business developers can use the solution to create applications that fit their unique business needs. And more than one developer, or a team of developers, can work on the same module at the same time.

    Of course, we need to ensure we are providing appropriate guard rails so our citizen developers are not able to accidentally expose data or create additional risks through the solutions they create.

    If you’d like to learn more about the latest in low-code/no-code and similar topics, check out the upcoming SKILup Day schedule. Or, learn more about the benefits of becoming a DevOps Institute Member.

  • A Funny Thing Happened On the Way to API Management

    A Funny Thing Happened On the Way to API Management

    As API use has exploded across IT departments large and small, the need to manage APIs as independent entities—both APIs themselves and calls to them—rapidly became apparent. We have been through this cycle several times before–ORBs, DCOM, object repositories … the list goes on.

    There are several competing goals when introducing some management structure into formerly as-needed remote calls occurs. We want to publish and advertise connections in the hopes we can avoid (or at least minimize) reinventing the wheel, while we also want to make access easier for app developers that are trying to use our APIs. Meanwhile, we want a centralized location where ops can monitor API usage, security can review APIs for compliance and “approved” APIs can be advertised or even hosted. Finally, we want to give internal developers access to approved external APIs that have been deemed appropriate and safe for the organization to use.

    As with previous iterations of this process, these goals, taken together, prove to be huge. There is a lot there to try and cover in an ever-changing API landscape. It is worse in this iteration, simply because of the proliferation of external APIs. Prior to REST, each iteration had more APIs, both internal and external, than the one before, and REST is keeping with that pattern, though now that we can look back, the external bit is a hyperbolic curve, not a line.

    So, it is interesting and cool to see the market fragmenting a bit. The entire package is a huge order that a few companies have taken on; I’ll leave it to you to decide if they’ve handled it well enough for your org. Others have taken one of two approaches—centralizing API access to external services that have been vetted and interfaced with by the vendor or enabling internal API developers and users.

    Now, we have a couple of manageable chunks to look into. I would even go so far as to say most vendors come from one or the other of these worldviews, and you should compare their “tilt” to your organization’s needs before purchasing an API management product. I would much prefer if we separated these markets into ‘API management’–those doing (or trying to do) it all, ‘API integration’–those simplifying and enabling access to approved APIs like Salesforce or ADP and ‘API enablement–those making writing, publishing, securing and advertising internal APIs easier. But I don’t get to make these calls, so we’ll have to watch it unfold.

    If your need is to streamline and protect your organization’s access to external platforms, and you use many popular services, start with API integration. If you need to get a handle on APIs that you are creating and using internally, start with API enablement. If you need both, and a centralized store of approved APIs regardless of source, look to API management. APIs have become central to our development efforts, and there will only be more of them in the future, so it is worth deciding where your org could use help and set out to research your options.

    Of course, you’re rocking it. DevOPs isn’t letting DEVops poke too many holes in the firewall; DEVops is cranking out solutions and devOPs is keeping them alive. This is just another tool to keep things on an even keel as more and more of the organization’s development is API-based.

  • Cribl Automates Observability Efforts

    Cribl Automates Observability Efforts

    Cribl today announced it has updated the Cribl LogStream observability tool in addition to generally making available a software-as-a-service (SaaS) offering dubbed LogStream Cloud.

    Nick Heudecker, senior director for market strategy and competitive intelligence at Cribl, said Cribl LogStream provides the means to aggregate log data collected from multiple platforms and applications in a format any observability or monitoring tool can access.

    The challenge with observability platforms today is they leave it up to IT organization to copy and paste data into the platform. LogStream automates that process via a log router that collects and optimizes data streams from existing installed agent software and then shapes and routes that data on to analytics platform.

    Version 3.0 adds LogStream Packs, a framework for accelerating deployments of the log router. Alternatively, DevOps teams can employ LogStream Cloud to build and share LogStream configuration models.

    Heudecker said the goal is to reduce the cost of achieving observability by reducing the time and effort required to set up and manage data pipelines. No customization effort on the part of a DevOps team is required to collect and send data from hundreds of sources to any number of destinations, noted Heudecker.

    While observability is a core DevOps tenet, achieving it has been a major challenge. In addition to deploying and managing agent software to instrument IT environments, there’s a significant amount of time and data engineering effort needed to set up the pipelines employed to transfer data to a platform where it can be analyzed. Cribl is making a case for automating those data engineering tasks using a tier of software that abstracts data into a format that can be consumed by multiple observability and monitoring platforms.

    It’s not clear yet to what degree IT organizations perceive observability to be an evolution of monitoring, or simply the next logical evolution of monitoring. One way or another, IT teams need a lot more context than they have historically been able to attain using legacy monitoring tools. Each platform employed within an enterprise typically comes with its own monitoring tools that are used by the IT teams tasked with managing it. Whenever there is an issue, however, IT teams will spend hours correlating data from multiple tools to ascertain the root cause of a problem.

    Observability platforms promise to reduce that time and effort by applying analytics to data collected from multiple platforms. The challenge now is finding a way to efficiently collect all that data.

    It may be a while before IT organizations start to abandon all the monitoring tools they have in place in favor of observability platforms. However, as IT environments continue to become more complex (thanks, in part, to the rise of cloud-native computing platform based on microservices) the need for those platforms becomes more acute. The issue is enabling DevOps teams to achieve that goal without having to wait for an overworked data engineer to find the time to build a pipeline.

  • How to Build a People-First Engineering Team

    How to Build a People-First Engineering Team

    When we think about successful technology teams, there is a tendency to primarily focus on the technology being created rather than the people who are creating it. While both approaches can lead to successful products, a technology-first approach is often the default when software engineers become managers without necessarily having the right tools to manage effectively. Becoming an engineering manager requires an entirely new skill set that involves different communication styles, time management, organization, tools and software, scoping work, learning how to effectively delegate and identifying the right people to complete the team.

    As a CTO, and having gone through this transition, I have noticed that the more a manager is able to be in sync with the people on their team, the less burnout they experience, the higher the employee retention rates and the better work their employees will do.

    While most people won’t be entirely people- or technology-focused, here is a quick overview of the differences:

    • A team-oriented manager knows the personal and professional struggles their people are experiencing, and there’s a sense that the team they run is all working together on a mission toward the same goal. When you talk to this type of manager, they will naturally gravitate towards giving status updates on the wellbeing of individuals.
    • Managers who tend to see the team through a more technical lens will instead identify parts of the project or technological or strategic roadblocks that are impacting the team.

    The same issues will be dealt with very differently by these different types of managers. It can be beneficial to record yourself when you are talking about your team to identify where your focus is, and evaluate if that feels right for you.

    Building Effective 1:1 Communication with Team Members

    An individual contributor is normally accustomed to arriving at weekly 1:1s and working with their manager to solve any problems they may have. When they become a manager, it becomes their role to coach team members to solve their own problems, in addition to building trust. This role change takes preparation, and, if you are unsure how to do it well, you may find it helpful to draft out an agenda that you can follow if needed. You can also gain knowledge about this process by talking to more experienced managers.

    What do you talk about in your 1:1 meetings? Even if you are more a technology-oriented person, 1:1 meetings with your team should be less about the status of projects and more about how the person you are speaking to is doing and how you, as a manager, can help. Having started as a software engineer myself, and now as a CTO, I have found the below questions to be helpful as I navigate how best to help my team thrive in their roles.

    1. “How are you doing?” If you are speaking with a team manager, another great question is, “How is your team doing?” You want to genuinely understand the status of the individual outside of the technical aspects of the job.
    2. “Do you have any feedback for me?”
      I always ask for feedback. For a while, they may not have any; but I often find they start with a small one like “Better options for snacks.” This is an excellent opportunity to build trust by improving the work experience. The next time, they will probably share more until you receive really important feedback, while also improving the relationship. The important thing to remember, here, is to respond to the feedback.
    3. “On a scale of zero to 10, how much fun are you having at work?”
      I really want people to have fun at work. You can ask them to give you a number between zero and ten. When they respond with a number, say, six, a great follow-up question can be to ask what is needed to change the situation so that number becomes a seven. How can you help get them there? I have heard a wide spectrum of answers to this question, ranging from easy to complex; from things like permission to be unresponsive for an hour a day to the desire to learn the skills needed to do a different type of job.
    4. “Where do you see yourself in the next two years and 10 years?”
      Helping your team define both long and short term goals is a really important part of being a manager. Specifically for engineers, it is a great idea to think about if you want to continue moving up in an engineering role, or if making the switch to a managerial role is a better fit.
    5. “How can I help you get to the next level in your career?”
      It is common to get sidetracked when balancing your day-to-day work with your personal life. You can also use 1:1 meetings to help your team members identify what they can do to hit their short- and long-term goals, and how to maintain a better work-life balance.

    Supporting Colleagues

    This last year has been tough for most people, to varying degrees, which has solidified the idea that we need to be there for team members during difficult times; this goes beyond the pandemic. That can mean sending their favorite meal via delivery, sending a care package or simply making it very clear that when someone on your team needs to step out or take time off, that is fine – no questions asked. To be able to be offline and not need to do a long handoff is so important when you are going through a hard time.

    During this move to remote work, as in many leadership teams, my company has gone against the cultural norm of keeping life and work separate. We have acknowledged that it is too difficult to do that right now, and very publicly recognized that everyone is going through something. It has felt like mental bandwidth is filled up with things going on outside of work, so as team members, it is good to make sure work doesn’t add to that pressure.

    In a way, this pandemic period has broken down barriers between work and “life” that has made it almost obligatory to focus on people first. From my perspective, this has been a breakthrough in how we can do business moving forward. By thinking about each individual on your team, everyone can be uplifted and heard, which can be the impetus to more innovation. If you are operating at your max with no slack in your system and little support from managers and teammates, you lose creativity. As innovative teams, this experience has been a reminder to build people-first principles into our cultures to promote a healthy work-life balance that lends itself to innovative solutions and better results.