devopsy.kinsta.cloud

Tag: bottleneck

  • 3 Lessons in Network Visibility from the A-List

    3 Lessons in Network Visibility from the A-List

    The network can be host to multiple performance killers. Here are three worth noting

    Like Ed Sheeran as a secret stormtrooper in “Star Wars: The Rise of Skywalker,” some cameos are hard to spot. They can appear out of nowhere halfway through the film or remain a complete mystery until behind-the-scenes footage reveals the masked actor.

    The same can be said for guest appearances on your network. It’s tough to spot the hidden malware, application interactions, network delays and other “cameos” popping up, especially when organizations today are grappling with the complexities of public, private and hybrid cloud infrastructure and multi-tier applications. To adequately support a robust and secure digital infrastructure, you need network and application-layer visibility.

    With movie awards season upon us, I invite you to sit back, relax—pop some champagne, if you’d like—and I’ll tell you about three cameos that can be easy to miss in the network environment, and what you should do about them.

    Hidden Malware

    At a pivotal moment in the Oscar-nominated “Joker,” the evil protagonist studies a tape to prepare for his appearance on a popular TV show. On the tape, he sees a comedian appear on the same show—played by Justin Theroux. It’s a blink-and-you’ll-miss-it cameo from Theroux, very much like hidden malware traversing the network without detection.

    Whether they’re sophisticated or non-targeted, high-volume criminally motivated attacks can move throughout the network without your knowledge and cause enterprises significant damage and cost, to the tune of $6 billion by 2021. Emotet, described by the Department of Homeland Security as one of the most costly and destructive instances of malware, likely comes to mind as one of the most pertinent examples. It has the ability to move laterally through the organization, spread other damaging malware and constantly evolve.

    Spotting hidden malware requires network visibility that spans the entire distributed digital infrastructure, as well as having access to network data to ensure visibility for all security and performance tools. Specifically, organizations need reliable access to data in motion to tap into traffic across cloud and on-premise infrastructure and traffic flowing across 5G and 100GB networks. This type of pervasive visibility allows enterprises to keep up with the speed at which networks and businesses run, as well as with security demands.

    Interactions Between Applications or Microservices

    When Brad Pitt’s character in “Once Upon a Time in Hollywood” unknowingly ends up at the Manson Family compound, he comes across Dakota Fanning, who plays a key member of the infamous cult. Their meeting is short but meaningful, and sets the stage for additional character development and exchanges down the line. Interactions between applications or microservices are just as meaningful, with 80% of network traffic consisting of such communication on average.

    Understanding these interactions is critical for guiding digital transformation (DX) efforts, which are a top priority for 87% of business leaders. Among the goals of DX are streamlining business processes, cutting costs, improving productivity and introducing new business models that redefine industries—or build new ones. These are ambitious goals, and they’re ultimately realized through new digital applications built on intricate microservices-based architectures that need to be managed and secured.

    Only with a clear view, and the right kind of data, can you understand the applications’ interactions, performance and security characteristics to accelerate DX. And the only way to get to this view is to look at the data in motion on the network. The ability to isolate specific applications or microservices communication streams for deeper inspection allows infosec teams to easily understand access patterns and put in place effective micro-segmentation strategies. Application developers, too, can benefit from this by better understanding communication bottlenecks between applications and microservices, as well as troubleshoot them.

    Traffic Bottlenecks

    The latest rendition of “Little Women” follows a non-linear storytelling structure, constantly jumping between two timelines. It can be jarring to viewers when the girls’ father returns from war, revealing himself as the actor Bob Odenkirk. While this moment is meant to be an intimate glimpse into a family coming together after a long separation, it brings the movie-watching experience to a standstill, since the audience associates Bob Odenkirk less with a serious scene and more with his previous comedic roles.

    Networks today are similarly prone to experiencing abrupt slowdowns or bottlenecks that limit network performance and waste bandwidth. Managing the network to ensure constant availability is a major challenge when data volumes are growing exponentially. It’s estimated that about 1.7MB of new information will be created every second for every human being on the planet this year. This data explosion paired with digital transformation technology investments and lightning-fast 5G and 100GB networks has led to a very complex enterprise infrastructure setup—and network blind spots across physical, virtual and cloud environments.

    The solution lies in having the right kind of data to help you understand network and application performance. By analyzing the network traffic, you get immediate actionable data points that can be used to address trouble spots.

    Whether we’re talking big screen or enterprise network, surprise cameos are common and they are easy to miss—and this is particularly problematic when things like malware and bottlenecks pose a threat to network security and speed. For the NetOps and infosec pros tasked with complex infrastructure development, implementation, maintenance and security, you need a clear line of sight into what’s happening within the network. Armed with insights on traffic patterns (both cloud and on-premises), delays in data flow and insecure areas, you can use this intel to write the screenplay (or plan) to support a robust and secure digital infrastructure.

    — Shane Buckley

  • Sometimes It is More Than Culture, It’s About Gains

    Sometimes It is More Than Culture, It’s About Gains

    Almost since the beginning, there has been a tug-of-war between whether better toolchains or increased communications were “real” DevOps. The answer to that question is, Yes.

    Better tools undeniably speed the delivery process and streamline operations. Anyone who claims differently is not living in the real world. Jenkins alone can make your development world a better place, and it is but one tool—with hooks into like a million more, but still one tool.

    Increased communications speed the delivery process and streamline operations, if they are frank and honest. A developer who knows that his choices for included dependencies can make life easier or more difficult on maintenance is far more likely to include that consideration in their choice, for example.

    Gains Are In the Doing

    But the rock-star gains that some shops have seen from DevOps? Are a bit of both, and something else. The sudden improvement of 4x delivery frequency most often comes from no longer avoiding that thing you don’t talk about, or the legacy piece of software that eats an increasing amount of time just being kept alive because it’s necessary.

    And I would argue that is the best place to start. I’ve touched on this tangentially before, so I thought I might be more direct.

    You have it. That thing in the process that is a bottleneck. No one talks about it. There are reasons—some because a few years ago you tried to resolve it and couldn’t; some because the bottleneck is a natural outgrowth of some requirement; some because “it’s always been that way,” But there is that step (or more often three or four steps) in the develop/test/deploy cycle that consumes a huge chunk of time to do little work.

    There are also those tools/processes so old that they consume an inordinate amount of resources. Most shops more than a few years old have them. I’ve worked in a couple of 100-plus-year-old companies and frankly, these bits of outdated software were part of the genotype of the org because they’d been in IT so long.

    Manual processes are known to take time, simply because they’re manual. Development itself, even when broken into smaller pieces and implemented piecemeal in the manner of Agile, still takes time. Testing, too, is still (or again, depending upon your background) a largely manual process. And security checks are also in a ton of shops.

    These are known. And if you’re approaching the problem from the “find the bottlenecks,” I would exclude them. Better tools or more automation can help all three, streamlining who does what can help test and security, but these aren’t the places that I’m thinking of.

    Aging Support Software

    I worked in a shop once that spent an inordinate amount of time keeping a “database” alive that was in existence before RDBMS. Yep. There was mission-critical software that used the database, so it was maintained as well as possible internally. Internally because the vendor no longer existed.

    Several analyses were done, and each time it was decided that the DB couldn’t be gotten rid of without a major rewrite of the systems it supported. But as time went on more and more resources were poured into those systems and the database. Eventually, it was forced from above that the database had to go. The estimate to do so came in at an insane number and the project was dropped. Meanwhile, an increasing amount of resources were being poured into the DB and the systems that supported it. All were outdated, and all required special skills that were also outdated. Soon the org was paying exorbitant rates for the few people who could work on the system to do so via contract.

    Finally, with hard dollars attached to the preservation of this system, a forced upgrade was a viable project, and it went forward.

    This is all pretty standard enterprise stuff, it’s kind of the way IT has functioned in the past. But had IT management gone to business units with improvements that could be offered during the upgrade, all of the resources poured into not upgrading over the years could have been used for new product/feature development. Identifying the fact that replacement was inevitable and wouldn’t get cheaper for waiting is the type of outdated software issue I’m talking about.

    Bottlenecks

    From the bottleneck side, there is often a “this monster log file must be reviewed by a human” step on the way to production, too. Think about simple automation. What is the human scanning for? Talk to those responsible and see if they can offer guidelines concerning how they would train a new person to review the file. Then write a small script to do it—not as a final solution, but to show that a person doesn’t have to if the review is well-bounded. People won’t trust it, so run it in parallel for a while, and learn from what it misses, point out what it finds. Long term, eliminate the delay of someone pouring over logs looking for the same things over and over. Take time to make it possible. It won’t be a ton of time, but it will reap gains. Just talking to the team that must accept the log output might reap gains.

    There isn’t a need to immediately install 15 DevOps tools or reorg all of IT before starting your DevOps journey. Just look for the places resources are being invested to little long-term gains, or where the process is held up in a manner easily eliminated. And keep rocking it.

    — Don Macvittie

  • DevOps and The Goal

    DevOps and The Goal

    Reading, “The Goal” was a pivotal event in my career.  More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book.  We were discussing the challenges of getting work done in a corporate environment.  I was grumbling about leading a web dev team in a company that did not embrace the software development discipline.  Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”.  It gave me a vocabulary and a roadmap for improving our throughput issues.

    Our throughput problems were a textbook case for development disaster.  When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration).  We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features.  Looking back, it is not surprising in the least that we were not successful.

    Not surprisingly, these issues led to a lack of trust with the business.  Overcoming the technical problems was straightforward using brute-force tactics.  We addressed the source code problem by recruiting a senior developer and empowering him to make improvements.  We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard.  Our tactics were basic, and they worked.

    The bigger challenge facing us was getting our business owners to accept a different SDLC model.  We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate.  The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release.  The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

    Using TOC as a guide, we redefined our SDLC from requirements through QA.  We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP).  De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time.  We then focused on improving the slowest areas, which seemed to be requirements and deployments.

    We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!).  The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA.  Some features would get done in a month and some would span several months.  Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

    Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome.  I really liked “The Phoenix Project” and many of the situations rang true.  I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides.  I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes.  Add a consistent release schedule and it looks a lot like DevOps.  Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.