devopsy.kinsta.cloud

Tag: planning

  • Looking Ahead, 2023 Edition

    Looking Ahead, 2023 Edition

    DevOps is in a perpetual state of change that keeps all of us engaged and busy, but makes a mess of things overall. Do you cloud-first? Are your APIs GraphQL yet? Are you doing SDN to connect to the VPN so you can put files on the CDN and build closer to the market? Sorry, had to do it. You get the point.

    The point is that the ground is forever changing under us, and often things that seem straightforward are not. Like “cloud-native,” including Kubernetes. Yeah, I get it, you mean software-defined infrastructure that works well with infrastructure-as-code (IaC); this field is confusing enough—calling it cloud is dumb when, these days, cloud has a specific meaning in the minds of IT.

    In our current round of change, a few patterns are showing up. Automation is king. Sometimes it is sold as AI/ML because those are the buzzwords of the day, but it is just continuing the trend of automation to make IT more responsive and predictable. We are rapidly approaching a point where the entirety of the application development/deployment process can be automated reliably and configured to create predictable results. By that I mean that development has more assistance writing code, test has more assistance running tests and processing results, deployment environments can be grabbed from application release automation/orchestration tooling and the final product all of that means that builds can be deployed with very little human interaction unless something breaks.

    Highly automated systems are a prerequisite of high-volume output, though – automobile factories are the easiest example of that statement. My problem is that we don’t seem to be asking “Should we be high-volume producers?” Does the average enterprise require non-stop development? I think that the initial answer is yes, and the more considered answer is “Only because the business wants more, consistently.”

    At some point, the gains we received from DevOps need to be looked at from the “What value is added?” perspective. And that is a business function with IT input. Before SaaS and DevOps made IT easier (to get something running; not minimizing our work here), the business had to consider carefully if it needed that application or that change to an existing one. Today, that consideration is pretty much “Because we want it, put it on the backlog.” But all of our new development adds to our technical debt. While the wave of automation we have been living through since “The Pheonix Project” was first published is keeping us ahead of the curve, we will inevitably need to consider flattening the curve. I suggest that the time is now.

    Getting started with this is actually pretty easy. For many organizations, simply stating, “We do not need two ways to do that” will yield benefits quickly. Other organizations will have to ask “Why can’t we use the functionality of existing tools/apps to do that?” and yet others will need to ask the business, “Do we even need to do that?” Prioritization still exists, and management does an okay job of pushing back at all of the organizations I talk to. Still, at some point it goes beyond “prioritize your wishlist,” and IT managers will have to start asking if a project is even necessary–even if doing so makes waves.

    Continue to rock it, but we’re past time to look for redundancy, synergy and misdirection. There is more complexity coming, and more sweet solutions to problems (both known and unknown) are right around the corner. But, as I’ve said before, simplify as you go because you need to be able to handle the next wave, and eventually, automation won’t cover the gap.

  • Three is the Magic Number: Planning and Executing a Successful DevOps Project

    Three is the Magic Number: Planning and Executing a Successful DevOps Project

    Diligent preparation, clearly defined workflows and watchful monitoring are key to delivering a high-performing DevOps project. DevOps is one of the hottest buzzwords in programming, but everyone seems to have their own ideas when it comes to structuring and deploying a DevOps team.

    In basic terms, DevOps is the integration of developers and IT operations workers into a single group, with the goal of increasing the speed and quality of software deployment. Its creation was a response to concerns over the segregation of teams, which led to communication breakdowns and a lack of cohesion.

    Since its inception, DevOps has grown in popularity, with 83% of IT decision-makers stating that they have implemented this style of working in some form.

    However, transitioning business focus in this way does not guarantee an immediate change in fortunes–this dynamic represents an IT culture shift, with increased emphasis placed on collaboration and interpersonal skills. For this model to yield returns, engineers should be open to new information, technology and ways of working.

    So, what vital steps must a DevOps team take to ensure the successful delivery of a project?

    Understanding the Needs of the Client

    The preliminary stages can make or break a DevOps project. If the team does not emerge with a vivid idea of what the client wants and how this can be delivered, the venture is doomed to failure.

    It is crucial that both the team and the client are willing to put the time in to understand each other’s goals, ensuring that their visions align for the steps to follow. This can be done through a series of meetings and workshops, where objectives are identified and both establish a good comprehension of how the final product should look.

    If executed correctly, the DevOps team will exit the project’s first phase with a well-defined brief and a crystal-clear understanding of what the client wants to see delivered. When this step is rushed, engineers will be held back by a lack of direction, increasing the chances that the finished product will not reflect the client’s hopes.

    Configuring the Environment

    This next phase is where the developmental process of the app begins, which is usually facilitated using a cloud-based solution. The team begins by preparing the environment’s aesthetic, working out the components it should contain and exploring how they should be configured to maximize efficiency.

    It is during this period that cybersecurity measures are added so that the final version of the app is as close to impenetrable as possible. This shows why a DevOps team needs a diverse range of skill sets: Cybersecurity experts are few and far between, but there should be at least one member of the group whose knowledge can be called upon to ensure protective measures are properly implemented.

    A successful developmental phase is underpinned by clearly structured workflows, with the project mapped out step by step and everyone involved understanding where they fit into the plan and how they can contribute.

    However, a DevOps project is rarely smooth sailing from start to finish–development is often disrupted by missed deadlines, bugs and conflicting duties pulling engineers away from their stations. For this reason, agile and multi-skilled team members, who can ably jump in and assist in various areas, are extremely important for ensuring continuity when workflows are derailed.

    Avoid Failing at the Final Stages of the Project

    Just because the development of a product is complete and the app is live, this does not mean it is time for the DevOps team to rest on their laurels and move on to the next project. It is essential that they continue to monitor the app in its early stages so that any bugs can be identified and immediately fixed.

    Engineers should be in place to see how much the environment is burdened and how it responds depending on the rate and number of new users. Any data obtained is a precious commodity and should be treated as such, with teams ordering and labeling results as soon as they are collected.

    Once the baseline data has been gathered, results can be analyzed to pinpoint which parts of the app may need recalibrating. After making these improvements, the final result should be a product performing at an optimal level while also meeting the client’s specifications.

    Don’t Rush the Project’s Process

    At the conclusion of all three phases, DevOps teams should have absolute confidence that the project is headed in the right direction and that they are fully equipped to execute the next procedural steps.

    A well-designed app is far easier to produce when a team is made up of specialists from an array of fields eager to collaborate and communicate with one another. Assembling the right personnel will aid the smooth transition between the stages of a DevOps project.

  • The Great Market Convergence is Coming

    The Great Market Convergence is Coming

    When you look around tech markets—almost any tech market—you can see an obvious overlap. Tons of markets that are complementary and that take different approaches to solving the same problem, or markets that have grown into each other over time.

    This happens in highly active marketplaces. People have great ideas and implement them but someone else thinks a different approach is warranted, and the two don’t directly compete until they have to. There is competition within a market, and then there is competition of markets. I think we are about to see a massive increase in competition of IT and, specifically, DevOps markets. I admit to trying to write this without veering into topics I’m currently covering, so bear with me.

    Let’s take dynamic app protection as a topic. You can do that through your DevSecOps vendor, you can do that through your WAF vendor, you can ask around about AAP vendors if your WAF vendor isn’t one, you can do that through a variety of plug-ins and tools offered for web hosting software like Apache or NGINX … Indeed, IDP is actually just attempting to protect your apps dynamically. I’m sure I’ve missed several other markets that have rushed to fill the gap where live applications were getting nailed by hackers. I chose one market, but pick one in DevOps and start digging. You’ll find the same crazy amount of overlap.

    The thing is, we don’t need fifteen solutions to one problem; we need one or two—a handful at most. And combining the fact that most of these markets are way overdue for consolidation (considering the stresses the global economy is placing on trade) and I suspect we’re going to see a fair amount of that consolidation happen in a relatively short period of time.

    This matters to you for a couple reasons. First, if you think one approach to solving a problem is significantly better than the others currently available, your organization should be investing money there by paying for products and services. “More of us should have bought Betamax” is not a great look. Second, someone needs to keep an eye on how these changes impact the IT environment. If one of your DevOps tool’s vendors is in a merger, inevitably one of the merged organizations takes over and everyone does things their way. That’s not necessarily bad, but can be a scary time if, suddenly, you get an email about a change to a system that’s core to your department. Be aware of what’s going on out there and at least have thought about how you would respond if apps—or an entire market—disappear or are significantly changed.

    And, honestly, because it is the apps we are always talking about—DevOps, mostly—that are most likely to converge with a nearby market, it can seem huge. But remember that the core systems went through this long ago, and while they may go through another iteration of “merge this with that” in the future, it won’t be a huge volume like what the currently very active and young markets appear to be about to suffer.

    So don’t worry too much about which make/gradle/whatever variants you are using or what languages, which overlap incredibly, but tend toward niches. Focus on market spaces still growing and expanding. Know what might be at risk, and have awareness of whether you need to care today or you still have time before the market in question implodes or merges with nearby ones.

    And own it. We both know that when the time comes, whether you were prepared or not, you’ll rock a solution out—because that’s what you do, and there is no one behind IT to hand off to. So keep making your organization look great and just stay aware of the wider IT market trends so you’re not caught completely off-guard.

  • Creating a Better Plan with Agile Data

    Creating a Better Plan with Agile Data

    Agile methods help companies respond to change, beat the competition and build high-quality products that customers want by aligning development work to business priorities. But what happens when an organization is unable to align strategy to execution or align its departments or teams due to internal silos? As organizations advance in agile maturity, so do their data analytic needs. Compiling and synchronizing the data of a few agile teams is significantly different than rolling up data across many teams or an entire portfolio.

    Without a clear picture of how teams are delivering their work, organizations may be flying blind when it comes to understanding the delivery of their products or services.

    See People, Time and Work in a Predictable Way

    Every team, department, and organization relies on data to make important decisions that steer the business. What many organizations lack is a way to clearly understand, prioritize and utilize the data they have. From an agile practice perspective, that level of visibility should be established early on in a data reporting structure—project, initiative and work hierarchies—and with the proper agile software solution.

    Setting data reporting standards helps teams understand how they are going to refer to, read out and analyze the data within their company. This type of data structuring promotes the standardization of data across teams, projects and work. But agreeing on a data infrastructure is only one part of data analysis. All of your organization, team and work-task information still needs a centralized system of record for proper utilization. Without a centralized database or agile software solution, data management becomes an extremely manual task. By establishing a data schema for your team-level, day-to-day work within a centralized system of record, all teams can start down a path of easier data accessibility, rollups, comprehension and insight.

    But, this type of reporting is not the end all, be all for most organizations. To truly have a full and comprehensive read-out of agile work, teams need to be able to slice their data any way they want to accurately reflect the work of the organization as a whole.

    How Does Data Discipline Lend Itself to Predictive Insights and Planning?

    Formatting data in an organized fashion has to be the first step in understanding the work that is happening across an organization. Without this first step, agile metrics become a tedious task of manual data rollups and missed opportunities. The hodge-podge of data becomes nearly unusable outside of each individual team, as the organization does not know how to compare health, status, risks, dependencies, cadence or velocity team to team.

    Getting an organization grounded on common data reporting practices has many benefits, such as:

     Everyone is Aligned to the Priorities

    With all teams seeing the work produced across an organization, teams keep focused on the most important tasks, align on dependencies and ensure they meet important delivery timelines. It also prevents unsanctioned “work for the sake of work” from happening, as all eyes can see the work in the system and how it impacts the overall business goals.

    Highly Prioritized Backlog

    Not only are teams able to align on the priorities, but they align on the prioritization of the work needed to meet and deliver on the business goals.

    Better Time Management

    With a common data management structure built into your agile software, time management, velocity and cadence planning become much easier. Leadership can see which teams are successfully delivering all their work during each sprint, as well as how much work they can deliver over time. This sense of time tracking and management gives executives a glimpse at predictive delivery. With predictive delivery comes more predicative revenue.

    Consistent KPIs

    If you ask several individuals what their idea of success is, you will get a different answer from each person. Looking at agile metrics is very much the same. If teams are given free rein to set up their own data structure and reporting, all reports and dashboards will be different. This makes rolling up the progress, status and health of initiatives time consuming and painful. With a governed data structure prebuilt in an agile software solution, all teams and departments within the organization can see the same type of reports and metrics all in one place. These teams then have the ability to analyze their data using standard reporting metrics from standard reports or dashboards.

    Transparency

    While no one wants to think that a project can go awry, it does happen. And it is much better to make course corrections as early as you know there is an issue, as opposed to waiting until it is too late. Giving all parts of an organization a centralized place to review progress in a standardized way promotes overall success.

    The Ultimate Goal: Build a Better Plan

    Standardized agile performance metrics that stem from an ordered data-schema empower organizational productivity, predictability, quality and responsiveness. With the insight that is gained from this level of data consistency—historical, real-time and health details—the pump is primed for deeper insights and advanced data manipulation.

    The Future is Predictive Planning and Big Data

    The old adage, “You only get out what you put in,” also holds true for how an organization uses its data. This is why that critical first step of organizing and housing your agile work in a similar structure in a centralized system is key. Without this simple alignment, advanced metrics and analysis are nearly impossible. And it is those advanced metrics that are the key to truly predictive planning. The future of planning lies in customized data reporting and manipulation with big data analysis. With big data, disparate pieces of information can be analyzed, effectively slicing and dicing the metrics any way an organization needs. Reports can be created to fit the shape of the organization, roles and teams that need them–how they need them. To do this, organizations need to input the data into a centralized system first! Analysis is only as good as the data inputting allows.

    — Brook Appelbaum

  • DevOps Future: Have a Break Plan

    DevOps Future: Have a Break Plan

    A big tenet of DevOps is, “Automate all of the things.” It’s generally a good policy, and definitely allows us to meet the delivery goals of DevOps. Taking the full-on, all-new technology case, an org could run full automation on the dev side and deploying with an application release automation tool to a fully automated container management system—fast, efficient, programmable, even portable. This level of automation absolutely enables DevOps.

    Until things break.

    Ignore the well-known and easily overcome cases such as, “That container is no longer responding,” and focus on the larger infrastructure environment. Do you have a plan for getting builds out if DevOps has taken you to many builds a day and your entire Jenkins environment just crashed? The easy answer is, “Switch to another Jenkins server,” but if you are big enough to have more than one, workload may be an issue, as might be the Jenkins environment itself. The usual example of Jenkins for client builds and separate Jenkins environment for server builds is a good example. If the client environment drops, you can’t just move it to the server build environment. Test tools and other tools for clients will have to be moved over, and almost certainly the build environment will have to be modified.

    Along those same lines: What if, instead of a few containers dropping, your container management system fails? (It’s software, it will happen.) What’s the plan then? Depending on the type of failure, it could take your entire application or application portfolio down with it. In most organizations, downtime is not something taken lightly, so what is the plan to return to normal function? Waiting until you’re in the middle of a disaster caused by a single bad disk, failed memory chip or an attack exploiting the infrastructure is a bad plan. We’ve been doing this for decades, yet we know software fails, and sometimes it fails spectacularly.

    Once the first rush of DevOps is through, processes are streamlined and automation is basically in place, it’s fair to take a deep breath and plan for how to overcome the break and total failure of each tool in your toolchain. Yes, those tools will change and you’ll have to do a little work each time you add/change one, but what is your preference: a slightly slower adoption of DevOps after the low-hanging fruit is gathered or confront a failure of infrastructure proportions without a plan?

    Long before computers, Benjamin Franklin is credited with having said, “By failing to prepare, you are preparing to fail.” That credo has been repeated often since. There is truth in that statement with any form of automation. That which is automated eventually goes awry. You can plan for it, or suffer the stress of trying to figure out what to do while it’s an emergency. I suggest having a break plan.

    — Don Macvittie

  • Lessons Learned: We’re Still Killing Ourselves with Innovation

    Lessons Learned: We’re Still Killing Ourselves with Innovation

    Since the widespread acceptance of virtual machines (VMs), we’ve been on a racetrack with changing technology, with little opportunity to stop and digest lessons learned. That’s not to say organizations haven’t done a good job of learning from experience and mistakes both; rather, the lessons learned are too frequently obsoleted by The Next Big Thing.

    While I have an abiding interest in several areas of DevOps, we’ll focus on one I’ve spent a lot of time around recently: provisioning.

    All Roads Lead to Rome

    Road signs for Rome
    Image Credit: Melanie Bateman via Flicker

    At this point in time, you can provision to VMs, “the cloud” (note that from an automated provisioning perspective this is anything but homogeneous, but the fiction helps here for a moment), containers, clusters and clustered container managers. While all end up giving you a working machine that has IP access, how each goes about it is different.

    That is a problem for automation, unless a company goes all-in on one of the above. Containers were going to save us from custom configurations based upon target—and largely have—but in DevOps they still leave us with an underlying deployment infrastructure that must handle where that container will land. The infrastructure doesn’t define itself, unless we teach it to, and that is still very much tied to the target. Anyone who has used both AWS and Azure knows that even two cloud environments can look very different to get up and running with automation.

    Server deployment tools are steadily growing their capabilities and are adopting the ability to spin up cloud instances alongside their ability to handle hardware/VMs. That’s a good thing, but it’s still relatively new.

    The thing is, we in DevOps are trying to do the exact same thing in every case: get a server configured far enough that our application provisioning tool (Puppet, Chef, Salt, whatever) can pick it up and get the server populated with our applications. One goal, many paths, and it is a rare organization these days that is taking only one of those paths.

    Plan Ahead, Forge Ahead

    The rate of change is not showing signs of slowing down—the re-emergence of clustering software in the light of containers is proof of that. I’m stoked about the idea of clustering, simply because it can take a layer out of an increasingly complex environment, but it changes the way we deploy yet again, offering new software, configurations and APIs.

    The defense mechanism I recommend is to look at what is available, consider how it does or will fit into your architecture and plan for it. DevOps is the area of IT most affected by these changes, so that’s where the planning should take place. But like all topics DevOps, this requires a lot of communication with Dev and stakeholders to make certain the platforms being considered are the ones that will best serve the needs of the company. But supporting all of the possible deployment mechanisms in a DevOps environment means investment in scripting and apps that do the same job for different targets. While that may be doable today if you are blessed with tons of DevOps man-hours, I cannot say often enough that it is building technical debt. Each of those different deployment mechanisms will need to be maintained moving forward, and while the idea of DevOps is always relatively simple, it has been my experience that the reality means some pretty complex code/script feats are necessary to make it work successfully and reliably.

    So build a list of platforms you aim to support, start working toward that list, then revisit in six months or so to make sure nothing has changed. Considering that the needs of most organizations change continually in light of technical advances, and the markets are changing from those same technical advances, six months is a good timeline for revisiting. Any longer than that and you may find yourself with a list of things to support that isn’t accurate and misguides DevOps automation efforts; any sooner than that and you will find you are shifting priorities too frequently to adequately support large installations.

    It’s About the Business

    In the end, for the vast majority of IT, the goal is simply to support the business in an efficient and secure manner. Find out what your organization needs and then go out and make it work. Our industry is lucky enough to have a metric ton of smart people, so making it work is not the issue; pinning down the goals is.

    And keep rocking it. For every error, there are a bazillion successes we don’t jump up and down celebrating, so do the same with those errors. Learn the lessons and move along. Pretty soon you’ll have fully automated deployment on the newer platforms most-suited to the organization, serving up the apps the organization needs.

    — Don Macvittie

  • Log Analysis, or Log Clog

    Log Analysis, or Log Clog

    Log Analysis has the ability to transform teams. It centralizes communication, increases responsiveness, moves efforts from reactive to proactive, reduces time to resolution TTR,  and allows increasingly complex environments to grow without fear.  But, if log analysis becomes the bottleneck, it reverses all these benefits.

    (more…)

  • Change your thinking about Change Management

    Change your thinking about Change Management

    Change Management, let’s face it, it’s usually a checklist item and a CYA tool. But in the world of DevOps where change is part of the culture and processes.  Change Management needs to be more than a cost center, and really a way to improve what you are doing. And there is no excuse not to.

    (more…)

  • The Delivery Pipeline is your DevOps Signature

    The Delivery Pipeline is your DevOps Signature

    Where is DevOps? Searching for it is like querying a rainbow for pots of gold. No single element in a development team’s lifecycle defines them as a DevOps-enabled organization. Were anything the rainbow and the gold, it would be the delivery pipeline. The deployment pipeline centers the team and the process, functioning as the signature of any DevOps shop.

    (more…)