devopsy.kinsta.cloud

Category: Doin’ DevOps

  • Technical Debt is Inevitable. Here’s How to Manage It

    Technical Debt is Inevitable. Here’s How to Manage It

    Eventually, your organization will accumulate technical debt. It doesn’t matter what programming language, software stack, computing style, host, or slick hybrid mix you use. It doesn’t matter if you’re using a waterfall or agile methodology. Ultimately, most technology becomes technical debt one day.

    “Technical debt is inevitable,” said Insight’s North American chief technology officer, Juan Orlandini. “Technical debt is like credit card debt. If you’re not smart about how you use it, your balance keeps increasing.”

    Today’s technological trends will one day fall out of favor, especially as talent moves on and older maintainers retire. And while this all might sound grim, acceptance is the first step toward change. Realizing this pattern of technical debt and accepting its inevitability may be the key to building future-proofed, evolvable software ecosystems.

    I recently chatted with Orlandini to understand some of the significant contributors to technical debt and how software maintainers can plan for and address it. Below, we’ll outline some of the major causes of technical debt and suggest methods to tackle it and prevent it from accumulating.

    Major Contributors to Technical Debt

    Enterprises face growing technical debt. A study from Outsystems found that 40% of IT budgets are devoted to addressing technical debt in large enterprises. To mitigate technical debt, we must first understand the main contributors to it. The causes of technical debt are wide and varied but can be distilled into a few common areas, said Orlandini.

    Unsupported software stacks. One major cause of technical debt is outdated or unmaintained software. Too often, a consumer will purchase a proprietary system that eventually becomes sunsetted, said Orlandini. Or, the vendor may evaporate during an acquisition or merger. When this occurs, the package goes unmaintained and leaves a mess on your hands.

    Trends in programming languages. Another obvious area is the continual evolution of programming languages. Even today’s most popular languages aren’t that future-proofed—Python replaces Perl, Go replaces Python, Rust replaces Go and so on. The same is true for frameworks. Ruby on Rails used to be in, but now Vue.js is a more hip framework.

    Aging cloud infrastructure Just because you’re in the cloud doesn’t mean you don’t have technical debt, reminds Orlandini. When early cloud adoption began, many core services we take for granted today, like authentication and authorization, weren’t offered by the cloud service providers. Some of the first-ever cloud migrations were stitched together with homegrown tooling, which may not be aging well. Or, organizations may have “lifted and shifted” legacy services into the cloud. Operators overseeing these older cloud-based environments will likely feel the pressure to refactor into more modern, cloud-native styles.

    The proliferation of microservices and APIs. Furthermore, cloud services retire features all the time. Thus, it can be challenging to keep up with these minute changes to retain backward compatibility for things like API versioning shifts. In the DevOps world, web APIs are the glue that connects our tools. But in an increasingly complex landscape, it can become easy to produce shadow IT and lose track of dependencies.

    Legacy physical infrastructure: It’s not just the software that becomes debt; the systems and hardware it runs on might also become obsolete. For example, the network itself might be full of old edge switches and wireless access points, says Orlandini. Or, an enterprise may hold onto aging on-premises servers. (This is often due to the depreciation benefits of using a capital expense accounting model as opposed to an operational expense model.)

    Other factors that could contribute to technical debt are things like rushed decision-making, skills and resource gaps and failure to implement DevSecOps.

    Ways to Tackle Technical Debt

    Technical debt is a threat to innovation, so how can we mitigate it? Well, if you don’t already do so, it’s a good idea to build technical debt into your budgeting, planning and ongoing operations, said Orlandini. “You have to manage it, expect it and be responsible with your technical stacks in the same way you are responsible with your financial stacks,” he said. Here are a few other ways to manage the debt you have and avoid accumulating more.

    Consider using AI to refactor legacy code. Generative AI could be leveraged to reactor legacy code into more modern programming languages. This could help automatically convert PEARL code, for instance, into JavaScript. Today’s large language models (LLMs) could help solve many of today’s problems. However, since they are built on a pre-existing body of work, they will use less trendy languages and might cause some technical debt in the process, cautioned Orlandini.

    Don’t over-rely on new DevOps processes as a cure-all. DevOps can accelerate the time to release features, but it does not, by its nature, eliminate technology changes, said Orlandini. Just because DevOps helps deploy an aging framework doesn’t mean it will maintain or evolve the debt. As frameworks go out of style, IT teams must continually assess their software stack.

    Remain lean whenever possible. Preventing the accumulation of unnecessary technology should be top of mind. In the DevOps world, there is a proliferation of individual services, and it’s easy to loop in a tool because it’s “the next best thing.” While sleek tools can help quickly integrate new functionality, looping in too much unnecessary tech could introduce sprawl concerns, said Orlandini. In general, he recommends keeping things as optimized as possible and carefully vetting the introduction of new technology. This includes carefully vetting new partners to ensure their business model works and that they themselves are future-proofed.

    Continually maintain updates of dependencies. Today’s software infrastructure is largely composed, at its foundation, of open source software. And if these packages aren’t mature and well-supported, you may be sitting on an unstable house of cards. It’s a good idea to continually audit your OSS use and respond to updates in a timely manner.

    Managing the Inevitable: Tech Debt

    Tech debt is like credit card debt—easy to get into and tough to get out of. And just like its financial equivalent, you shouldn’t go too far into technical debt. Thankfully, we’ve covered some best practices to keep in mind that should help spot key problem areas and reduce bloat before things get out of hand.

    As a final note, consider asking: How much stuff are you using and why? It’s become a trope, but the principle of KISS (keep it simple, stupid) bears repeating in software architecture dialogues. Because if you reduce the surface area, it will make ongoing maintenance easier and reduce the potential of cyberattacks.

    The subject of technical debt is a multi-faceted issue with many factors to consider. For related insights, you can also watch the conversation between Techstrong Group Chief Content Officer Mike Vizard and Juan Orlandini on Techstrong.tv here:

  • Supercharging Ansible Automation With AI

    Supercharging Ansible Automation With AI

    Whether the latest incarnations of generative AI will live up to their seemingly great promise remains to be seen. This does not always happen with new technology, but the torrent of AI advancements is promising disruption across nearly every industry. Instead of posing a threat to human roles, AI now aims to make tasks easier, faster and more efficient. “This is precisely what we have been trying to achieve with Ansible and automation through generative AI models,” according to Richard Henshall, senior manager of Ansible Product Management at Red Hat. “At the end of the day, AI is another form of automation, and while we do not have thinking computers yet, we do have highly advanced and powerful models that can automate access to data.”

    Case for Integration

    Picture two scenarios: Consider a user who is already skilled in using Ansible. Their limitation is often the amount of time it takes to find the right way to do something, as Ansible is very broad and encompasses many different technologies. The user may be an expert in one or a few of these technologies, but not all of them. In this case, AI can act as an assistant, helping the user find answers more quickly and efficiently. For example, if they want to automate something in AWS, the AI can help them find the necessary resources, and the user can then tweak and curate the automation to create a final product that can be distributed to a large number of people. “This essentially superpowers the individual user,” said Henshall.

    Another type of user may not know how to automate, but they do know how to use a specific technology. “They may not automate that technology because it’s difficult for them to learn something new,” Henshall added. “AI capabilities can help these users get started with automation more quickly, enabling more people to write automation and manage both new and existing technologies.”

    In both cases, the purpose of automation is to help users solve problems. AI serves as an accelerator for these capabilities. “Once users have created automation, the AI model can be trained on the organization’s DNA, including best practices, security posture, policies and enforcement points,” Henshall asserted. This allows the AI to become a co-worker, empowering users and helping them maintain and manage their technologies more effectively.

    “As the AI model learns from the organization and its users, it can help train junior engineers, who will eventually become senior engineers, continuing the cycle of knowledge transfer and improvement,” Henshall predicted. “In this way, generative AI technologies can be integrated into the Ansible ecosystem, benefiting users and organizations in a wide range of contexts.”

    Solving Problems With AI

    As the DevOps leader of cloud testing organization LambdaTest, Shahid Ali Khan has led the company through open-source engagements and investments in next-generation automation solutions, addressing key challenges with Ansible. “We are dealing with a very large scale inventory in terms of geologically separated data centers and multiple OS. Managing this inventory on the basis of these characteristics becomes a hefty task and requires continuous efforts from the team,” he said. “With AI, Ansible can analyze the inventory and automatically group the hosts based on their characteristics, such as their operating system or their geographical location. This makes it easier to manage the inventory and apply changes across multiple hosts at once.”

    Khan also highlights the issue of auto-healing in the industry. “Auto-healing has always been a problem, and while tools in the market have somewhat addressed it, they still require defining errors and categorizing fixes,” he said. “With current AI technology in place, Ansible can detect and automatically fix issues on the basis of historic data in the inventory without the need for manual intervention, reducing the risk of human error.”

    “Inventory forecasting has always been a pain in order to calculate how much inventory we will need based on usage and on what fronts like location and OS we need to scale,” Khan shared. “AI here with Ansible can help detect the usage and forecast based on exact demand.”

    Benefits and Challenges for the Ansible Community

    In the Ansible community, which consists of people who want to solve problems, the benefits and challenges of AI-driven automation are similar to those faced by individuals in their jobs, just on a different scale.

    Community members want to see how generative AI can help them accomplish more tasks, whether or not they are part of an enterprise. Having access to a community model allows them to contribute to training the AI, ensuring its usefulness, as they may eventually work in an enterprise setting during their careers. “The access to community services enables them to take advantage of the benefits offered by AI-driven automation,” Henshall remarked. “It is a mutually beneficial relationship between the community and AI-driven automation.”

  • Mastering DevOps Automation for Modern Software Delivery

    Mastering DevOps Automation for Modern Software Delivery

    DevOps automation services are a crucial aspect of modern software development that helps organizations to improve their software delivery processes and keep up with the pace of innovation. With DevOps automation, developers can automate repetitive and time-consuming tasks, enabling them to focus on writing code and delivering features faster and with higher quality. By automating processes such as continuous integration, continuous delivery and infrastructure-as-code, organizations can reduce costs, improve collaboration between teams and respond quickly to changing business needs and customer requirements.

    In this article, we will discuss the basics of getting started with DevOps automation services, including popular tools and best practices that organizations can use to implement a successful automation strategy.

    Agility and Efficiency for Smart Digitalization

    Emerging technologies like multi-cloud, IaC, AI/ML and IoT have only increased the importance of DevOps automation, as they bring new challenges demanding agility, flexibility and efficiency in software delivery processes. Multi-cloud and hybrid cloud environments, for example, can benefit from automating the deployment and management of software across multiple cloud environments. Similarly, AI/ML and IoT require organizations to collect and process large amounts of data in real-time, which can be challenging without automation. DevOps automation can help organizations automate data pipelines and analytics, enabling them to process data faster and more accurately.

    Here are some benefits of DevOps automation:

    ● Speed and Efficiency: DevOps automation enables organizations to deliver software faster and with higher quality. By automating repetitive and time-consuming tasks, developers can focus on writing code and delivering features.
    ● Agility and Flexibility: DevOps automation provides organizations with the agility and flexibility to adapt to changing business needs and customer requirements. Automated processes make it easier to release software updates and respond quickly to market demands.
    ● Cost-Effectiveness: DevOps automation reduces the costs associated with software development by reducing manual labor and minimizing errors. It also enables organizations to use cloud resources more efficiently, which can result in cost savings.
    ● Improved Collaboration: Bringing the different stakeholders along with devs and operations, DevOps fosters a collaborative approach toward enhanced CI/CD. By automating processes, teams can work together more effectively and share knowledge and feedback in real-time.
    ● Scalability: DevOps automation enables organizations to scale their software delivery processes quickly and efficiently. Automated processes make it easier to deploy software across multiple environments and platforms, which is essential for organizations that need to scale rapidly.

    Getting Started With Efficient DevOps Automation

    Implementing DevOps automation requires a well-planned strategy, best practices and the right set of tools. In this section, we will discuss how to successfully implement DevOps automation in your organization.

    Understanding DevOps
    Before you begin with automation, it is essential to understand the principles and practices of DevOps. DevOps is a culture that emphasizes collaboration and communication between software development and IT operations teams to build, test and release software more quickly and reliably. There are some common tools that DevOps teams use for their work, collaboration and communication:
    ● GitHub: The Git repository hosting service used for version control and source code management functionality, especially for distributed projects.
    ● JIRA: A project management tool that helps teams plan, track and manage software development projects.
    ● Slack: A communication platform that enables real-time messaging, file sharing and collaboration.

    Choose the Right Tools
    Automation is at the heart of DevOps, and selecting the right tools is crucial. Choose tools that fit your requirements, budget and infrastructure. Some popular DevOps automation tools include:

    ● Jenkins: An open-source automation server that enables continuous integration and continuous delivery (CI/CD) pipelines.
    ● GitLab: A web-based Git repository manager that provides CI/CD capabilities, project management, and collaboration tools.
    ● Puppet: An open-source configuration management tool that automates the deployment and management of software across multiple environments.
    ● Chef: A configuration management tool that automates the deployment and management of software across multiple environments.
    ● Ansible: Open source tools for automating software provisioning, deployment and configuration management

    Identifying Processes for Automation

    First, list all the critical processes within your organization that can be enhanced with automation. Start with the repetitive and time-consuming tasks that can be automated, such as code testing, deployment and monitoring. There are a number of popular tools that can help with such automation:

    ● Selenium: An open source testing framework that enables automated testing of web applications.
    ● Docker: A containerization platform that enables the creation, deployment and management of containerized applications.
    ● Kubernetes: The open source container orchestration tool is essential in the DevOps pipeline for orchestration and automation of the deployment, scaling and management of containerized applications.

    Creating a Plan

    Once you have identified targets for automation and selected tools, create a plan for automation that includes a roadmap and milestones. Start with small projects and gradually expand the automation to cover all aspects of the software development life cycle. Some popular tools for strategizing these life cycles are:

    ● Trello: A project management tool that enables teams to organize and prioritize tasks using Kanban-style boards.
    ● Asana: A project management tool that enables teams to track and manage tasks using to-do lists, calendars and timelines.

    Training The Team

    Automation requires new skills, so it is essential to train your team to use the tools and processes effectively. Provide training and resources to help your team learn new skills and techniques. Some popular education and training resources include:

    ● Udemy: An online learning platform that provides courses on DevOps automation tools and techniques.
    ● Coursera: An online learning platform that provides courses on DevOps and related topics.
    ● Pluralsight: An online learning platform that provides courses on DevOps automation tools and techniques.

    Implementing Continuous Improvement

    A continuous feedback loop is essential for DevOps’ iterative process and continuous improvement. Integrate mechanisms for evaluating your automation processes, identify areas for improvement and make necessary changes. Here are two common tools used to do this:

    ● New Relic: An application performance monitoring tool that provides real-time visibility into the performance of software applications.
    ● Splunk: An operational intelligence tool that enables organizations to analyze and visualize machine data in real-time.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    Citi
    New York, New York
    DevOps Engineer – Vice President
    $137,610 to $206,420

    SimplyHired.com

    Metrum Research Corp.
    Remote
    Senior DevOps Engineer
    $135,000 to $195,000

    CareerBuilder.com

    Peraton
    San Diego, California
    DevOps Systems Engineer
    $112,000 to $179,000

    Dice.com

    Stamford Technology Solutions
    Redmond, Washington
    Site Reliability Engineer (SRE)/DevOps Engineer
    $100,000 to $110,000

    LinkedIn

    TIAA Bank
    Jacksonville, Florida
    Senior DevOps Engineer
    $92,500 to $154,100

  • How to Choose an Observability Data Pipeline

    How to Choose an Observability Data Pipeline

    A complete full-stack observability platform combines data from multiple sources to gain holistic insights into system behavior. In fact, organizations without a full observability solution could find themselves facing departmental silos since teams with different data sources cannot share insights.

    While a key step in observability is the ability to combine logging data with metrics, companies that have developers viewing logging data and their metrics elsewhere are not collaborating on their data in the way observability requires. The solution is an observability data pipeline that best fits your business needs.

    What is an Observability Data Pipeline?

    An observability data pipeline does two things—centralizes data from a number of different sources and provides teams with the right tools to turn data into business insights. Observability data pipelines are typically user-friendly so that you can easily set up and manage your data.

    Observability data pipelines integrate data from a range of sources, including AWS, FluentD and Kubernetes. They have capabilities for searching and viewing log data and notifying engineers through email and even Slack.

    An observability pipeline ingests logs, so they can be viewed in a log viewer. Pipelines can also analyze data in real-time, allowing engineers to quickly spot critical security issues before they affect critical systems.

    There are a number of full-stack observability platforms, including Coralogix, Splunk, Elastic and even options like Azure Log Analytics and Google Cloud Logging. Coralogix, for example, is a popular observability data pipeline that can analyze data in real-time using Streama and use Kafka Connect to integrate data from a variety of sources and extract trends.

    Business Benefits of Observability Data Pipelines

    Using observability data pipelines has the potential to transform the way a business does DevOps. Let’s go through a few key benefits of this technology:

    ● Reducing Cost of Observability Data
    Many organizations use systems like Cloudwatch to analyze logging data. However, these often charge per GB of logging data analyzed, raising the question of whether Cloudwatch is really cost-efficient. And it’s not just Cloudwatch. For many systems, logging costs can be increased due to slow query times, noisy logs and the wrong type of storage. Observability data pipelines help organizations manage high volumes of logging data and move away from expensive legacy platforms. This enables enterprises to increase their observability for a modest price.
    ● Reduced Onboarding, Increased Problem-Solving
    Pipelines are straightforward to learn. Unlike options such as ELK, which can be quite specialized and difficult to maintain, observability data pipelines are easy to use and easy to set up. This reduces the time needed to onboard new engineers. Less time onboarding means more time solving problems and serving clients, which increases business revenue.
    ● Better Data Shaping and Merging
    Data from different sources is often stored in multiple formats and structures. Because observability depends so much on well-integrated data, it’s vital to transform this data into standardized formats. Observability data pipelines use log parsing to store key features of logging data for search and analysis.
    ● Reducing MTTD/MTTR
    Having an optimized, user-friendly observability data pipeline reduces downtime by helping engineers solve problems faster.
    ● Enabling DevSecOps With Observability
    We’ve written about the rising prominence of DevSecOps, a paradigm that involves shifting security left in the development life cycle. Key to DevSecOps is the fruitful collaboration between DevOps and Security. This can only be achieved by high levels of observability. Observability data pipelines facilitate this with their ability to integrate data from multiple different sources and present it intuitively. This means security teams can apply their expertise to log analytics that would otherwise have remained siloed.

    Wrapping Up

    Observability, the ability to use logs, traces and metrics to “picture” the internal state of a system, is vital to any DevOps team. Observability data pipelines can help by integrating and presenting data from different sources. This mitigates the effects of organizational silos and allows DevOps teams to solve problems faster.

  • 8 Hot Takes: Will We See a Monolithic Renaissance?

    8 Hot Takes: Will We See a Monolithic Renaissance?

    Is the DevOps world slowly backing away from microservices and to a monolithic renaissance? After years of proselytization about the benefits (Flexibility! Resilience! Support faster release speeds!) of microservices and serverless architecture, there are signs of a potential backlash against the singularly fragmented microservices architectural approach.

    The argument in favor of rolling back to monolithic architectures feels similar to the cloud repatriation debate. Just as with a lot of cloud-first transformations—often in lockstep with them, actually—many organizations moved headlong toward microservices without a case-by-case business or technical rationalization for the change. As the shiny newness of microservices has dulled under that approach’s daily exposure to real-world deployment and business challenges, the monolith isn’t looking so bad in a lot of situations.

    The monolithic versus microservices debate boiled over the last few weeks in the wake of the publication of a case study earlier this month from the Amazon Prime Video team. The analysis from Marcin Kolny, senior software development engineer for Prime Video, reported that his team was experiencing scaling bottlenecks in its audio/video quality inspection tooling, which was architected as a distributed system using serverless components. By rearchitecting the tool as a monolith, they were not only able to increase scaling capabilities but drastically reduced the infrastructure costs around that tool.

    “Moving our service to a monolith reduced our infrastructure cost by over 90%,” Kolny wrote.

    The announcement elicited a fair degree of hyperbole on the topic, but it also spurred a very organic airing of the pros and cons of monolithic, microservices and serverless architectures.

    The following are 10 different perspectives spurred on by this online debate from a range of CTO and engineering experts. Some of them are diametrically opposed to one another, many are overlapping in their opinions. But as a body, they point to the general (and very pragmatic) consensus that there simply will never be a one-size-fits-all approach to software architectures.

    Microservices a Siren Song for Complication

    “It’s clear that in practice, microservices pose perhaps the biggest siren song for needlessly complicating your system. And serverless only makes it worse. Replacing method calls and module separations with network invocations and service partitioning within a single, coherent team and application is madness in almost all cases.”

    —David Heinemeier Hansson, creator of Ruby on Rails and CTO at 37signals (maker of Basecamp and HEY)

    Architectures Aren’t Good or Bad

    “Application architectures are neither good nor bad, nor are they appropriate for every use case. Organizations need to step back and consider carefully what it is they’re trying to achieve and which architecture might best serve that need instead of choosing the most “modern” architecture because, well, it’s in vogue.”

    —Lori MacVittie, distinguished engineer in office of CTO at F5Networks

    Distributed Architectures Will Suit AI-Driven Future

    “In the ongoing debate between monolithic and serverless architectures, it is important to acknowledge that both approaches have their merits and considerations. Monolithic architectures have been prevalent in large human-centric organizations due to their management advantages and historical use cases. However, as the world continues its rapid progression towards a more AI-driven future, the landscape is shifting towards architectures that embrace fluidity, autonomy, and distribution.”

    — Reuven Cohen, veteran CTO and entrepreneur, mentor for TechStars

    Monolithic and Serverless Each Have Pros And Cons

    “Over the last 15 years, everyone has wanted to move to serverless microservices-based horizontally-scaled architecture – but it is not ideal for every application’s specific needs.  As with any choice in life and in business, there are always tradeoffs;  you must evaluate and choose the appropriate architecture based on your business and technology needs. Another important dimension to this is workforce skills for application architecture.

    —Hemang Davé, client technical leader and CTO at Kyndryl

    The Amazon Rearchitect Wasn’t Even Truly a Monolith

    “The problem is that they called this refactoring a microservice to monolith transition when it’s clearly a microservice refactoring step, and it is exactly what I recommend people do in my talks about serverless first. I don’t advocate “serverless only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container as part of a larger serverless event-driven architecture, which is what they did. If you built it as a microservice to start with, it would probably take longer (especially as you have to make lots of decisions about how to build and run it) and be less able to iterate as you figure out exactly what you are trying to build.”

    —Adrian Cockcroft, longtime DevOps advocate and veteran architect with stints at eBay, Netflix, Amazon, and Battery Ventures, among others

    One Tool as a Monolith, Maybe, But Not All of Amazon

    “Microservices, while duplicative and inefficient from a compute standpoint, allow you to release stuff really quickly, which is a tradeoff that most organizations at scale are willing to make.

    Microservices exist to reduce communication pathways—not to solve technical scaling challenges. At a certain point, teams stop functioning due to the communication overhead. A seven-person team has 21 communication pathways, whereas a 100-person team has 4,950. I work with large retailers selling billions of dollars a year online, and putting everything into a single monolith just doesn’t work. Could the entirety of Amazon.com be deployed as a single monolith? No way.”

    —Kelly Goetsch, chief strategy officer at commercetools

    Architectural Tension Always Exists Between Speed and Maintainability

    “Speed to market and speed to develop does not always equate to long-term scalability and maintainability. You must actively balance your investments across these two critical pillars to build a viable product. Active rebalancing and reconsideration of the stack and technology mix for your FinOps and BizOps context is both art and science.

    Cloud-native distributed systems paradigms are here to stay. The Prime Video folks have mistakenly labeled a properly factored, highly coupled data-intensive processing step a monolith—it’s at best a rocky outcropping in their distributed microservices forest. “

    —Ajish George, managing director of cyber data & analytics for State Street

    The Shiny Thing is Not Always the Best Thing

    “The news that AWS discovered that serverless and microservices aren’t for everything is surprising. Not because it didn’t work out for that use case; but because the wrong use cases have been so well-known for the serverless/microservices pattern and should have been obvious to Prime Video. It’s worth remembering: That shiny new thing isn’t necessarily a better way of trapping mice. Right tool for the right job.”

    —Doug Vanderweide, director of cloud architecture for Avanade

  • How to Secure C-Suite Buy-In for Your DevOps Efforts

    How to Secure C-Suite Buy-In for Your DevOps Efforts

    You could be forgiven for assuming DevOps is the modus operandi only for big modern enterprises like Tesla and Uber, but this is not the case. DevOps builds on earlier approaches, such as Agile development and Lean methodology, by emphasizing the importance of collaboration between development and operations teams and the use of automation to streamline the software development process.

    So, while the term “DevOps” is relatively new, the principles and practices that it encompasses have been in use for many years.

    In a traditional waterfall model, the software development process is split into distinct stages, with each stage being completed before moving on to the next one. The development and operations teams work in separate silos, with little interaction between them until the end of the development process when the finished product is handed over to the operations team for deployment and maintenance.

    Benefits of DevOps

    In contrast, a DevOps model emphasizes collaboration and communication between the development and operations teams throughout the entire software development life cycle. The goal is to create a culture of shared responsibility for the success of the project, with both teams working together to ensure that the software is developed, tested and deployed quickly and efficiently.

    DevOps also emphasizes automation, with the use of tools and processes to streamline the aforementioned project tasks. This automation helps to reduce the likelihood of errors and allows the teams to focus on higher-level tasks.

    Overall, the key differences between DevOps and a traditional waterfall model are the emphasis on collaboration and communication, the use of automation and the focus on shared responsibility for the success of the project.

    In short: Speed, time to market, agility and cost savings.

    What’s Driving DevOps Transformation?

    Any company, regardless of size, can release faster with DevOps. Look at the agile FinTech success stories; look at the generative AI success stories. In terms of cost savings too, it’s not ‘How much am I paying for 10 people, 20 people, 100 people…?’ From a DevOps standpoint, cost means penalties avoided from failing to release on time, project delays and project poor quality.

    Overall, development practices are evolving. Some clients call DevOps ‘zero rework’ development, others refer to it as ‘shift-left development.’ Whichever term you prefer, the context is the same: They want faster release and to reduce duplication of work. The way to achieve ‘shift-left’ or ‘zero rework’ would be through DevOps transformation. All of the individual pillars of DevOps concern shifting left, causing them to release faster and achieve their cost optimization.

    However you refer to it, DevOps is mainstream because of its scalability benefits, as well as the agility to react to unforeseen situations during development, QA or operations. This equates to competitive advantages and future-proofing.

    Challenges of Implementing DevOps

    Every business is different, but many of the challenges are the same. One of the main challenges is understanding your company’s specific blend of pain points on its road to DevOps transformation. Common challenges include:

    Vision – DevOps is not a top-down initiative. The role of executives in DevOps is to define a vision; if the vision is not clear, you won’t have buy-in. You can have the best tools and technology but if you don’t have buy-in, then it won’t work.

    Investment – DevOps is a change in every aspect of people, processes, technology and culture. You can’t just bring in this change with zero investment. It’s not unusual for costs to go up for the first couple of years after moving to DevOps because you’re transitioning from manual testers and normal developers to fungible team members that come at a premium. The cost advantage shows up after at least a year as rework goes down and the quality of release incrementally increases, leading to a better customer experience. So, it’s self-funded from the efficiencies it will bring in … but only if you persevere.

    Celebrating Success – Too many people try to boil the ocean with DevOps transformation. This leads to frustration, misaligned teams and, eventually, poor results. If you start small with DevOps transformation, it soon shows its superiority as a business model. This inspires your leadership and your executive teams to see the benefit of DevOps and buy into it by osmosis rather than taking the word of that executive with a big idea and a big purchase order.

    Culture and People – This is more than just a technology play. In fact, technology is the easy part. People, processes and culture all need to transform to implement DevOps. You need to move from a project to a product-based methodology. From a people perspective, you can’t bring in manual testers or manual validators to do DevOps. Any successful DevOps implementation has to get this right.

    How to Implement DevOps: The Hub and the Spoke

    Again, start small. You could start with only continuous integration and then move on to add continuous deployment. If you start over aspirationally, you are more likely to fail and go back to the old ways. As you expand DevOps gradually, you can follow the hub and spoke model.

    In the hub and spoke model, the hub is a lean team responsible for setting up the tools, automation and best practices. The hub is responsible for defining what is to be executed. The spoke, on the other hand, is responsible for execution. The spoke is the extended team and their individual projects which they are responsible for executing.

    The hub is also responsible for enablement by defining what processes, best practices, tools and pipelines should be used. Teams in the spoke will use the pipeline to promote their goal from development to QA, from QA to staging and from staging to production.

    Next-Gen DevOps

    The evolution of DevOps continues even for early adopters, especially with the introduction of AI and generative AI. How do you use AI to validate your requirements? How do you use AI to decide what infrastructure is needed and when? How can you use AI to optimize your test sets? Next-gen DevOps is about smarter DevOps and intelligent DevOps.

  • OpenAI to go Open Source — Elon Musk was a ‘Huge Idiot’ ¦ Mojo Risin’

    OpenAI to go Open Source — Elon Musk was a ‘Huge Idiot’ ¦ Mojo Risin’

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Sources say it’s an open source future for OpenAI, and improving AI perf with Mojo. (more…)

  • What’s the Difference Between DevOps and Platform Engineering?

    What’s the Difference Between DevOps and Platform Engineering?

    The DevOps space is buzzing with a new trend—platform engineering. Platform engineering has been gaining steam over the last few years but has really peaked in interest lately. In fact, according to a study conducted by Puppet Labs, 51% of organizations have adopted platform engineering or plan to do so in the next year. But what exactly is platform engineering, and how is it different than DevOps?

    Below, we’ll explore the nuances between these two concepts. We’ll review what DevOps and platform engineering are and see how they’re similar and dissimilar. We’ll also outline the benefits of platform engineering and suggest some best practices to make the most of this emerging discipline.

    What is DevOps?

    DevOps is a broad topic, but it generally involves efforts to increase software release efficiency and frequency. Some of the hallmarks of the DevOps discipline include reducing siloes between teams and increasing collaboration. DevOps is often directly associated with various tools that automate the process of moving code from a local environment into production, also known as continuous integration and continuous delivery (CI/CD). Monitoring, incident management and configuration management tools also support the DevOps process.

    “DevOps reflects more of a philosophy and a way of working,” said David Sandilands, senior solutions architect for Puppet. DevOps enables more of a shift left mentality. Instead of having operations teams “guarding the gates,” developers work in partnership with Ops. In doing so, groups can decrease build time significantly, he said. This style helps engineers incrementally and rapidly release new features, replacing old waterfall development cycles.

    The ultimate goal of DevOps is to automate the software life cycle and shorten the feedback loop. Thus, DevOps takes a life cycle approach to shipping applications more efficiently, streamlining phases such as build, release, configure, monitor, plan, create and verify. Microservices architecture and containerization also often play a crucial role in enabling DevOps.

    What is Platform Engineering?

    Platform engineering is all about developing a self-service internal platform that serves the needs of developer consumers. Although DevOps opened things up and empowered individual developers with advanced automation, it left organizations with scattered tooling and processes, especially at large enterprises. For example, various infrastructure-as-code (IaC) configurations might exist for similar functions within the same organization but be unknown to engineers in different departments. Not to mention, overseeing a growing DevOps pipeline means added responsibilities around continual updates and security.

    Platform engineering places more control in the hands of a central platform team rather than individual developers. These teams typically treat the platform as a product. They do so by evaluating the needs of developers across an organization, building out internal self-service tools that suit the needs of most consumers and maintaining the platform.

    The 2023 State of DevOps Report found that this self-service approach typically handles areas like deployment, operation of infrastructure, monitoring production applications and security and compliance. Other capabilities, such as abstracting Kubernetes or overseeing secrets management, may also play a role in these efforts.

    Benefits of Platform Engineering

    • Improve developer experience: Platform engineering can enhance the experience a developer has with their software build and release systems. A better developer experience can do loads to improve morale and create a more productive engineering atmosphere.
    • Standardize DevOps practices: By bringing together common DevOps approaches into a shared platform, organizations can begin to standardize with reusable build processes and automated infrastructure. This can unite an organization on common runtimes, observability methods and deployment processes. This can streamline the introduction of DevOps practices to an organization.
    • Secure the DevOps pipeline: Various vulnerabilities exist within tooling that comprises modern CI/CD chains and cloud-native open source software. By outsourcing DevOps responsibility to a shared platform team, the organization can coalesce around more secure practices and avoid overlooking important updates or holes.
    • Supply additional guardrails: Platform teams might also assist in the acquisition of new tools as well as outlining and enforcing policies for software infrastructure. While this may come at the cost of customization, additional guardrails can help avoid shadow IT and governance around internal development. Letting platform teams handle it could make maintenance easier across an organization.

    Platform Engineering: Under the DevOps Umbrella

    I don’t see platform engineering as replacing DevOps. Instead, I view platform engineering as falling under the umbrella of DevOps since platform engineering takes DevOps processes and makes them more reusable and self-service. Both use a lot of automation and share many of the same end goals.

    However, there are nuances between the two approaches that reflect the recent trends in the IT market. With DevOps, the market was more interested in shifting responsibility left and empowering individual developers to oversee the entire software life cycle. Now, as sprawl and shadow IT emerge, the pendulum seems to be shifting back toward a bit more centralization and governance with platform engineering.

    Platform engineering is typically more of an in-house development initiative. So, what are some best practices IT divisions should remember to get the most out of platform engineering? Hope Lynch, senior director of platform at Cloudbees, shared some great strategies to consider that revolve around treating the platform as a product. Doing this involves accepting feedback from developers, not controlling everything and always maintaining a focus on developer experience.

    Hopefully, by following these tips, organizations should have a better grasp of the emerging discipline of platform engineering and have the confidence to begin introducing platform-based concepts within their own environments.

  • DevOps Security: Your Complete Checklist

    DevOps Security: Your Complete Checklist

    In a fast-paced environment like DevOps, your security strategy needs to be even more agile. While moving fast is great for delivering application features and functionality, speed shouldn’t come at the cost of security.

    DevOps security incorporates security strategies directly into your DevOps pipeline, creating a culture of pre-secured software deployments that work seamlessly for the end user. However, this requires a complete shift in your operational policies and security mindset. You need to start viewing security as an integral part of your DevOps lifecycle instead of a separate department.

    Here’s a checklist of DevOps security strategies for your development teams to deliver secure and reliable applications faster:

    Your Complete DevOps Security Checklist

    1. Analyze your development process for security hazards.
    The first step in implementing a DevOps security strategy is finding and fixing loopholes in your existing development process.

    For instance, continuous delivery and development are key in a DevOps environment. However, if each change doesn’t go through a code review, it has the potential to break your application.

    Deploying changes without passing them through a security checkpoint is like driving at high speed on a road with many curves. Each curve (change) is a potential hazard, and the danger increases when the driver does not know a car is coming from the other side (bug).

    Therefore, you must turn the spotlight on your development process and find those curves.

    2. Create a common goal between development and security teams.
    Traditional roles of developers and security teams might look like they are at crossroads — Developers write code, and security teams find fault with their code. However, in a collaborative environment such as DevOps, security and development must work hand-in-hand to deliver a secure and seamless experience to end users.

    Your organization should have a common goal of integrating security directly into the development process. Therefore, encourage open and regular communication between development and security teams. Further, establish joint milestones and review them to ensure there are no silos between both teams.

    Once these teams get used to this collaborative approach, your deployments will become faster because fewer things will break.

    3. Integrate security procedures in the development phase.
    Automation is the lifeblood of DevOps security. Integrating scanning tools into development workflows is critical to your entire process.

    For example, automated tools can scan the code, network and infrastructure to identify security vulnerabilities and provide recommendations for remediation. With this, development teams can fix a few security concerns themselves before passing them on to the security team for a final check.

    4. Add observability to your pipeline.
    One of the biggest challenges of the distributed nature of DevOps operations is visibility. How do you efficiently monitor a large number of operation nodes spread across several different locations?

    Observability helps you do this by using logs, metrics and traces—i.e., the outputs of your application to determine internal application health. With observability tools, you can see much deeper into the operations of your systems and identify the exact details of any security concerns. You can also easily identify potential bugs and fix them before they cause problems.

    5. Make changes in small, incremental units.
    It is infinitely easier to test code in small units and deploy them as minor but incremental changes than handling large, unwieldy blocks of monolithic code. Deploying gradual changes means you can resolve vulnerabilities faster.

    6. Scan and secure your entire DevOps pipeline.
    A large part of the security hazards in your DevOps process comes from unsecured containers and operations nodes. According to a study from ThreatStack, 94% of organizations say containers represent a security risk.

    However, the risk is not limited to containers.

    Other services like APIs, third-party tools like Docker and imported code pose a potential security risk to your processes. This makes it necessary to scan all your operations nodes periodically for vulnerabilities. And once you’ve cleared them, remember to lock the door behind you by requiring strict credentials for access to your administrative dashboards.

    Ticking Items Off Your DevOps Security Checklist

    If you implement most of the security guidelines on this checklist, your organization will have a solid security foundation. However, DevOps security is not a one-time process.

    You will need to continuously integrate security protocols into each part of your DevOps life cycle. Furthermore, you will need to re-train your employees and invest in new technology that makes your security pipeline smoother, such as observability.

    However, since this entire process makes your applications more secure and compliant while shipping them to market faster, investing in DevOps security is well worth it!