devopsy.kinsta.cloud

Category: Doin’ DevOps

  • DevOps and ITIL Integration: Driving Collaborative Agility in ITSM

    DevOps and ITIL Integration: Driving Collaborative Agility in ITSM

    Businesses face continuous pressure to deliver high-quality IT services efficiently. One emerging integration to meet these goals is between DevOps and IT service management (ITSM), more commonly known as DevOps+.

    DevOps+ is becoming more than a trend and has emerged as a powerful collaborative approach to meet these challenges. By combining the strengths of DevOps’ agile software development practices and ITSM’s structured service management approach, organizations can foster better collaboration, enhance agility and improve overall service delivery efficiency.

    DevOps and ITSM Symbiosis

    DevOps and ITSM traditionally operated as separate frameworks with distinct focuses. DevOps emphasizes rapid software delivery and collaboration between development and operations teams. On the other hand, ITSM-based ITIL (information technology infrastructure library) provides best practices for ensuring service stability, reliability and adherence to governance and compliance.

    Integrating DevOps and ITIL creates a symbiotic relationship, allowing organizations to leverage the best of both worlds. DevOps+ encourages a cultural shift, breaking down silos and promoting cross-functional communication. This collaboration enhances the entire software development lifecycle, leading to faster feedback loops, accelerated innovation, and increased responsiveness to business needs.

    Enhancing Agility and Speed

    DevOps+ integration brings greater agility and speed to IT service management. Organizations can streamline the release process by automating deployment pipelines and implementing continuous integration and continuous delivery (CI/CD) practices, reducing time-to-market for new features and updates. Agile development methodologies enhance the adaptability of ITSM processes to meet evolving business requirements effectively.

    Ensuring Stability and Reliability

    While agility is crucial, it must not compromise service stability and reliability. ITIL’s best practices provide a strong foundation for maintaining service quality. By incorporating ITIL processes such as incident management, change management and problem management, organizations can effectively manage risks, reduce service disruptions and ensure the stability of critical IT services.

    Leveraging Automation and Orchestration

    Automation is central to both DevOps and ITIL. Through integration, organizations can leverage automation and orchestration tools effectively. Automated testing, infrastructure provisioning and configuration management improve efficiency and consistency across the ITSM lifecycle. This enables teams to focus on higher-value tasks, reducing manual errors and operational overhead.

    Emphasizing Customer-Centricity

    DevOps+ puts the end-user experience at the forefront of service delivery. By continuously seeking customer feedback and measuring service performance, organizations can better align their offerings with user expectations. A customer-centric approach fosters higher satisfaction, loyalty, and competitive advantage.

    Breaking Down Barriers to Change

    Change management is vital in ITIL, ensuring that modifications are planned and controlled. In the context of DevOps+, change management becomes more adaptive and responsive, facilitating a culture that embraces change as an integral part of the development process. This promotes experimentation, reduces resistance to change, and accelerates innovation.

    Challenges of DevOps and ITIL Integration

    Despite the numerous benefits of DevOps+ integration, organizations may encounter challenges during implementation:

    Cultural Shift: Merging DevOps and ITIL requires a cultural shift. Overcoming resistance to change and fostering collaboration between traditionally separate teams can be challenging but is crucial for success.

    Process Alignment: DevOps and ITIL have different processes and terminologies. Aligning these methodologies while preserving their individual strengths can be complex.

    Skill Sets: Integrating DevOps and ITIL necessitates upskilling teams in both domains to operate effectively in the integrated environment.

    Tooling and Automation: Adopting suitable tools and technologies that support DevOps+ practices and integration is vital but may require significant investments.

    Metrics and Performance Measurement

    Measuring the success of DevOps+ integration is essential to understand its impact and effectiveness. Key performance indicators (KPIs) can help monitor progress and identify areas for improvement.

    Deployment Frequency: The frequency of software deployments indicates the efficiency of the integrated approach. Higher deployment frequency implies faster and more reliable releases.

    Mean Time to Resolve Incidents (MTTR): MTTR measures how quickly teams can resolve incidents and restore services after disruptions occur. A reduced MTTR demonstrates improved service reliability and faster incident resolution.

    Customer Satisfaction Scores: Measuring customer satisfaction provides insights into how well the integrated approach meets end-users’ needs and expectations.

    Continuous Improvement and Feedback Loops

    Continuous improvement is at the core of DevOps+. Establishing feedback loops and conducting regular retrospectives empowers teams to identify areas for improvement and implement iterative enhancements to processes and practices.

    Implementing DevOps+ integration requires a well-structured roadmap. For example, assessment and planning help leaders evaluate the organization’s current ITSM practices to understand how DevOps and ITIL principles align and define integration vision and goals.

    Additionally, your continuous optimization efforts allow you to continuously monitor performance metrics, gather feedback, and conduct retrospectives to identify improvement areas, leading to better team collaboration and communication. Fostering cross-functional collaboration between development, operations and other relevant teams is easily accomplished. To meet these goals, encourage regular meetings, schedule joint planning sessions and enable shared documentation to ensure everyone is aligned on goals and progress.

    When this happens, leverage automation tools to streamline manual processes and increase efficiency. Implement continuous integration and continuous delivery pipelines to automate software deployment and testing. Automation enhances consistency, reduces the risk of human errors and adds a new layer for measuring and evaluating to continuously measure the performance of the integrated approach using defined KPIs. In this vein, you can regularly review progress against the initial vision and goals to ensure the integration aligns with organizational objectives.

    Next, as the DevOps+ integration proves successful in pilot projects, gradually scale and expand it to other teams and departments. Share the success stories and best practices to encourage broader organizational adoption.

    Conclusion

    In conclusion, the integration of DevOps and ITIL, known as DevOps+, marks a transformative approach to IT service management. By combining the agility and rapid deployment of DevOps with the stability and reliability of ITIL, organizations can drive collaborative agility and achieve efficient service delivery.

    DevOps+ integration requires overcoming cultural barriers, aligning processes and upskilling teams, but the rewards are immense. With clear metrics to measure success, organizations can continually optimize their DevOps+ approach to meet customer needs and adapt to evolving market demands.

    As technology evolves, DevOps+ integration will remain at the forefront of cutting-edge IT service management, empowering organizations to embark on a journey of innovation, efficiency and excellence.

    By embracing DevOps+ as a guiding philosophy, organizations can cultivate a culture of collaboration, continuous improvement and customer-centricity, laying the groundwork for a prosperous future in the dynamic world of IT service management.

  • Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

    Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

    Multi-tenancy—where a single instance or multiple instances of one or more applications operate in a single shared infrastructure environment—is not a new concept. First used in mainframe hardware, timesharing enabled a single massive piece of computing iron to be sliced up and used by multiple applications on a single hardware instance. This simplified both the hardware landscape as well as the skills and resources needed to manage environments. With the rise of software-as-a-service (SaaS) in today’s modern private and public cloud environments, multi-tenancy is now a common and critical function. It allows cloud companies to scale and customers to save money by essentially pooling resources. As more applications demand more and more data, multi-tenancy is now a core function at the database layer.

    But when you have a lot of different applications running on the same infrastructure and accessing the same database, how do you prevent one application from impacting the data performance of another? And how do you determine from the underlying database infrastructure which tenants get what infrastructure and at what performance level?

    Here’s a look at the driving forces behind today’s real-time data needs and how they impact multi-tenant data architecture management.

    SLAs Are at the Heart of Multi-Tenant Data Management

    Everyone wants faster performance and more resilient systems. We’re impatient, so reliable, real-time performance delivers large, competitive business benefits. However, every tenant does not require the same level of performance. Actual performance needs may be plenty fast for a user or machine without requiring the fastest execution path possible with the available resources.

    Service-level agreements (SLAs) ultimately govern the resources delivered to a tenant. Real-time may need to be measured in sub-milliseconds in one instance (i.e., for a stock trade) but in sub-seconds for others (i.e., in a Zelle funds transfer). This is where developers, architects and operators partner with a business to understand its actual user or system needs and to build a system that can deliver the SLA using the minimum amount of resources possible.

    Mixed Workloads and the ‘Noisy Neighbor’ Problem

    As soon as workloads are shared, there’s the potential for one application to quickly grab and use resources in a way that impacts other applications. Often called a “noisy neighbor,” the impacting applications can dramatically interfere with real-time response time SLAs when multi-tenant apps share an infrastructure base. For example, one node in a database cluster becomes heavily loaded, and the performance of other nodes begins to be impacted, causing SLAs to be missed.

    Typically, multiple virtual machines (VMs) are hosted on a single physical machine or node but share system resources (i.e., processor, network, storage, etc.) with the other VMs. It’s possible for applications to overuse those shared resources, impacting other applications and sometimes even the database itself. Such overusage of the infrastructure layer (in this case, sharing compute, storage, network queues, etc.) leads to poor database and end-to-end system performance. It’s like plugging a 100-amp appliance into a 50-amp circuit. All the circuit breakers will trip, and the entire system comes to a screeching halt!

    Strategies to Maximize Performance for Database Multi-Tenancy

    Configuring and managing the database itself (not just infrastructure) with multi-tenancy in mind can maximize performance and avoid problems like noisy neighbors. Isolating databases in your multi-tenant architecture can help minimize non-database problems that can be the root cause of poor performance and missed SLAs.

    Start by picking databases that allow applications to isolate their data access based on sets (or tables in relational parlance) that can have storage limits attached to them. When combined with rate limits for transactions on a per-user basis, applications can be guaranteed their quota of runtime resources. Using the per-app quota configurations for items like storage and TPS (transactions per second), the database can evaluate the total capacity available in the system and ensure that enough resources are reserved and available to ensure that the SLAs of all applications are met all the time.

    Note that unexpected events that cause rebalancing of data within the distributed database, such as node and network failures, require additional resources. The database system will subtract that resource capacity from what is available for apps so that the system is not overcommitted during failure situations.

    Enterprises also need very good automation and observability to fix immediate problems quickly and have the requisite visibility into long-term trends that impact performance and SLA compliance. Otherwise, by the time a human notices a problem, the SLA may already have been missed and the business impacted.

    Good observability can help you understand how data is used and create those automations. As more applications access and use the same data set in different ways, you need modern observability tools integrated into (or interoperable with) the underlying database.

    With a good understanding of how your data-rich applications interoperate with other tenants in mixed-load environments, you can begin to deploy these strategies to make sure you conform to all required SLAs and keep all that data working hard for the business.

  • How to Build a Data Platform for Self-Service, Ad-Hoc Analytics

    How to Build a Data Platform for Self-Service, Ad-Hoc Analytics

    In a digital world, data is often the differentiator between success and failure. Whether it’s defending against cybersecurity threats, improving application performance, or resolving full-blown outages, data is critical to the modern DevOps team.

    In fact, there are plenty of situations where DevOps engineers, SRE teams and other observability-focused departments need to explore reams of data rapidly and flexibly. An SRE trying to figure out the root cause of latency may need to dissect the movement of data across endpoints to find the malfunctioning one. Another engineer may need to compare historical and current performance to find anomalous behavior. Lastly, a DevOps engineer may have to dissect user performance metrics to understand the scope of a slowdown—whether it is global or regional.

    Every use case requires dashboards that can provide detailed data through a wide variety of visualizations, including choropleth maps, stack areas, pie charts and bar graphs. Ideally, a dashboard will enable teams to isolate dimensions, apply filters, and dive into data for deeper insights.

    Unfortunately, not all dashboards can support this type of fast, flexible analysis. Many still utilize outdated technologies, which lack the flexibility, agility and scalability required to seamlessly explore data. Many were also designed without the urgency that today’s data requires—after all, in previous years, most uses for dashboards (such as internal reporting) were not time-sensitive.

    Other dashboards are constrained by templates, which offer a finite array of widgets, tools and drill-down capabilities. These dashboards may feel unwieldy and sluggish, unsuited for a rapidly evolving situation like an application outage.

    Requirements

    To truly fulfill the promise of self-service, ad-hoc analytics across huge datasets, teams need to work in real-time—and so they need a database capable of timely responses.

    When an application goes down, an SRE may not know what to look for and, hence, needs to quickly comb through lots of data. While some dashboards rely on workarounds for faster queries, such as pre-aggregations, precomputing, or rollups, this is not possible in this instance because the SRE simply won’t know what to look for. After all, they can’t necessarily predict what will go wrong, and even if they work on assumptions from previous failures, this current issue could be much different.

    Therefore, dashboards must provide plenty of functionalities and visualizations. Users should be able to filter data by time, isolate variables like location, and zoom into specific time intervals with a few clicks. Dashboards should also accommodate diverse data types, including intricate parent-child relationships and nested columns.

    Further, dashboards must offer a depth of insight. An online streaming media platform may need to assess user metrics (such as latency or load times) across different devices, operating systems, and regions to fine-tune performance. A cloud provider has to monitor their physical hardware for high temperatures, slowed network switches or devices, and other anomalies.

    Because data is now so important to success, many more people across a company, including data scientists, product managers and external users, themselves require data-driven insights. In these situations, dashboards must handle the increased user traffic and query activity, maintaining fast responses even under load. This is especially important considering that a single-user operation (such as a zoom) will require multiple queries on the backend to execute.

    A database also must scale seamlessly. If an organization’s environment generates millions of events an hour, that equates to billions of events per day or week—challenging for any database to ingest, store, analyze and query. In fact, many databases cannot successfully provide fast response times while managing large datasets and high query volume. As an example, transactional databases (OLTP) can often query rapidly but cannot execute analytics at scale, while analytical databases (OLAP) can analyze massive volumes of data but not at speed.

    Apache Druid for Independent, Ad-hoc Data Exploration

    This is where open source Apache Druid comes in. Combining the scale and advanced analytics of an OLAP database with the speed of an OLTP database, Druid offers swift, real-time data exploration.

    Upon ingesting data, Druid makes it immediately available for querying and analysis, removing the need to first batch or aggregate the data in some way. In addition, Druid natively integrates with streaming technologies like Apache Kafka and Amazon Kinesis, removing the need for workarounds or connectors.

    Druid powers interactive visualizations, providing millisecond response times, enabling more versatile exploration, expanding the range of available dimensions and filters, and even maintaining subsecond speeds in the face of surging user and query volumes.

    Druid’s unique architecture also enables easy scaling. By separating key duties among separate node types—data nodes for storage, master nodes for data ingestion and availability, and query nodes for executing queries and returning results via the scatter/gather method—Druid ensures that nodes can be independently scaled based on need. Afterward, Druid also automatically rebalances traffic to ensure consistent performance.

    Salesforce: A Druid Success Story

    Salesforce pioneered the customer relationship management (CRM) space, serves 150,000 customers worldwide and earns billions in annual revenue. 

    The Edge Intelligence Team is the division of Salesforce that tackles the massive task of ingesting, processing, filtering, aggregating, and querying the entirety of their log data — anywhere from billions to trillions of lines daily. Each minute, Salesforce ingests 200 million metrics, while each day, Salesforce processes five billion daily events globally. In total, Salesforce accumulates dozens of petabytes of data in their transactional store, five petabytes of logs in their data centers, and almost 200 petabytes in their Hadoop storage.

    Salesforce teams use Druid to unlock real-time insights into product performance and user experiences, diving into large datasets instantly. Anyone from engineers to account executives can query a wide variety of dimensions, filters and aggregations to better understand trends, troubleshoot any issues that arise and set strategies for the future.

    By using Druid’s compaction abilities, Salesforce also decreased the number of Druid rows by 82%, leading to an overall reduction of their storage footprint by 47% and accelerating their query response times by 30%.

  • 5 Effective Git Workflows to Streamline Your Development Process

    5 Effective Git Workflows to Streamline Your Development Process

    In the fast-paced world of software development, efficient collaboration and version control are vital for successful project management. Git, a distributed version control system, offers various workflows to facilitate seamless collaboration among team members. Each workflow is designed to address specific development needs and enhance productivity. In this blog post, we will explore five popular Git workflows and discuss their significance in the software development process. 

    Centralized Workflow

    The centralized workflow is a simple and straightforward approach commonly used in small teams or projects. In this workflow, a central repository acts as a single source of truth for the project. Developers clone the repository, make changes in their local branches and push their changes to the central repository for integration. This workflow is easy to understand and implement, making it ideal for beginners. It promotes a linear development process, making it easier to track changes and maintain code consistency. 

    Feature Branch Workflow

    The feature branch workflow is widely adopted in larger development teams. It enables parallel development and promotes the isolation of work. Each feature or task in this workflow is given a specific branch. Developers work on their features, branch off from the main branch, and submit updates to the remote repository. Once the feature is complete, it undergoes code review and is merged back into the main branch. This workflow allows for better collaboration as developers can work independently on separate branches without interfering with each other’s progress. 

    Gitflow Workflow

    The gitflow workflow is a robust and well-defined branching model that is particularly useful for projects with multiple release cycles and long-term maintenance. It incorporates two main branches: The “master” branch for stable releases and the “develop” branch for ongoing development. Developers create feature branches from the “develop” branch, work on their features, and merge them back into “develop” for integration testing. Once the code in “develop” is ready, it is merged into the “master” branch for a new release. This workflow ensures a structured and controlled release process, making it easier to manage versioning and hotfixes. 

    Forking Workflow

    The forking workflow is commonly used in open source projects and promotes collaboration among multiple contributors. In this workflow, each developer creates a personal fork of the main repository. They make changes in their forked repository, create a pull request to propose changes and undergo a review process. The changes can then be merged into the main repository by the project maintainers. This workflow allows for a clear separation of duties and provides a controlled mechanism for accepting contributions from external developers. 

    Release Flow Workflow

    The release flow workflow is well-suited for projects that require frequent releases and hotfixes. It combines the concepts of the feature branch and gitflow workflows. The main difference lies in the way releases are managed. In this workflow, a dedicated release branch is created for each release. Once the release is ready, it is merged into both the “master” and “develop” branches. This workflow allows for efficient bug fixes and maintenance, ensuring that critical updates are quickly propagated to the main development branch. 

    Why Are These Workflows Needed? 

    Efficient Collaboration: Git workflows provide structure and guidelines for collaboration, enabling developers to work in parallel without conflicts. They facilitate seamless communication and code integration, fostering teamwork and productivity. 

    Code Consistency and Quality: By following established workflows, developers can ensure code consistency and maintain high-quality standards. Workflows enable code review processes, catching potential issues early and reducing the risk of introducing bugs into the main codebase. 

    Traceability and Version Control: Git workflows provide a clear history of changes, allowing developers to trace back and understand the evolution of the codebase. They enable easy rollbacks and provide a reliable version control system, safeguarding against accidental data loss. 

    Streamlined Releases and Hotfixes: Workflows like gitflow and release flow are specifically designed to streamline the release process, making it easier to manage versioning, hotfixes, and long-term maintenance. These workflows ensure that stable releases are delivered promptly and critical updates are quickly deployed. 

    Conclusion 

    Git workflows are powerful tools that help streamline the software development process. Whether you’re a small team or a large enterprise, adopting the right workflow can significantly improve collaboration, code quality, and overall productivity. From the centralized simplicity to the structured gitflow model, each workflow serves a specific purpose and caters to different project requirements. By leveraging the appropriate workflow, developers can navigate the complexities of version control and empower their teams to deliver exceptional software. 

  • Microsoft kills Python 3.7 ¦ … and VBScript ¦ Exascaling ARM on Jupiter

    Microsoft kills Python 3.7 ¦ … and VBScript ¦ Exascaling ARM on Jupiter

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: VS Code drops support for Python 3.7, Windows drops VBScript, and Europe plans the fastest ARM supercomputer. (more…)

  • The Ultimate Measure of Software Development Effectiveness

    The Ultimate Measure of Software Development Effectiveness

    There is a famous quote by Peter Drucker that states, “You can’t manage what you can’t measure.” In our industry, measuring how quickly we can get working software into the hands of our customers has become a mantra for agile development teams, and it’s even being talked about in the boardrooms.

    Customers measure the value of software by how quickly it can solve their problems. Customer needs change constantly. Just like us, they are responding to the needs of their own customers. Reacting quickly to customer feedback and turning that feedback into working features is key to customer satisfaction and retention.

    Having these directives in mind, it’s no wonder our teams are being asked to deliver working software sooner. Often, the main roadblock in achieving this goal lies not in developing the features but in getting them deployed.

    Let’s focus on three areas that determine your team’s deployment performance, which, in turn, directly impacts your software development effectiveness.

    1. Frequency of Deployments

    When I meet developers, I ask them a simple question: How often do you typically commit your code? A typical answer is a few times a day or even a few times per hour. However, if you ask the same person a question about how often their application is deployed to production, the answer varies widely.

    Some will say weekly or daily or still – very rarely – even a few times a day, which is software development nirvana. However, a more common answer is monthly or quarterly. I’m also not surprised if someone says that they release annually. They say it with an honest level of pride and accomplishment.

    Let’s put these two things – code commit and deployment frequency – into a broader perspective. If the goal of measuring software development effectiveness is getting working software into the hands of the users, it doesn’t matter how often you commit your code but how often you can give your users new value.

    Delivering small improvements to your customers has more impact on client satisfaction than giving them a big gift (new release) once a year.

    2. The Control of the ‘Launch’ Button

    Triggering a deployment includes someone instructing a deployment, pushing a “deploy” button, or committing triggered deployments. How many people in your organization can promote the application to a production environment? If this privilege is limited to a single person or just a handful of people – we have a problem.

    There are many reasons you might want to limit access to production. It could be because you don’t have safety measures built into your deployment process, such as role-based access control. You may believe in the delusion that you can protect your organization just by having fewer people with access to the production environment.

    The reality is that artificially limiting user access to production significantly reduces your chances of deploying frequently. Imagine if the “person in charge” must take a sick day or is traveling. Remember that access to the production environment is also required for bug fixes when your organization applies a fix-forward versus rollback strategy.

    3. The Level of Deployment Automation

    Measuring software development performance is directly correlated to the level of process automation. You can’t manually deploy to hundreds or thousands of customers in a reasonable amount of time. The manual process does not scale, as time is the one thing we can stretch.

    Ask yourself: Is your deployment process manual, partly automated (where parts of the process are automated, such as scripts that can be triggered to perform application deployment tasks), or can you claim that you fully automated your deployment process where there is no manual intervention, apart from push-button approvals.

    The more you automate your deployment process, the more productive your development team can be. For complex deployments, you may need to target many customers, sets of infrastructure or even physical locations. You might need to apply different configurations or hit agreed maintenance windows for customers or regions. One way to accelerate software delivery for these scenarios is tenanted deployments.

    Tenanted deployments capture these differences and apply them to a shared deployment process, allowing the complexity to be automated. You can manage tenanted deployments with tags, variables and customizations while reducing the cognitive load for team members.

    There are many ways deployment frequency impacts software development effectiveness. When you deploy more often, you reduce deployment risk, increase quality and get feedback sooner. One action you can take to reduce batch sizes is to increase deployment frequency. Smaller batch sizes will also allow you to recover faster when a problem arises.

    Remember that the longer the time between deployments, the more changes and points of failure build up.

    As you think about ways to increase software development effectiveness, think about the deployment process as a way to get close to the ultimate measure of developer success–getting the working software into the hands of customers sooner!

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2023 – November 6-9, 2023.