devopsy.kinsta.cloud

Category: DevOps Practice

  • Why Legacy Application Development is a Challenge

    Why Legacy Application Development is a Challenge

    Enterprises seeking to modernize their legacy mainframe applications often cite the key reason as not being able to evolve quickly enough to support the changing needs of the business. This is also a driving force behind the increasing trend toward moving these legacy applications onto other platforms, either by attempting to rewrite or re-platform them.

    If this was a simple process however, the transition would have happened years ago. So, why is it so difficult to modernize mainframe applications? It is surely not the programming language. COBOL and other traditional languages are simply programming languages. They may not be considered “cool,” but neither are many of the other scripting or development languages, or configuration syntax programmers deal with day in and day out.

    So, what actually makes legacy application modernization challenging? After countless conversations with several outsourcers, system integrators and mainframe customers, four major issues came up time and time again.

    Development Tooling–Low Productivity on Mainframe

    The traditional environment for a programmer, working with COBOL or PL/1 on the mainframe, has few of the productivity features today’s programmers expect: smart cross-references, on-the-fly syntax checking, autocomplete for variables etc. Attempts have been made to bring the experience closer to that of a modern JAVA programmer, but there is no escaping the fact a mainframe exists in the background with all its quirks. This divergence in programmer productivity unfairly tends to stick to the programming languages on the mainframe. With the result that language choice gets conflated with developer tooling in decisions to make mainframe development more agile.

    Open-Source Availability–Limited Access to Innovation

    Looking for solutions to business requirements among the many popular open-source projects is one of the fundamental reasons why development processes in general have become more agile. Clearly, if less code is needed to support any requirement, the faster that requirement will be satisfied. Very few open-source projects have been implemented and tested on mainframes. Perhaps even more troubling is the difficulty of integrating a Linux-oriented solution with legacy applications in a natural way.

    Pipeline Integration–Separate Pipeline and Processes for Mainframe Development

    In a similar vein to developer tooling, modern development pipelines do not fit naturally with the way in which mainframe applications move from development into production. Build processes, testing, source-code management and promotion are integrated and streamlined within a modern development pipeline. Specifically, modern tools enable developers to promote locally-tested changes and have those changes flow through an automated testing pipeline, with steadily increased levels of integration with existing applications and data. The pipeline is facilitated by the tight integration of tools such as GIT, Eclipse, Jenkins, Containers, etc. to leverage the best practices of CI/CD. All working in harmony with an agile development organization to get application updates to production faster. The mainframe is not as efficient, mainly due to nascent support for many of these technologies, alongside the costs of running many of the automated processes repeatedly.

    Testing–Limited Autonomy and Automation

    Testing, as part of a modern development pipeline is a highly automated and scalable process. The ability to launch containers with a single click is a fundamental reason why development is faster today than in the mainframe era.

    Development expectations for testing extend to being able to launch a complete testing environment on the developer workstation, without recourse to systems administrators, for the recurring small unit tests intertwined with the development itself. Due to the physical hardware dependency of mainframe development, this testing autonomy can be highly impractical for legacy applications.

    Furthermore, large scale-tests, which can be regularly, automatically and cost-effectively scheduled in modern, container-based cloud environments, require extensive budgeting and planning for legacy applications.

    Consequently, all testing of legacy application developments is extended by the need to involve mainframe systems administrators, and schedule expensive mainframe resources.

    Solving the Issues

    To address the issues above, a solution is required which breaks the software lock-in that ties a legacy application to the mainframe. It is possible to rewrite or re-platform existing applications, although this can be time-consuming, costly and risky. A more beneficial approach is to rehost the mainframe applications by migrating them to the cloud, without recompilation or conversion of data types. This method replaces the proprietary, mainframe hardware and software dependencies with a set of Linux libraries, requiring less time and fewer resources to execute.

    Linux libraries can be created to include features that recreate the behaviors of the operating system, databases, security and language runtime that previously tied the application to the mainframe hardware. In addition, the incompatibility of the mainframe instruction-set with x86 hardware can be managed via a virtual machine.

    If the mainframe is represented as a set of libraries in the same way as any other more conventional application dependency, then the entire problem of legacy application development is reduced. Any workstation or commodity server can fully virtualize the mainframe environment, in several instances in parallel–if needed–thanks to containers. And with total elasticity when on the cloud.

    Migrating to the cloud makes it possible for a developer to spin up a full test environment for legacy applications on their workstation, in the same way as they would for a Java or C# program. The entire development cycle for legacy applications can follow the same pipeline as for other applications. The same agile models, and CI/CD DevOps policies with strong emphasis on shift-left testing can be employed. The same automated scale testing is possible. The same exploitation of open-source projects can be considered.

    Once rehosted, the word “legacy” can be dropped from the description of these applications. They simply become applications written in COBOL or PL/1. Everything else about their ongoing enhancement is identical to those applications written in modern languages.

    The highlighted issues that businesses seeking agility are facing, are only issues that exist on the traditional mainframe. Migrating legacy applications to a software defined mainframe, running on an x86 architecture provides a modern dynamic environment in which to develop, innovate, integrate and test applications.

    — Dale Vecchio

  • Maintaining Exceptional Quality Despite Shrinking Delivery Deadlines

    Maintaining Exceptional Quality Despite Shrinking Delivery Deadlines

    My first article in this series focused on the need for continuous improvement across three key dimensions of software development–velocity, quality and efficiency. While these are all equally important, in this piece we’re going to focus more specifically on quality.

    The past few months—what you might call the summer of outages—have been troublesome for many of the internet’s biggest players. Google, Apple and Cloudflare, as well as Facebook, Twitter and LinkedIn, have all experienced bouts of unplanned downtime, sometimes even multiple bouts in quick succession. Is this all just a matter of coincidence and bad luck? Perhaps, but peeling back the layers reveals an increasingly painful reality of modern DevOps life.

    While the root causes of these outages were varied, several were blamed on software glitches. This suggests software rollouts may be happening before they are comprehensively battle-tested and ready for production-level systems. This is never a good idea, especially when you’re dealing with high-profile services supporting millions of users worldwide. In these cases, software-related stumbles are that much more damaging.

    When software is rolled out prematurely, a cardinal rule of development is broken in that we, the end-users, become the unwilling guinea pig for poor-quality applications. No matter the pace of rollouts, an unwavering focus on quality needs to be maintained, and automation, integration and measurement can be the keys to ensuring this.

    Automation

    Automation refers to a pre-programmed process that executes tasks with minimal human intervention. The more you automate, the higher your quality should be. Avoiding manual intervention decreases the likelihood of less optimal techniques or human mistakes, both of which can lead to poorer application quality and customer satisfaction. Essentially, automation enables the best techniques and practices to be quickly and easily replicated.

    Consider the example of automated unit and functional testing. If you can automate the execution of the best testing scripts incrementally across code units and functions, you can test more accurately, not to mention more quickly. The benefits will increase exponentially with the additional layers of testing you can automate. We recently worked with a customer who realized an overall ROI of 467% from their investment in automated unit testing.

    Integration

    Modern applications are increasingly componentized, spanning multiple platforms. Consider an e-commerce application, where the front-end may reside on a system of engagement (such as a web server) while the back-end transaction ultimately occurs on a system of record, such as a mainframe.

    High-quality, end-to-end digital services ultimately depend on all critical components being universally and automatically included and prioritized in DevOps pipelines. Similar to automation, this is because any time manual effort is introduced into the integration process, you increase the chances for error and consequently, the risk of a particular code segment lagging behind others.

    The end result? More defects making it into production in the sprint to the finish line. It’s like the famous I Love Lucy chocolate assembly line episode, which offers a humorous visual interpretation of the chaos that erupts once human intervention is introduced into a process.  Ideally, seamless toolchain integrations, across multiple application-leveraged platforms, can be utilized to maximize quality outputs leveraging DevOps technologies you’re already using, such as JIRA, Jenkins and SonarQube.

    Measurement

    Improving quality depends on development teams constantly improving their methods. This can be achieved through ongoing measurement against key performance indicators (KPIs).

    The challenge is establishing relevant KPIs, and we have seen many development teams fail at this. The most effective KPIs are those that translate to real business impact. For example, “number of bugs released into production” is a commonly used metric, but it does nothing to convey the bottom-line cost of those bugs. A more effective metric would be the cost of resolving defects, with success being declared when the organization reduces the amount of money spent on fixing bugs.

    The cost of fixing a bug is much less when it takes place earlier in the software development lifecycle (for example, in the coding and unit testing phases) than when the software is near or already in a production state. When a development team can consistently show it is reducing costs through earlier identification of bugs, this puts them in a stronger position to justify investments in better testing and analysis tools.

    Conclusion

    We’ve long known about the challenges DevOps teams face to balance velocity and efficiency with a high degree of software quality. It’s like asking a circus performer to sprint across a tight rope. It may seem unfair, but the need to release high quality software faster and faster, while staying within budget, is not going to abate any time soon.

    The recent spate of outages has taught us that even the biggest and the strongest in the industry are not immune to immense speed pressures, and the quality stumbles that often result. Automation, integration and measurement can help development teams in their ongoing quest to meet the seemingly conflicting requirements of improving quality, velocity and efficiency—at the same time.

    — Rick Slade

  • Challenges of Designing API-Driven Experiences

    Challenges of Designing API-Driven Experiences

    As SaaS and mobile application development–and cross-software integration and communication–continue to become increasingly important for organizations across all industries to remain competitive today, APIs play a critical role. In fact, APIs dominate digital experiences today, with an average of 220 new APIs published every month, representing a 30% increase over the previous four years.

    As more organizations shift to an API-first development strategy to further drive innovation, partnerships and rich end user experiences, the proliferation of demand and consumption of APIs can bring its own challenges, if not properly executed from the get-go. Amidst the ever-increasing rush to develop and publish an API to remain competitive, we see several common issues arise–and often only after the API is rolled out–that negatively impact integrations, future partnership opportunities and end user experiences. Read on for a look at these common challenges and how to address them before you hit publish on your next API.

    Infrastructure Costs

    One of the biggest challenges when it comes to API design is the ability to manage costs, in particular infrastructure costs. From managing multiple gateway servers and instances, to building an entire API management program from the ground up, we see many common infrastructure approaches that organizations deploy today when designing APIs that lead to unnecessary and often unwieldy expenses.

    Critical to avoiding spiraling infrastructure costs is leveraging a single gateway to deploy, govern, secure and deliver global API traffic across various data centers. By using this approach, organizations can reduce man-hours spent managing and synchronizing servers; increase reliability by eliminating the need to manage multiple gateway instances; govern APIs without needing to build and maintain a separate API management platform; and eliminate regional gateway replication.

    By considering an API gateway at the onset of a design project to ensure API centric operations (such as authentication, authorization and throttling) are happening at the edge, organizations can proactively control costs and avoid financial surprises.

    Scale

    To a developer, there’s nothing worse than having a wildly successful API that can’t scale to meet the demand.

    Addressing scalability early on in the process can help define both early adoption, future success and the lifespan of the API. However, organizations commonly overlook the need to manage scalability in several ways–including planning for surges in API traffic and managing the volume of API requests consumers can make–which can lead to damaged user experience.

    Developers should look to a combination of load testing, authentication, throttling, quota management and API caching at the edge to improve and predict traffic, while preventing infrastructure from being overwhelmed by requests to ensure availability and reliability for consumers. Building in load testing early on can help a developer determine the traffic volume the application can withstand when surges in requests occur.

    Quota management can help enforce business service level agreements and limit the number of API requests that a partner is allowed to make. When it comes to API caching, some good places to start include: any resource accessible via HTTP GET, static data, immutable responses, infrequently altered or predictable responses and frequently requested data.

    Security

    While the commercial value of API development is clear, the reality is APIs can come with great cybersecurity risk. For one, APIs can provide a glimpse into the back-end of an application implementation and even the database it is connected to, providing hackers with new avenues for attacks. As the communication bridge between multiple applications, APIs left unprotected can also increase an organization’s attack surface and expose it to downtime and malicious attacks, including unintended misuse by legitimate users.

    Critical to protecting APIs is incorporating security at the infrastructure level with a multi-pronged approach that includes leveraging an API gateway to easily validate, authorize and control the access of legitimate API consumers (and block legitimate users), and endpoint protection against malicious traffic. Additionally, rate limiting–which puts caps on the number of requests per minute or second that API consumers can make–can prevent adversaries from overloading origin in an attempt to bring down API infrastructure in the form of a DDoS attack.

    Organizational Decentralization

    As speed and agility increase exponentially and API development evolves from microservices to nanoservices with individual team members owning what feels like ever-shrinking components, it’s easy to see how teams can organizationally splinter. While autonomy is important to making the development process flow quickly, decentralization is creating major governance and communication issues within many organizations today.

    This lack of governance creates its own challenges related to configuration visibility, changes and duplications across implementations which could lead to wasted resources, data leakage and more. Organizations should consider a single gateway that can expose APIs across multiple, distinct implementations but also provide suitable isolation so as to not affect the performance of other APIs when requests occur, while also providing separate control pages and the ability to set up distinct authorization and access rules for each API. The idea is the APIs are managed in the same decentralized way that is required to meet speed and agility requirements under a centralized infrastructure and governance policies.

    Strategic API design is not a simple task, but it is absolutely critical to organizations that desire long-term API success. By seeking cost-effective infrastructure, tapping tools to ensure scalability, building security protocols, practices and policies, and taking steps to more centrally manage the API design process early on will ensure longevity for the API, driving strong partnerships and, ultimately, providing better user experiences.

     

    This article was co-authored by Anthony Larkin, director of product marketing at Akamai. Anthony leads the go-to-market strategy for Akamai’s performance product portfolio. For more than 11 years at Akamai he has been dedicated to helping businesses remove barriers and unlock their potential to better engage users through web and mobile applications.

    — Sid Phadkar

  • DOES19 London: Sharing DevOps Lessons Learned

    DOES19 London: Sharing DevOps Lessons Learned

    There’s a long-held belief in the DevOps space, “If you’re not moving forward, you’re dying.” But if you’re a DevOps veteran, you know that one of the best ways to orient yourself to keep moving, regardless of where you’re at in your DevOps journey, is to listen to stories of others—both their successes and failures.

    Through story, we are given the opportunity to better understand the landscape of each phase of our growth and evolution, as well as to develop a deeper knowing of the challenges impacting our large-scale DevOps transformations.

    This is why it’s always a unique opportunity to listen as thought leaders share their first-hand experiences each year at the DevOps Enterprise Summit.

    Recently, in London, I continued our series speaking with some of the keynote speakers in our Digital Anarchist newsroom. There I was joined by two such thought leaders in the space: Ross Clanton, formerly executive director at Verizon, and Andy Burgin, platform engineer at Sky Betting and Gaming.

    Ross Clanton has been one of the keynote speakers at DOES right from the beginning. This year, he gave a “DevOps confessional” style presentation of real stories of real failures and the leadership lessons, which he highlighted during our chat.

    This isn’t Andy’s first time at DOES, either. This year, he shared several of the lessons learned from Sky Betting and Game’s time in its agile transformation since 2011, including sharing a practical approach to how company has structured teams, managed work and even the way it owns products and services.

    Below the interview you can find the presentations from Ross and Andy at this year’s DOES London event.

    In addition to this conversation, don’t miss the other 30 or so interviews we have from the conference on the Digital Anarchist site.

    And, don’t forget! DevOps Enterprise Summit Las Vegas 2019 is happening Oct. 28-30 at The Cosmopolitan Hotel. Be sure to register here if you haven’t already and join us for a premier event for the technology leaders of large, complex organizations implementing DevOps principles and practices. You can also preview the event by watching presentations from past DevOps Enterprise Summits.

    — Alan Shimel

  • DevOps Needs Collaboration and a Safe Place for Success

    DevOps Needs Collaboration and a Safe Place for Success

    When it comes to DevOps adoption, “[many organizations] tell a similar story,” said Scott Van Kalken, systems engineer at F5 Networks. It takes more time and effort to develop the right culture than you probably expect.

    Van Kalken has a long history in tech, including development and operations roles, and has spent several years practicing DevOps. He is part of the open source and meetup communities, and organizes some of the largest DevOps events in Australia.

    “DevOps typically starts in a corner of the organization, where people want to deliver high-quality software really quickly,” said Van Kalken.

    When that team gets good results, the organization decides to scale up its DevOps efforts, but tradition and silos get in the way.

    “At least two things are needed to overcome those obstacles and achieve DevOps success,” suggested Van Kalken.

    Collaboration and Safe Places

    The first thing you need to achieve DevOps success is a collaborative team environment. One of the central ideas of DevOps is you don’t throw software over the wall between functions, such as development and testing, so there needs to be an understanding and acceptance that everyone in the team is on the same side.

    Second is the provision of a safe place to fail and the realization that failure is just an iterative step on the path to implementing the best ideas.

    A financial services company that Van Kalken works with was adopting GitOps but needed to convince the service management team this was the right move.

    “Initially, it was quite challenging because service management thought it [GitOps] was uncontrolled chaos, even though that isn’t the case,” he said.

    Its support was gained after running a series of workshops that showed the underlying processes were actually the same, even though the tools used to implement them were changing.

    Service management is still the final gatekeeper, but it now approves releases in the repository rather than on paper.

    Maturity

    “There is a maturity in accepting change does involve risk,” said Van Kalken.

    Things such as GitOps provide a quick and easy way of reverting to a previous release if something goes wrong. So the “safe place to fail” idea can be implemented in a way that protects the organization in a practical sense as well as staff members in a psychological (and career protecting) sense.

    Another way DevOps can protect the organization is that frequent releases generally involve relatively minor changes, which in turn have a smaller blast radius in the event something goes wrong. Again, this is about having the maturity to recognize risk and dealing with it, rather than trying to avoid it in the first place.

    A sometimes overlooked aspect of increasing the release cadence is the effect on users, who are being asked to cope with frequent changes.

    Van Kalken pointed out that increasing the frequency does not necessarily mean multiple releases a day–it could be from two releases a year to three or four, if that’s what suits the organization.

    Not all changes have a direct impact on users. Some are under the hood, improving performance or addressing rarely-encountered bugs. But when users’ interaction with the system is changed, he suggested canary deployments as a way of checking the acceptability of the new approach among a larger pool of users than those brought into the development process, before it is released to the entire user population.

    “This approach also has a place in DevSecOps as another way of limiting the blast radius,” he said.

    Accepting Failure

    Perhaps the biggest challenge to an organization’s culture is the adoption of chaos engineering because if you’re going to kill a container or flood a network with data in order to check that the wider system can cope, you absolutely need to be in a safe place to fail.

    You also need to realize everyone involved–including developers, security people and those involved in the business side–wants to achieve high availability, and that means designing systems that can handle (partial) failures.

    It isn’t enough to do this in pre-deployment testing. Ongoing testing provides confidence that production systems will continue to cope with such failures.

    “It’s pretty cool when you get it right,” said Van Kalken. However, “your culture has to be OK with this iterative approach.”

    “DevOps is really just a methodology to achieve a better outcome,” stated Van Kalken. That involves collaboration, iteration and embedding security and other considerations up front. But the wider organization needs to understand and tolerate this, and the risk that goes with it.

    “Education, awareness and experience can all contribute to dispelling the traditional view that risk and failure are inherently bad,” he suggested.

    Are the Metrics Appropriate?

    A related point is that whether one outcome is better than another, depends on the metrics adopted. “[Metrics are] probably the biggest things organizations need to change,” said Van Kalken.

    For example, a customer who ordered a pizza doesn’t care whether a particular system involved in taking the order, cooking the pizza and then delivering it, achieved a certain uptime target. They just want the pizza to arrive promptly.

    When that doesn’t happen, he believes a collaborative approach is needed to determine why the delivery was late, exactly what caused the problem and how to quickly roll out a fix to stop it recurring.

    When it comes to practical advice, Van Kalken emphasized the need to get people together early in a project, otherwise there is a risk that sub-groups will demonize each other. In addition, you get more buy-in and everyone accepts they have to help with the pedaling, not just sit back and be passengers.

    “If people aren’t talking, you have a bad culture,” he said. But he acknowledged it can be difficult to get past existing adversarial relationships. Breaking the ice between teams that haven’t previously engaged with each other can also be tricky.

    Returning to an earlier theme, he stressed the importance of adopting an analytical, rather than punitive, approach when failure occurs. Everyone involved must know the organization accepts risk and understands the consequences.

    — Stephen Withers

  • How to Implement an Effective CI/CD Pipeline

    How to Implement an Effective CI/CD Pipeline

    DevOps bridges the gap between development, operations and IT services teams efficiently. To foster a DevOps culture, implementing the right DevOps tools with the right DevOps process is essential. Continuous integration/continuous delivery/continuous deployment (CI/CD/CD) help developers and testers ship the software faster and safer in a structured environment.

    Unlike the traditional software life cycle, the CI/CD implementation process gives a weekly or daily update instead of monthly or quarterly. The fun part is customers won’t even realize the update is in their applications, as they happen on the fly.

    A Brief Insight About CI/CD/CD Terminology

    • Continuous Integration: CI is a process in which developers integrate new codes into a shared repository several times a day. This is a better approach than the traditional one, where the developer will build new codes in isolation and then integrate into the main repository at the end of the project lifecycle. The primary goal is to detect any integration bugs at the initial stage so it can be rectified quickly. It triggers a new build whenever a new code merges with the existing main repository. Test runs are performed against these new builds to check for any breakage.
    • Continuous Delivery: At the end of the CI, CD comes in. CD ensures to automate the software delivery process and commits to deliver the integrated code into the production stage without any bugs or delay. CD in the DevOps implementation process, helps developers merge the new code with the main branch consistently so they can build an instant software-ready product. It is responsible for checking the quality of the code and performs tests to check whether it can release the functional build to the production environment.
    • Continuous Deployment: The most critical part of the automation occurs at this stage of the delivery pipeline. Whenever there is an important change in the code, the corresponding build and deployment occur simultaneously. It is achieved through continuous deployment process where it allows to perform live deployment changes for every code modification that passes through the CI stage. During this stage, there is no manual intervention right from the initial code until the code is in the production stage.

    Benefits for Implementing CI/CD

    The ultimate aim for any software application is to reach its potential customers faster than ever before. This is exactly what the CI/CD pipeline offers to any business model that leverages a software process which reduces the risks in each build and helps the end product to reach the customer quicker.

    • Achieve Faster Feedback Through CI Tools: For every committed code, the corresponding tests are run simultaneously to avoid any link breakage at later stages. Faster feedback helps to check the quality and impact of the code.
    • Greater Visibility: With CI/CD pipeline setup, the whole process of new builds, test results and any issues with new builds can be analyzed. This transparency allows the developers to know which changes in the build led to the brokerage and to avoid them in the future.
    • Early Bug detects: Conducting various types of automated testing allows to identify any bug issues at an early stage without any last minute surprises. These automated tests along with few manual test runs, help to fix any issues that may arise.

    The Different Stages of CI/CD Pipeline Implementation

    To get the full benefits of the modern DevOps environment, embarking on a scalable CI/CD DevOps implementation is important. It lays a strong foundation for building an agile DevOps for your business.

    Tools for the CI/CD Process

    Tools are the magic wand that help automate a successful CI/CD pipeline and ensure a faster software delivery.

    • Open-Source Tools or Proprietary: Depending upon the budget and the client’s requirement, business can opt for open-source tools such as Jenkins for automating the CI/CD process.
    • Cloud or Self-Hosting Services: To self-host your CI/CD application, tools such as Jenkins and Cruise Control help to setup, configure and monitor the application effectively. Otherwise, you can get SaaS providers to host on cloud services.
    • Build Status: Choose a tool which embraces the transparency and visibility of the entire process through appropriate notifications, through email or other type of communication.

    Possible Challenges of CI

    • You might need a discipline strategy to address an issue by the CI/CD pipeline, otherwise the whole fast feedback is in vain. 
    • Slower builds eventually take place as newer builds keep adding to the product.
    • Failure to train the resources to adapt to new tools and patterns.

    Wrapping Up

    In this era of faster software shipping and building quick releases on the fly, adopting a more robust project cycle that accelerates the delivery pipeline is crucial. Implementing DevOps services is the optimal way to get the effective collaboration between cross-functional teams through the CI/CD process.

    — Jay Elango

  • Common Pitfalls of Software Development and Tips to Avoid Them

    Common Pitfalls of Software Development and Tips to Avoid Them

    In business, no one wants to plan on failure. However, when it comes to software development, failure is more common than most managers want to admit. In particular, many managers and executives fear large software-based projects are especially likely to fail due to the complex nature of these projects and the nuanced reality of the deliverables.

    While there is no doubt a software development plan is complex, there is no reason to be so readily consigned to failure. In fact, it might be argued that software is so important to your operations that failure should be viewed as unacceptable. However, this aggressive approach in theory needs to be backed up with proper execution in practice. That is why you need to know the common pitfalls that befall most software projects. This is the only way to make sure your company avoids them.

    Vague Project Requirements

    One reason so many projects fall short of the mark is that these projects fail to define the mark adequately. Your software development project cannot succeed until you know what you really want it to do. This means your project specifications need to be clear from the very start. Your goals should be defined first, and then these goals should be consistently reiterated to team members throughout the process. This includes the scalability of the project. The development team needs to know what features should be included in order to produce a final software solution that actually meets your company’s needs.

    This is all part of developing a versatile software development plan. A good software development plan will anchor your project from start to finish. A poor one will inevitably leave you vulnerable to failure.

    Unrealistic Timeline

    The timeline matters to the company, which is why some companies set aggressive timelines as part of their software development plan. However, while these timelines sound good in planning sessions, they may not make much sense in the daily development process. It is pivotal to think carefully about a timeline in order to account for development, testing and more. All parties should be consulted in the development of the timeline. This includes high level executives, project management and developers. This varied input should allow the project to accommodate the needs of management alongside the needs of developers. This is a form of risk management because an insufficient timeline will lead developers to cut corners, leading to a lackluster result.

    Unbalanced Staffing Commitments

    As you define your software requirements, you will need to think about the team members who will bring your plan to fruition. If you do not assign enough people to the project, production will fall behind. However, if you put too many people on the project, things can start to get messy. Overstaffing a project can easily lead to miscommunication, and it is harder to keep a large team of people unified. This can also lead to redundancy and a lack of consistency if too many duties are spread out. For these reasons, it is important to assess your staffing situation correctly from the start. Be sure to include your project management team in the final staffing decision.

    Insufficient Testing Procedures

    In all your planning, it can be easy to neglect testing. The reality is testing should be a critical part of your plan, accounting for a significant amount of your timeline and resources. Testing is important because it allows the development team to iron out any problems with the software. All software will have bugs in its early versions, and these bugs can be caught and fixed with sufficient testing. Testing is the only way to make sure your software works the way it needs to work. If you implement deliverables that do not perform, then your workers will be flummoxed, your consumers will be disappointed and your security could be at risk. Therefore, allot plenty of time for testing throughout the entire development plan, allowing for each component to be tested prior to going live.

    Poor Hands-On Leadership

    If you really want to make sure your project goes well, then make sure you have good people in place to lead it. Your project management team is absolutely critical to successful software development. A good project manager will internalize all parts of your plan. This includes the budget, the timeline, the goals and the staff. Your project managers need to facilitate communication, manage schedules and oversee adjustments. This key leadership role is your software development plan in action. Make sure you hire the person with the right skills and the right experience. Then, make sure that person knows exactly what your goals are to achieve success.

    These pitfalls are common for a reason. However, with a little foresight, it is possible to keep your project and your company from disaster.

    — Shane Zilinskas

  • Chaos Engineering for ITOps

    Chaos Engineering for ITOps

    Chaos engineering (CE) is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. This approach is becoming commonplace in software development and operations (DevOps) practices. But how would its application extend to ITOps? CE for ITOps offers a similar framework for stress-testing a technology platform to understand its weak points and performance pitfalls under heavy pressure.

    CE tends to be used primarily in DevOps during bug testing: setting up experiments to run software under different conditions, such as peak traffic, and monitoring how it functions and performs. This becomes increasingly necessary in cloud-based systems where failure to understand extreme load responses could result in runaway cascade failures or, worse yet, spinning up thousands of extra nodes handling error conditions while not doing any actual work. These same principles, applied to IT operations management (ITOM), help define a functional baseline plus tolerances for infrastructure, policies and processes by clarifying both steady-state and chaotic outputs when extremes are reached.

    Applications in IT

    The theory of CE in DevOps gained early traction at Netflix as they moved from physical to virtual infrastructure, with the team that implemented it on AWS breaking off to form Gremlin. However, CE is not typically used in ITOps, because ITOM has historically been separated from development (generally, IT monitors system dynamics, and when a problem occurs, engineering change management or ITSM is brought in to remediate the issue).

    With the growth of containerization in cloud applications today, IT infrastructure looks more like development environments than classical multi-tier architectures. But the limitless scale of the cloud means failures can also be limitless: Microservices are well-served by testing elasticity and scalability, data flows and resiliency through stressing the system to the edge of its tolerances and fixing their shortcomings before a public crash.

    One, Two, Three…Chaos

    Implementing chaos engineering for ITOM provides a systematic approach to identifying weaknesses in a microservices-world. In a monolithic environment, you have visibility into performance and event metrics that may be lost with microservices designs. As a result, the need for operational insights becomes even more critical when scaling to unknown workloads.

    Netflix’s Chaos Monkey grew out of CE principles from their own cloud-native community, meant to address the gaps in common dev tools’ abilities to manage extreme complexities. This methodology is extendable to infrastructure and helps to set guardrails on platform behavior as a whole.

    Here are five fundamental steps to follow in order to bring this thinking into your team’s ITOM.

    Define the Current Steady State

    Performing baseline analysis is a standard concept in capacity planning, upgrade strategies and other high-impact functions. Start with something relatively simple (and small) so you don’t get overwhelmed by the data, or risk interfering with the business if something goes wrong (such as security Red Teaming). For example, monitoring CPU and network utilization, which are common bottlenecks in any IT shop.

    Define Optimal Conditions

    There’s how your system generally operates, and then there’s how it should operate; these typically aren’t the same thing. CPU utilization and network latency are always affected by application efficiencies, hardware conditions and a host of other factors. Create a standard that outlines what engineers should expect on a normal day, on an easy day and on a very hard day. These are the control groups, and the extreme day will be the stress test.

    Form a Hypothesis

    Where will the system break? If you’re running an application scenario such as doubling the peak traffic that even your worst day so far has seen, will your CPU maintain optimum utilization (or will the container provisioning engine smoothly deploy additional nodes) as in the variable control groups, or will it spike so severely that processes grind to a halt because there isn’t enough memory or network bandwidth left to manage the load?

    Execute a Real-World Event (But Contain the Blast Radius)

    Do something extreme, such as taking down a firewall that severs connectivity to one internet service provider. This will confuse the application as it tries to respond to requests with repeated failures, ramping up CPU processes as errors return from a dead network endpoint. Log events will mount, filling the database and saturating the backbone.

    Validate the Hypothesis

    What happened? Monitor utilization and network throughput during the test and see where the system fell over. Is it what you expected, or did something never previously considered take place? Did new chaos erupt from the fissures in your infrastructure? Stabilize, document and remediate.

    Never Stop Not Being Afraid

    Stressing a system to its absolute max—and a little bit further—to see where things go wrong allows you to understand steady-state behavior and error-handling, so you can fix it before something breaks in new and unexpected ways. What do traffic spikes look like? What are real-world events and their impacts on your organization?

    CE is not just for DevOps. It should be a systemic practice for load-testing (out of your comfort zone) to the point of failure. It’s a responsibility for more than microservices deployments and applies to all sorts of disciplines within the IT organization.

    — Michael Fisher

  • The Secret to Digital Transformation is Human Connection

    The Secret to Digital Transformation is Human Connection

    We know digital transformation is no longer an option for today’s organizations–it’s a requirement. End-users have come to expect an easy, digital-native experience, whether they are ordering a pizza or automating an entire network infrastructure, which is why businesses and governments are throwing trillions of dollars at new technologies and related services.

    Organizations must build and deliver exciting new software applications, faster than ever before. They must continuously innovate, adapt and stay ahead of the never-ending demands of digital data, experiences and channels. These demands have contributed to the meteoric rise of cloud computing, DevOps methodologies, open source technologies, predictive analytics, AI and many other innovations.

    However, even with all these investments, more than half of all new software or IT projects still fail. In fact, according to a study by Forbes Insights, 75% of executives say they’re still waiting to reap tangible benefits from disruptive technology. Here’s why: Technology alone does not solve digital transformation. It doesn’t address some of the most fundamental issues that inhibit or enable transformational success nor the most critical dependency, which is people.

    To truly transform, organizations must go beyond just technology to embrace the human elements of this evolution. We’ll look at six factors driven by the human side of the business, and how each of these can help, not hinder, digital transformation when properly addressed.

    Organizational Culture

    One of the biggest challenges for companies as they embark on digital transformation is the needed cultural change. The technology decisions, while daunting, are easier compared to the cultural shift organizations must undertake to truly become digital leaders. Top down mandates won’t work, because for it to be successful, digital transformation needs buy-in from all involved. People need to connect to the change and feel they can have an impact, or at the very least, know their job is safe amid the change. Also, to achieve the speed of change and innovation, organizations need to empower everyone to make decisions, not just a select few.

    Embracing an open culture can help with this. In an open culture, digital leaders look at their organization more like a community of individuals, all contributing to something bigger than themselves. Anyone can participate, lead and suggest a change or improvement. This methodology helps for the best ideas to win out, regardless of where they come from. Employees feel part of the process and ultimately, it empowers them to work toward the shared goal. Let me be clear, this isn’t a democracy or a lack of clear decision making power. However, it is using more open decision frameworks that move everyone toward a common goal, which brings us to the next big human factor.

    An Inspiring Vision with Metrics of Success

    It is often said to start with the end in mind. With digital transformation, this is absolutely true. For digital transformation to be successful, everyone in the organization needs to understand, and ideally agree with, the destination. Visions are not just words—they are driving the priorities, investments and decisions made by every team and employee. Without a clear vision, organizations can’t create an executable business strategy. The key is to make sure you have a vision that clearly describes what your organization will be like in several years–and then make that vision measurable.

    All good visions come with goals attached to them. Yet according to research from Gartner, 50% of organizations have no digital transformation metric or way to measure progress for digital transformation. Set up clear, ambitious targets–stretch goals the entire company can work toward. These metrics allow organizations to clearly check on how they have progressed and where they still need to get to.

    Cross-Organizational Teams and Processes

    Another cultural barrier to digital transformation is siloed behavior and workstreams. Often, teams refuse, or just don’t know, how to break out of their silos. Digital transformation requires collaboration and breaking down boundaries, as data, insights, applications and processes need to flow seamlessly throughout an organization. It all goes back to having a vision and an open culture. Without a culture of trust and a shared agreement on the end goal from across the organization, you won’t be able to break down these silos.

    With the vision and culture in place, adopting DevOps and agile processes is a great way to initiate this. It helps bring together business, technical and other people from across an organization to solve a specific problem or figure out how to better work together. Often, this takes a courageous leader, who doesn’t measure their value by the power or budget they hold, to broker these cross-organizational relationships and new teams.

    People Development

    In talking to IT and business leaders all over the world, they are faced with a common challenge: skill set gaps. This includes both soft skills, such as communications or management, and hard skills, such as technical capabilities. According to Red Hat’s survey on Global IT Trends & Priorities Research, the biggest blocker to an organization’s digital transformation efforts is people. More specifically, the survey found IT talent or skills gap was preventing organizations’ efforts from being successful. Technology and digital processes are changing so quickly, it’s hard to keep up. This is creating a digital skills gap in organizations across industries and around the world.

    Rather than hoping to hire people with the skills and talent you need, I encourage organizations to develop the people you have. Take your loyal employees who already fit your culture and believe in your vision and invest in them. Also reward them for taking risks. Digital disruption does not happen by standing still or encouraging small moves–it takes big bets. Leadership must also be willing to stand up for employees who try new ideas and push the boundaries.

    User Experience Focus

    According to CareerFoundry, which manages a Web UX design school, the cost of a bad user experience is high. Really high. Its research estimates bad user experience costs e-commerce companies $1.4 trillion in lost revenue.

    As we focus on the human connection part of digital transformation, user experience takes the front seat and is a top priority. We need to start with the user. What are they trying to do? What is the pain we are hopefully solving? Too many times organizations start with the cool technology and slap on the user experience as almost an afterthought. By starting with the user, projects are more likely to give users the feeling they are interfacing with a human being rather than a digital thing—and ultimately, will succeed because of it.

    Personal Connections

    One of the downsides of the digital era is we don’t have enough face time. We believe video conferences provide the same benefits as face-to-face meetings. We put our trust in digital handshakes instead of the real thing. However, to succeed at a company-wide digital transformation, you can’t rely on digital communication alone. Ironic, right?

    One key reason for making face time is to build trust. As humans, we need to see someone, look them in the eye, shake their hand, respond to their cues. All of this helps us trust the other person. To put your organization through a major transformation takes trust–across the organization, but especially of the leadership. This means leadership needs to get out there and talk to employees, customers, partners and users. Take the time and the effort to make your transformational efforts relate to the humans doing the work and buying your products or services.

    The Bottom Line

    The most common mistake organizations make is believing technology alone can solve their problems. Technology is still needed for a successful digital transformation. In fact, I am not in any way advising we all sit around and have a lovely conversation and hold hands to solve these massive technology issues. However, technology must be combined with human connection.

    In other words, it’s not just what you use to transform, but the how and the why of your transformation. The human elements require hard work across culture, hiring, people development, leadership and communication.

    Human connection is the killer application for digital transformation.

    — Margaret Dawson

  • Top 3 Challenges of DevOps Adoption

    Top 3 Challenges of DevOps Adoption

    Customer expectations continue to relentlessly rise as new technological opportunities emerge. Today, organizations demand products and application features are delivered and implemented wherever and whenever needed. Everywhere you look, execution is at risk of falling behind imaginations and business imperatives. That is, of course, unless DevOps is starting to wield a strategic and philosophic influence.

    DevOps has been one of the hottest enterprise trends for some time now, and a crucial concept in helping companies satisfy customer demands by bringing teams together to fuel better collaboration and innovation.

    According to Deloitte, organizations adopting DevOps see an 18% to 21% reduction in time to market.

    By removing the waste that impedes software development, operations are streamlined, and businesses can react faster to market demands. At the same time, automation enables companies to align business objectives with processes to immediately recover from IT failures. Deloitte also reports the speedy, DevOps-influenced, customer-friendly release of new products and services led to a 20% revenue increase.

    While that is all good, DevOps is still a new topic to many. Large scale adoption is generally slow. It is important to note DevOps is not simply the implementation of new tools. It is about rebooting deep-rooted cultural habits, transforming old processes and ensuring all changes are made for the right reasons. Don’t embrace DevOps for the sake of it. Only proceed if you have clarity and purpose.

    Breaking of the Deep-Rooted, Cultural Habits

    DevOps is not a team. It is a culture of collaboration that yields a set of best practices and operational cultures. Sometimes companies wanting to adopt a DevOps approach are met with resistance from employees uncomfortable with change. Leadership needs to come from the top. Executives must support employees in dismantling the status quo and breaking down silos hindering successful implementation. Motivation rises when results are positive, concrete and observable. There is no magic formula but, if you get the right leadership and culture in place, chances are DevOps will land with resonance. Gartner reports 88% of businesses believe team culture is one of the top impacts on an organization’s ability to scale DevOps.

    Changing Well Defined Processes to More Efficient Ones

    The best way to identify inefficiencies is to map processes and recognize what is and isn’t working. Start by looking for waste–areas where resources are being exhausted without delivering real value. A silo-influenced lack of communication can significantly slow down businesses and there is a risk each department will think DevOps is not their responsibility. Some may be resistant to implementing change, which will cause unnecessary lags within the business. It is vital to understand it is not just the responsibility of software developers to ensure the processes work for them. DevOps is a process that brings development teams and other IT stakeholders together to achieve one common goal: delivering faster, higher quality work to market.

    Beware of DevOps for DevOps Sake

    With DevOps, companies can streamline and automate every task. That doesn’t mean it’s right for every element of every business. Often, companies will pick a couple of departments to trial a DevOps approach with–usually the development team. The temptation is then to continue scaling across the organization as soon as positive results are apparent. The issue here is that different departments all have different needs and challenges and, as we’ve already said, effective DevOps rollout requires an overhaul of processes and ways of working. Organizations need to be careful and think about specific business needs, instead of just being engaged in a rudderless trend chase. Know your objectives before engaging in sweeping overhauls. Keep learning and, above all, stay tuned to customer needs.

    — Rodrigo Albuquerque