devopsy.kinsta.cloud

Category: Enterprise DevOps

  • 5 Steps to Ensuring Validity of Your Business-Critical Data

    5 Steps to Ensuring Validity of Your Business-Critical Data

    Bad data can lead organizations to make mistakes. If your organization isn’t continuously ensuring data is accurate, you can never be sure the business decisions you’re making are smart.

    So, how do you continuously validate your business data without consuming excessive resources? The answer starts with your data professionals, who are entrusted with massive volumes of internal, customer and partner data. A breakdown in database performance can consume valuable time finding a fix. Also, data that’s not properly tested and validated can lead to poor business decisions. IBM estimates poor quality data costs the U.S. economy $3 trillion per year.

    Understanding where to implement data validation tests is a vital step to ensuring accurate data. Data testing is traditionally a development-focused initiative that involves testing smaller units of functionality and asserting what the expected results will be within a development or test database environment. Data validation, on the other hand, focuses on production databases and involves validating larger processes. These processes can include Extract, Load, Transform (ELT) and Extract, Transform and Load (ETL) batches, data and application integration, vendor data feeds and data exports for partners.

    While data validation appears to be a massive undertaking, making small but important changes now will set you up for future success. Consider these approaches to incorporate testing and validation into standard database platform operations.

    Conduct Unit Testing

    Test small units of functionality in isolation from other parts of the system. Your database management team will be able to offer temporary or mock-up data stores that will allow data professionals to test smaller subsets of data.

    Conduct Integration Testing

    Verify units of functionality combine to produce useful results. This will mean taking a careful look to make sure data gets from point A to point B with the intended transformations. Breaking down these processes into bite-sized pieces is the key to testing business rules throughout integration processes.

    Automate Testing

    The keys to building quality test plans include being able to standardize tests, repeat them on a schedule or within a continuous integration/continuous delivery (CI/CD) pipeline and make testing results transparent to stakeholders. Developing an automated process is fundamental to taking a proactive approach as you incorporate testing and validation into regular processes.

    Monitor Production Databases

    Don’t stop at testing and validating data only when updates are deployed. Validating that data is correct should be an integral, ongoing process and will ensure confidence in the data.

    Validate Movement

    Validate counts, balances and totals anywhere data is transferred from one system to another. Monitor that values fall into expected ranges, domain values are adhered to and data heuristics are met. Also, evaluate validation results over time to determine overall data health. Is the data optimized for business needs? Are there holes in the efficiency process that should be addressed immediately? Leverage historical validation results for baselines to identify exceptions and determine a path forward for continuous monitoring processes. Make sure you have the needed capabilities for ongoing testing, validation and monitoring processes for your database environments.

    The journey to a proactive approach to database testing—and away from a traditional reactive or ad-hoc approach—will come with its challenges. Bad data affects companies of all sizes in all industries. Once you understand where to implement data validation tests and start following these best practices for testing and validation, you’ll be well on your way to stopping bad data from compromising your organization’s ability to make sound business decisions.

    — John Welch

  • Better Together: Why Software-Development Toolmakers Should Embrace Integration

    Better Together: Why Software-Development Toolmakers Should Embrace Integration

    It’s getting harder for businesses that are building technology tools to get attention, traction and ultimately widespread adoption. You can’t just launch an amazing new developer tool and hope developers will find you; providers need to zero in on a specific use case that empowers developers while also ensuring those developers will still have access to the broad ecosystem of other tools they need to do their jobs.

    The more forward-thinking companies in enterprise SaaS, public cloud computing and artificial intelligence are increasingly partnering with companies they might once have considered competitors to elevate the presence of all their brands among end users. By meeting in the middle, and integrating with a common-ground platform, providers join their customers where they are, on the platform that makes the most sense for them.

    Instead of forcing customers to choose, providers can take advantage of powerful network effects, where their tools become more valuable because they’re part of a network. It’s one of the oldest concepts in computing: Metcalfe’s Law states the value of a network is equal to the square of the number of users on that network.

    But building that network yourself can be time-consuming and costly, as many of those who’ve done it alone have discovered. That construction project can easily distract providers from their own core competencies, forcing them to spend time and resources on essential but undifferentiated development work.

    Network Effects

    The mobile side of tech has been operating under these principles since the launch of distribution platforms such as the App Store and Google Play, which introduced mobile developers to millions of potential customers. Those customers want seamless connections between their favorite apps and services to streamline their digital lives. They don’t want every app developer to create their own marketplace with incompatible payment methods, unfamiliar interfaces or strange hoops to jump through.

    The explosion of enterprise software vendors enabled by cloud computing and software-as-a-service models has created similar patterns. Cloud-native developers have embraced distribution channels such as the AWS Marketplace as the best way to find their customers, and enterprise companies such as Box and Dropbox are making integrations with rival products and office productivity tools (like G Suite or Office 365) an important part of their product strategies.

    “What seemed unfathomable merely a decade ago is now becoming commonplace as Fortune 500 companies are mixing and matching best-in-class technologies—from upstarts to cloud mainstays like Salesforce, Workday and ServiceNow—to power their business,” said Aaron Levie, Box CEO.

    But the work required to enable those integrations is substantial.

    Growing companies focused on building best-in-class tools for developers don’t always have the in-house expertise required to strike partnership deals with platforms and support customers with a world-class billing system. Likewise, end users often need to do their own due diligence when considering whether or not a promising new development tool will play nicely with their existing tech infrastructure.

    Go to Market

    Marketplaces are the best way to solve these problems. Service providers need distribution and integration with other development tools to reach end users, and end users want the peace of mind of knowing they’ll be able to properly evaluate a tool under consideration through a certified clearinghouse.

    Companies can certainly build API-level integrations on their own for one-off partnerships or integrations where performance isn’t a major requirement. However, when performance is vital, such as with databases or infrastructure monitoring tools, a more deft touch is required.

    These seamless, low-latency connections are only going to become more important in a multi-cloud world as companies use the power and scale of Kubernetes to build tech infrastructure that spans several different cloud providers and perhaps even their own on-premises servers. True service interoperability will require common frameworks to make sense of all those connections and allow developers to mix and match their favorite tools to create the technology stacks of the future.

    Enterprise software silos are a thing of the past. The future of software development will rely on tools that blend the best available services—no matter where they were born—and link developers, service providers and platform companies in flexible and dependable ways.

    The companies that figure out how to integrate best-of-breed software development tools with the leading platforms and services of our time will win. If you’re just starting out, let someone else handle the complexities of integration and distribution while you focus on creating the best developer tool you can.

    — Jevon MacDonald

  • 5 Best Practices for Managing Open-Source Components

    5 Best Practices for Managing Open-Source Components

    In today’s connected world, software development environments focus a lot on faced-paced development. Organizations that adopt agile development practices such as DevOps and use Open-Source (OS) software and components to their advantage have a much better chance of keeping up with demand and shorten the Software Development Lifecycle (SDLC).

    However, incorporating OS components into applications does not come without risks, as they often contain bugs or vulnerabilities, and are highly exposed to exploits. In this article, we will cover what is Open-Source Software (OSS) and OS components, discuss the importance of security for open source and cover five best practices for managing open-source components.

    What Is Open-Source Software?

    Open source refers to any software with accessible source code that anyone can modify and share freely. You can head over to the Open Source Initiative (OSI) and see a detailed list of criteria for the definition of OSS. While OSS allows many more modifications over proprietary software, you must still accept the terms and conditions defined in its license.

    Some estimations suggest OSS adoption saves $60 billion dollars for consumers per year, while about 96% of all commercial applications contain some components or elements of OSS.

    The Benefits of Open-Source Components

    Open-source components are software units that meet the definition provided by the OSI. Modern development environments put a lot of pressure on developers to build and deploy applications more quickly. To successfully achieve their goals within short software release cycles, developers frequently use OS components.

    The Importance of Security for Open-Source Components

    OS components are incorporated into the vast majority of software and applications. Even though they allow developers to be more efficient and get more done, they can also introduce new vulnerabilities and jeopardize the integrity of the entire code.

    Since OS components are developed in a distributed system, they are often more vulnerable to exploits than proprietary code. Some of the reasons for this are:

    • Distributed development: Compared to a distributed OSS developed by many individuals, it is much easier to track and manage a code developed in-house and ensure you follow policies and protocols.
    • Fast release cycles: OSS development is much quicker and agile, which means that very often, by the time a new software release passes all the quality and vulnerability checks, it has already become obsolete with the release of a newer cycle. This makes it almost impossible and in most cases useless to run the necessary tests and ensure the safety of using the code.
    • Lack of quality standards: OSS elements do not pass the same quality and standard checks as proprietary code. Thus, unless you evaluate each OS component before implementing it, you can often incorporate low-quality code that is exposed to vulnerabilities, lowering the overall quality of your code.
    • The double-edged sword: When members of the open-source community discover a vulnerability, they publicly share the details of the problem in online forums to help developers learn about the risk and how to patch it. Unfortunately, this solution is often a double-edged sword. While this is a step in the right direction, it exposes the details of the vulnerability to threat actors as well, who can exploit it against systems that have not yet patched the vulnerability.
    • Lack of security measures: The problem with OSS is even greater for smaller companies, who prefer to use it to save time and resources but lack the necessary security measures to ensure the components they are implementing are secure enough to be implemented into the code.

    Now, let’s take a look at the five best practices for OS components management.

    Form a Policy

    Establish a clear policy on how to incorporate and manage OS components. This step will give usage guidelines to your developers and ensure OSS elements are incorporated responsibly without damaging code quality. Additionally, if you are facing problems with OSS elements, this policy can act as a playbook and give instructions to your security teams on how to handle the threat.

    Carefully Choose Software

    It is important to choose the right software for you. While it might be tempting to use many kinds of software and enjoy all their possible benefits, you have to consider your requirements and ensure the software you use can answer all your demands. You should consider the amount of support you require and the risks you might face. One of the most crucial aspects is how secure the software is. A good way to measure the software’s security is the Open Web App Application Security Project (OWASP) dependency-check, a utility tool that identifies project dependencies and checks if they contain any open-source vulnerabilities.

    Track and Update OS Components Regularly

    Track your OS components throughout the entire product life cycle and ensure transparency to find critical security vulnerabilities. Updating OS components versions frequently is typically the best way to deal with bugs and vulnerabilities.

    Forking

    One of the best features of OSS is (as long as you keep up with the policies) you can modify the source code and customize the software to fit your needs. Thus, you can take software intended for specific use-cases and modify it to provide a better solution for your situation. However, this process can be intensive and take effort and time. Thus, before you attempt forking, consult with other community and team members to ensure you have the right idea and execution.

    Use Automated Tools

    Cybersecurity is highly time-sensitive—the first moments of dealing with an attack are the most critical. Automated tools, such as Security, Orchestration, Automation and Response (SOAR), can help you identify threats quicker and make mitigation efforts much faster to minimize and dramatically minimize the attack’s potential damage to your system. You can also use automated tools to enforce policies and check your OS components for vulnerabilities, making the process much faster and more secure.

    Wrap Up

    OSS and OS components have many advantages for developers and facilitate agile development. However, the flip side of their ease of use is the danger they pose in terms of open-source vulnerabilities. To mitigate this risk, you should incorporate the right practices for managing OS components into a clear strategy. This will allow you to enjoy all the benefits of OS components without exposing your application, network and systems to unnecessary risks.

    — Ilai Bavati

  • Why COBOL Still Matters After 60 Years

    Why COBOL Still Matters After 60 Years

    In the world of DevOps, the Agile Manifesto and countless other application delivery frameworks, one has to wonder how an anachronism such as the programming language COBOL (common business-oriented language) has managed to remain relevant. After all, COBOL has its roots back in the 1940s, and was built upon Rear Admiral Grace Hopper’s work on the FLOW-MATIC programming language (a text-based programming language). Hopper, who worked as a technical consultant on the FLOW-MATIC project, is sometimes referred to as the grandmother of COBOL.

    Last century, COBOL quickly became one of the most used programming languages in the world and it is still quite popular, claiming more lines of code in active use today than any other programming language. It offered a huge advantage over the other programming languages of the era. One that can be summed up as portability. From the outset, COBOL was the first popular language designed to be operating system-agnostic. In other words, it could be run on numerous different systems with little change made to the source code. What’s more, as a text based language, COBOL used English words and phrases to make it easier to understand and also made it somewhat self-documenting. 

    While much of the programming world has moved onto JavaScript, Python, PHP and even C and C#, COBOL can not be ignored. It continues to evolve and remains relevant, even in the age of the cloud. Simply put, it still has some chops to it. Take for example:

    • COBOL is a Language of Business: It is still used by the majority of major banks and insurers. The language can be found in widespread use across numerous other sectors, such as retail, healthcare, government and automotive. For most of those businesses, no other programming language has displaced COBOL and it remains as the back-office business language of choice.
    • COBOL is Extremely Portable: From the outset, COBOL was designed to be operating system agnostic, meaning the same applications developed under it could run unchanged across many different platforms. That allows COBOL developers to focus on building applications, instead of worrying about the intricacies of operating systems. That fuels DevOps and agile by allowing developers to test and deploy across a wide range of supported platforms.
    • COBOL Continues to Evolve: It supports and integrates with numerous platforms and works with operating environments such as Linux, mainframes, Windows and UNIX. What’s more, COBOL also integrates with Java, C#, Docker containers, mobile, .NET and JVM as well as Azure and AWS cloud environments. It can be used today to build cloud, containerized and managed code apps. And continues to evolve to meet the challenges of the future.
    • COBOL is Easy to Learn and Understand: It was created with the objective of making it easier for programmers to work with computers and each other. That objective has translated into a language easy to learn and understand by using English phrases and commands. The readability aspects of COBOL make it self-documenting, along with an ease of use not found in other languages. What’s more, the language costs less to maintain and its readability means it is easier to maintain.
    • COBOL is Future-Proof: It has continually evolved and now works with industry-standard IDEs, meaning programs can code with COBOL using familiar tools that support teams. COBOL developers are able to use the language alongside contemporaries, without having to forgo the latest IT tooling and processes.

    While many in the DevOps world sell COBOL short, there are many reasons why the language remains popular today. It continues to meet the needs of business and embraces many of the changes the cloud and new techniques can throw at it. COBOL is even now embracing APIs, micro services, the cloud and distributed platforms, making it a worthwhile tool for business today.

    — Frank Ohlhorst

  • Cloud Security: Self-Healing Infrastructure With Fugue

    Cloud Security: Self-Healing Infrastructure With Fugue

    Misconfiguration is the No. 1 cause for data breach in the cloud. Fugue helps to eliminate data breaches, as it fixes anything that gets drifted by providing baseline drift detection and automated remediation. Fugue is all about cloud security. It identifies security risks and compliance violations and ensures they’re never repeated.

    During this interview, Josh Stella, co-founder and CTO of Fugue, explained how his company offers developer-friendly tools that tell you if you are “breaking a rule and how to fix it,” providing visibility and control for all DevSecOps teams. With the self-healing cloud infrastructure, Fugue corrects misconfiguration and works on a whole configuration without having to write code or automation scripts.

    Check out the interview below, which also includes a demo of how the technology works by Stella himself.

    — Alan Shimel

  • The Bedrock of Digital Transformation

    The Bedrock of Digital Transformation

    Trends come and go. I’m old enough to have been persuaded, twice, about the good sense of flared jeans. I had a space hopper. I had a Betamax video player. Fads and cool ideas are a constant white noise, vying for our attention. Some will stick, some won’t. And that’s true in technology. No-one admits to owning a C5, but plenty of us built PowerBuilder and Delphi apps. A few of us tinkered with Smalltalk.

    But genuinely smart, enduring ideas find a way of sticking, of evolving, and adapting. No-one these days thinks of a phone as a wired device stuck to a wall which you could only use if you asked someone to connect you. And no longer do you have to employ anyone to run in front of you with a flag as you drive your motor car. Like cars and phones, good ideas endure and in the world of computers and IT, the COBOL language is no exception.

    Very few organizations of any size are not being impacted by today’s digital era and the seismic changes being seen in the market place. Entire industries have been disrupted beyond all predictions in the last decade or so. Indeed 88% of the Fortune 500 have dropped off that list in the last 50 years, as those who fail to keep up with the pace of change are being replaced by those who can adapt.

    But adapting isn’t easy. Making fast progress toward new digital objectives is often hampered by practical barriers that are individually difficult to solve, particularly in the case of the existing technology landscape already in place. Things such as vendor lock in, complex business processes, interdependent apps and data, inefficient delivery processes, technical constraints and skills issues.

    Additionally, the technology landscape is forever changing, but only growing in complexity. Today, large global enterprise businesses typically run a hybrid environment, which is a consequence of adopting multiple waves of technological advancements over time. Each individual advancement offered opportunity and progress, but eventually results in a highly complex and inefficient technology environment. Most organizations are pursuing digital transformation programs with a view to reduce complexity, create better flexibility and reduce total cost of ownership.

    There has been tremendous value created in business applications and data and there is a lot to be gained in digital transformation by leveraging that value as much as possible, in transforming it with the minimum cost and risk. Much of this value is encapsulated in systems built some time back, and which have evolved with the business. Oftentimes, using COBOL technology. COBOL was invented way back when, but has adapted to support ongoing business technology demands and rigors.

    Having an appreciation for where your organization is today, along with an awareness of what needs to be achieved in regard to the required updates, the process used and the infrastructure considered, can assist with plotting the pragmatic steps to develop a plan that delivers transformational value. With some of the most critical systems being written in COBOL, these core systems continue to power a variety of fundamental processes in modern business.

    The risks of ripping and replacing systems are measured as an average IT project failure rate of 71%, which amounts to a $1.7tr cost in a single year. These are dreadful numbers and no organization wants to waste that sort of time and money and reputation. There is now a growing realization, from many years of experience in those more radical transformation projects, that their success rate is relatively low, planned costs and timescales typically grow exponentially, and the risks to successfully continue business during such a transformation are unacceptably high.

    Enter stage left a more pragmatic approach: modernization. This concept prioritizes on protecting the value of what companies already have, exploiting advancements in technology to deploy and integrate applications in new ways and to surface and integrate data in new ways, with the minimum change to what already exists. Creating portability and operational independence for applications and data is becoming recognized as the fastest and most future proofed path to building a platform for digital transformation. Market influencers such as Gartner and IDC have commented on it, and most of the global systems integrators are industrializing their modernization practices to support companies successfully transform. So what is the underlying technology acting as the catalyst for this modernization and transformation?

    COBOL enters its seventh decade of helping run great chunks of the world’s largest IT systems entirely thanks to its staying power; its adaptability.

    COBOL’s endurance owes a lot to its original design that make it readable and portable, meaning it can be picked up by new staff quickly, and deploy wherever it is needed. What kept it in such demand owes more to the ongoing evolution of the language itself, and the tooling surrounding it. COBOL works alongside, integrates with and supports containers, cloud, JVM, .NET, JSON/REST, micro services, modern IDEs and other technology typically associated with more contemporary names. COBOL technology plugs directly into a DevOps toolchain that is adopting continuous delivery practices for some of the economies’ most important IT systems.

    COBOL is, in fact, a 2019 technology; just with a 60-year heritage.

    Next time you pick up your mobile to make a call, or sit in your car to drive to work, ask yourself how you feel about how old that idea is.

    — Derek Britton

  • The Barriers to Robotic Process Automation, Part 2

    The Barriers to Robotic Process Automation, Part 2

    Enterprises are rushing to embrace robotic process automation (RPA) but, as we covered in The Barriers to Robotic Process Automation, Part 1, they’re also hitting considerable challenges along the way. While we cited prominent barriers to the effective implementation of RPA in part one, here, we will provide practical ways organizations can overcome these barriers.

    One of the biggest causes of RPA failure is the lack of executive-level sponsorship for such initiatives. When it comes to RPA success, getting management’s attention–and keeping it–is essential. One way to do this is to focus on RPA quick wins and efforts that have a measurable return on investment.

    Next, IT leaders need to understand how significant digital transformation is to AI technology.

    “An efficient digital transformation strategy will eliminate internal IT silos through integration and automation, leading to opportunities for AI projects,” said Marcel Shaw, AI expert and Federal Systems engineer at Ivanti. Shaw pointed to how organizations that interface with customers will see opportunities for chatbots that leverage natural language processing technology, thereby reducing demand on operators while enhancing the user experience.

    “IT leaders will be able to take advantage of machine learning technology that will support and enhance complex automated processes that will fulfill customer requests from beginning to end, without intervention from a human analyst,” said Shaw.

    Such benefits could make RPA an easy sale to executive leadership. Other benefits that can be used to sell RPA to leadership include cost savings, more efficient processes, improved process quality and free staff to focus on higher-value tasks.

    “The only thing holding AI back at this time is a lack of experience with the technology. AI will take many years to develop. Some estimates point to a 75-year maturity lifecycle for AI. AI technology is designed to learn. As it learns, it will correct mistakes and mature, much like a human has to learn,” said Shaw.

    Next up: The high level of investment needed for effective, automated and intelligent robotic automation.

    “There’s a great deal of catching-up to be done before most companies can even consider building out their own advanced developments with AI,” said Salvatore Stolfo founder and CTO at Allure Security. “It may be more cost-effective and faster to partner with a software vendor that has already put the time into dedicated research and development to build an AI-powered solution, and then customize that solution for your particular use case.”

    Simha Sadasiva, CEO at Ushur might agree. “Many intelligent automation solutions on the market today require a heavy IT investment. This can include hiring an outside third-party just to manage the platform they’ve invested in,” said Sadasiva.

    “Those attempting to manage the technology in-house are finding that it requires personnel with sophisticated developer-level knowledge, which is in short supply today. These issues are causing some companies who were initially interested in exploring intelligent automation solutions to abandon their projects altogether,” added Sadasiva.

    Sadasiva said one possible way to solve this is to evaluate potential zero code platforms. “These are SaaS-based intelligent automation solutions born in and live in the cloud,” said Sadasiva.

    Finally, and while perhaps self-evident is often overlooked: know what needs to be achieved with robotic process automation.

    “An organization needs to identify what the problem is that they are trying to solve with AI. This needs to be the most important pain point to them at that point in time. Inherently AI requires data to be able to train the machine to learn,” said Ben Mercer, co-founder at personalized web experience provider Personify XP.

    “The challenge for some organizations, when it comes to accessing data, is that valuable data is stored on legacy systems and difficult to extract. In cases like this, it may make more sense to train the AI on new data, than fighting with legacy systems that may delay or reduce the ROI of the deployment,” said Mercer.

    Effectively managing data, AI and the resulting automation requires staff with strong skills and experience in these areas.

    “Regarding skillset development, most of the skills needed can be learned, but you need to find different types of team members who have a great comprehension of data and database management and secondly a development background,” said Jesse Tutt, CEO at mattress company Gotta Sleep.

    Finally, chances are if an enterprise is planning to do something there’s already someone who has tried and has lessons to share. “Every automation project has likely been done by someone else already. Tips and tricks are freely available from others who have more experience so networking on Linkedin, meetups, and conferences are key,” adviced Tutt.

    — George V. Hulme

  • When Shift Right is Shift Left

    When Shift Right is Shift Left

    In some areas of IT, we utilize the idea of shift right quite a bit. Testing and security are two areas where it comes up often. The idea is that tracking late in the process–usually while the application is in production–leads us to find more insidious issues that might have remained hidden, and gives us a kind of quick reaction force to mitigate issues.

    All true, and it works relatively well. Some shops use this concept to push off necessary testing, but for the most part it allows us to keep moving while assuring the system is essentially correct. But the idea of catching issues late in this iteration might be misguided. It does do just that, but in the world of cyclical iterations, shift right is also shift left for a future iteration. The idea being that we’re catching something today that will be resolved for the next iteration. And presumably for every iteration after that.

    We often put a temporary fix to resolve the in-production issue, then list it as something to be more permanently resolved in a future iteration. Adding to a future iteration’s to-do list is the ultimate shift left. And that matters. We’re catching it late for this iteration, but early for the next one. And as a result, overall quality increases.

    Standardizing this process would be the next step. If it’s an easy fix–a minor change to code or config–it’s shifted right. If it’s a larger issue that will need to be considered in a future iteration, it is considered shifted left for that iteration. It’s an input, not just a fix. And an input that deserves a high priority if it reflects a negative impact upon the running system.

    In the daily grind, this is pretty much irrelevant. We have a problem and say, “there, that’ll keep it going. Put in an issue to make a better fix and move on.” But in the next iteration’s prioritization, it is far more relevant, “we have this massive pile of things to do, what makes this iteration?” Should include a heavy weighting for things that are limping along on a manual fix. Often, the existence of a workaround reduces the weighting, and these things don’t make the cut. But permanent fixes mean you don’t have to hustle around fixing the same old problem each time the app is deployed, and that’s greatly what DevOps is all about.

    There is an interesting dichotomy between shops/groups with a high iteration rate and those with a much slower cadence. The cost of not permanently fixing these issues is too high when you’re deploying frequently, so they tend to be handled this way already. But slower iteration shops/groups often will let these fixes slide in favor of new functionality. The burden this creates is not worth a shiny new bit on the UI. Start slating permanent fixes for production issues as highest priority, don’t add steps to deployment because you failed to resolve the more fundamental issue.

    DevOps requires speedy resolution, but long-term, any type of development effort requires adequate resolution. With iteration after iteration, you are paying a price for putting off fixes, so don’t. Shift them left in the next iteration, and spend that time of each release on something more productive.

    — Don Macvittie

  • Agile and DevOps Success Needs Continuous Testing

    Agile and DevOps Success Needs Continuous Testing

    One of the central ways to progress an agile transformation is to introduce continuous testing. With all such changes, it is cultural as much as technical. The expected outcomes are less broken builds and reduced time from break to fix, as well as a yardstick for measuring the team’s ambition for improvement.

    The Testing Bit and the Continuous Bit: Testing should be part of any agile developer’s skillset; the shift is to apply it to the whole build pipeline. Continuous implies automation, with the build featuring as a living thing whose state is always changing. The infrastructure options have expanded with the advent of cloud computing and virtual containers.

    Let’s say your teams’ developers work on components, independently buildable and testable chunks of the whole app or service. They receive stories and tasks to improve a customer journey and then add functionality to implement these. And when done, they check their work into your team’s code repository.

    Starting with One Developer, One Laptop and One Test

    The initial testing focus for a single developer (or pair) working on code will be the unit test and the coverage test. Whether writing the test first or code first, the unit test pokes the bounds of the new code and the coverage test checks that all code paths are exercised.

    As an example, maybe the developer has implemented the “customer wants to deposit money” story on an accounts component of a banking system. The tests should check what happens if a user deposits zero, negative or fractional money and the coverage confirms the tests have wandered down all possible the code paths of the new feature. In this way, coverage tests also prevent a developer from adding code without introducing the corresponding tests.

    At this point, it is worthwhile making sure tests are run as part of the build so every time the developer builds the code all the unit tests are run. This is a fairly standard thing for most build systems to do. The bottom line is the chance of checking-in breaking code is greatly reduced.

    Now the Bigger Picture

    The new code worked on your developer’s laptop, the tests passed and they checked-in the code. But in a busy enterprise team, other developers are also working on the same component, maybe even on the same file. This is no different from working on a book chapter with several authors. Your developer worked with a stable snapshot of the code. It is quite possible changes were made by others that conflict with the changes your developer just made. So how do we proceed?

    This is the purpose of the central build. The job of the central build is to check out all code for that component, build it and run all associated tests whenever new code is checked in. In today’s world, the likely path would be to create a container on a cloud server, where the component can be built from scratch, and then all the tests run against it. Traditionally, a tool such as Jenkins would watch for git check-ins and kick off a build. The result is the team always knows if the overall build is green (working) or red (broken). A developer must understand the central build is the true source of truth, not what is happening on their laptop. So, the latest check-in leads to a correctly built component. What next?

    There are three general types of testing needed to develop a software system. Tests that make sure new features work correctly and don’t break the component, tests that make sure the components still work with each other and those confirming the customer journeys your component participates in all work as expected. There would be no point in making a deposit function without a customer journey test that proves a customer can deposit cash with the system.

    The second type of testing, usually referred to as integration testing, needs an environment where all current successfully built components can communicate with each other. Today, that may mean some form of container orchestration with the system’s components placed and managed in their containers. Tests of a customer journey may involve emulating the customer’s manipulation of a website or mobile app, depending on how the application or service is accessed. These are the tests stakeholders are most interested in as they confirm what has been requested in the stories actually work. As with integration testing, these tests may be run less often.

    I Mock Thee

    Mock (or fake) services and data are a necessary part of the testing process, and usually, must be maintained and extended as part of the testing. A feature story may require a developer to spend more time developing or extending a mock service than writing the code it helps test. For example, a mock account with some transaction history is needed to test a new deposit function.

    Live environments can never be entirely simulated in test environments. Sometimes a live service uses a paid for third-party service, and it would be too expensive to use in testing. It is usually the case that the development staff are not permitted to work with live customer data.

    The Build Is Alive

    All the structures mentioned are just products of one simple attitude: “How do I know if this works?” For continuous testing to thrive, engineers should always question whether every part of the system is being tested and whether the tests are being evolved at the same pace as feature development. When tests or mocks are missing, an agile team refers to technical debt. This is as it implies, something that must be paid for later.

    When team members realize they cannot just check-in code and go to lunch without checking the central build, that is a good sign. Information radiators such as large screen displays of the build state should be used to get the extended team to understand the state of the build is everyone’s concern, throughout the lifecycle.

    — David Eastman

  • Why ITOps Is Moving to the Cloud

    Why ITOps Is Moving to the Cloud

    There’s been a lot of movement in the IT operations management (ITOM) business lately, from the acquisition of SignalFx by Splunk to the PagerDuty IPO, and all signs point to a Datadog IPO in the future. What’s with all this consolidation? I believe we’re seeing the rise of a future-state of ITOM; that is to say, it’s the rise of SaaS-based ITOM. And it’s easy to see why.

    In my previous consulting career as lead enterprise systems architect, our team had an impeccable record in designing and implementing well-architected hybrid infrastructure solutions. We maintained an immaculate record and near-flawless customer satisfaction record. By project sign-off, our job was always done. And yet, returning to the same solutions six months later told a different story entirely.

    Well-architected solutions are similar to human bodies: They are perfect when they’re born but need constant care and feeding. These same solutions that satisfied SLAs, exceeded expectations and transformed organizational efficiency can easily degenerate, and just like our bodies have their nervous systems to monitor, brains to send alerts and tissue to self-heal, well-architected systems need operational maintenance to keep them humming.

    The traditional approach for solving this eternal need was and still is to design and implement well-architected IT operation management (ITOM) solutions, around a well-architected infrastructure. And yet, this is a self-fulfilling paradox because the ITOM solution itself needs the same care and feeding.

    There’s a Problem On-Premise

    ITOM is a broad term encompassing application and operating system (OS) performance, alerts, log management, notification, asset configuration, incident management and more. It typically involves the purchase of a suite of on-premise point tools addressing each need, and then to develop an internal framework to help those tools interoperate in a meaningful way. While that is possible conceptually, the facts on the ground reflect a very different reality:

    • Multi-vendor tools are often not designed to work together.
    • Creating an internal logical framework that orchestrates various teams and technologies can be very complex in large enterprises.
    • It’s near-impossible to create technical integrations flexible enough to adhere to inevitable organizational and technological changes that will affect this logical framework.
    • Predicting cost-of-ownership is nearly impossible since each tool is controlled by a different vendor, and the internal integration effort is often unknown.
    • Predicting the cost of the manpower required is also very difficult, as each tool requires its own set of specialists, in addition to integration specialists to make it all work together.
    • Upkeep is often overwhelming, as vendors offload software patches, upgrades and on-premises hardware costs to the customer.

    In the face of all these challenges, the end result is often unrealized value, overwhelmed operational teams, loss of service and inability to accommodate new technologies resulting business service disruptions.

    Why Cloud? Why Now?

    It’s historically been near-impossible to build a traditional ITOM platform on-premise. Vendors typically sell a collection of white-labeled tools cobbled together by acquisitions, and this is far from a platform. The complexity and the rate of technological change make it difficult to provide consistent quality and value across the various product lines. This puts the ITOps team squarely in the middle of a pickle: How can they ride the wave of a changing environment without relying on static tool suites?

    The Future is Flexible

    Enterprise ITOps has been stretched now more than ever. There is a serious skills gap, shortage in IT workforce and ever-increasing technical complexity. Time and resources are precious, and Enterprise ITOps need simplicity and predictability along with flexibility and control. Enter SaaS ITOM. By moving the ITOM function to a SaaS orientation, the responsibilities, workloads and daily tasks can transform according to the needs of the organization:

    Keeping Up with the Business

    SaaS ITOM can keep up with technological change, and keep pace with cloud, DevOps, artificial intelligence and more. In the world of SaaS, change is an accepted constant and not an inconvenience. What’s more, SaaS ITOM is infinitely more consumable than the tool suites of legacy past, and that reduces the learning curves associated with running ITOps.

    Keeping Up with Industry Needs

    A SaaS ITOM platform will be able to deliver a framework that’s both flexible and governed and can accommodate technical and organizational complexities. This agility is a feature of modern SaaS. SaaS ITOM can also integrate features running on a single code base supported completely by the SaaS vendor, who will absorb maintenance and upgrade cycles, freeing considerable and valuable time back to the operator. All of this results in a more predictable total cost of ownership, improved service quality and more value to the business user.

    It’s not news the world of IT is moving to the cloud. It is news, however, that cloud can offer such transformational benefits in ways we’ve never seen before.

    — Wael Altaqi