devopsy.kinsta.cloud

Category: DevOps Practice

  • The Role of DevOps in the Financial Services Industry

    The Role of DevOps in the Financial Services Industry

    We are living in a fast-changing environment where unexpected technological, economic and political changes can be expected and where it is necessary to adapt quickly to these changes. This statement is especially true in the financial services sector, where there is a greater need to exceed the expectations of customers and provide uninterrupted service.

    From small trading firms to the financial industry players—big banks and exchanges—all are racing to deliver features and content to customers more quickly, and opting for DevOps tools, practices and ideas to solve the problems arising in financial systems.

    In DevOps methodology, the development and operations engineers work together in the entire service life cycle, starting from design via the process of development to production operations, maintenance and support. A DevOps approach is helpful for the adoption of various technologies such as blockchain, which is the technology behind cryptocurrencies such as bitcoin wallet. In this article, I will discuss the applications of DevOps methodology in the finance industry.

    How DevOps is Transforming Financial Services

    As said earlier, financial services companies face new challenges in their day-to-day operations in meeting the standards laid down by the unicorns. To support today’s digital age, many financial services companies are modernizing their back-end systems and processes.

    In addition to using legacy systems and code, they can face difficulties in meeting governance requirements, complying with regulations and alleviating risk for better security. These concerns make the software innovation a slow and complex process.

    But, over the past few years, the financial services sector is leading the transformation when it comes to adopting latest technologies and modern software delivery practices including DevOps, continuous delivery (CD) and Agile.

    Finance executives view DevOps as a way to bring value to the market in a more safer and efficient way. DevOps allows financial services to enhance the quality and cadence of application releases while addressing compliance, security, risk and governance strategies. Now, let’s dig deeper into the ways in which is transforming the finance landscape.

    Mitigating Potential Security Threats

    In the financial services industry, compliance and security are major concerns and have been the major setbacks in replacing legacy practices of software development and delivery. In the initial days, DevOps practices were seen as major risk to security and the enhanced velocity of software releases it allows was viewed as a threat to security, governance and regulatory controls.

    In spite of the initial setback, some enterprises that adopted DevOps have shown consistently that these practices actually reduce potential security problems and help companies locate issues and tackle threats more quickly.

    Due to this many companies have embraced the automation practices that enable and enforce compliance, security and audit requirements. Today, financial services companies view DevOps as a resource for security.

    Automating all the Processes

    Automating the processes all through the software delivery pipeline is another essential to enhancing resource utilization, product quality and developer productivity, and allow cross-project and enterprise-scale management and visibility in financial services. For any continuous delivery or DevOps initiative, automating all the processes is becoming a major requirement for enabling scalability and speed, ensuring compliance and achieving higher product quality.

    DevOps offers enterprises with an efficient approach to alleviate risk from software releases and aid the ever-evolving demand for frequent application updates. By managing all the release processes in a single centralized platform and automating the complete end-to-end pipeline, these companies gain visibility into the progress of all DevOps processes and releases and make sure they can optimize, govern and control their delivery pipelines even as they evolve.

    Driving Cultural Transformation

    DevOps focuses on creating an environment and a culture where the business, operations, quality assurance, development and other stakeholders work collaboratively toward a shared goal. While optimizing the technology and tools to scale DevOps adoption, embedding security practices and automating the life cycle are the major steps. DevOps is a mindset and any change in culture takes time.

    This change takes time and even the financial services industry that is regulated heavily is implementing DevOps and leading the change from both the cultural and technology standpoint. Thus, the financial services companies have what they need to keep pace with the unicorns, modernize their benefits and completely harness the benefits of DevOps.

    Orchestration

    For many financial services companies, the four basic software delivery pipeline stages (release, deploy, test and build) consist of numerous processes that hold within them millions of jobs and thousands of tasks executed by a large number of people. These toolsets, processes and the people are often present in silos, making things even more challenging.

    While existing in silos, these people may become skillful at understanding and enhancing their part of finding the solution, but from an organizational perspective, there is no end-to-end visibility and orchestrating the complete pathway is impossible. This leads to a lack of governance, errors, delays, manual handoffs and fragmented processes.

    As the IT operations and development teams are often geographically distributed in these large organizations, it is important to enable shared visibility and control into the different processes, tool chains and infrastructures. Optimizing the process of software delivery shouldn’t be attacked as a separate set of problems but viewed as utilizing a system-level approach, much as the manufacturers of cars apply lean manufacturing principles all through their production lines.

    The implementation of DevOps is spreading rapidly in financial services sector due to the value that this global, systems-level approach enables. DevOps accomplishes this by scaling the latest IT process across the entire organization in order to enable an end-to-end, robust pathway to production.

    Eliminating friction between teams and breaking down functional silos by orchestrating every aspect of the software delivery pipeline to create repeatable, predictable processes that can operate frequently with minimal human intervention prevents many of the challenges distributed companies face.

    Conclusion

    Even facing strong opposition in the beginning, many organizations took a risk implementing DevOps and have benefited substantially. Today, many technologies such as big data analytics are evolving rapidly and are implemented by many organizations around the world. According to a Peoplehr post, data analytics is used to enhance various organizational activities. Those companies that embraced analytics as part of their activities have gained huge benefits from it. In the same way, any new technology such as DevOps has to be embraced by organizations in an effective manner to enhance their overall productivity.

    DevOps implementation eliminates the security problems before they become a major threat and scales the IT processes across the whole enterprise, thus allowing an end-to-end and vigorous pathway to production. Every day, more organizations are switching to DevOps and not looking back. The change to DevOps culture may take time, but it will revolutionize the activities of an organization, including financial services.

    — Sujata Rani

  • As Business Speeds Up, Testers Must Adapt

    As Business Speeds Up, Testers Must Adapt

    It should be no surprise to anyone working in software today that delivery speed is the name of the game. Irrespective of business sectors or B2B/B2C focus, global competition in application development and delivery has flooded the market. Due to the increased number of options in the marketplace, users expect an excellent experience from your application or site—and if you cannot meet expectations, they’ll find it elsewhere.

    A company’s ability to be highly responsive to (if not proactive about) customer’s needs is also valuable for adoption and branding. Established companies understand that delivering high quality in their products and services is core to brand value.

    This shift isn’t always easy for testers. Moving to a faster pace of work can be unsettling when your job is all about assessing the perceived quality of the software. Testers like to be cautious and check everything twice. The current work environment has made testers more germane than ever in software and IT organizations.

    Testers help build confidence in the quality of an application, and this perception of quality is what typically drives user adoption. If two applications have more or less the same functionality but one is faster, more reliable and has a better UI and design, you can guess which one will be more popular.

    Setting the Foundation

    To thrive in this new world, testers must learn to prioritize and focus on the most important needs of the team’s delivery goals versus testing every last mouse click. Testers will need to quickly incorporate feedback from multiple places, and work well with stakeholders across the development organization and the business. Efficiency is imperative, and that means increasing the use of tools whenever possible.  The following framework to develop a world-class testing organization when your company is moving faster all the time can make the transition a little easier.

    1. Understand business goals. While this may sound obvious, testers haven’t always had to think about the bigger picture. With more competition from everywhere, everyone on the development team should have a solid understanding of customer needs, business goals such as to drive new revenue or grow customer loyalty, and how the competition stacks up for the solution in development.
    2. Define a testing strategy. This exercise begins with understanding the business goals as they relate to the product or project and, at a more granular level, the goals for each sprint. The testing strategy should include a description of the overall project, priorities for testing mapped out into phases, timelines, testing approach, tools and methods to use and how progress will be measured.
    3. Develop a risk management plan. What will happen if those business goals are not achieved? For instance, the business stakeholder might say the project must go live in four months with new functionality. But is this an arbitrary goal or one that is embedded in an actual customer commitment? What will be the financial impact if the deadline slips? Which enhancements are critical and which are nice to have from the user perspective? Once those particulars are documented (or discussed), the testing team can create the appropriate plan, prioritize and determine what resources are needed to execute it.
    4. Resources. Testing teams with mandates to move faster will need support in the form of new tools, training and even new hires. Software that testers (as well as others) use can include DevOps and CI/CD tools, ALM tools, reporting and analytics, UI and API testing tools, systems for recording and logging manual testing, tools for usability feedback and more. This can get overwhelming for startups and other companies that don’t have the budget to purchase new software. Fortunately, there are many credible open source tools that can fit the bill. This reality is also why development teams must understand well enough what customers really want so they can prioritize on which features must be perfect and which areas of the application could meet a lower bar of quality.

    A Word of Caution

    Testers are well-equipped to make all the changes required for the faster pace of modern software development. This shift has been underway for a few years now, and many testers are embracing the change and the elevation of their role on engineering and IT teams. However, as testing relies on more automated tools, there are a few things to consider:

    1. Avoid tools mania. Testers, like many technical people, love to try out the newest toys, but it’s important to justify a tool’s adoption before acquiring it. With too many tools, there’s just too much noise and complexity. That will counteract any speed and efficiency gains that automation supplies.
    2. Automation tools do not replace testing. It’s not necessary nor helpful to automate every testing task. Testers should view automation tools as enablers, not a replacement for the function. Automation is wonderful for reducing effort and giving testers time to work on value-added activities that can boost the overall user experience and usefulness of an application. Be sure to baseline test objectives before recommending an automation solution.
    3. Institute process around automation. Ensure you have defined a proper process in sharing artifacts that are produced by your automation tools. Artifacts such as scripts must be versioned with proper comments and stored in a central repository instead of on individuals’ machines. Defining and enforcing good development practices such as check-in/check-out process, scripting style guides and object organization with automation will ensure a higher return on effort in the long run.

    In a few years, this point may be moot. But there are still many testers coming into new roles from the waterfall mindset who may struggle in the transition.

    Testers once lived simply in the silo of sluggish procedures. They knew exactly when they would be called upon for action and how they would complete a task. Now, though, testers must put on their business assurance hats and work side by side with developers, operations managers and others early in the process. They will have to learn new ways to test, such as with behavior-driven development and test-driven development. They need to get more technical, given the push toward test automation.

    What hasn’t changed though is the tester’s quest for quality in software. Great testers will add the most value when they can be proactive about alerting others to brewing problems before they present themselves to the user.

    — Mush Honda

  • Reducing Risk in Applications Using Docker Containers

    Reducing Risk in Applications Using Docker Containers

    Docker. It seems like in this day and age you are either using Docker containers or you are going to use Docker containers. If you haven’t jumped on the bandwagon yet, check out a previous article, “Docker: The New Ordinary.” If you are on the wagon or are thinking about it but have concerns about their security, it’s time to read on.

    José Manuel Ortega (jmortega.github.io ) is a software engineer and security researcher in Spain. I recently watched his presentation online titled, “Testing Docker Images Security.” He gave an overview of typical Docker deployments, explained the attack surface and threats, presented how to detect vulnerabilities and outlined a couple of best practices. In short, his advice will help you learn how to better secure your Docker containers.

    (New to Docker? Read this paragraph; all others skip ahead.)  If you aren’t sure what Docker is, José offered this explanation: “Docker containers wrap a piece of software in a complete file system that contains everything it needs to run: code, runtime, system tools, system libraries—anything you can install on a server, regardless of the environment it is running in.”

    That is, containers are isolated but share an operating system and, where appropriate, binaries and libraries. Docker provides an additional layer of isolation, making your infrastructure safer by default. This makes the application life cycle faster and easier to configure, reducing risks in your applications.

    For starters, José laid out Docker’s default mechanisms for security:

    • Linux kernel namespaces
    • Linux Control Groups (cgroups)
    • The Docker daemon
    • Linux capabilities (libcap)
    • Linux security mechanisms like AppArmor or SELinux

     

    Screen Shot 2018-04-08 at 7.17.58 AM

    José walked through others tools, add-ons, best practices, etc., to increase Docker container security. I will cover most of them here.

    Docker Inspect Tool. The Docker Inspect Tool is built into Docker. It provides information about the host name, the ID of the image, etc. and it comes up when you start Docker.

    Docker Content Trust. It protects against untrusted images. It can enable signing checks on every managed host, guarantee integrity of your images when pulled and provide trust from publisher to consumer.

    Screen Shot 2018-04-08 at 7.19.01 AM

    Docker File Security. Docker files build Docker containers. They should not write secrets, such as users and passwords. You should remove unnecessary setuid and setgid permissions, download packages securely using GPG and certificates, and try to restrict an image or container to one service.

    Container Security. Docker security is about limiting and controlling the attack surface on the kernel. Don’t run your applications as root in containers, and create specific users for testing and policing the Docker image. Run filesystems as read-only so attackers can not overwrite data or save malicious scripts to the image.

    José provided a useful checklist to check the security of a Docker container, but it’s not a short one. Remember, if you are going to deploy hundreds or thousands of these containers, you’ll want to ensure consistent handling of security concerns to keep the hackers at bay:

      • Do not write secrets to Docker files
      • Create a user
      • Follow version pinning for base images, packages, etc.
      • Remove unnecessary setuid, setgid permissions
      • Do not write any kind of update instructions alone in a Docker file
      • Download packages securely
      • Do not download unnecessary packages
      • Use COPY instead of ADD
      • Use the HEALTHCHECK command
      • Use gosu instead of sudo whenever possible
      • Use no-cache (if applicable) when building
      • Enable Docker Content Trust
      • Ensure images are free from known vulnerabilities
      • Ensure images are scanned frequently throughout your DevOps pipeline
      • Ensure your images, packages are up to date
      • Use file monitoring solutions for image layers (if required)

    Auditing Docker Images. You can scan your images for known vulnerabilities with a wide variety of commercial and open source tools such as:

     

    All of these solutions can be integrated in one element of your CI/CD pipelines — some can be integrated in multiple places.

    José explored these solutions and best practices in more detail and offered up technical implementation tips in his full talk, available for free here. If you are working on or interested in Docker security, his talk is worth your time.

    Thirsty for more DevOps discussions led by practitioners (not vendors)?  You too can binge watch 99 other talks from the All Day DevOps conference here.

    — Derek E. Weeks

  • Code Delivery: Pipeline Automation for the Speed to Innovate

    Code Delivery: Pipeline Automation for the Speed to Innovate

    Many developers steeped in the world of agile startups view continuous delivery pipelines as an accepted standard requirement for software development. Yet many companies, particularly large enterprises with traditional infrastructure, still struggle to make this approach a standard part of their development process.

    Whether your company is an enterprise looking to make continuous delivery pipelines a standard project element to increase agility and speed time to market, or your firm is looking to simply implement code delivery pipeline best practices, optimizing code delivery pipelines through DevOps-based automation will increase developer output and strategic contributions and deliver substantial business value.

    Efficient code delivery pipelines that balance security and agility can be achieved by using automation coupled with cloud infrastructure. Together, these technologies and processes can deliver big dividends in the form of lower development costs, higher-quality results and faster times to launch. By focusing on optimizing a delivery mechanism for value versus just delivering more code, code delivery pipelines can meaningfully and measurably impact business objectives.

    Continuous Delivery in Modern Development Environments

    Continuous delivery is a design practice wherein developers use automation to accelerate the software delivery process. It is a high-value process because it enables programmers to cut the overall cost of software development. Faster delivery of high-quality code makes it easier for companies to improve customer satisfaction, capture new markets and gain a competitive advantage.

    To optimize this development model, companies need to automate their build, integration and testing processes as well as provisioning and retiring infrastructure components.

    Let’s look at a typical scenario. Code delivery starts from a feature or bug request filed with a developer. The developer takes existing code, modifies it, does local testing and commits it to the code repository. The code in the repository is then tested by QA and tagged for production, where IT or operations professionals deploy the code to production servers.

    This creates loops at the local testing, QA, and production stages of the process. Each of these loops has a cost, which is directly affected by developer productivity and how long it takes to execute each loop.

    The local testing loop executes most frequently, so it’s critical to make this part of the testing process fast. Production bugs are the least frequent but most expensive for businesses when they exist. Consider the cost of a significant disruption to the workflow, or the negative customer experience, or the inability to deliver a service, or security vulnerabilities. Imagine if you could keep delivery time and costs under control while minimizing total effort at every stage.

    Ideally, code should be pushed in smaller batches. The downside to this approach is it creates overhead and room for error as more and more things move through the process. To reduce the overhead of too many pushes, automation can play an important and meaningful role. Indeed, automation makes it cheaper, faster and more reliable to do things in smaller batches and helps streamline delivery.

    Stories from the Front Line: Automated Code Delivery Pipelines

    DevOps automation such as automated code delivery pipelines can be more easily achieved when coupled with cloud technology. For example, Flux7 recently worked with a Fortune 500 enterprise to set up an automated code delivery pipeline using AWS CloudFormation, AWS CodeCommit, AWS CodePipeline and Jenkins. Developers would use a Git flow methodology, and once they merged code to the master branch in CodeCommit, polling would alert to a new commit. CodePipeline would create a CloudFormation change set and queue it up for review by the Ops team. Once the Ops team approved the change set by manually pressing a button, CodePipeline would trigger a CloudFormation stack update. This allowed for very high full-stack agility while not losing central governance.

    Many leaders in technology organizations are moving to Agile and DevOps as methods of improving the speed of delivery, of which an automated code delivery pipeline is foundational. Optimized development pipelines deliver significant value by minimizing the time and costs of local test, QA and production processes and by promoting code from one development stage to the next.

    When you engage in this process, you reduce the time it takes to identify problems and their causes. And when you automate development steps, you dramatically reduce the cumbersome and time-consuming tasks that slow down the release of software code.

    Impact to Your Business

    Efficient developer workflows in a cloud-based environment are the foundation of pipeline optimization. Their importance relates to the need to:

    • Control ever-increasing developer labor costs. Your monthly developer labor cost is almost guaranteed to exceed the monthly cost of your cloud server setup. Keeping your developers working efficiently means you get the most out of that labor and can focus that on what projects are bringing the most value to the business.
    • Get a clear understanding of developer activities. When we interview developers and IT team members, what emerges from these discussions is astonishing. Companies learn that they really don’t know as much as they thought they did about their employees’ day-to-day activities and workflow.
    • Find developers with the required skill sets. Hiring the right talent is costly and time-consuming. If you can help your developers work more efficiently, you can avoid the search costs. And, building a reputation as a company that supports developers helps attract new talent.
    • Accelerate time to market. It’s possible to improve the performance of less-productive developers by accelerating their work process with optimized code delivery pipelines and by building best practices in through automation.
    • Avoid rework-related slowdowns. Rework comes at a cost of time and effort. Automating development processes can avoid many of the human errors that cause rework.

    An effective code delivery process is highly valuable as it reduces risk from each release, ensures that rogue code is not introduced and helps organizations effectively manage global deployments. Most importantly, optimized code delivery pipelines can significantly impact business objectives and business outcomes, making DevOps a strategic contributor to achieving business imperatives.

    — Aater Suleman

  • 200 Billion Downloads Can’t Be Wrong

    200 Billion Downloads Can’t Be Wrong

    Laurie Voss, COO and co-founder at npm (@seldo), tweeted recently that JavaScript packages downloaded from their repository has topped 4 billion. On an annual basis, that would be more than 200 billion downloads. Wow.

    Just as open source software turns 20 years old this year, these numbers are a testament to the incredible magic that happens when communities of developers openly share innovations.

    Laurie estimated that 12 million developers were responsible for the downloads. That means, on an annual basis, each JavaScript developer will download nearly 17,000 components. Imagine the time saved by not needing to code each of those functions from scratch simply by downloading a package in a millisecond from the internet.

    Software Supply Chains at Work

    What we’re witnessing here is a software supply chain in full effect. Open source projects contribute packaged code to the community, place it in public warehouses, and it’s then consumed by development teams around the globe to create new front-end, back-end or mobile applications for all of us. Every development team on the planet now utilizes a software supply chain that operates at insane speeds with massive throughput.

    In a second tweet, Laurie wrote, “Since we have ~12 million users, that means the average user installed the neatly round number of 1000 packages in 7 days. In reality, most individuals installed very few packages, and a bunch of CI build boxes installed 10s or 100s of thousands of packages each.”

    Essentially, there is an army of robots (automated CI tools) downloading billions of components from the web. While this type of automation is efficient, by itself it does not serve teams well.

    Repositories to the Rescue

    At Sonatype, we saw similar behavior by developers in the early days of Maven. Individual developers would download Java components directly from Maven Central. Maven users sitting right next to one another in the same room and on the same team would download the same versions of the same components needed for their builds, repeatedly. It was for this reason, among others, that our founders invented the Nexus repository manager.

    While repository managers have been used for quite some time in the Java community, Laurie’s post tells us that adoption of the technology still lags in younger, less mature, packaged code ecosystems.

    Companies looking for repository managers can look toward solutions like e-npm, an on-premises technology exclusively for JavaScript packages. For companies with polyglot development teams, universal repository managers provide full-function, (sometimes) free support for npm and 12 other component types, including Docker containers.

    Not All Components Are Created Equal

    I have often said that if you have 100 developers, you have 100 front doors open into your organization. If you have 1,000 developers? Then there are 1,000 front doors.

    If you dissect Laurie’s comment further, you will also realize that any developer can bring any package into your organization at any time. While this does make them more efficient, it also points to security and governance concerns. Every component was developed by someone else, donated out to the internet and consumed in the millions. But you have no true idea of the source or origin of many of the components.

    Research I have done in previous years at Sonatype demonstrated that 5.5 percent (1 in 18) of components downloaded by repository managers had known security vulnerabilities. While the percentage is not very large, keep in mind that a repository manager will only download a component once. Once cached, future downloads of that component are unnecessary and the components can be reused infinitely by that team.

    Sonatype analysis of more than 40,000 Nexus repositories reveals that the average repository holds more than 1,600 components. Deeper analysis of the 1,600 components housed in the average repository manager found 192 security vulnerabilities were present among the components (some components having more than one security vulnerability).

    Within software supply chains, repository managers and private container registries represent procurement gates into the development organization. The gates can be left wide open where component flows are not governed or they can represent opportunities for quality and security checkpoints that ensure defects are not passed downstream.

    What’s in Your Repository Manager is in Production

    Using data from more than 133,000 websites, researchers from Northeastern University showed that “37% include at least one JavaScript library with a known vulnerability. From a per-library perspective, at least 36.7% of jQuery, 40.1% of Angular, 86.6% of Handlebars, and 87.3% of YUI inclusions use a vulnerable version.

    “Alarmingly, many sites continue to rely on npm packages like YUI and SWFObject that are no longer maintained,” the researchers continued. “In fact, the median website in [NU’s] dataset is using a library version 1,177 days older than the newest release, which explains why so many vulnerable libraries tend to linger on the Web.”

    What is Wrong With 200 Billion Downloads?

    Nothing from an innovation standpoint. It’s awesome and we can all celebrate this achievement with Laurie.

    At the same time, it gives us pause to reflect upon other concerns of our day, including cybersecurity threats, technical debt and wasteful context switching.

    — Derek E. Weeks

  • Bridge the DevSecOps Experience Divide with Cross-Functional Teams

    Bridge the DevSecOps Experience Divide with Cross-Functional Teams

    Last week’s DevOps Connect event at RSA Conference offered up a ton of wisdom and real-world examples of the power of DevSecOps. With its best-ever attendance after several years and impressive participation levels, this year’s event stands as a good harbinger for the growing interest that the security community has in vesting itself in the DevOps phenomenon.

    But there’s still a lot of work to do in helping security professionals shift their mindsets and fully understand what DevSecOps is all about. Questions from the audience offered evidence of some of the trepidation and misconceptions that still exist in the security community.

    For example, after one of the first talks of the day, an audience member asked about a typical point of confusion for those new to DevOps—namely, the misunderstanding that the melding of disciplines and the requirement that DevOps team members flex their skills across traditional IT boundaries also means that everyone in IT must be renaissance men and women with a range of in-depth knowledge about all IT specialties.

    “Not everyone can know .NET, not everyone can know Python, not everyone can know firewalls,” the audience member said. “I’ve been doing this a long time and it takes a lot to become an expert in any one of these categories. And yet I’m hearing, ‘Oh, everyone needs to know it all.’ They can’t, so how do you deal with that?”

    This was a valuable question to revisit here, because the response from one particular speaker offers up an extremely important tenet that everyone in the DevSecOps world needs to keep in mind. The answer came from Paula Thrasher, a longtime DevOps evangelist in one of the hardest spaces to implement cutting edge IT processes: federal IT. As she explained, one of the biggest success criteria for a successful DevSecOps transformation is the ability to build fully engaged cross-functional teams. Individuals working within the DevOps are not meant to know everything. Rather, different specialists need to be paired up so the complete team has that full body of knowledge at its fingertips.

    “It’s the fidelity of your collaboration within the cross-functional team that makes this successful,” explained Thrasher, who is director of digital services for General Dynamics, a large federal IT integrator.

    For example, she pointed to one particular team she has seen in her work that is the epitome of a solid cross-functional team.

    “My teams all have mascots, and so one of our teams is the Tigers. The best quote I’ve heard from a team about this topic came from a new employee who said, ‘If one Tiger knows it, they all know it,’” she said. “This is a team that has a crazy diverse background—there’s developers, there’s ops people, there’s a former networker on there—and it doesn’t require everyone on that team to know everything. They collaborate so well that if there’s a network vulnerability, the network guy knows it and tells everybody.”

    This is the why Thrasher sees cross-functional teams that include security people as a big success criteria for DevSecOps.

    “I usually say the first thing should be if your teams aren’t cross-functional, start with that. You don’t have to change anyone’s boss, you don’t have to change anything else about who you report to, but make sure your teams represent all of the skills you need to deliver,” she said.

    Not only will the teams be able to deliver on all of the necessary skills when they’re working as a unit, there’s also an ancillary benefit: In the process, they’ll be training each other, and each team member will end up gaining a more diverse set of skills—and perhaps become renaissance men and women in the process.

    “They might have been a graduate of a cyberprogram and they’re next to a graduate of a CIS-type program and they’re next to a math major and they’re next to a hardware engineer, and they’re all kind of new because they’re recent graduates, but they’re learning with each other and we’re going to train them with some more senior engineers that we self-select to be good mentors,” she said. “This is how you are giving people the skills to work with this next-generation of automation and technology, both in the security realm and the Ops realm and the Dev realm.”

    — Ericka Chickowski

  • Trust and Safety in the Age of Automation

    Trust and Safety in the Age of Automation

    “As with any new technology, it’s really important that we be thinking now about how to do [it] ethically and responsibly.” —Robert High, Vice President and Chief Technology Officer, IBM Watson

    Here’s something to think about: According to an article in The Wall Street Journal, more than 30 countries have defensive weapons run by automation. South Korea has robotic sentries along the DMZ, while Israeli robots patrol the border at Gaza. Russia has robotic vehicles armed with 30mm cannons and anti-tank guided missiles that are well-equipped to battle it out with NATO forces should the need arise.

    Got your attention? Wait, there’s more.

    The SeaHunter is a U.S. military vessel that can travel the high seas on its own for months without a crew to feed or protect. It’s completely unmanned. And, it costs only $20 million per boat, less than half of the $49 million it costs to fire 59 Tomahawk missiles. At that price, the SeaHunter is practically throwaway.

    The SeaHunter, an unmanned oceangoing vessel, can remain at sea for months at a time.

    The implications are profound. The value proposition is hard to resist, Why spend billions on building and manning traditional warships when, for the same expense or less, you can have a navy of predominantly robotic ships patrolling the oceans of the world? And, you can have robotic aircraft, capable of automated takeoff and landing, using these ships as mobile bases. An air force untethered to any land mass is a game changer, particularly when non-state actors can get their hands on a few of them. It sort of brings new meaning to the word “pirate”—no need to board a vessel by force, just get the access credentials.

    And, at the core of it all this modern weaponry is automation and growing capabilities in AI. That should give you something to think about.

    We in DevOps are all about automation. It’s our raison d’être. Our fondness for, and dedication to, automation is admirable. We really have changed the way tech works. Yet, many of us think of our activities as mundane. It’s akin to the nonchalance of a Major League baseball player. They’ve hit the ball so much that many players have forgotten how skillful one needs to be just to foul off a bad pitch. Most people on the planet can’t hit a Big League pitch, let alone make it go foul by intention. But for the pro, it’s no big deal—it’s just part of a day’s work.

    It’s the same for us. We’re so accustomed to automating our world that it’s become no big deal. But, it is a big deal. It’s a very big deal.

    The easiest thing in the world is to trivialize the incredible impact our work has on others. After all, we’re “just working on” an ecommerce site, a dating app, or IoT endpoint for wired bicycles, right? Those of us working in the defense industry might grasp the significance of the work, but for most of us, it’s business as usual.

    Here’s the raw truth: Those of us doing the automation are changing the world. Each line of code we write is intelligence that alters the digital infrastructure of some part of the planet. The code we write will live on well after we’re gone, whether it’s from a company or from the planet. Are we really clear about the impact we have? Yes, some of our tech superstars are asking us to pay attention, but how many of us really are?

    Every day that passes we give automation more power. It might be a little piece of power, such as letting only authorized users access to an application. It might be significant power, such as determining whether a person qualifies for a loan to buy a house. It might be enormous power, such as manipulating political forces toward a desired behavior. Or, it might be the ultimate power: Deciding who gets to live and who gets to die. But, regardless of degree, it is power and we have it. And, we continue to make automation more powerful. As surprising as it may sound, despite appearances otherwise, we’re the ones the world trusts intrinsically to do our work safely and responsibly. A lot of the world’s safety depends on how we do our work. Political leaders don’t write the code. We do.

    Originally,  I was going to call this piece “Death and Destruction in the Age of Automation” and write about how we’ve taken automation way beyond commercial IT into the realm of life-and-death decision-making. I decided otherwise. The world does not need another gloom-and-doom piece, nor does it need another piece of sensational journalism. We’d do better to focus more on the positive things we’ve achieved. Technology is doing a whole lot more good than bad.

    But the fact remains that technology has accompanied death and destruction since Roman catapults assaulted the frontiers of ancient Europe. The thing that’s really different now is that the technology thinks—or, at least, it’s doing a really good job of emulating thinking. And, we’re the ones making it think better. Thus, we should ask, Are we making technology think in ways that are safe and trustworthy? Are we acting in ways that are safe and trustworthy?

    These are big questions. Do most of us really care about them? Or, is it easier to accept our work as mundane, with little consequence in the Big Picture? The nice thing about mundane is that it doesn’t require a lot of ethical contemplation. But, when you consider that the script that deploys an updated container to a Kubernetes cluster is not that far from the one that deploys an unmanned warship to a regional hotspot, things look different. And, contemplating the implications thereof becomes more than an academic exercise in a computer science ethics class.

    More than 2,000 years ago the philosopher Socrates said, “The unexamined life is not worth living.” Given the immense role that technology plays in today’s world and the power it wields—power that continues to move further away from our direct control—we might do well to say, “The unexamined technology is not worth doing.” It’s something to think about, while we still can.

    — Bob Reselman

  • DevOps Transformation: Leave No Ops Behind

    DevOps Transformation: Leave No Ops Behind

    In adopting DevOps, it’s easy to forget what you know

    Have you ever found yourself struggling to explain something that you know how to do without a moment’s thought? I’m not talking about the amygdala, the part of our brain that initiates the automatic “fight or flight” response. No, I’m referring more to the muscle memory-type behaviors that we’ve collected through life and, likely, pay little attention to while executing.

    Things We Take for Granted

    Muscle memory is awesome. Through the magic of muscle memory I can type fast (occasionally with okay-ish accuracy). It enables me to drive in peak-hour traffic without having a nervous breakdown, and it’s the only way I can ski down a tree-lined path without ending up in hospital.

    It’s through training, practice and repetition that we can carry out an increasing list of mundane tasks without being distracted from the bigger picture. In contrast, consider the expanding influence of DevOps across organizations, and how we’re pushing developer principles and practices onto operational facilities that have never been exposed to such. We’re asking people to break a habit of a lifetime and to think like a developer practically overnight. This can create some significant speed bumps in your operation transformation toward achieving continuous deployment pipelines.

    Type Mismatch Error

    On one side we’ve got infrastructure operations, where we’ve seen two decades of little change in the way services are provisioned and managed. Heck, for the most part, spreadsheets are still the source of truth—spreadsheets without versioning!

    There’s no “commit log” trail, no integrated testing and definitely no consistency across the vendor technologies that must be managed. Imagine what root cause analysis after an outage looks like? A team huddle for a round of “Who touched it last?”

    To software engineers, it must seem like the Wild West but, it’s important to acknowledge that this problem is largely the fault of the technology vendors. Until the principles and practices of DevOps shone a very bright light on operational inefficiencies, infrastructure operations weren’t challenged to increase the speed of business. Hence, the technology vendors hadn’t been pressured to evolve.

    A New Hope (Yes, Like ‘Star Wars’)

    Most infrastructure vendors have responded to the DevOps calling by re-architecting products in support of programmability. Many of them have even done an alright job of it, adhering to RESTful standards and publishing API schemas. However, there’s one key element that’s missing, and will be the cause of further challenges if not addressed.

    To an experienced developer, operating with declarative interfaces, idempotency and continuous integration is practically muscle memory. But to infrastructure operations, it is completely foreign. Progressing with the kind of operational transformation required to achieve continuous deployment is possible but only though closing this operating model gap.

    “Give me a Swiss army knife, and MacGyver I do not become,” said I at DevOps Enterprise Summit 2017.

    As the lines between development and operations increasingly blur, be super-aware of the development principles and practices that are just a part of life for the software engineers. To close the operating model gaps, run workshops for your pals in operations. Teach them all the fancy words. Show them how to manage their config files and architectural notes in a code repository, so they can familiarize themselves with the tooling.

    Most importantly, leave no ops behind!

    — Nathan Pearce

  • How Python is Transforming the DevOps Landscape

    How Python is Transforming the DevOps Landscape

    DevOps is the combination of operations and software development which reduces or almost eliminates the disconnect between systems administrators who run the infrastructure and software developers who develop applications.

    During software development, when the Agile methodology is utilized properly, a bottleneck often occurs during the frequent operations and deployment phases. New fixes and updates are produced so rapidly in every sprint that the infrastructure teams can be swamped with deployments. To eliminate few of these issues, the operations personnel and application developers are asked to work closely to automate the code delivery to production from deployment.

    As DevOps is an approach for handling change and agility, engineers must master multiple languages. The Python programming language is one of the most crucial components of the DevOps toolchain. Many DevOps teams utilize it for building web applications for visualizing custom data, building custom utilities and more.

    Ansible and other tools are written in Python, which means you can build custom scripts or modules to automate your tasks, for example, or do other things.

    Reasons for Using Python for DevOps

    The accessibility and flexibility of Python are reasons why it is a preferred language for the DevOps toolset. If you are working in the role of DevOps, you must require an adaptable skillset; you must know how to work with multiple languages. For anyone looking to explore new tools and languages, and are curious about technology, the language can form a solid foundation, as it doesn’t require the commitment levels that a specialist language requires.

    In addition, it is a great scripting language, and scripting refers to automation. A few major configuration management tools, including SaltStack and Ansible, are written in Python underscores. This shows how useful the language is when it comes to orchestration and infrastructure automation.

    Python and Ruby are compared often, and with good reason: Both are highly accessible and utilized in applications developed by many organizations. They also feature in the DevOps toolset. We can do almost all the things with Ruby that we do with Python. But, when it comes to syntax, Python gains the upper hand, as it is much more direct than the syntax of Ruby.

    In the realm of learning to debug problems and code, Python has the advantage over Ruby, as it takes a much more direct approach to programming. Its directness and simplicity are invaluable if you are working in DevOps environment and agility is highly important.

    Conclusion

    As a growing number of engineers and developers take on decision-making responsibilities, Python can be the stabilizing language. And not only for DevOps—the language also can be used for developing blockchain, which is the underlying technology for various cryptocurrencies such as bitcoin wallet. As it can be utilized in various ways, it enables you to be open to broad range of technical possibilities. Python is the go-to programming language for the DevOps landscape.

    — savaramravindra

  • Monitoring Your Data Center Like a Google SRE

    Monitoring Your Data Center Like a Google SRE

    Back in the early ’00s, when Google was beginning to expand its portfolio of services beyond search, it encountered a combination of challenges. Some of these emerged from familiar, classic disconnects between developers and operations folks or between IT services and line-of-business owners. Others were brand new, never-before-seen failure modes that arose from providing services on novel cloud platforms—and doing so at planetary scales. 

    To confront these challenges, Google began evolving a discipline called Site Reliability Engineering, about which the company published a very useful and fascinating book in 2016. SRE and DevOps (at least the contemporary version of DevOps that’s expanded into a vision for how IT operations should work in the era of cloud) share a lot of conceptual and an increasing amount of practical DNA; particularly true since cloud software and tooling have now evolved to enable ambitious folks to begin emulating parts of Google’s infrastructure using open source software such as Kubernetes. Google has used the statement, “Class SRE implements DevOps” to title a new (and growing) video playlist by Liz Fong-Jones and Seth Vargo of Google Cloud Platform, showing how and where these disciplines connect and nudging DevOps to consider some key SRE insights.   

    Understanding Google SRE Principles

    First, some basic principles: 

    Failure is normal – Achieving 100 percent uptime for a service is impossible, expensive or pointless (e.g., given the existence of masking error rates among your service’s dependencies). 

    Agree on SLIs and SLOs across your organization – Since failure is normal, you need to agree across your entire organization what availability means, what specific metrics are relevant in determining availability (called SLIs, or service level indicators) and what acceptable availability looks like, numerically, in terms of these metrics (called the SLO, or service level objective). 

    Use agreed-upon SLOs to calculate an “error budget” – SLO is used to define what SREs call the “error budget,” which is a numeric line in the sand (e.g., minutes of service downtime acceptable per month). The error budget is used to encourage collective ownership of service availability and blamelessly resolve disputes about balancing risk and stability. For example, if programmers are releasing risky new features too frequently and compromising availability, this will deplete the error budget. SREs can point to the at-risk error budget, and argue for halting releases and refocusing coders on efforts to improve system resilience. 

    This approach lets the organization as a whole balance speed and risk with stability effectively. Paying attention to this economy encourages investment in strategies that accelerate the business while minimizing risk: writing error- and chaos-tolerant apps, automating away pointless toil, advancing by means of small changes and evaluating “canary” deployments before proceeding with full releases.     

    Monitoring systems are key to making this whole, elegant tranche of DevOps/SRE discipline work. It’s important to note (because, remember: Google isn’t running your data center) that this has nothing to do with what kind of technologies you’re monitoring, the processes you’re wrangling or the specific techniques you might apply to stay above your SLOs. In short, it makes just as much sense to apply SRE metrics discipline to conventional enterprise systems as it does to 12-factor apps running on container orchestration. 

    Applying Google SRE Principles

    So here are a few things Google SRE can tell you about monitoring, specifically: 

    Alert only on failure, or on incipient failure – Alert exhaustion is a real thing, and “paging a human is an expensive use of an employee’s time.” 

    Monitoring is a significant engineering endeavor – Google SRE teams with a dozen or so members typically employ one or two monitoring specialists. But they don’t busy these experts by having them stare at realtimecharts and graphs to spot problems: that’s a kind of work SREs call “toil”—they think it’s ineffective and they know it doesn’t scale. 

    Post-hoc analysis, no magic – Google SREs like simple, fast monitoring systems that help them quickly figure out why problems occurred, after they occurred. They don’t trust magic solutions that try to automate root-cause analysis, and they try to keep alerting rules in general as simple as possible, without complex dependency hierarchies, except for (rare) parts of their systems that are in very stable, unambiguous states (their example of “stable”: when they’ve redirected end-user traffic away from a downed data center, systems can stop reporting on that data center’s latency). Elsewhere, their systems are in constant flux, which causes complex rule-sets to produce excessive alerts. One exception to this general rule about simplicity: Google SREs do build alerts that react to anomalous patterns in end-user request rates, since these affect usability and/or reflect external dependency failures (e.g., carrier failures). 

    Heavy use of “white box” monitoring – Google likes to perform deeply introspective monitoring of target systems grouped by application (called business service monitoring in Opsview Monitor). Viewing related metrics from all systems (e.g., databases, web servers) supporting an application lets them identify root causes with less ambiguity (e.g., is the database really slow, or is there a problem on the network link between the DB and the web host?) 

    Four golden signals – Because part of the point of monitoring is communication, Google SREs strongly favor building SLOs (and SLAs) on small groups of related, easily understood SLI metrics. As has been widely discussed, they believe that measuring “four golden signals”—latency, traffic/demand, errors and saturation—can pinpoint most problems, even in complex systems such as carrier orchestrators with limited workload visibility. It’s important to note, however, that this austere schematic doesn’t automatically confer simplicity, as some monitoring makers have suggested. Google notes that “errors” are intrinsically hugely diverse and range from easy to almost impossible to trap, and “saturation” often depends on monitoring constrained resources (e.g., CPU capacity, RAM, etc.) and carefully testing hypotheses about the levels at which utilization becomes problematic. 

    Conclusion

    The bottom line is that good DevOps monitoring systems need to be more than do-it-yourself toolkits. Though flexibility and configurability are important, more so is the ability of a mature monitoring solution to offer distilled operational intelligence about specific systems and services under observation, along with the ability to group and visualize these systems collectively, as business services. 

    — John Jainschigg