devopsy.kinsta.cloud

Category: Blogs

  • Repex – Regex Management Simplified

    Repex – Regex Management Simplified

    Our Versioning Complexity

    Cloudify is a Python shop.

    Our REST service is Python.

    Our Workflow Engine is Python.

    Our Plugins are Python.

    We have different version formats and different dependencies across different types of files which need to be changed when a version is updated.

    This is complex, to say the least. We didn’t want to manage this manually having more than 60 repos to take care of when handling version updates.

    Why not sed?

    • sed is a bitch, isn’t it? It’s unmaintainable and unmanageable. While this is solely my opinion, handling hundreds of files, each change in a different context without any safety net, makes sed something you should walk away from.
    • sed isn’t configurable.

    Why not Jinja?

    Using Jinja templates, while safe, was out the windows pretty much instantly as we needed the repos to be usable out of the box or we’d be forcing users to change templates so that they can use our product.

    The Solution

    We turned to Regex. Yes, there is the regex problem. You’re probably familiar with the famous quote:

    –Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

    While this can be true, it also depends on how you use regex.

    I believe we’ve done this well.

    Repex

    We developed repex.

    repex allows you to specify the structure of changes you need to perform in YAML format:

    variables:
  replace: 3.1.0-m2
  version: 3.1.0-m4

paths:
  - description: demo thing
    path: repex/tests/resources/mock_VERSION
    match: '"version": "3.1.0-m2"'
    replace: "{{ .replace }}"
    with: "{{ .version }}"
    to_file: repex/tests/resources/mock_VERSION.test
    validate_before: true
    must_include:
      - date
      - commit
      - version
  - description: handles core modules version in all core repositories
    type: setup.py
    path: cloudify-.*
    excluded:
      - my/excluded/path
    match: version=('|")(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)('|")
    replace: (\d+)(\.\d+){1,2}(dev|(\w+\d+)?)
    with: "{{ .python_core_version }}"

     

    You define a list of paths you’d like to iterate over, provide it with what you’d like to search for, what you’d like to replace, what you’d like to replace with, and some more parameters and off you go.

    For example, the first object in the above configuration will:

    • Look for a file in the path repex/tests/resources/mock_VERSION (regex)
    • In the file, look for `”version”: “3.1.0-m2″` (regex)
    • Attempt to replace 3.1.0-m2 with 3.1.0-m3 only if the string was found during the matching stage and if the strings `date`, `commit` and `version` were found in the file.
    • Write to the output file repex/tests/resources/mock_VERSION.test

    The second object will:

    • Look for `setup.py`(regex) files under `cloudify-.*` (regex) and not under `my/excluded/path`.
    • Look for `version=(‘|”)(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)(‘|”)`(regex) in the files found.
    • Replace `(‘|”)(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)(‘|”)`(regex) with a variable which is not harcoded and therefore must be provided using the API.

    Layers of protection

    repex attempts to provide layers of protection and comfort:

    Safe!

    • You can first `match` whatever you want to match so that you’re sure you’re only replacing something in the exact context you wish to address. If `validate_before` is true, replacement will only occur if one or more matches were found.
    • You can verify that only files containing very specific strings will be addressed by listing them under `must_include`.
    • You can exclude files or directories you don’t want to search in by listing them under `excluded`.

    Don’t take this the wrong way though – regex is regex. You make a mistake, you pay for it. All repex grants you is the ability to have everything organized in one place, with some layers of protection to keep you from making some trivial mistakes.

    Comfortable

    You can declare every single directory or file you’d like to address in one manageable YAML.

    Of course, we needed to create some kind of naming convention so that we don’t have to configure each and every file separately.

    Accepts Variables, both hardcoded and via an API

    One of repex’s strongest features is that it supports using variables for string replacement. The above example contains a {{ .version }} variable.

    The API allows you to provide a dict of variables which will be placed in placeholders. You can either hardcode a variable in the variables section or use the API to send a dict of variables to be used. If a variable is hardcoded in the YAML and you use the API to send a variable with the same name, it will overwrite the hardcoded one. In addition, validation will occur upon variable replacement to make sure it was replaced.

    Functional API

    The API provides three basic functions. I’d rather not rewrite repex’s documentation.. so just refer to this.

    Basically, the three functions allow you to decide the level of granularity in which you’d like to handle the changes. It will allow you to perform actions in different stages of the replacement process.

    For example, we don’t really use the highest level of the API which just iterates over all objects in the YAML. We wrote a Cloudify specific wrapper which can be used as a reference implementation. It enables validation after every replacement. So, for instance, after every blueprint specific replacement, we run a validation that the blueprint is valid vs. our DSL-parser.

    Repex is available on PyPI. You can install it by running:

    pip install repex

    We might, at some point, provide a CLI for repex. Also a logging feature might be implemented to allow for JSON formatted log messages to be generated so that they can be transported for analysis.

    After using repex throughout our entire build process for the last five months or so, it has proven to be extremely useful. We can now follow our entire version update process and easily identify problems by looking at one YAML file and one execution log file. We’re planning on using repex to replace additional types of data (not only versions).

    We would appreciate feedback and, even better, pull requests.

    Hope you find this useful.

  • DevOps Perspectives from Pink15

    DevOps Perspectives from Pink15

    I recently had the privilege of covering the Pink15 conference on behalf of devopsy.kinsta.cloud. Pink Elephant is a renowned leader in ITIL and ITSM education, consulting and conferences. Now in its 19th year, the Pink conference brings together attendees from a wide range of vertical markets to learn and share about IT service management and related topics. My mission was to take the pulse of DevOps in the ITSM enterprise community, many of whom represent the operational side of IT.

    Let me start by saying that this was a well-organized and executed conference. The attendees that I spoke with unanimously agreed that the sessions were relevant and the peer networking invaluable. And while there was not a specific Devops or Agile track, the core concepts were directly or indirectly present in several of the sessions.

    One particularly interesting session was on “Enterprise Change Management in an Agile, Fast Moving World” from a large retail organization. The presenter did a good job of describing how they are adapting their categories of their changes to meet the growing requirements for a more agile change management process. At present, only 5% of changes are categorized as “fast track” – an even more expedited route than what is normally considered a standard change. This organization’s goal is to exponentially increase their ability to fast track more changes in response to increasing demand. This session was a good indicator of how enterprise IT is recognizing the need to adapt for continuous delivery and more automation.

    More than one attendee shared their appreciation at being able to interact with other IT organizations about specific ways to bridge the practices between Dev and Ops so as to make service management more agile. One person mentioned that while some of the sessions did not specifically call out DevOps or Agile, presenters often had to address these in depth during the question and answer period. There was particular interest among attendees in learning how to engage developers in processes such as Event, Demand, Capacity, Availability, Change and Release Management.

    Some attendees expressed concern about the perception that ITSM and ITIL have no place in a DevOps culture or that DevOps would somehow invalidate the investments that many organizations have already made in establishing better processes for managing services. Kaimar Karu, the ITSM Product owner for Axelos (who owns the IP for ITIL) strongly believes that ITIL is absolutely a key player in the realization of DevOps as a philosophy. ITIL processes are not specific to operations – they enable the entire lifecycle of a service or product from development to deployment and into operations. ITIL was meant to be adopted and adapted; nowhere does the guidance recommend that the processes be complex, bureaucratic, lengthy or manual.

    So what does all of this mean? While DevOps is not yet fully understood, organizations are taking tangible steps to make ITSM processes go faster, integrate with other practices and leverage automation in order to deliver more business value. Wouldn’t you call that DevOps?

  • IBM InterConnect: HM Health Solutions shows how continuous delivery helps cure healthcare IT headaches

    IBM InterConnect: HM Health Solutions shows how continuous delivery helps cure healthcare IT headaches

    As a spin off business unit of health insurer Highmark, HM Health Solutions is tasked with helping not only that insurer but also other third-party insurers across the industry to optimize systems that run core functions like benefits design, member enrollment, claims processing and customer service. And according to key members who opened up about the organization’s on-going DevOps transformation at IBM InterConnect this week, continuous delivery will increasingly play a role in HM Health’s ability to do that.

    Utilizing IBM Urbancode, the business has been able to improve the application delivery process by giving application teams more control over how they stand up new environments, better visibility into builds and deployment , and much speedier configuration management. As a result, not only are they able to deploy more quickly, they’re keeping their developers happier and better engaged, with a higher rate of productivity.

    “We save time and a good amount of it,” says Jim Neumyer, senior application developer for HM Health Solutions. “The (app) teams have the ability to look at their builds, configure their builds how they want to do it, strip down the builds with our standard templates and go very quickly from an SCM check-in to deploying in very few steps.”

    As he explained, the teams are able to stand up their own environments, they’re no longer hampered by lengthy approval processes. Streamlining of approvals was perhaps the biggest time saver, Christina Felts, team manager of open solutions engineering, explained that one team said they experienced a savings of an hour of deploy time per app—something that can add up on release nights dealing with dozens of applications.

    “One of our app teams was able to setup a new environment in less than one day, which previously we couldn’t do. So it could take a couple days before they got everything set up in that new environment,” Felts. “And it also eliminated the need for them to work with my team and put in a request and have to wait up to a couple days turnaround to get configurations done.”

    Additionally, Neumyer says they can also easily roll-back when necessary. Perhaps even more impactful, though, has been the visibility piece in overall enablement of developers, Neumyer said.

    “So the teams can check everything.  If they have an issue, if they don’t know what’s going on in their environment they can go out there and look.  It takes me out of the mix,” he said. “They can see it very clearly and easily and try it again.  And most of the issues can be resolved on their own.  And we can automate things a lot easier.”

    HM Health Solutions says this is only the start of their DevOps journey—as things stand, it’s still a waterfall development shop moving toward Agile—but the early successes have garnered more buy-in from across the organization. Neumyer recommended to colleagues in the audience to follow the baby-steps approach that has worked for his organization when it comes to getting started with continuous delivery.

    “The biggest takeaway for me is to take small bites,” he says. “Don’t shove the ice cream cone in your mouth all at one time.  When you take small bites you’re able to tweak things very quickly.”

  • How Deploy Frequency Impacts Infrastructure Stability

    How Deploy Frequency Impacts Infrastructure Stability

    I recently took some time out from reading the Internet to read a very specific piece of research I found on the Internet from IDC – its “DevOps and the Cost of Downtime” to be exact. Sponsored by AppDynamics, this fascinating read included a variety of statistics that are helpful in understanding the impact of (or a lack of) DevOps on organizations given certain financial and competitive impacts.

    I won’t bore you by repeating what you can so easily read yourself, but I have pulled out a few key stats that are worthy of further investigation – that of the failure of  stability on the business and the predicted increase in deployment frequency.

    • The average hourly cost of an infrastructure failure is $100,000.
    • The average hourly cost of a critical application failure is $500,000 – $1M.
    • The average number of deployments per month is expected to double in 2 years.

    Interestingly, that prediction on deployment frequency is a good thing and carries with it the possibility of making sure the costs associated with the other two rarely (if ever) happen.

    We’ve already gotten to a point where “speed” with respect to DevOps is effectively equivalent to “time to market”. But there’s another measure of “speed” that’s just as critical, particularly when we apply it to maintaining stability in the infrastructure. That measure is MTTR (Mean Time to Resolution).  This measure is very important when you consider that 60% of production failures are caused by human error or lack of automation according to research from ZeroTurnaround. A goal of DevOps, then, should be to reduce that number as much as possible and introduce more stability into the environment. Frequency of deployment can actually be of assistance in realizing that goal.

    So how does frequency of deployments help do that?

    possible points of failureFirst let us consider the number of possible points of failure that might cause downtime – both in the application and core network infrastructure. Then let us consider that each of those possible points of failure also have multiple potential causes of failure. A single service in the logical critical data path within the core service infrastructure, for example, can easily cause “downtime” that, in the case of IDC, would be characterized as “infrastructure failure” and run a tab of $100K per hour.

    If that application is a critical business application, the tab runs much higher per hour.

    Obviously the speed with which we restore stability (service) in such an event is a factor in the overall impact to the business.

    This is where DevOps comes in and where operationalization can play a significant role in mitigating the impact of downtime by aiding in faster time to resolution.

    Consider that one of the most difficult tasks in the network is not provisioning or configuration, but troubleshooting. To be fair, troubleshooting in app dev, too, is one of the more time-consuming and frustrating tasks, consuming 75% of a developer’s time (according to research from InitialState). That’s close to the 65% of network engineers who are dedicated to “break and fix” tasks, who spend their time troubleshooting network issues up and down and all along the network stack. Needless to say, if you just deployed ten or fifteen different configuration changes splayed across the core and per-app service infrastructure and suddenly something is “wrong”, it’s going to take some time to figure out exactly which configuration change caused the problem in the first place.

    The relationship between number of changes and frequency of deployment is no secret; if you’re deploying changes more frequently then it’s likely there are fewer of them. With an average sprint cycle of 3-4 months, the amount of change is logically not going to exceed that of a cycle that takes 12-18 months. The same holds true for the infrastructure; more frequent deployments should mean fewer changes).

    Logic (and math) tells us that the fewer lines of code and configuration that have changed, the less time it should take to identify the culprit should something go wrong, thereby hastening the time to resolution. Assuming that an agile, frequent deploy rate results in fewer changes to the environment it stands to reason that this approach should also reduce both the number of failures and the time to resolve any that do crop up. Frequency is a significant factor  in reducing failures by increasing its overall stability.

    This is backed up anecdotally by Puppet Labs findings in its “State of DevOps” in which organizations adopting a DevOps approach reported 50% fewer failures and a 12x improvement in time to resolution. That’s stability at work and it’s driven in part by the frequency of deployments and its positive impact on MTTR.

    So let’s apply this approach (and logic) to the rest of the services along the critical data path in the data center infrastructure, i.e. apply it to application services like caching, web application security, load balancing, identity management and application access control.  Automation to reduce misconfiguration caused by human error and smaller, more frequent changes that enable a faster time to identity the inevitable issues that will be introduced. No methodology or approach is going to completely eliminate errors, after all. There’s a limit to how perfect we can get and it’s pretty much a Zeno’s Paradox kind of problem. While we keep solving current issues, more config and code is being deployed that is statistically likely to contain yet another error, and so and so forth.

    But the goal is not necessarily to eliminate all errors (though that’d be nice, it’s simply not practical and it’s the primary reason why our goals for uptime are called “5 9s” and not a “1 with two zeros”) it’s to reduce the number of errors and their impact on downtime. And that is something we can achieve without running afoul of Zeno and his paradox.

    Smaller, more frequent deployments can result in a more stable environment in which downtime is reduced to a level consistent with acceptable risk.

    DevOps, whether applied directly to infrastructure or not, will play a role in improving its stability which in turn has a direct and very real impact on the business by eliminating (some) downtime thanks to more frequent deployments.

  • Streamlining no matter how many speeds your IT operates at

    Streamlining no matter how many speeds your IT operates at

    IBM InterConnect 2015 took over the Mandalay Bay and MGM Grand conference centers this week with an estimated audience of more than 20,000 attendees. The event is filled with DevOps evangelists, hybrid cloud experts, and IBM customers converging to share information in presentations and panel discussions.

    On Monday I had an opportunity to attend one such panel discussion titled Mobile to Mainframe Experts: DevOps Best Practices for Systems of Engagement and Systems of Record. Yeah, the title doesn’t exactly “pop” but I didn’t let that dissuade me. The panel included a number of respected names in DevOps such as Carmen DeArdo from Nationwide Insurance, and Rosalind Radcliffe and Sanjeev Sharma from IBM.

    In spite of the title of the panel discussion most of the conversation revolved around the concept of two-speed IT. In any organization—no matter how streamlined—there are going to be different processes or teams that function at different speeds. Applying the Pareto Principle, even if the organization figured out a way to operate more efficiently there would still be 20 percent doing 80 percent of the work and 80 percent doing the other 20. It would just be a matter of degree.

    The problem is that if the speeds are too different problems arise. First, it’s just a waste of resources to have skilled workers sitting idle waiting for a different group or process to catch up. Second, those workers will often get antsy and go rogue—writing code and introducing features that weren’t part of the initial requirements and aren’t part of the planned testing. Now you’re just adding chaos that disrupts the workflow even further.

    According to the esteemed panel the goal is to find ways for those processes and teams to function at their own pace without it impeding the overall efficiency —hence two-speed IT. Rather than trying to force a square peg into a round hole, or getting frustrated that the square peg can’t just be round in the first place, you have to find ways to maximize efficiency with the teams and processes you have.

    Carmen DeArdo from Nationwide Insurance applies another term as well: variable-speed DevOps. In essence two-speed IT frames the challenge, and variable-speed DevOps is the set of processes and tools organizations can utilize to address the challenge. Removing some of the friction between different groups and enabling the organization to function more effectively is one of the primary benefits of DevOps.

    When I spoke to Rosalind a couple weeks ago she stressed repeatedly that the foundation of effective DevOps is breaking down silos and finding common ground. That doesn’t mean all the teams have to be restructured under one organizational unit, or even that everyone has to adopt the exact same tools and processes. It means figuring out how they’re different and how they’re alike, and determining ways to make the most effective use of the common ground where their similarities overlap. It provides a starting point to get the ball rolling, and then things can mature and evolve from there.

    In the session at IBM InterConnect Rosalind agreed that variable-speed DevOps is a great approach, but cautioned attendees regarding how it’s used. She told the audience that variable-speed IT should be driven by business needs, not by tools or processes. In other words, you have to stay focused on the business requirements and the need to achieve business goals and then find ways to do that. You can’t try to shrink or adapt the business requirements and goals to fit the processes and tools you already have.

    DevOps tools and processes can—and in most cases should—speed processes and teams up. One of the hallmarks of DevOps is speed. Even as organizations embrace DevOps and introduce more automation, there will still be different teams and processes functioning at different speeds—and a need to implement variable-speed DevOps to enable those different speeds to exist in harmony.

  • InterConnecting the Internet of Things

    InterConnecting the Internet of Things

    In every direction one turns in the so-called Internet of Things there is fragmentation. There are many programing languages, devices and, more recently, a disparate set of IoT development and management platforms that are all vying for the market and developer’s attention. They hope, to become the standard in IoT in the months and years ahead.

    One of the biggest obstacles to widespread IoT adoption has been the difficulty attracting developers to use the low-level programming languages on which most run. “IoT has been really hard to work with for a long time because a lot of it has required working with the Arduino tool set, for example, which is a lot of C++,” says RedMonk senior analyst Donnie Berkholz. “A lot of developers these days aren’t very interested in working in low level languages anymore,” he says.

    Enter the Ethernet IoT Starter Kit for the IBM IoT Foundation. The kit from chipmaker ARM, Freescale, and IBM aims at the industrial Internet of Things apps, and is based on ARM’s mbed operating system, the Freescale-designed development board, and BlueMix – IBM’s cloud service.

    According to the ARMmbed blog, the starter kit for the IBM IoT Foundation provides a slick user experience, and makes it possible to get data from the board sensors to the IBM BlueMix cloud within minutes. “It is particularly suitable for developers with no specific experience in embedded or web development, as it provides a platform for learning new concepts and creating working prototypes,” the announcement states.

    “After the initial out of box experience, the infinite possibilities of cloud applications can be explored through IBM’s production grade BlueMix platform, in which deployment and device management is as simple for one device as it is for one million. The starter kit hardware can be modified and extended to explore the device design space, and a finalised design can be taken to production using the mbed SDK and HDK,” according to the announcement.

    There were additional IoT announcements this week at the IBM InterConnect 2015 conference. Enterprise cloud provider CloudOne announced the launch of their CloudOne Internet of Things (IoT) Lab. The CloudOne IoT Lab hopes to be a platform for real-time collaboration to build IoT solutions. According to CloudOne, it will shortly build a technical center in Chicago, IL to house the companies technical team and provide a space for customers to get hands-on experience with the CloudOne platform, including IBM continuous engineering, asset management and analytics technologies.

    “The CloudOne IoT Lab allows our team to brainstorm IoT-enabled solutions with customers in real-time. This will speed delivery of connected products while embracing and extending existing customer investments in partner technologies, data and processes,” John McDonald, CEO of CloudOne, said in a statement.

    Also this week at InterConnect 2015, stream network PubNub announced that it is partnering with IBM to bring realtime data streaming and messaging capabilities to BlueMix.

    PubNub demonstrated what it called realtime secure device connectivity, powered by its easy-to-use data stream APIs, at the conference. And, according to PubNub, IBM BlueMix customers will be able to leverage PubNub to securely and reliably deliver large scale realtime IoT, web, and mobile solutions.

    There has been much movement in the IoT space in recent weeks. Earlier this month, the Raspberry Pi 2 was announced, with operating systems being provided by both Microsoft and Canonical. The increased processing power of the Pi 2 means in can be used in more smart devices from robots to drones.

    Many of the recent announcements hope to make devices, data, and development tools more accessible to more developers. All of that is a must if the IoT is to reach its full potential soon.

  • To Be or Not to Be – Shadow IT

    To Be or Not to Be – Shadow IT

    Is it being an innovator, or demonizing business decisions to use other vehicles, and cry foul?

    Sun Tzu, the famous Chinese strategist and philosopher, once said, “If ignorant both of your enemy and yourself, you are certain to be in peril.” These words ring true to the quandary that traditional IT organizations find themselves in today: on one side, hordes of tech-savvy business users are deploying and engaging technology services at an alarming rate, often without any input or validation by the IT team. On the other side, there’s an aging on-premise infrastructure that requires frequent care and stresses the limited resources and budgets of IT staff. In between, sits a group of technology professionals that want to support their users while maintaining the appropriate level of compliance, risk management and security — fondly known as “Traditional IT”.

    Security teams, a part of traditional IT organizations, have begun demonizing any non-sanctioned technology use – dubbing it “Shadow IT – taking a somewhat close-minded view as to how software and services are being procured and utilized in today’s organizations. In the legacy world, IT Security could block or veto projects based on its merit and risk. Business teams had no choice but to work within their constraints, as the security teams controlled the firewalls — the proverbial gates to the kingdom. Nothing entered or left the organization without being blessed through the firewall, which left IT Security in a very advantageous position. Fast-forward to today and we see a drastically different dynamic taking place in organizations: DevOps and business teams are driving buying decisions, or making them without IT approval.

    Why is this happening?

    If you look at how various teams are structured in most organizations it becomes obvious. IT departments, a legacy construct best suited for organizations with few technology-savvy users, are best suited with a role as a service provider. Meaning, they build infrastructure, deploy software and services, and operate utilities as a service to their user base. This could be users in the form of bankers, call center employees, or even software engineers. There is no minimum or maximum level of technology expertise required on behalf of the user, because the IT department is there to offer support and guidance as necessary. This is very much a “thou shall not use technology not provided by IT” point-of-view, because IT is ultimately responsible for the support and user experience of said technology.

    However, IT departments are fundamentally based around annual budget cycles and 3 to 5 year amortizations of capital expenditures (servers, network equipment, etc.). As the quality and variety of software, hardware and SaaS solutions has increased ten-fold over previous decades, we have quickly entered a buyer’s world where multiple satisfactory solutions can be obtained quickly. This is counter to the traditional IT pattern of planning purchases in advance, leaving ample time for procurement haggling. In as little as a year, entire service providers can spring up with scalable, resilient, superior technology solutions — often affectionately referred to as disruptors. These disruptive technology companies move fast, gather feedback from their customers, improve their product/service and deliver new products. If you just spent a million dollars on servers that you plan on keeping for the next three years, it is nearly impossible to justify leveraging a fleet of cloud servers as an alternative just a few months later. The cloud servers could be the right solution for your business — but the pain of wasted capital is too great for most organizations to shake, much less the implied defeat or mistake in judgment by IT management.

    DevOps teams, on the other hand – and much to the chagrin of IT and InfoSec teams, thrive in the chaos of moving quickly and breaking from tradition. This is not done maliciously, but rather out of necessity to reach a peculiar and demanding audience. Years of “buy it now” instant gratification have conditioned the end-user to expect fast, high-quality and competitively priced products. This holds true for the Internet generation now responsible for the buying decisions of everything from inexpensive productivity tools to enterprise-grade communications solutions. Decisions are not made solely on who can deliver the highest SLA or who had the best benchmark score. Agile teams now look for the best product fit, defined by a set of conditions ranging from automation integrations to direction and vision of the vendor. This wide range of criteria falls well outside the traditional IT buying spectrum, and thumbs its nose at the legacy buying practices. DevOps teams buy subscriptions to services that they can change or cancel at any time. They procure products that don’t even have user interfaces beyond an API and JSON output. They buy based on “what can we do with this” rather than “what can this do for me”, which is reminiscent of maker culture.

    A new breed of tools that enable the DevOps teams to do great things while catering to the IT Security team has emerged. DevOps teams tend not to budget for security, and are often too fast moving to build all the requisite capabilities themselves. IT Security is desperate for a place at the table to ensure the safety and security of organizations and its users. This sets the stage for new, emerging technology companies to capitalize on the widening gap that has occurred as the industry shifts from the traditional datacenter model to one of private, hybrid and public clouds. This means no more 3-5 year investments in a security technology and no 3-5 year guaranteed relationship for security vendors.  Emerging companies also have to raise their game to meet the needs of these organizations and prove to be the partner of choice. The uptake to date has been slower as the industry has been slow to detect and understand this shift. Many organizations have legacy investments and don’t want to bring additional complexity to their lives by adding new panes of glass. DevOps teams may directly integrate raw data feeds from new products into their toolkit, but IT teams need solid integrations to existing infrastructure to make the best use of these new products. Without this, they can’t keep pace with the rapid deployments and changes made by DevOps teams, and are left nearly blind to the dynamism of their infrastructure risk and security posture.

    So what is the solution?

    It’s simple. By integrating security more deeply into the DevOps toolchain, and operationalizing security practices the same way we operationalize infrastructure and code issues. This means there is a space to not only bring cloud-centric data down to the legacy investments through connectors and integrations, but also an opportunity for IT Security to enable DevOps by automating common security practices. If the practices and behaviors of Security teams were operating at or beyond the pace at which DevOps teams deploy, then a collaborative environment can be built with security detection, notifications, and even mitigations act as a “safety net” in agile environments. This makes a lot of sense when you realize that the biggest threat to these agile organizations is not malicious hackers or state-sponsored attacks, but rather human error, policy failure, lack of transparency into security practices, and lack of consistency over time.

    About the Author/Tim Prendergast

    evident-headshot200X200With well over two decades of experience pushing the limits of technology, Tim set out to create the first “next-generation” security company, Evident.io, focused solely on programmatic infrastructures (cloud). Tim co-founded Evident.io to help others avoid the pain he endured when helping Adobe adopt the cloud at a massive level.

    Tim’s prior experience includes leading technology teams at Adobe, Ingenuity, Ticketmaster, and McAfee. He holds over 15 years of security experience; 8 years of AWS security experience, with 3 of them defending the Adobe AWS infrastructure from inception to production as an AWS Certified Solutions Architect.

    Connect with Tim: LinkedIN | Twitter | Evident.io

  • Four Steps to Free Your Organization From Scheduled Maintenance

    Four Steps to Free Your Organization From Scheduled Maintenance

    Whether you’re an operations engineer or a customer, scheduled maintenance is often viewed as a given when delivering software as a service. While far better than surprise outages, planned downtime is still tough for everybody involved– customers will always have off-hours access needs, and the low-traffic times you want to perform maintenance are often the same times your operations engineers and admins would like to be sleeping. On top of this, scheduled maintenance implies your system is less reliable than you think, because you’re afraid to change it during the workday.

    How do we make scheduled maintenance better for everyone involved?

    Planned maintenance is a non-starter at PagerDuty, since our customers don’t know when their monitoring tools will fire alerts. We’ve had to focus hard and innovate to deliver continuous uptime for our service, and Doug Barth, my colleague and a member of our operations team, recently did a talk at DevOps Days Chicago on the strategies we used to ditch scheduled maintenance altogether, and replace it with iterative maintenance that doesn’t compromise your entire system.

    1. Deploy in stages

    Your deployments need to be rock-solid. They should be scripted, fast and rolled back quickly, as well as tested periodically to ensure rollbacks don’t lag.

    They also need to be forward and backward compatible by one version. It’s not an option to stop the presses when you push out a new version. Red-blue-green deployments are crucial here, as they ensure only a third of your infrastructure undergoes changes at any given time.

    Lastly, stateless apps must be the norm. You should be able to reboot an app without any effect on the customer (like forced logouts or lost shopping carts).

    2. Avoid knife-edge changes

    Use canary deploys judiciously to test rollouts, judge their integrity and compare results. These test deployments affect only a small segment of your system, so bad code or an unexpected error doesn’t spell disaster for your entire service.

    During his talk, Doug suggested a few practical ways to accomplish this:

    • Gate features so you can put out code dark and slowly apply new features to a subset of customers.
    • Make sure you’re testing the changes on a representative sampling of your customer base: different account sizes, usage patterns, and configurations.
    • Find ways to slowly bleed traffic over from one system to another, to reduce risk from misconfiguration or cold infrastructure.
    • Run critical path code on the side. Execute it and log errors, but don’t depend on it right away.

    3. Make retries your new best friend

    Your system should be loaded with retries. Build them into all service layer hops, and use exponential backoffs to avoid overwhelming the downstream system. The requests between service layers must be idempotent, so that you can reissue requests to new servers without double-applying changes.

    Use queues where you don’t care about the response to decouple the client from the server. If you’re stuck with a request/response flow, use a circuit breaker approach, where your client library delivers back minimal results if a service is down—reducing front-end latency and errors.

    4. Don’t put all of your eggs in one basket

    Distribute your data to many servers, so that no one server is so important you can’t safely work on it.

    At PagerDuty, we use multi-master MySQL clusters, which help with operations and vertical scaling. We also use decentralized, linearly scalable databases like Cassandra for our alerting pipeline: since transactions aren’t gated on the behavior of a single node, we can do operational work during the day.

    Put together, these four strategies set an organization up to ditch annoying weekly downtime and help admins and operational engineers sleep more, worry less and maintain better—all ahead of schedule. Not only is this great for your team’s quality of life, but it also helps you deliver a great customer experience– no matter where your users are or what time it is.

  • IBM InterConnect: Disruption and the stakes of DevOps

    IBM InterConnect: Disruption and the stakes of DevOps

    It’s only a day into the main sessions at IBM InterConnect and it’s already clear that the most persistent theme that’ll be impressed upon the 21,000-strong crowd this year is one of business disruption. Woven throughout programming about cloud, mobile, analytics and DevOps innovation, the case studies already presented yesterday offer a testament to the disruptive power of IT transformation—and there’s more to follow in the next few days. Most important to the DevOps community, these stories don’t just show the hows of making good on DevOps, continuous delivery and iterative IT strategies. They also show the whys driving DevOps momentum.

    Sometimes the DevOps community can get so fixated on the intricacies of orchestrating DevOps practices and processes that it misses the forest for the trees. Ultimately, DevOps is about helping the business completely shake up product delivery models, customer interactions and core business processes through a makeover in how IT delivers its services. And so, it becomes clear through this disruptive message that the show this week offers an informative glimpse of what the stakes are for DevOps transformations.

    Throughout the day, IBM talking heads and a number of big-name customers explained how its become a eat or be eaten digital world across all industries as new technologies introduce new modes of doing business.

    “If you don’t believe that your industry is changing, then you’d better step back and take a look at what your competitors are doing,”  said Robert LeBlanc, senior vice president of cloud and software solutions for IBM, “because there are going to be disruptors and the disrupted. I’ll tell you that it is much more fun to be a disruptor than disrupted.”

    It’s a real concern in the financial sector, according to Citi’s head honcho for consumer digital experience, Heather Cox. In the general session yesterday, she explained that her two biggest challenges are caused by disruptive technology forces. The first is facing up to new entrants into the field of financial services that has had barriers to entry diminished by increased mobility and connectivity.

    “We have new people stepping into our landscape every single day,” said Cox, chief client experience digital and marketing officer for global consumer banking at Citi. “The payments and banking space is  being fundamentally transformed. It has been said that people need banking , they don’t necessarily need banks.”

    Almost equally challenging is the fact that her customers’ expectations for digital interactions are being rewritten by forces completely outside of her industry’s sphere of influence.

    “I no longer look to my competitors to try to determine where we need to go or how we  need to think about innovation,” she said. “I have the likes of FitBit, Uber, Nest–things that have nothing to do with banking but have everything to do with setting our clients’ expectations.”

    In order to meet head-on the kinds of challenges Cox outlined, IT executives are going to need to do a better job taking advantage of the data and the tools already available to them, explained Pascal Eymery, vice president of strategy and business development customer support

    “Look around 360 degrees, because I’m sure you have customers and suppliers and employees who have tons of data ready to be shared to create value for others,” he told the crowd, explaining that speed is paramount, especially as cloud and mobile technologies put continuous delivery in the realm of possibility. “One very good thing with new technology is its capability for agility; with cloud and mobile you can develop very quickly, you can try something and then you can scale fast if it works or just try something else if it doesn’t.”

    Fundamentally, both IT and line-of-business executives grappling with the decision to start a DevOps journey should ask themselves a big question, said Gary Gruver, president at Large Scale Practical Agile.

    “If you could start to say yes to all of the business requests coming in to IT and become two to three times more productive, would that matter for your business?” said Gruver, who explained in his InterConnect session yesterday how he was able to achieve that kind of productivity gain through DevOps practices while managing a team of 800 developers in HP’s printer firmware division.

    “As executives leading large organizations, if you felt like you could deliver 2x to 3x productivity as opposed to trying to go out and hire two to three times more people and ruin your cost structure, there probably isn’t  anything more important that you could be focused on.”

     

     

  • Dibbe Edwards, VP Rational Software Development at IBM, On Standardizing Tools, Process, and Technology to Build DevOps Success

    Dibbe Edwards, VP Rational Software Development at IBM, On Standardizing Tools, Process, and Technology to Build DevOps Success

    Dibbe Edwards, VP Rational Software Development at IBMWe had the pleasure, for IBM InterConnect 2015, to speak with Dibbe Edwards, vice president, Rational Software Development about the success IBM has had in implementing DevOps processes within their organization. In her current role, Edwards is responsible for leading Rational’s development business that includes application lifecycle management and reporting, quality and requirements management, systems development and architecture management, and integration and open software development across both the IT and Systems domains.

    Here is our conversation with Edwards about IBM’s internal DevOps transformation and the importance of, and challenges associated with, standardizing on tools, process, and technology to build DevOps success.

    devopsy.kinsta.cloud: Can you tell us a little about what your presentation will be about on Tuesday? It’s about how IBM has led its own internal DevOps transformation, and I think that is something that would be of great interest to devopsy.kinsta.cloud readers.

    Edwards: From an IBM perspective, we’re moving aggressively towards adopting DevOps practices. We have a larger software group, DevOps Center of Competency. The group is really taking these practices and the transformation efforts and rolling them throughout IBM. The session and the talk will kick off about that, and transformation at scale across the entire IBM company.

    devopsy.kinsta.cloud: In a presentation you gave at DevOps Enterprise Summit, you highlighted the challenge of standardization as one of the top three challenges in an enterprise move to DevOps. Could you share a little about what you meant by that for our readers, and maybe some of the lessons learned at IBM as you worked to standardize your people, processes, and toolsets.

    Edwards: We talk about how there are three elements to any transformation. That’s the process or the ways of working. There’s the culture of getting people to think differently and work differently. And then there are the tools. It has to be a combination of those three things that will drive any sort of successful transformation, you can’t really do one very well without the other.

    So standardization, or coming to a common way of achieving DevOps practices, is an important aspect because it gets people speaking the same languages. You’re able to have, then, a single version of the truth in terms of the viability of the project and where you’re going and you know that you’re communicating effectively. That’s been an important part of us being able to not only get this off the ground, but be able to scale it up at a company the size of IBM.

    devopsy.kinsta.cloud: What were some of the challenges that IBM encountered working to standardize? Did you identify those processes that worked and saw that they could be applied horizontally and then slowly put them in place across the different groups? How did you get buy-in to do that?

    Edwards: I would say the culture and the ways of working, or process, go hand in hand. I think if you don’t view that that’s the case, than you are in for a real challenge. If you try to just write down a bunch of processes and apply them in an organization without acknowledging and understanding the culture of that organization you are going to run into a lot of challenges. And getting buy-in from the organization is also a very essential part of it.

    What I’ve done in many of the organizations to drive towards a more standard way of working and modifying the process is to get the teams personally involved. What we did as a part of our transformation in both the web organization as well as the Rational team is start with a look at our current process and look at what was working and what was not working.

    We had set our business objectives. We wanted to deliver more quickly. We were delivering once per year. We wanted to deliver on a quarterly basis to speed up our time to market and get more valuable capability in the hands of our clients. We wanted to eliminate waste. That meant spending more money on innovation and less money on maintenance.

    What we did is we looked at our process from beginning to end. That meant everything from how we worked with our product management team, who help us make our business decisions and set our investment priorities, to a complete analysis of our development process, our testing process and our deployment process. We looked at those and we said, “What are the biggest inhibitors to us moving fast?”

    We built very specific plans. We actually established work group leaders to go and bring in other members of the team to determine how to go about addressing any inhibitors. We just started picking the biggest inhibitors, the ones that were our biggest pain points and put plans in place to address those.

    I think one of the important things about this is that you have to recognize that in order to improve the situation, you have to prioritize those investments along with your product investments. From our perspective, it certainly wasn’t something that could happen overnight. You also can’t stop the delivery train while you’re doing all this stuff. So, it has to be prioritized alongside the other work that you’re doing.

    Then the key thing is once you establish what your key focus areas are and you have the buy-in from the teams and recommendations from the different work groups, you begin to implement those things. At the end of each one of your sprints, you reflect on how you did and reflect on the improvements that you’ve been able to make and you identify your next set of inhibitors.

    It really then becomes a focus on continuous improvement, gaining momentum and reporting on whether you are really removing the inhibitors that you set out to remove.

    devopsy.kinsta.cloud: Are there a specific set of challenges around a large organization standardizing on toolsets?

    Edwards: One of the things that we’re big believers of in IBM and even within my organization and other parts of IBM is that there are certain things that you need to standardize on. If you’re managing a large program, you need to have the ability to, like I said, have a single version of the truth on the status of the project. So, things like how you do the tracking of your work, your dashboards and how you report on things, how you track your defects. The definition of a sev-1 defect should be consistent across teams that work together.

    But then underneath that, there’s flexibility that we give our teams in terms of tools that they might want to use. As an example, we support Git with our products. Sometimes we acquire companies and they want to use Git as their source code management repository. We’re okay with that. We just need to have this single sort of way to govern and manage the projects so we know that we’re meeting our business commitments and quality commitments that we make to the business.

    Another example is, from a testing point of view, I use in my organization Selenium for some of our automated testing because it was a solution that worked best for the type of Web-based situation that I have. We use some page object frameworks, which is a principle that is supported in Selenium. It helped us to be more efficient.

    The message I wanted to come across is that standardization is important with tools, but it doesn’t mean that every single thing has to be from the same vendor and it doesn’t mean that you can’t bring in things that work really well for your particular team. That’s another thing, I think, that helps from a cultural point of view is that there are some things that people can be pretty passionate about or have been effective for them in the past and bringing them into this overall governance structure is something that has helped us also sort of break down some barriers around people wanting to move forward with new approaches.

    devopsy.kinsta.cloud: What helps drive the cultural shift? I would imagine success begets success. When teams start to see success with other teams it probably makes it easier to get buy-in horizontally through the organization.

    Edwards: That’s true. There are multiple aspects of that as well. One of the key things that I have always believed in with the team is that you need to have some quick wins because, like you said, success breeds success. If you can get some good momentum under your belt with something that really has a positive impact, then it will get other people to buy-in.

    As an example, one of the things I talk about is our build time. It was a very frustrating thing that touched every developer and every tester in our organization because it took them a really long time to get a build. That was viewed as one of our key inhibitors. We spent quite a bit of concerted effort focusing on eliminating the amount of time that it takes to get a build. In many organizations, that can be an inhibitor.

    We had to invest in multiple different ways, but our initial leap was from 36 hours to 12 hours, which is huge, and along the way we had incremental improvements. Every time it got faster, it built more momentum and more excitement on the team and they could see immediate productivity impacts in a positive way for themselves.

    Another key area for us was implementing the use of patterns in our test organization. We found we were spending a tremendous amount of time setting up the test environments where we needed to actually validate our product and customer-like scenarios. One example is it took us 30 hours to setup an environment because we were doing it manually.

    We were able to implement patterns, which took us some time to build, but once we had those in place, we were able to deploy those environments in an automated way in three hours. We could then spend that 27 hours doing useful, constructive testing and being able to be more creative with our testing and finding more problems and being able to test so much earlier.

    Those are examples of things that were real momentum gainers for us.

    devopsy.kinsta.cloud: That’s fantastic. I think you’ve shared a lot of useful information for people who are thinking about going down this path. I would like to ask you about one objection, or belief, that repeatedly comes up in my conversations with enterprises. They’ll say that they can’t embrace DevOps in their organization because they don’t have the resources. They’ll say “Well, that’s Netflix or that’s Amazon doing those things or that’s IBM, a software development company doing that. We can’t do that because our core competency isn’t software development.” What’s your response to those types of positions?

    Edwards: Definitely. I think as I look at this, I’ve spoken had, myself, almost 50 briefings last year just talking to clients about this topic. People in every single type of business, everything from manufacturing to banking to insurance to retail, everyone is under a similar pressure: “How do I respond more quickly to the market? How do I make my company more competitive?”

    The reality is that IT is a very critical part of that today. There was a point where maybe IT wasn’t a differentiator. But today, IT is absolutely core. So, what is this all about? It’s about being able to be more agile and interact more rapidly with their clients and provide value-added capabilities to them and get their feedback. I think that certainly the business need is there.

    Now, as you start to think about some of the companies that aren’t running their business off of this, we talk a lot about some of the challenges: like how do you get the stakeholders brought in to want to participate in doing this type of development? What I’ve seen with many clients that are not “IT companies” is that they usually start with a pilot of some project that’s a reasonable size. And they have agreement from their business stakeholders and from the IT organization and they bring all their testers and developers and their operations and their business stakeholders together and they run some pilot projects.

    When they do that, the types of results I have heard insurance companies and banks have from such pilot teams have been outstanding. They’ve been able to get wonderful feedback from their stakeholders around how much more the application that was being built addressed their business needs. How they got into production so much faster.

    The types of results were excellent. That has helped, I’ve found, to drive the momentum to other parts of those of businesses. Because at the end of the day, just like the rest of us, it’s about return on investment. It’s about having confidence that what you’re doing is the right thing. This process is really about helping to shine the light on that. And using these types of DevOps approaches is about doing that.