Category: Enterprise DevOps

  • Dev loves Open Source. Ops needs Scale. Now What?

    Dev loves Open Source. Ops needs Scale. Now What?

    Crossing the divide from Dev-Ops to DevOps:

    At the heart of the divide between Dev and Ops are the very different needs and constraints of each organization. Traditionally their day-to-day lives are very different. So are their war stories. And so are the metrics that are traditionally used to judge Dev and Ops. Predictably, the two organizations can end up internally competing with one another over (among other things) resources and IT automation tool selection. Part of the driving force behind the DevOps movement is to reduce this unproductive internal competition. Even with the increased levels of trust and communication that DevOps fosters, IT Operations teams will still have different criteria for IT automation than their colleagues in Dev.

    My company is considering implementing (or has just implemented) an open source configuration management tool. That’s great for Dev. But as far as I can tell my job, and my team’s job, is about to get a LOT harder

    ~ VP IT Operations

    Dev loves Open Source:

    Take configuration and release automation as example. Like so many other areas of enterprise software, many developers naturally gravitate to open source options. It’s no wonder that 90% of code in modern applications is open source – after all, open source provides developers with

    • Immediate gratification: Just Google it! Choose the right key words to describe your situation and you might find a good match.
    • Community: Don’t reinvent! Find out what have others done.  Opinions and suggestions might be plentiful across your favorite user groups. Maybe someone has solid sounding advice. Maybe someone even has specific settings or scripts that they have used. If so, you might be in luck.
    • Free: Enough said.

    But while Dev is drawn to prebuilt open source components that they can then modify,

    Ops has completely different needs including scale, stability, support, security and speed.

    Let’s look at each of these in turn:

    • Scale: It’s great that the code works in Dev and in Test. In Operations, I need it to work everywhere, all the time. And we need to add 10s of thousands of servers with no new resources.
    • Stability: Don’t force me to rebuild my infrastructure to be stable in a production environment.
    • Support: Someone, somewhere needs to be on the hook for this software, with enforceable SLAs. This cannot be left to chance. I need a phone number. Now.
    • Security: Failure is not an option. Our financial future and reputation is on the line. We cannot allow our customers to become vulnerable due to security issues. According to a study by Forrester Research, consider:
      • “One out of every 16 downloads of open source component requests are for a component with a known vulnerability.”*
      • “31% of companies have had or suspect a breach in an open source component, gaining control over open source risks is essential to improving security.”*
    • Speed: We need quick time to market and low total cost of ownership that comes from quick learning curves.

    In the context of configuration and release automation, can we satisfy both Dev and Ops?

    Yes. Here’s why:

    A new generation of application configuration automation solutions have been introduced in recent years. (To be transparent my company, OrcaConfig is one of these).  These solutions cater to Ops’ need for fully supported, central, secure control, ecosystem visibility, workflow automation and compliance enforcement – all at enterprise scale and without relying on open source, community driven scripts.

    But what about Dev’s needs?

    I will answer a question with two questions.

    1. Does Dev need open source?
    2. Or do they want the benefit that open source often brings?

    If those sound like product requirements document (PRD) type questions, they should. In an era with tightening constraints on budgets and headcount, selecting the wrong tool based on incorrect assumptions is a major self-inflicted wound.

    These next generation DevOps solutions provide Dev with application-centric modeling (after all it is all about the application), templated workflows (immediate gratification) and versioned full stack configurations (time and money savings) while providing accountable, knowledgeable support (community is great. Community and support is even better.)

    While there are no silver bullets in the drive to DevOps success, my hope is that we can dispense with the open source “requirement” and instead focus on needs and results.

  • Why NoOps is a DevOps disaster waiting to happen

    Why NoOps is a DevOps disaster waiting to happen

    Ever since the first hunks of Big Iron hit the streets, we’ve been struggling (and failing) to keep up with the demand for software applications.

    Now it’s getting harder. Not only do we have to satisfy an insatiable appetite for new apps and services, but the nature of the business palate is changing. Business now demands a continuous flow of software innovation over back-office application support – or using the haute cuisine analogy – they want IT to serve a classy degustation menu versus just warming up a ready-made TV dinner.

    All this bodes well for the fast iterative style agile, with fully autonomous development teams empowered to increase the rate of software deployments.

    Sound great, but what does this mean for IT Ops?

    Many would argue that IT Ops is a completely defunct or redundant function. Even credible analysts and thought leaders have touted NoOps (No IT Operations) as a viable option. Some even suggesting that it accelerates DevOps collaboration since by removing the Ops function you remove all friction.

    There appears some validity to these arguments. After all, if developers can codify operational functions into their activities, why do we need the disciple?  Perhaps this logic explains why for some, IT Operations as a function is becoming disenfranchised; no longer having a voice into critical digital transformation initiatives – or if it does, just the sound of “No” – the last deployment bottleneck.

    The death of IT Ops has been greatly exaggerated

    All this NoOps thinking is flawed for a couple of reasons.

    Firstly, however operationally awesome developers think they are, building resilience, maintainability and supportability is not always top-of-mind. Worse still, these elements might be neglected if management is fixated on rewarding developers according to ‘speeds and feeds’. Secondly, even if these elements are addressed, they’re often conducted at the end of development cycles or bolted on after problems are discovered – that’s like baking a cake, forgetting the sugar, and then trying to compensate with a sickly sweet chocolate sauce.

    In reality, great operations engineers are best equipped to help incorporate operational excellence into all practices. After all they have years of experience supporting every new wave of technology – from Mainframes to Microservices. What must change, however, is how their expertise is developed and shared.

    • Become cloud connoisseurs – Like our development colleagues, operations’ emphasis will shift from on-premise to cloud. This means ensuring Ops expertise is fully leveraged to support the business nirvana of delivering high-quality digital customer engagement at scale. So if you’re an IT operations professional still configuring QoS policies on routers or manually provisioning development environments, it’s probably time to skill-up in PaaS, Amazon EC2 and containers. Either that or go and find a gig with a cloud service provider.
    • Embed Ops In Dev – This means being less separatist and silo’d and more inclusive and unified. Rather than focus on technical diagnostics and reacting to failures, the new operations professional will apply systems thinking to holistically look at how business applications and the all-important customer experience can be improved. This is easier said than done with skeptical development teams, so it’s essential that monitoring feedback is automatically shared and incorporated into all practices – like for example, agile sprints
    • Embrace Lean thinking – this involves being less interrupt-driven – fixing one technical problem after another, to working side-by-side with other teams to ensure a constant flow of value is delivered to business with all waste removed. As with our development colleagues, elements of agile (and common sense) thinking will come to the fore – with terms like “never done” and continuous improvement becoming established elements of the new operations and performance management mantra.

    Perhaps the biggest change IT Operations will undergo is how it interfaces with other teams. Here, IT Ops will morph into providing sets of easily accessible and repeatable processes other teams will use to bake quality into everything developed and tested. To this end, operations will transform from a separate “keeping the technical lights on” function, to a high-value to a craft that outfits the software factory and staff with new capabilities they need to ensure speed and quality at scale.

    The new age of Agile Operations “craftsmen”

    For IT Operations this means letting go of many traditional admin tasks – not just running things, but participating in crafting new automated processes. So rather than routinely monitoring applications in production, the Ops in DevOps will ensure this capability is available in pre-production. Rather than mundanely provisioning servers from change requests, new teams will equip the organizations with a complete release environments that support program level goals.

    Operating as DevOps “craftsmen” makes complete sense because it moves expertise out from behind the production curtain. The organizational focus of IT operations positively changes too — from being technically good at describing and fixing problems, to being awesome at prescribing improvements that drive better business outcomes.

    Great organizations exploit their business models with flawless operational execution. With these models now being constantly re-shaped by software applications, IT Operations is more important than ever. Relegating it with NoOps thinking isn’t an option.

     

     

     

     

     

     

  • Avoiding Database Migration Pitfalls with Continuous Software Delivery

    Avoiding Database Migration Pitfalls with Continuous Software Delivery

    When people talk about continuous software delivery, they often focus on the delivery mechanism itself, such as automation tooling or branching strategies. What is frequently overlooked is the simple truth that continuous software delivery is worthless without continuous service availability. In other words, if you’re going to frequently update your service, your customers shouldn’t frequently experience disruptions.

    Despite all good intentions, the cause of many unintentionally-disruptive deployments are database migrations. This makes sense, as all other components of a service infrastructure are fairly easy to swap out or to roll back. Software directory symlinks can be easily changed, EC2 instances can be easily started, DNS entries can be easily updated, but performing a breaking change on a multi-million row table in a database that’s under heavy load? Not so easy.

    Successful Migrations

    There are three necessary requirements for a database migration to be considered a success:

    1. The delivery of the service was not disrupted or degraded
    2. The database migration must be repeatable (re-entrant)
    3. The software version can be rolled back to the previous release if needed

    The first seems obvious, but can be surprisingly difficult to achieve. The chief reason for this lies in the vagaries of how different database technologies, and even versions of the same technology, handle things like table locking during an ALTER statement. The other reason is a hidden complexity within your application code. For example, a JOIN statement that causes a frequent query to be blocked by a seemingly unrelated table change.

    The second criteria means that if a migration fails for some reason, it cannot leave the data in such a state that the migration cannot be run again – essentially, running the same migration twice should not result in a negative outcome.

    The third criteria absolutely precludes ever doing a “one way” release where rollback is made impossible due to data destruction. It’s okay to have a release that rules out rolling back several releases; without that, the database would become a giant pile of discarded data. That said, it’s never a good idea to paint yourself into a corner with a release that you cannot possibly revert.

    Avoiding Service Disruption

    The simplest way to avoid service disruptions is to never perform a database migration. But, that just isn’t feasible, so how do we avoid stepping on a landmine during a migration?

    First and foremost, implement a policy of strict database migration reviews. This doesn’t need to be burdensome or time consuming. It just means that someone who is best equipped to understand the impact to the database technology actually gets eyes-on the migration before it is run. This is a role often served by your database administrator, chief architect or most senior developer.

    Another good idea is to set up a hook in your code versioning repository, or a test that’s part of your build process, that examines database migrations and looks for certain keywords – and the lack of other keywords. For example, it might “fail the build” if it finds an ALTER TABLE statement that is not accompanied by the word CONCURRENT.

    If you’re dealing with a lot of legacy code, or a monolithic application with significant complexity, then there are two very important safeguards you can put in place: load testing and SQL statement logging.

    Load testing means having a development environment setup where you can apply a database migration to a database that is under load, i.e., experiencing a level of usage that roughly approximates your production database load and which contains an amount of data similar to production. Then, while your database migration is running, you can run an application smoke test or similar to see if production-like load combined with the migration results in any unexpected slowness or errors.

    Remember, all database migrations are fast when they are the only thing running on your database, and faster yet when the testing database is a tiny fraction of the size of the real thing. Going into production feeling confident that a migration took only ten milliseconds to complete on a half-empty database with zero active connections is a fool’s errand.

    When you have a poorly-understood or monolithic application, it can be a very enlightening experience to enable SQL statement logging on your production database server. Remember that MyISAM table that never seems to change and against which it should be safe to run a giant blocking migration? SQL statement logging might reveal that every single page load executes a long-forgotten JOIN statement that scans that entire MyISAM table. Or that the user login process looks in that table for reasons nobody can remember. This is also an area where using a solid application performance tool like New Relic can make a very big difference.

    Repeatable Database Migrations

    This one sounds simple but can actually be a bit tricky to implement. Consider the following scenario, you have a table that contains three columns: id (integer), username (string) and password (string).

    As it currently stands, the passwords are stored in plaintext in the database, and your application does a simple string comparison between what the user enters and the results of a SELECT statement. The next version of your application is going to hash the password entered by the user, and compare the hash against a hash returned from the database by SELECT.

    There are two possible ways of getting from where you are now, to where you want to be – one is dangerous, and the other is safe.

    The dangerous way would be to create a database migration that iterates through the password column and replaces each plaintext value with a hash of that value. Basically a SELECT; md5(); UPDATE; loop. This is dangerous for a number of reasons:

    If the migration stops before completing, the database will be left in an unusable state where neither the old nor new code can make use of the entire table.

    1. If the migration fails to complete for any reason, it cannot be run again without corrupting all the previously-migrated values.
    2. If the new code has to be reverted to the old code for any reason, the table is now in a state that cannot be used by the old code.

    The safe, and repeatable way of performing this migration actually involves three releases:

    Version 2.0 migration adds a new column (pass_hash) and migrates password to it, and new application code must INSERT and UPDATE to both

    1. Version 2.1 application code must now use only the pass_hash column, and ignore the password column.
    2. Version 2.2 migration removes the password column, and new application code continues to use only the pass_hash column.

    You can rollback from the first release (2.0), as all the original, unencrypted password data still exists in the database, and the previous code release won’t care about the new column (pass_hash) being present. You can also run that first migration multiple times – just have it remove the pass_hash column if it exists prior to creating it.

    Next, you deploy code (2.1) that makes use of the new column only. You might be tempted to bundle the database cleanup migration with it, but don’t – if you need to rollback for any reason, you won’t be able to if you’ve removed the password column. It’s always a good idea to delay breaking changes, like a column deletion or dropping a table, by at least one release.

    And finally, run a migration (2.2) that drops the password column, if it exists. That latter condition is so you can run the overall migration more than once if needed. If the password column exists, remove it; otherwise, merrily continue with the rest of the release. Note that at this point in time, the live application code is ignoring the password column entirely. This is important, since the current application code will be using the database while your migration is running. In other words, any migration must be safe to run under both the current and the about-to-be-deployed code version.

    In case you hadn’t already guessed, this approach to migrations does take a bit more conscious thought and effort than the old, dangerous way. You’ll need to keep track of “deprecated” bits of your database for future removal. And you’ll need to occasionally write code to intelligently update two columns or tables in lockstep, and then remember to remove that code in the future. That said, it’s definitely worth it in the long run when you’re able to release frequently with zero downtime.

    Release Rollback

    Nothing produces longer outages and greater negative customer impact than being unable to rollback or revert a release. If you’ve accidentally deployed a major bug into production, you want to quickly revert to the previous version that didn’t have that bug, and then reproduce and fix the bug at your leisure in your development environment. Being unable to reproduce the bug in your development environment is not a valid reason to fail to rollback a release, and it’s an entirely separate discussion apart from migrations.

    The essence of rolling back is speed. It must be a fast and reliable process. The number one way to ensure it is anything but fast and reliable is to have a breaking or one-way database migration in the release you wish to roll back. Thus, anytime you are creating a new database migration, ask yourself two simple questions:

    1. What would happen if I ran this migration with the current application version?
    2. What would happen if I ran this migration twice?

    If it would change the database in a way that would cause the current application version to explode, then it needs fixing. And if it would change the database in such a way that it can’t be run again, then it needs fixing.

    The End Results?

    In a nutshell, this approach to migrations means that each migration will be very small and very simple. Expect to get from state A in your database to state B via many safe steps, instead of one giant, dangerous step. Your customers, and your application uptime, will thank you!

    About the Author/Nate Lindstrom, VP solutions engineering

    highres_250316251Nate Lindstrom is the VP of solutions engineering for NS1, an intelligent DNS and traffic management platform with data driven architecture and a unique Filter Chain routing engine. NS1 is purpose-built for the most demanding, mission-critical applications on the Internet. NS1’s comprehensive platform technology leverages infrastructure, application, and network data to make intelligent routing decisions in real time, ensuring optimal application performance and reliability.

     

  • TASC used pipeline orchestration to onboard DevOps

    TASC used pipeline orchestration to onboard DevOps

    I often talk about how development teams ignore their pipeline. It is easy to focus on specific tools and process, but by and large the pipeline that contains them is not being treated as its own entity. Your entire process is something that should be managed, iterated over, and measured just like the code within it. While pipeline oversight is rare, there are some companies doing it, and TASC is one. At the DevOps Enterprise Summit I was able to sit down with John Gildenzoph, a DevOps Engineer at TASC, to better understand how they took a holistic strategy to adopt DevOps.

    See John’s #DOES15 talk here and his interview with devopsy.kinsta.cloud’s, Alan Shimel along with Automic’s CEO here.

    TASC is a large health benefits provider based out of Madison, Wisconsin. They have a technology team of about 100, including Developers, QA, and Ops. Their development is supporting both web and mobile applications. The mobile application is a critical tool for their customers to submit claims and manage their information, and a core web application that serves both customers and employees. After listening to John talk, I discovered that TASC found their path to DevOps by addressing the pipeline holistically.

    The problem – an inefficient pipeline

    Around 2011 TASC realized they were struggling with an outdated development process, based on the waterfall methodology. According to John, ‘deployments were manual, inconsistent, and time consuming, and infrastructure was monolithic, static, and inconsistent.’ In addition to slow processes their was no transparency into what was going on and how processes were executed. This resulted in an inability to scale, and did not support the objective they had to package and deliver apps faster, and at a higher quality.

    The solution – automate the entire pipeline

    They could have made some incremental changes to their processes to make them faster and more efficient, but this would ultimately lead them back to square one with all the same issues to fix sooner or later. They needed a new approach, one that can adapt and scale as their needs change over time. Especially with the rate that DevOps tooling is changing. John and team realized automating the entire pipeline is the only viable strategy.

    “How do we move all of this stuff forward at the same time, not just the development cycle? We wanted to move everything forward together so we did not paint ourselves into a corner” – John Gildenzoph

    It did not happen overnight, first they needed to know what they already had, and start somewhere. The first step they took was to automate parts of their development, QA, and deployment cycles. They started by automating builds with Jenkins. Jenkins being the standard in build automation, they didn’t want to reinvent the wheel and make a build automation tool themselves. These quick wins allowed them to start somewhere, and set them up for bigger wins down the road.

    The next step was to automate the entire pipeline. They realized that their needs will change over time, requiring new tools frequently. The same tools may not always be suited for the job, and new tools may rise to replace existing tools. Because single do-it-all tools tend to overpromise and underdeliver, building for the future required a ‘rip & replace’ approach to tools that makeup the pipeline. This being the case, TASC wanted to take a long term approach to managing and automating their pipeline. To achieve this they needed a pipeline orchestrator. This would make it easy to add and remove components from the pipeline as needed, and give them a holistic view.

    “we can re-evaluate … we have no real dependency on a tool. We just made a move from SVN to Git. I only had to modify a handful of SVN commands with Git and I was done.”

    To orchestrate the entire pipeline, they opted to use Automic Release Automation, an enterprise wide orchestrator. A tool they already owned but were not fully leveraging – “we basically used it like Cron to move files around” John explained. The new implementation of Automic not only abstracted their existing production deployment but also made the interfacing and execution of that deployment process simple enough for non-operations users to handle. It enabled deployment to production environments which required a sophisticated workflow to be automated.

    While Automic could in some cases be seen as a competitor to Jenkins. In their environment considerations for existing business applications was important, a place where Atomic apparently excels, which provided a greater breadth than Jenkins alone could provide.

    In addition to Automic, they are not afraid of leveraging tools. They incorporate many tools to help with automation, but also governance like Sonatype Nexus a component and artifact repository. Sonatype helped how they packaged and delivered artifacts and simplified the consistent deployment of those artifacts. And Docker to get closer to full-stack deployments and supporting their micro-services setup of around 20 services.

    The outcome – Faster delivery, higher quality

    The automation was not the only benefit. TASC found that automation is a great way of driving process change through the organization.  By taking the time to examine the current process and asking the hard questions of why they were doing certain things actually brought a lot of things to light and in the process of automating they actually improved the core processes.

    The new orchestration engine led to increased visibility into development, better QA process, and more stable infrastructure. Easy workflow monitoring meant that even business stakeholders could now gain visibility into the pipeline. All this combined to drastically reduce deployment times to less than an hour, and with zero downtime.

    The only way to break the cycle of inefficiencies in a software delivery pipeline is to automate every cycle within it – Dev, QA, and Ops. This can’t be a half-hearted, or partial effort, it needs all hands working together to overhaul every section of the pipeline. While this may sound daunting, the payoff is worth every ounce of energy spent on it.

    Adopting a pipeline orchestration tool like Automic that allows you to ‘rip & replace’ tooling whenever you see fit might not be an immediate need, but helps sustain the environment in the long term and ensures you can adapt to what comes. This takes your focus off the tools, and puts it where it belongs, on your software delivery strategy. By doing so it’s easier to adopt DevOps and enhance your pipeline in the future.

    TASC made the transition to modern development without calling it ‘Devops’. This goes to show that as long as you’re concerned about delivering higher quality software faster, a Devops-like methodology is where you’ll naturally land. Pipeline orchestration might be a new concept, but from an execution standpoint it is not new at all. It has been executed successfully within even large environments, and is the easiest way to make sure your pipeline supports current and future development.

  • Making Over the Mainframe for DevOps

    Making Over the Mainframe for DevOps

    Introduction

    As a discipline, DevOps emphasizes mitigating any and all constraints in order to roll out high-quality, high-performance software, faster. Constraints can be man-made – such as lack of communication between developers and IT operations teams – or technical in nature, such as an overcommitted testing environment. DevOps teams are only as strong as their weakest link. Ironically, the strongest computing platform available in the

    world – the mainframe – can be an Achilles heel in DevOps.

    Since the 1990s, the mainframe has been a target of many jokes, being far less “en vogue” than its modern, distributed architecture counterpart. But the reality is, no computing platform comes close to matching the speed, reliability, security and cost-efficiency of the mainframe. When managed properly, the mainframe can unleash tremendous competitive advantage as the transaction-processing engine at the heart of modern applications. But if it is not properly evolved and left to languish, the mainframe can quickly rear its head as an unwelcome bottleneck.

    This article will explore the current state of the mainframe; why mainframes have the potential to hamper even the most nimble DevOps teams; and how mainframes must continue to modernize in order to keep up with the pace of DevOps and today’s hyper-competitive application delivery landscape.

    Mobile Means The Mainframe Remains Firmly Embedded

    In contrast to many industry prognostications, the mainframe isn’t going anywhere, due to the unparalleled strength of the platform and the billions of lines of code that dictate much of an organization’s business processes. Consider this:

    • IBM’s new z13 mainframe is capable of handling 100 Cyber Mondays every day, 365 days a year.
    • The volume of CICS (mainframe-based) transactions on any given day dwarf the number of Google searches, Twitter tweets, YouTube views and Facebook Likes combined, according to IBM.
    • Mainframes currently process 30 billion business transactions per day, and enable $6 trillion dollars in card payments annually.

    Most organizations find that moving off the mainframe is too risky, too costly and too time-consuming – and with the technology advances in the platform, just plain unnecessary. Numerous instances have shown mainframes to be more cost-efficient than distributed commodity infrastructures, because reductions in commodity server pricing have not kept up with the growth in computing loads spurred by mobile.

    Mobile computing loads are growing at a dizzying pace. Recent research from the global payments industry shows that more than 27 percent of global online transactions are now conducted on mobile devices. And increasingly, these mobile transactions need to pass through mainframes. Consider a mobile banking application, where users snap a picture of a check in order to instigate a series of steps leading to a deposit. Ultimately, this transaction takes place on the mainframe and must connect through it.

    How Mainframes Can Pose a Challenge to DevOps

    Traditionally, mobile/distributed teams work in very different silos from mainframe teams. Each with their own distinct culture. Unfortunately, the characteristics of mainframe systems can cause an overall software development effort to encounter slowdowns, or even come to a screeching halt.

    As one example, consider a mobile developer working on a hybrid application needing test data that resides on a mainframe. Developers and testers must be able to access test data quickly and easily, in order to adhere to compressed go-live schedules. Any challenges to accessing this data – such as not understanding how and what data flows through the program – leads to unnecessary delays. At a micro-level, this negatively impacts the effectiveness of DevOps. But at a higher level, ongoing obstacles like this can quickly snowball, and in an era where “software defines the business,” the organization’s competitive position can be compromised.

    What’s more, mainframe experts (primarily baby boomers) are expected to retire en masse over the coming years. According to industry research, 55 percent of today’s enterprise apps still touch the mainframe, and more than five billion new lines of mainframe code are being added each year to a base already exceeding 230 billion code lines. A sudden loss in the expertise responsible for this intellectual property can expose a company to major risks, or even disasters, and managing it cannot be left to chance.

    Evolving the Mainframe to Keep Up with DevOps

    For organizations looking to distinguish themselves through cutting-edge mobile transactional applications, the mainframe can be a key competitive differentiator. But in order for this to happen, it must be continually nurtured and non-mainframe trained resources must be enabled to support it.

    Over the past year, mainframe ISVs have driven tremendous advances in this area. These ISVs are adopting Agile development processes themselves, in order to help their customers bring the mainframe into the DevOps fold; extend the value of their legacy investments and in the process, become more competitive in today’s fast-moving markets.

    These advances include:

    • Windows-like interfaces and functionalities (for example, copy and paste) on the mainframe: This allows modern developers to work in an environment they’re familiar with, while providing tools that enable them to easily and quickly complete such tasks as dragging and dropping mainframe data from one host to another. Mainframe-specific expertise is no longer required to complete simple tasks.
    • Program and data visualization on the mainframe: These capabilities enable developers – both mainframe and non-mainframe experts – to better understand program and data interdependencies; find and fix mainframe code issues and build the highest quality mainframe code possible.
    • Insights into the performance of code created with the world’s most common programming languages, such as Java: Java developers can truly “write once and run anywhere,” including the mainframe, and ensure the smooth connectivity of applications traversing numerous platform types.
    • Visibility into the often-complex interactions between mainframe programs: This makes it easier and faster for developers to understand, modify and troubleshoot even the oldest, most complex and/or poorly documented mainframe code. Developers can shave days or even weeks off mainframe software updates, while reducing unintended mistakes.

     Conclusion

    When barriers to enabling DevOps on the mainframe are removed, the positives benefits extend beyond creating a more agile DevOps team. Newer generations of developer talent become attracted to mainframe careers, drawn in by the prospect of working on the some of the most exciting, new mobile transactional applications that feed the world’s economy. And with everyone working in a common, visual, intuitive and modern environment, the knowledge transfer between newer developers and more experienced generations of mainframe experts becomes more fluid and seamless, protecting much of a company’s DNA – its mainframe code.

    The mainframe has a very real place in modern application delivery and DevOps. However, mainframe users require support in bringing fast and agile DevOps best practices to bear on their mainframe-resident applications. With new advances, the mainframe can successfully evolve from yesterday’s computer, to an innovation engine suited to modern DevOps teams.3

    About the Author/Chris O’Malley

    Chris OMalley2Chris O’Malley is CEO of Compuware. With nearly 30 years of IT experience, Chris is deeply committed to leading Compuware’s transformation into the “mainframe software partner for the next 50 years.” Chris’s past positions include CEO of VelociData, CEO of Nimsoft, EVP of CA’s Cloud Products & Solutions and EVP/GM of CA’s Mainframe business unit.

    Follow Chris on Twitter: https://twitter.com/chris_t_omalley and LinkedIn: https://www.linkedin.com/in/christophertomalley

  • To Be Continuous w/Special Guest Kris Gale: Effective Team Structure for Continuous Delivery

    To Be Continuous w/Special Guest Kris Gale: Effective Team Structure for Continuous Delivery

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Kris Gale about effective team structure for continuous delivery. This podcast is brought to you by Heavybit, a 9-month program for developer startups.

    Screen Shot 2015-11-13 at 9.33.45 AMKris is co-founder and CTO of Clover Health, a unique health insurance plan focused on driving down costs and producing improved health outcomes. Formerly, Kris was VP of Engineering at Yammer.

     

    Edith Harbaugh, CEO LaunchDarklyEdith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

    Paul Biggar, CEO CircleCIPaul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     


    [soundcloud url=”https://api.soundcloud.com/tracks/232834454?secret_token=s-3aRuL” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]


    Edith: So, Kris, what do you like best about continuous delivery?

    Kris: So, I like continuous delivery, because I think it’s sort of the first step of building a learning organization. It’s sort of a fundamental capability that a team needs to be able to learn collectively about what it’s doing in the world. The first step is being able to sort of ship increments of value continuously.

    The second, I think, is to be able to measure the result of any incremental change you’ve made, so you can back out of a bad path that you’re on, or sort of refine what you’re doing and know that quantitatively.

    The next step then, is to move from a sort of historical corporate planning, so this is a our six-month milestone, our two-year roadmap, our marketing commitments for Q3 of next year, into sort of an assumption that if we are shipping continuously and learning continuously because we’re measuring what we’re doing, whether it’s quantitative or qualitative measurements.

    The next step is really being able to assume that in your planning, and know you’re going to have more information in the future than you had in the past. So your plans that you made six months ago are necessarily lacking information than plans you would make today would have, so sort of institutionalize that.

    And then kind of the last part, and this is all kind of enabled by continuous delivery, the last part is building organizational fluidity.

    So as an organization, whether it’s your project planning structure, or your accountability to your work flows, are fluid, so if you learn something new, you’re not locked into, oh we have a sixth person, I don’t know, sign-up flow team, and we just discovered that we have no levers for moving engagement right now in the sign-up flows, so we’re committed to that and we’ve got this implicit prioritization that’s built into our org chart. So, to have organizational fluidity where you’re adapting the organization to what you’re learning continuously.

    Edith: Oh. You know that’s fascinating. Paul and I have talked a lot about, I think, more the mechanics of continuous delivery. We put it as kind of the last step to software org, when you’re really talking about how that changes entire organization beyond software. So, this would be a great time for you to introduce yourself, and tell us more about you.

    Kris: Yeah, so I’m Kris Gale. Most recently, I was the VP of Engineering at Yammer. I heard you talking about Yammer in a previous episode. I’d love to dig into any of that, if you’ve got questions. Currently, though, I’m the co-founder of a company called Clover Health. So, what we’re actually doing is offering health insurance for seniors, with the idea that we can fill gaps in care, and actually prevent hospitalizations.

    We’re not trying to be people’s doctors, but we’re trying to look across their consumption of health care and say, “Oh, here’s an opportunity ‘to inflect the probability with which ‘this population of people will be admitted to the hospital.” So, for example, making sure they get treatments regularly, get follow-up visits with specialists, that particular bio-markers don’t deviate outside of a specific range. And all this stuff is really interesting to me, because the health care industry as a whole doesn’t change what it does continuously.

    We’re sort of locked into what people learned in medical school, or what people learned on their first day on the job at, say a big health insurance company, or something like that. But, to be able to power workflow with software, you’re able to continuously iterate and continuously refine and optimize the intervention protocols that are leading to these prevented diseases.

    Edith: Yeah, I mean, I heard that you were pretty good at data loops, and that this is just basically using data loops for good, I think is how you say it.

    Kris: See, the way we were talking about data loops, we were actually talking a little bit before we walked up here, is this idea that there’s something you intend to do. There’s data that you collect in doing it. There’s your objective assessment of that data to determine whether you’ve added or removed value or improved something.

    And then closing the feedback loop, to get back now to actually modifying what you’re doing, and form find out new data. So, that’s sort of the data feedback loop, is from doing something to measuring it and deciding to do something differently the next time.

    The tighter you can close that feedback loop, the more opportunities you have to optimize in a short amount of time.

    Paul: So, it’s kind of what you might call the shipping loop, if you implicitly believe that data is a key part of shipping.

    Kris: Yeah, yeah, and I would say that I think I do.

    Edith: No arguments here.

    Paul: Well, I think it is interesting, because I think that a lot of people don’t actually have measurements as part of their feedback loop, and especially, I think this especially happens in a small organization, where you have a start-up that sort of comes out of a founder’s vision, or something like that.

    And so, one of the questions that I’m very interested about is, you were doing a big thing, with lots and lots of people. And now you’re doing a very small thing with, I presume, a small handful of people. How much of what you’ve learned still applies at that very small scale?

    Kris: 

    So, yeah, a lot of the project workflow strategy that we came up at Yammer was really about how do you maintain that agility, that velocity of value delivery, and ability to react to new information, and not be locked into, “Well, this is just the way we do it,” in the face of new information. At scale.

    So, Yammer’s whole process was really about, how do you take these advantages we had, when there were like four or five of us in the really early days, and not have that big bureaucratic company, “Oh, we know we should be doing something better, “but we’re not,” because of momentum, inertia, bureaucracy, all these things. Like, how to break that. So, what’s cool about starting over with a much smaller team is, we’re just there again. We’re back at that five person size.

    Edith: You’re a lot bigger than five people.

    Kris: We are now, yeah. So, like nine months ago, we were. We’re like 65 people now, already. It keeps growing really, really fast. It’s been really helpful to me to have been through the process of scaling a software engineering team and seeing what that looks like. So, it’s like, “Oh, I’ve seen this problem before, “not worried about this.”

    Paul: So, when you started, did you immediately set out organizational goals, and mission, and values, and that sort of thing that would help the company scale it?

    Kris: Yeah, no, that’s a great question. I’m obviously a really big believer in the way we approached this problem space at Yammer, but I didn’t want anything to be dogmatic. I didn’t want anything to be–

    Paul: You wanted it to grow organically.

    Kris: Yeah, I wanted it to grow organically, to a certain point, but there have definitely been times where I stepped in and I said, “Nope, I’m buying a physical board “to track our projects, “because, just the way we’ve been doing this “with sticky notes, “I know that it needs this other dimension “of information capture. “I know that we’re suffering because it’s not there.

    “So, I’m gonna actually pull the boss card and do this.” There have only been like three or four times I’ve done that in the last year, but it’s things where, like– Okay, I’ve seen where this goes off the rails, if we don’t address this right now.

    Paul: And is that a result of knowing, sort of directionally, how the company should be doing things, or setting things up in some way that you didn’t get, that you didn’t end up in these bad places?

    Or was it all sort of retroactive, and you know, a couple places where you pulled the boss card?

    Kris: Yeah, it was mostly, like, observing patterns, saying, “Okay, we made this mistake three or four times, “and I’m noticing that our process or our approach here “is divergent from a known good approach “that I’m aware of.

    “So, let’s at least–” I still wanna give the team full empowerment to iterate and own the process themselves, but let’s at least just pull the boss card, get it back on the rails, and then say, “Okay, let’s see how this works.” Just give everybody a chance to work in a way that I know to be okay for a few iterations, and then give them the keys again, and say, “Okay, take this where you want.”

    Edith: Also, Kris, what I’m dying to know now is, what are those three or four key things where you gotta play that card?

    Kris: Yeah, so, one is single project focus. Really, really important to me for software engineering teams. So, what I mean by that is, if you are on a project, that is your only accountability. You are not being split between three deliverables.

    The other is that everybody work in cross-functional teams. So, that means not giving, for example, I might tell you, “Build out the back end for this feature, and I’ll do the front end.” So, in order for us to do that, we would have to agree to the abstractions ahead of time.

    And again, I’m pretty opposed to planning, and locking in planning, because the information that we have now, when we’re trying to lock in those abstractions, is going to necessarily be incomplete.

    Edith: I love that quote, because it’s so true. I think you put it more, it’s like you always know more in the future. It seems so obvious.

    Kris: Yeah, I was actually– So, we had Parker from Pivotal Labs over at the office yesterday, and he said that there was this saying they had a Pivotal, that I really, really liked. He said somebody there would always say, and I wish I could remember who it was, would always say,

    “You will always make better decisions with more information, and you will always have more information in the future.”

    This is so much more concise than any way I’ve ever said that. But I think it exactly captures it.

    Edith:

    But, on the other hand, my old boss, Gregg Brockway the founder of TripIt, he always said, “A good enough plan today is far superior to a perfect plan tomorrow.”

    Kris: Absolutely.

    Edith: He’s like, “You just gotta start moving.”

    Kris: Yeah, but I think the key is not locking yourself into that “good enough” plan. To allow yourself the mutability of changing that plan over time, so you’re not constrained by that.

    Edith: Yeah, but that was TripIt. I mean, I think he’d seen so many, because he came out of the travel industry, where people just got paralyzed. Like completely paralyzed. “Oh, we need to plan for five years, “and what business travel will be like then.” It’s like, well, nobody knows.

    Paul: I think the kind of key point to that is that in order to get that information, you have to get moving today with whatever information you have available. Otherwise, you’re just not gonna have that future information. You’re gonna be in the same place.

    Edith: Well, look, sometimes, though, it’s painful. Sometimes you move backwards, then. Like, you don’t always move forward. Sometimes you start moving, and when you’re looking back, you’re like, “Actually we took, like, three steps back “before we started moving forward.”

    Kris: It can definitely happen.

    So to complete that thought, when we assign teams, we would always assign a cross-functional team with a business objective. So, it would never be I give you a back-end task, and I take the front-end task. It would be collectively, let’s accomplish this business goal.

    So hopefully, we’re iterating day by day on the abstractions. So, we are collectively accountable to one another, and to this business goal we’re trying to deliver. Not onto some specific set of tasks that we broke off when we first decided to assign out the work.

    So, that’s one of the other ones, is always cross-functional teams. Always single project focus within those.

    Paul: So, there’s an interesting thing that you’re talking about there. It’s not just cross-functional teams, but it’s also that those cross-functional teams get put together for a specific project, is that right?

    Kris: That’s absolutely right.

    Paul: So, are those teams teams that will have worked together in the past? Is there explicitly trying to make sure that the people know how to work together?

    Kris: Yeah, so this is one of my other core principles that I am pretty, pretty set on, which is rotation, is this idea that cross-functional teams are rotational and ephemeral. So, that would mean that maybe the three of us come together to ship a feature.

    We’re all within the same organization, but when we’re done shipping that feature, we’re gonna break up.

    You might be the back-end person that goes onto another iteration of this feature area, or maybe, you know, the two of us work on some other feature. Maybe it’s three completely different people. But constantly rotating people around, I think is really important in an organization.

    I have totally heard the arguments about, you know, you can build high trust teams if they’re stable and stay together. But I’m trying to optimize for trust across the entire organization. So, across Yammer, we had 220 people in engineering. They would all rotate through these two to 10-person cross-functional teams together.

    Which meant that you, basically within three to six months, would work with a pretty significant portion of those 220 people, and would start to build trust across the larger organization. Whereas had we just torn off four people and said, “You’re a team,” they would– And I’ve seen this throughout my career in engineering, where it’s like, “We do everything right, and that team has no idea what they’re doing.”

    Edith: It’s their API’s are bad.

    Kris: Yeah, yeah they use this weird language, or this approach, or we use this style guide, or whatever. So, it sort of forced that rotation, so that we’re all sharing best practices among one another and building, maybe lower trust, maybe not getting to that very, very, very highest level of trust among a group of four people, but across the entire organization, getting to like 40% or 60% of what’s possible versus 0% if you don’t ever work with those people.

    Paul: So, interested in structurally how that ends up, because one of the things you got when you split people up by team is you end up with a sort of microservices architecture.

    And this team works on this repo, or this project, or whatever works on this service or this API. And when you have those teams that are forming and splitting up and they’re ephemeral, and it’s only a small period of time, architecturally, what does that do to your application or your architecture, you know, across your entire product?

    Kris: Yeah, no, totally. It was actually very intentional that we ended up with that kind of architecture for working this way. For two years, we had a big monolithic Ruby on Rails code base, and for two years we said, “This is a big problem. We should break this up.”

    And everyone agreed, and everybody was on board with breaking it up, and two years later, we had never done it, because we continued to work in such a way that supported that sort of architecture.

    So, the key for us was to say, well one of, the– So, something to dig into here a little bit, is these cross-functional teams are made up of representatives of what we call functional teams. So, front-end engineering might be one.

    Paul: Right, right.

    Kris: We had one called Core Services, which I actually owned the service-oriented architecture that you’re describing. Or there’s little micro-services that collectively made up our service-oriented architecture.

    Paul: And this is sort of a matrix style, where you’re a member of the team, but you’re also a member of the back-end team, or you’re a member of the front-end, or the core services, or whatever.

    Kris: Yeah, so I might be a member of core services. My manager is one of the core services managers. But all the work I do is going to be within the context of the cross-functional team, with the front-end engineer, maybe a client’s engineer, whoever’s necessary to get end-to-end with a value proposition.

    And then, I’m always going to be a core services member, when I go into these cross-functional things, but it’s the fact that all the members belong to functional teams that makes that team cross-functional.

    For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.

  • 4 DevOps strategies to stop your application having sand kicked in its face

    4 DevOps strategies to stop your application having sand kicked in its face

    While recently thumbing through some old comic books, I got a laugh from all the quirky advertisements. Fantastic promotions of awesome gadgets – like x-ray specs and radio wrist watches. Each designed to extract a gullible kid’s hard earned allowance. Of course they were all too good to be true and probably left many kids disappointed — and maybe like me, just a tad cynical.

    I spied one ad that actually got me thinking about DevOps. It’s the famous “kick sand in your face” strip, which to those unfamiliar offered “98 pound weaklings” a sure fire program to build muscles, beat up bullies, and make lots of new friends down at the beach.

    In IT we also have weakling apps we wish could be stronger. Services so brittle and fragile that any proposed change, update or enhancement is seen as risky – perhaps even career threatening.

    So is it possible to beef up these veritable data center weaklings? Can we also find a way to turn the tables and make our apps stronger?

    Well yes we can, but only with some new thinking. But definitely not (if like kids reading comics), we get sucked in by outrageous DevOps claims, along with a raft of gadgets and tools that never quite live up to our expectations.

    Here are four things to consider as you look to toughen up your apps:

    1. Problem fixing is on the decline – successful businesses finally get that mobile and cloud is the only way to engage customers at scale. This means IT operations won’t have nearly as much stuff to keep watch over in the future. While some mourn the loss of visibility, it actually forces us to do a much better job at what’s really more important – ensuring an awesome customer experience. IT Operations as a discipline will no longer be judged on how effective it is at fixing problems, but on its ability to detect service issues before they impact customers. Therefore app toughness means less after-the-fact causal determination and more analytics for pre-emptive strikes on emerging apps problems – especially those about to bite the business.
    1. Issues now lurk in the “shadows” – developers now have lots of control. With the ascendance of agile and new techniques like test-driven-development and infrastructure-as-code, activities like build, configuration management and provisioning can be codified into work streams. This leads many to argue that operations as a function is defunct (or at least soon will be). After all, if operations doesn’t need to be involved, why is the function needed at all? Unfortunately this “shadow Ops” logic neglects to consider an inconvenient truth – basically that performance and supportability are not top-of-mind issues for developers. So while we have faster lead times, the reality could be a steady stream of weaker apps. That’s what you call “failing fast”, but not as DevOps intended it.
    1. Agile Operations is about “Craftsmanship” – since developers can’t always be counted on to instill quality, its incumbent on IT Operations to help teams build these capabilities. This doesn’t mean trying to impart years of ops knowledge, but actually crafting this expertise into automated solutions – solutions others will use to improve the performance and supportability of applications in context of the actual task being conducted. This could include:
    • Making more accurate performance data available during development so as to code and test against realistic conditions
    • Integrating change management processes with release automation so as to activate change approvals and co-ordinate all workflows – essentially ensuring compliance while eliminating typical delays from manual handoffs
    • Providing earlier guidance to developers on code defects by incorporating standardized, consistent and scalable monitoring into every new release stage
    • Working with business analysts and development to gain usability and customer experience insights from newly released mobile apps
    • Collaborating with developers and service managers to make applications easier and less costly to maintain and support – essentially ending the 3:00am head scratching and blame games!

    Looking at these examples it’s obvious the new IT Ops craftsman won’t linger behind the production curtain. On the contrary, they’ll be the ones working across the entire software pipeline; focused squarely on acquiring new skills and building solutions to drive better business outcomes. Those who blissfully remain in denial, preferring instead to watch monitoring consoles and build out physical infrastructure have a lot to lose.

    1. Application resilience is so yesterday – With on-premise applications the traditional logic has been to associate reliability with preventing and/or recovering quickly from problems. But highly distributed, scalable, cloud-native apps are very different, so our approach also has to change. These applications are the epitome of complexity (only on steroids), introduce many more moving parts, and will service unknown demand – meaning failure is inevitable. Therefore design and operational practices must move beyond break-fix, to ensuring inevitable failures have a minimal effect on customer experience and that each application release ‘learns’ from the experience – yes, actually gets stronger.

    While the old 98 pound weakling comic book advertisement is kind of hokey, many businesses are getting sand kicked in their faces by digital disruptors. So before it’s too late, start re-engineering the operational practices needed to toughen up your applications. Just remember that while rapidly developed apps now deliver your business muscle, they’ll atrophy just as quickly if you neglect quality and performance.

  • A 6-pack for building a business case for CD

    A 6-pack for building a business case for CD

    There are all sorts of compelling business benefits awaiting those who adopt Continuous Delivery. It has the potential to reduce operating costs, reduce defects and redundant work, improve efficiency, and result in a much faster delivery speed. By releasing applications to market faster, you boost your chances of success and gain a competitive advantage. But how do you get from the theory to the practice?

    If you really want to scale up and realize the benefits of CD, then you’re going to need money to retool, retrain, and change the existing culture. In order to secure the budget you need, all of the key players in your organization have to buy into the change. A strong business case is essential. Here are six key areas to look at when you build yours. 

    Production incidents

    A natural place to start with building a business case for CD is with production incidents, because they’re very visible. When you perform a root cause analysis on an incident, you’ll often find that the software worked fine in the development environment. By standardizing the environment configurations you can reduce the number of incidents, free up resources from the provisioning process, and reduce wasted time.

    It’s possible to calculate the cost of data center downtime for your company, and you can use root cause analysis to find out where provisioning errors were to blame. From that, you can extrapolate the potential savings made by reducing these outages with standardized environment configuration. You also have to consider the reduction in wasted time and effort, resources that can now be productively employed. 

    Deployment failures

    Take a look at your past deployment failures and work out the average cost to the business of rolling back the deployment, diagnosing the cause of failure, and remediating. By automating deployment you can cut these costs completely.

    Manual testing

    To employ an API-based testing model that can be integrated into your delivery pipeline will pose an upfront cost, but that cost per build decreases with each new build. With manual testing, the cost rises with each new build. Automated testing gives you better coverage, earlier. If you can find defects earlier they tend to be quicker and easier to fix, which reduces the potential knock-on impact in terms of production incidents, and, ultimately, reduces costs. You’re also cutting manual labor costs and overhead.

    Manual build process

    If someone on your team is writing scripts to automate your builds, there’s a labor cost involved and they may cause a potential bottleneck. If you can switch to a centralized delivery pipeline where the build is automated and which development teams can customize for their needs, you can reduce the time and effort required. You also free up the developer who would have written the scripts to work on improving the product instead. At the same time, you’ll boost oversight, control, and reliability in the build process.

    Better analytics

    A lot of development time is wasted on building features that don’t add value. According to research by Forrester, only around a third of the ideas implemented result in a measurable improvement in the customer experience, a third have no obvious effect, and the final third actually have a negative impact. By building analytics into your code, and constructing a strong feedback loop, you can find out what really improves the customer experience and focus your efforts there. The earlier you can get feedback, the easier it is to prioritize and maximize the value you’re adding.

    Focus on the big picture

    Most business cases might focus on some of the benefits we mentioned above, and that’s because they’re easier to measure. But it’s the intangibles, such as improved competitiveness, goodwill, and revenue, which are harder to quantify, but which have the greatest impact.

    You want to present the potential cost savings, but this is about more than cost cutting. The long term benefits filter through as you ramp Continuous Delivery up. Faster feedback loops and a greater understanding of your customer are laudable aims for any business. You’ll have to delve into specifics to generate the persuasive numbers you need, but it’s important that you keep those intangible benefits front and center as part of the business case you’re constructing.

    About the Author/Andrew Phillips

    AndrewPhillipsAndrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

  • DevOps Impact on IT Operations Management

    DevOps Impact on IT Operations Management

    Most of the focus of the DevOps movement is on the application development to deploy cycle, and that’s appropriate.  When folks talk about the IT Ops side of DevOps, you’ll hear talk about IT process management tools like Puppet, Chef, and Ansible that are used to automate the deployment, update and remediation of software and VMs.  Yet, DevOps is widely acknowledged to be more than just one process—it is a culture of collaboration, enabled by automation that results in high quality outcomes.  As such, more IT disciplines are looking at how to apply DevOps principles to their domain.  These domains include those that can be put under the umbrella of IT operations management (ITOM): network engineering and operations, disaster recovery, and infrastructure operations management.  Many of these disciplines and organizations are currently highly siloed in traditional IT organizations, and tool suites are from an era that precedes the current emphasis on API automation.  Some may wonder whether these IT arenas can adapt and adopt DevOps.  The more pertinent question may be “can IT organizations afford not to?”

    ITOM is a huge topic, and there’s no way to do an exhaustive treatment of how DevOps is impacting it in one article.  However, we can at least take a look from a couple of angles.  The first angle is to consider DevOps impact in terms of functional areas of operational management.  ITOM can be roughly broken down into managing provisioning, capacity, availability and performance.  Of these, provisioning is probably the area where DevOps has had the most impact.  This only makes sense, since without automated provisioning of infrastructure and application components, it would be very hard to achieve the continuous processes that are a hallmark practice of DevOps teams. The trend of all IT infrastructure to become “software-defined” aligns with this provisioning automation movement.

    Once we stray outside of provisioning, the reach of DevOps becomes more sparse.  In the win column for DevOps is Application Performance Management (APM).   Five years ago, there was a tremendous dissatisfaction level felt by engineers trying to understand application performance.  So much so that a #monitoringsucks meme developed.  Since then, APM has significantly modernized and moved profoundly towards DevOps-friendly approaches.  Open Source Software (OSS) options such as the ELK stack (ElasticSearch + Logstash + Kibana) are seeing increasing adoption.  Meanwhile, commercial SaaS offerings from New Relic, AppDynamics, BMC and Riverbed are offering nextgen, highly API-enabled options.  Note that even though APM is considered a category of ITOM, it’s certainly the most dev-centric.

    Infrastructure monitoring has lagged by comparison.   The typical operations center arranges itself in a multi-level skills hierarchy.  Practices are shaped around monolithic, GUI-driven tools, and manual processes are the norm.  In fact, it’s fair to say that much of the institutional knowledge in many ITOM practices is literally “muscle memory.”  That is to say, knowing where to click on the screen.  Stovepipes of these hierarchies are formed around different types of infrastructure–compute, storage and network.  Collaboration between stovepipes is the exception, and ‘hot potato’ troubleshooting routines are the rule.  Rather than emanating from organizational DNA, collaboration often requires top-down pressure, usually to respond to crises. On the tools side, monolithic, enterprise software deployments and appliance-based architectures are common.  These products have been primarily designed to deliver a GUI experience.  Their APIs are usually second or third class citizens from the vendor point of view, and aren’t meant for end-users.  Rather, APIs primarily exist for vendor to vendor partnership integration and high cost custom integration projects. Clearly, this picture is not DevOps nirvana—so how is DevOps impacting it?  The answer is, so far, incrementally.

    One of the clear signs that an IT arena is getting primed for DevOps is cloud and open source disruption, and by disruption, I mean going beyond narrow, point tools to general purpose platforms.   Those same open source tools used for APM are making their way into systems management, and to a lesser degree in network management.  We’re also starting to see “green shoots” of change in systems and network management tools, with new SaaS offerings, though we’re far from a tipping point in these segments.  However, an interesting challenge for true conversion to cloud-based DevOps approaches is the need to address a Big Data problem.  The amount of data generated by IT infrastructure and networking equipment is now so voluminous that traditional systems and approaches can’t keep pace.  Legacy systems that have their architectural provenance in the 1990’s are based on scale-up models, and were never built for today’s data volumes.  Even cloud-based solutions may not necessarily be scoped to address IT Big Data. Until ITOM faces this technology challenge broadly, it will remain an impediment to transformational change.

    About the Author/Alex Henthorn-Iwane

    AlexAlex Henthorn-Iwane is the Vice President of Marketing at Kentik . Henthorn-Iwane has more than 20 years of experience bringing new technologies in networking, security and software to global markets. Henthorn-Iwane leads the global marketing strategy for Kentik and help the company bring its innovative story and solutions to network-dependent organizations around the world.

    Prior to joining Kentik, Henthorn-Iwane was VP of Marketing at QualiSystems, where he oversaw worldwide marketing at the industry’s leading supplier of DevOps cloud orchestration software for hybrid IT infrastructure, and Packet Design Inc., a provider of network management software founded by serial entrepreneur Judy Estrin. Prior to that, he held product management roles at CoSine Communications, Corona Networks, Lucent Technologies and Livingston Enterprises.

    Through his roles at QualiSystems, Packet Design, CoSine Communications, Corona Networks and Lucent Technologies he has acquired expertise in cloud computing and the opportunities presented through virtualization. He has written for Embedded Computing, Virtual Strategy Magazine, Datamation, SDN Central, Datacenter Knowledge and InformationWeek.

    Connect with him on LinkedIn, Twitter, or SlideShare.

  • To Be Continuous: Does your company have to be all in to do Continuous Delivery?

    To Be Continuous: Does your company have to be all in to do Continuous Delivery?

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about the spectrum of continuous delivery and where continuous delivery will and will not work.

    Edith Harbaugh, CEO LaunchDarkly

    Edith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

     

    Paul Biggar, CEO CircleCIPaul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     


    [soundcloud url=”https://api.soundcloud.com/tracks/231204216?secret_token=s-45TTy” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]


    Edith: So Paul, does one have to be all in to do continuous delivery?

    Paul: So I think the obvious answer is no. And in particular, you know that the answer must be no because nobody starts at all-in, or at least if you’re starting a new project you can be all in, and many projects are not all-in.

    So someone starts somewhere and they say, “Now we are going to do continuous delivery in some way,” and so they have to go from zero to full continuous delivery along some way.

    Edith:

    People have this conception that continuous delivery just means pushing any time you want, and a lot of very old school development shops did that. They would just hot patch and production constantly.

    Paul: Right. So, I mean, are we calling that continuous delivery? I feel that the idea of SSH-ing into the server is not continuous delivery somehow.

    Edith: But people have this misconception because they’re like, “Oh, I had a continuous delivery before this was a thing,  I just patched it production.”

    Paul: I’m going to put a lot of money on the line here and say that if you define continuous delivery as “I SSH into the production server,” then you are not doing continuous delivery right. In fact, you are doing it considerably wrong.

    Edith: What exactly are you doing wrong then? What steps are you skipping?

    Paul:

    The whole point is that you have this automated process for releasing software. And SSH-into the production server is the very opposite of that.

    That’s not to say you shouldn’t do it in the occasional emergency. And occasionally, if we have some sort of catastrophe, we will SSH into the production server, replace some code, execute some code on the server to keep things as they are and prevent customers from noticing or from things from going down.

    But that isn’t the same as a continuous delivery pipeline or a release workflow, or an automated release workflow, which is essentially what continuous delivery is.

    Edith: So continuous delivery isn’t just shipping stuff out to production, it’s more having all the processes behind it.

    Paul: It’s very much the process. So when you think about can you be a little bit continuous delivery, I think what you’re looking at saying is there a lightweight version where we get some benefits of the continuous delivery as we go between all and nothing?

    Edith: Can you be, do you have to be all in, or can you be a little bit continuous?

    Paul:

    I think the very obvious instance of a little bit continuous is where you deliver the code where you have the automated release process builds some sort of artifact and then someone manually deploys that by a single click.

    Or, in fact, where a build happens and test happens and the remaining automated release process is triggered by a human saying, “Now we are going to release,” and then everything else is scripted, and so they just have one input, which is now.

    Edith: Oh, that’s really interesting. So do you think to truly be continuous it has to all be completely automated with everything always pushed?

    Paul: No, no. I don’t think so. I think it largely depends on the size of the team and that sort of thing. But I’ve seen a couple of examples where our customers were asking for particular things, and it indicated that they had sort of semi-continuous workflows.

    And one example of that is a team that only wants to do continuous delivery during work hours.

    Edith: Yeah.

    Paul: And there was another team, which only wanted to do continuous delivery at midnight because that was the only time that they didn’t have enough customers on their site to be able to clear the caches.

    Edith: Yeah.

    Paul: Now if they wanted to go to complete continuous delivery they would have to say, “We are going to clear a subset of the caches,” or have some other caching policy that allows us to continue or that allows us to deploy at any time, but where they are is basically were semi-continuous.

    Edith: I had a really interesting chat with Douglas Squirrel, who actually listened to our podcast, and he does a lot of consulting for companies in England, and he said he goes to a lot of companies that have all the tooling to be continuous.

    Paul: Okay.

    Edith: But they don’t actually use it.

    Paul: And what does that mean exactly?

    Edith: They have set-up continuous integration, they could, in theory, deploy whenever they wanted but they still, for whatever reason, still cling to their old schedules.

    Paul: Okay. So, I was looking at this very interesting article on the First Round Capital blog.

    Edith: Oh Jocelyn’s.

    Paul: Yes, that sounds right.

    Edith: Jocelyn Goldfein, who is at Facebook, right?

    Paul: Yes, that’s right. Yes. So, what she talked about was this idea that delivery schedules are useful in certain contexts. And in some cases velocity matters, and that’s kind of the Facebook case, and in some cases predictability matters a lot more. And if you’re delivering enterprise software, for example.

    An enterprise is not going to allow you to continuously deliver your product into their firewall or into their enterprise. What they’re going to want is every quarter a new thing.

    Multiple times a year preferably, but each time you release to them it creates a new workflow, and they want to be able to audit it and see a change log and validate that things are, they might have a change process internally.

    And so it’s not necessarily going to be a good idea to continuously deliver there. But a team that is building for that can do a continuous delivery process internally where they produce all of the artifacts and they do a complete release on every single push. Even if that is not released.

    Edith:

    There are a lot of reasons why an enterprise doesn’t want to take new features constantly, and a lot of it comes down to training. If there are big new features that need to be rolled out to people then they have to take on the cost of training.

    Paul: Right.

    Edith: It was actually interesting. I chatted with a guy from Square and he talked about how they really try to limit the amount of pushes they put out to the actual cashiers.

    Paul: Okay.

    Edith: Because of the training cost. I mean, a cashier at a coffee shop is doing a gazillion other thing and they can’t just suddenly pop up some new functionality and expect them to grok it or even take a tour.

    Paul: Right. That’s an interesting idea.

    Edith: So, although they have all the processes in place to do a lot of changes, to do continuous delivery, they actually really limit what they push out to the true end users.

    Paul: And it’s funny that the people who have far, far more users, someone like Facebook, can just do those changes because they don’t necessarily value an individual user on a…

    Edith: You’re walking down a path.

    Paul: No, I’m not saying that they don’t value their users. It is safe to piss off an individual user. Less so for someone like Square. And as you go up the value chain, the more they’re paying you or the more involvement they have to have in the product decisions, the less able you are to piss them off.

    Edith: Facebook is actually a fascinating example because at any time they’re going to be pissing off somebody. It’s just a matter of, it’s kind of like Apple.

    Ya know?

    Paul: Right.

    Edith: They’re always pissing off somebody, it’s just who.

    Paul: Did you hear about today at Facebook where all the feature flags went on?

    Edith: It was LinkedIn.

    Paul: Oh, was it LinkedIn? Okay.

    Edith: Ya. Well, it probably has happened to everybody that’s done feature flags. There’s the day that that happens.

    Paul: Right.

    Edith: And, it’s ugly, yet. Facebook is fascinating. I actually, I wrote a blog post about Facebook and how they use feature flags. They have a product called Gatekeeper.

    Paul: Right.

    Edith: That controls everything, everything. And after they had a huge PR disaster, i think it was in 2011, with their news feed, they actually do really care if they piss off a lot of people.

    Paul: Right, right.

    Edith: And that’s why they implemented Gatekeeper where they do a lot of 1% roll-outs.

    Paul: So, I remember when I was setting up Circle I sat down and talked to my friend who worked at Facebook and talked about their workflow and that sort of thing, and their release process was that code got released on Tuesdays and everyone who was releasing code, or who had code in that bit that was going out, had to be available during the release process.

    And the release process didn’t turn on any features because that was all done by Gatekeeper, but the code itself had to go out. And they had to validate that everything stayed the same and everyone had to be there. And so they had a fairly continuous delivery process but were still, manually there was a release going out.

    Edith: Well ya. You want to have some sort of enforcement that people look at stuff and care.

    Paul: Care in what sense?

    Edith: So, the example I heard about also at Facebook is it’s so large that there’s just no way that all the groups can coordinate with each other. So literally what they do is when they roll stuff out with Gatekeeper they roll it out internally to Facebook-only employees and that’s how they find out if stuff breaks. It’s not like at a smaller company like CircleCI.

    Paul: Right. But there’s a difference between the code being rolled out and the feature being rolled out.

    And they don’t do, I don’t know what the story is nowadays, but at the time they certainly didn’t do continuous delivery on code. Or continuous releases or whatever you might call it.

    Edith: I don’t actually work at Facebook, but what I did hear was that– that they just literally can’t coordinate. When you’re a small company you can still coordinate in a HipChat or a Slack room like, “Hey, we’re pushing this now, start looking for changes.”

    Paul: Right.

    Edith:

    But with Facebook, the only way they know if something is broken is if somebody complains.

    Paul: So there’s an interesting discussion about microservices at AWS. I guess they didn’t call them microservices at the time, but having fully service oriented architecture.

    So, one of the things they talked about is how when you roll, or when you continuously deploy a service, you might break other services. So, the thing that actually breaks may not be in any way related to the people who’s alarms start going off. And, in fact, there may be multiple steps away from who’s alarms are going off.

    Edith: Google has a similar system, and I heard that they monitor 200 plus different metrics. And one of the ones they monitor is just do people say, “Gmail sucks more?” on Twitter.

    Paul: Oh. Interesting.

    The thing that I’m getting at in terms of how little or much can you continuously deliver is, if you are Google or you’re Facebook and you have the monolithic code repos, the thing that you’re releasing is it’s not the same release process as someone like Amazon who has what I would describe as pure, continuous delivery.

    Every service does it’s own continuous delivery and your dependencies may be changing at any time and you don’t actually know what’s going on. Versus Facebook, we are taking this multi-gigabyte binary and we’re uploading it to a set of servers, and sure, we’re slowly releasing and maybe we’re doing that multiple times a week or multiple times a day.

    But it’s not quite the same as Amazon releasing every 11 seconds. And I think we can see, even between these multiple large companies who have incredibly advanced release process, that you could say that they are clearly at different spectrums along continuous delivery.

    Edith: So, what do you think is suitable for, let’s say, a medium company with, let’s say, 10 developers?

    Paul: I mean, there’s a lot of different things going in. Who is your customer base? What is the level of safety that you need? Let’s suppose that it’s a consumer web app. If it’s a consumer web app, I would expect that that company should be continuously delivering all the time. Especially if it’s slightly okay to break things.

    Edith: Yeah.

    Paul: Or at least, you break things you can kind of roll them back or whatever within a couple of minutes. A team of that size might not have the resources to make their continuous delivery process perfect. But it seems likely that they would have more success shipping with continuous delivery than shipping without it. So lower risks, higher velocity. That sort of thing.

    Edith: It’s funny how continuous delivery has permeated even the VC world. Jason Lemkin, the guy behind SaaStr,

    Paul: Right.

    Edith: He tweeted yesterday that one of his due diligence questions was how,”How often can you roll back, and does it work?”

    Paul: Right.

    Edith: And it’s funny that I think of that as a very technical thing, like, “can you roll back?” But to the VC is seen as a sign of technical sophistication.

    For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.