devopsy.kinsta.cloud

Tag: development

Reality Check: Bringing Continuous Delivery Dreams to Life

Every developer shares the same dream: to write a piece of code or app that changes everything.

Whether it makes your company do things more efficiently, attracts hordes of new customers, simply makes existing clients happier or maybe even changes the world a little bit, that code is what we strive for. Some of us don’t merely harbor that dream; many of us actually achieve that moment of brilliance. For those who have, it’s remembered as a moment of pure joy.

So, where does this magical process begin? First, you talk to your manager and, if you’re lucky, she or he agrees that it is a fantastic idea and starts to sell it internally. Sometimes the idea is so good that you get the go-ahead on the same day and even get a small team to work on it!

Then, soon thereafter, reality kicks in.

You need data, inevitably from applications running somewhere in the cloud and/or some other data from on-premises data stores. You need to interface with APIs, some well-documented, some not. You need some user experience work done, and the standards your company has adopted don’t really match your vision.

You also need to interface with many of the standard services your company has already implemented because, after all, your app doesn’t run in isolation—you work with existing billing systems, a CRM system, a fulfillment system … and all of them run on servers that can be anywhere. Soon you find yourself fighting for test data, systems to test with and applications and APIs to interface with. Before you know it, you’re banging your head against the wall.

Fortunately, your teammates are flexible, creative people. And because your internal systems seem to work against you, you go around them. You download a trial version of some testing software; you install a copy of some internal systems on a virtualized test server (if you can find one) and start working with these.

Since your app needs to be snazzy and snappy, you download some open-source software to help you test the performance on mobile devices. After three sprints, you’re now ready to move your app to an environment where you can start performing real tests!

Most times this is the moment when the hardest reality kicks in. The copies of the systems you tested with were not up to date. Data has changed. APIs have changed. The real system responds three times slower than the copy that sat on the server right next to you. The data in the cloud has changed. The performance testing tool you used is not available in the official testing environment. … The list goes on.

This doesn’t even include the difficulties of moving all the components of your new app to different environments. Trust me, this is not a fictional nightmare scenario; it is reality and I see it happen all the time.

Now, what if you had the test-data generators necessary to create the data you need and a server that ran virtualized services, mimicking all the (up-to-date) external systems you need including the APIs to talk to them—services that would allow you to say, “What if this application responds three times slower when I request data?” or, “What happens if this system suddenly goes down?” What if, when you’re done building, you could simply migrate the app and all the components that go with it to another (test/QA) system with one click?

Everybody wants to support people with good ideas and turn these ideas into working software without the frustration that very often distracts and demotivates people. To do so, you need to create the infrastructure that supports this. Is it difficult? Not really, the tools to automate services, create test data (and test scenarios) and migrate complete applications from A to Z are available.

However, just like a great app, this too starts with a dream—the dream of allowing people with great ideas to work on these ideas without the overhead of an infrastructure that allows them to focus on the coding and not on the infrastructural issues; the dream to create a software factory where the creation of the app takes 90 percent of the time and the infrastructural burden only 10 percent, not the other way around.

If you’re serious about developing software and want to create an environment where the infrastructure allows people to develop and release software faster instead of working against them, find the tools that support it and jump in, because they exist today.

Enacting this change now and maybe the next time someone comes to you with a great idea you can be confident that they will have all the support needed to get there efficiently.

— Marcel den Hartog

February 17, 2017
4 Top Tips for Developing a New Feature

As a backend developer at CA BlazeMeter, I recently had the chance to work on a big feature for our web app. The feature required a massive change on our backend code, a big change on our frontend and multiple migrations. You know what I’m talking about—it was one of those projects that developers fear.

But, I gained a lot of valuable experience from the project. Not only did we deliver a kick-ass feature, I also learned the importance of teamwork and patience when developing something new. Armed with those two values, and the right tools and technologies, you can deliver a new feature quickly and efficiently.

Here are a few more things that I learned that should prove helpful whether you’re developing a major feature or one that only requires a minor code change.

Tip 1: Plan to understand the development requirements

The process of development starts way before you start coding, with planning. After the product manager decides what needs to be developed, R&D needs to understand the cost: how many days (or weeks) of development the feature will take, how many people will be needed and which APIs are required.

To do that, the product manager needs to conduct at least three “grooming” sessions with backend, frontend and QA leaders. The grooming sessions start with the product manager presenting the product flow and requirements.

Since no one wants to read a lot of text (except perhaps the awesome readers of this article), the product manager should prepare mockups. They don’t have to be designed or colorful, but they do need to show the team what the feature looks like in his or her vision. During our grooming sessions, we used Balsamiq, which lets you create screen sketches for how the product is supposed to look, feel and work.

Editing screen sketches using Balsamiq

After the initial presentation, the sessions should include a lot of open discussion—asking questions, checking for answers and examining compliance, etc. —until everyone is satisfied and understands what the feature requires from them and from others.

This is the time and place for frontend, backend and QA to give their opinion about the schedule and to alert and flag any potential problems in development or extreme use cases. It’s the role of the product team to answer questions and change the flow and requirements until there are no R&D rejections.

Afterward, the frontend and backend leaders need to create a feature development plan, sitting down to determine what data they need to transform the product vision into a real thing. Or in other words, which APIs are required for each flow or button, how long each step is supposed to take, including buffers, and the order of development. APIs are the agreed “contract” between the development teams. After determining the contract terms, each team can proceed individually.

I recommend you write your APIs in Swagger, which turns YAML files into detailed HTML pages with all the APIs.

Swagger Editor

This plan needs to be transparent to everyone working on the project. In fact, it’s even better if each project participant can add and describe scenarios. Tools such as TestLink and Atlassian do just that. Then, when the code is ready, developers can go over the list and see everything is covered.

We will need to stick to this schedule later on, especially when new demands unrelated to the feature come in, such as fixing bugs. Sticking to this plan is the best way to distinguish the wheat from the chaff of the development process.

Start writing automation ASAP

After planning, we want everyone to start working, and not only the backend. To do that, the backend can prepare mocks, either through outsourcing services such as JSON servers, which show inputs and outputs, or by writing dummy code.

I mentioned earlier that we used Swagger to write APIs. We also used it to generate a server in NodeJS/Python, like this:

Generating a JSON Server through Swagger

From this moment on, frontend can begin writing and building screens and QA can start writing API automated testing, by using tools suh as JMeter. At this point, frontend and QA have the response codes and response bodies; they know which features to put where; and which APIs they need and how to use them.

QA also know which data they’re missing, such as field names and field data. Sure, the backend code isn’t ready yet and all of the automated exams will return “red” instead of “green,” but the main issue is making sure everyone is as prepared as possible.

This includes load and performance testing. You can use open-source tools such as JMeter, which is enhanced by CA BlazeMeter. By uploading our test script into, configuring the number of virtual users and ramp-up time, determining geo-locations to test from and analyzing KPIs to determine trends and the health of our system, we can ensure our product is delivered to our customers without crashing.

This is also true about sending code to QA: Developers should test their code themselves and ensure that when they commit their code, it works 99.9 percent of the time.

By starting automation as soon as possible, you can minimize future bottlenecks in the development process.

Ensure the team communicates and works together

Like I said before, teamwork is the most basic requirement when creating a new feature. This is because new questions arise during development and because requirements change. Therefore, everyone in the team needs to be in sync all the time, from the product manager and the designer to the developers and QA.

Open communication lines are crucial because everyone is working on the same product at the same time, but each developer or QA should be working on a different part of the product and on different functionalities. Communication ensures each person’s work is compatible with the rest of the team’s work. This saves time, prevents frustration and redundancy and ensures maximum professionalism.

To make sure everyone knew what was happening, we used Slack and opened a dedicated channel for the creation of this feature. We also had daily (short!) standups where we discussed what we have and what is missing.

Opening a Slack channel

Communication also includes team and assignment management. JIRA, our primary issue tracking tool, offers Kanban-style, Scrum and fully customizable boards, which show the progress of each development task, including the developer assigned to it, and his or her exact status or phase.

This helps you understand what other people are working on, and just as importantly, where people are stuck. If you can, get everyone to work in an open space and put a large screen in the middle of it. This lets you stay in sync at all times.

We used JIRA to open and manage bugs as well. This was helpful when multiple QAs found the same bug. It’s important to carefully choose which tickets are opened and put them under one epic, so everyone stays focused.

JIRA Dashboard

Another important aspect of communication is ensuring everyone uses the same terminology, as sometimes people might use the same term to describe different things or different terms to describe the same things. To prevent mixups that could waste your time, make sure everyone is on the same page. For example, do you say “system admin” or “<name (ex. BlazeMeter)> admin”?

Work on the same development environments

It’s important that everyone works on identical development environment s and even on the same version of the tools and libraries. Otherwise, it could cause an (in)compatibility mess, including a lot of unnecessary back and forth and even might cause bugs to pop up. This environment needs to be as similar as possible to production, but only on a smaller scale.

Conclusion

Now look at the feature that came out of all that work here. I’m so grateful that I had the opportunity to work on it with an awesome team.

I hope these best practices help you the next time you develop a new feature. Don’t forget to keep going agile after development and delivery by running tests in a continuous environment such as Jenkins for every commit.

CA BlazeMeter works with Jenkins, ensuring you can load test in the cloud and analyze your data while shifting left. You can also use different APM tools, such as New Relic, AppDynamics and CA APM to alert you about errors before QA, which saves time.

— Dor Atias

February 7, 2017
Puppet Launches 2017 State of DevOps Survey

It’s that time of year again!

We at Puppet have kicked off the 2017 annual State of DevOps. survey. Now in its sixth year, we have once again teamed up with DevOps Research and Assessment (DORA), founded by our dear friends Gene Kim, Jez Humble and Nicole Forsgren, to survey technology professionals worldwide in an effort to better understand the health and habits of organizations adopting DevOps. We’re pleased to see that the report now serves as the de facto gauge for the DevOps community to find out how IT organizations are doing and how they can improve.

One of the most critical aspects of the survey is that it manages to translate the processes between technical practices and choices into actual business outcomes and helps organizations understand the real benefits of DevOps adoption. Through analysis of each year’s survey responses, we’ve been able to quantify IT performance. From last year’s report, we know that high-performing IT teams are pulling away from the pack in terms of throughput. They deploy 200 times more frequently, have 2,555 times faster lead times, have a three times lower change failure rate and recover from failures 24 times faster.

Survey Says …

This year, we’re interested in learning more about infrastructure architecture patterns within organizations. We’re not just asking how the services are designed, but we also are curious to see if teams are free to move at their own pace and if they are largely self-sufficient. Can they easily make large-scale changes to their architectural design?

We’re asking questions about team experimentation and whether teams are able to evolve the design of their architecture as they work and if they are required to interact with a lot of people across the organization. One of the things to emerge from the agile and DevOps movements is that the more we can enable teams to work quickly and self-sufficiently, and increase speed without reducing quality, the better it is for everyone.

This year, we expect to see proof that high-performing IT teams are those who have been enabled to operate with a much higher degree of autonomy, and that their quality metrics remain stable or even exceed those of high-performing IT organizations. We also expect the findings to confirm that early DevOps adopters are still seeing exponential benefits compared to their colleagues who haven’t yet embraced DevOps. The longer your organization has been adopting DevOps practices, the more it becomes muscle memory and reinforces positive behavior within them, which has unparalleled benefits. As any IT professional working in a large company knows, there’s an awful lot of waste and inefficiency and we’re not anywhere near peak DevOps penetration. With each passing year, survey results show DevOps adoption is a key contributor to enhanced productivity and efficiency. We expect this year’s results to be no different.

We’ll be publishing the results and our analysis in June, and have no doubt it’ll be worth the wait. Stay tuned!

About the Author / Nigel Kersten

Nigel Kersten is CIO of Puppet. He came to Puppet from Google, where he was responsible for the design and implementation of one of the largest Puppet deployments in the world. At Puppet, Nigel was responsible for development of the initial versions of Puppet Enterprise, and has since served in a variety of roles, including head of product, CTO and CIO. He has been deeply involved in Puppet’s DevOps initiatives, and regularly speaks around the world about adoption of DevOps in the enterprise and IT organizational transformation.

January 19, 2017
Welcome to the Continuous Delivery 101 Podcast

No matter what industry you’re in or how old your business is, your software often determines your success in today’s application economy.

Applications are the face of your business—both the way it performs and the speed with which you can incorporate new ideas and innovations, driven by the efficiency of your software development life cycle—determines whether your company can survive in this competitive market.

Software doesn’t just support the business today, it drives the business—and your organization’s livelihood in this application economy comes down to your ability to quickly deliver world-class capabilities. As a result, more organizations than ever are turning to the practice of continuous delivery (CD) to improve their ability to plan, build, test and release applications to market.

With this increased focus on continuous delivery methodologies and practices, we’ve established a new podcast that highlights the importance of utilizing CD, along with adoption techniques and other best practices.

Dubbed “Continuous Delivery 101,” the series features some of today’s leading experts in the field discussing where they’ve seen CD success, across some of the world’s top organizations and across multiple industries.

The first episode in this series covers CD fundamentals and the software evolution that has made continuous delivery so critically important.

Beginning in the ‘60s, this first wave of this software evolution saw mainframe systems brought in to automate business tasks. A second wave saw the emergence of ERP/CRM-based processes, so we could improve how we run contact management and transaction processes. Today we’re in the third wave, which started with the widespread adoption of mobile and web technologies, with all of these paradigms intersecting in perfect convergence.

Businesses that have mainframe and distributed systems, that have CRM and ERP system, that have web fronts and mobile apps, are utilizing these continuous delivery tools to innovate faster than ever before. These organizations are being forced to reinvent how they communicate with customers and how they deliver goods and services to stay relevant in the market. This third wave, where business is now driven by IT, is the biggest disruption that we’ve seen.

To hear a lot more on this topic, check out the first episode of the Continuous Delivery 101 podcast via YouTube or the iTunes store, where you can find it on the Podcast App by searching for “Continuous Delivery 101 podcast.”

Software delivery, fueled by CD, is evolving at a furious pace. Stay tuned for more expertise and best practices.

— Scott Edwards

January 18, 2017
Automic Helps DevOps Transformation Sparkle
Working closely with Automic, an international wholesaler and retailer of jewelery transformed a department of project managers, developers and operations responsible for all of the customer-facing websites into a DevOps organization. As a result, project delivery was completed with 90 percent fewer delays, and first customer-facing releases of projects could be delivered in less than half the time. The usability of its websites increased significantly, resulting in more customer traffic, longer average time spent on the sites and increased revenue.

Haphazard Development

Prior to the DevOps implementation, new projects were delivered by “waterfall” project management means. The cyclical resource planning process was based on the project portfolio and business priorities. Project managers then planned project milestones working closely with business stakeholders, business analysts and application developers. Analysts created specifications and developers were assigned to work on packages and tasks to implement them.

Different web technologies were used with varying levels of dev and ops expertise in each technology. The Operations team members usually fulfilled requests made during the course of a project as part of their overall daily tasks—for example, to provision a new environment, deploy a package or change the environment configuration.

This approach had a major disadvantage. Delays in one project impacted the overall resource planning and could have a cascading effect on other projects. Delays in projects could be for multiple reasons, such as waiting for business stakeholder impact, frequently changing requirements, unexpected sick leaves of developers or poor estimates. Other frequent causes included delays in operations tasks such as a deployment or provisioning of a new environment. These delays were caused by unaligned planning as well as a mismatch in priorities between projects and operations. Besides delays, some developers were underutilized, which also was exacerbated because they were specialized in a technology or component.

DevOps Transformation Solution

Working closely with the customer, Automic restructured dev and ops by:
1. Switching the development from a project to a product mode (cross-project development teams based on capabilities rather than assignment of individual developers to projects).
2. Moving project managers from development micromanagement to focus on major milestones and deliverables to the business.
3. Enabling some project managers as product owners (project management remained necessary).
4. Providing developers with agile methodologies through initial training plus ongoing coaching (six to 12 months).
5. Providing ops with an agile methodology.
6. Organizing both dev and ops into multiskilled teams focused on individual products rather than technologies.
7. Aligning ops into delivery teams, initially part-time and later full-time.
8. Giving delivery teams the end-to-end responsibility for their products, including their operations.
9. Standardizing the technology stacks within teams (but not across them) over time.
10. Delivering projects into production (or alpha, beta) in slices (releases) instead of just once to get early feedback and improve.
Results: Fewer Delays, Better Usability

By standardizing, this customer has transformed its dev and ops teams, creating an agile, scalable environment that can drive the business forward. The company achieved the following specific results:
- Project delivery are now completed with 90 percent fewer delays. First customer-facing releases of projects now are usually delivered in less than half time.
- The usability of websites has increased significantly, resulting in more customer traffic, customers remaining longer on the sites and increased revenue opportunities.
About the Author/Michael Schmidt

Michael Schmidt heads DevOps Practice at Automic, a management and technology advisory service established to help Automic customers to prepare for and transform their organizations toward DevOps. Michael spent his carrier as consultant, entrepreneur and in several product-related leadership positions. He has a passion for building lean IT organizations. Michael holds a joint MBA from New York University, London School of Economics and HEC Paris and a Master of Computing Science.
January 5, 2017
Breaking the Cycle of Dependency
While technology is an important component of DevOps and accelerating delivery, there are other aspects of process and culture that are critical to keeping the cycle moving to realize the goals of reducing lead time to become more responsive to the needs of the business.

If you go to most agile teams and ask the question of what is stopping them from going faster, they will typically respond that they are waiting for “something.” Something can include:
- Waiting for work to enter their backlog
- Waiting for environments
- Waiting for another development team to provide some service or capability that is needed
These obstacles include:
- Large batch sizes
- Lack of continuous flow of work into Agile teams’ backlogs
- Calendar-based gated releases
- Test environment complexities
- Environment availability
This manifests itself in the current state as shown below as the vicious cycle. When teams are only delivering system releases a few times a year, then the business believes it needs to pack everything it can think of into that release, since waiting for the next release incurs a delay of months. The leads to large complex releases with many dependencies. This, in turn, slows down the development and testing of these releases and requires a lot of release coordination, resulting in long lead times and the inability to release frequently. And this leads into the cycle repeating.

So what are the countermeasures needed to break this cycle? They include:
- Smaller batch sizes
- Architectural, design and development decoupling techniques
- Increased automation
- Readiness-based release capability
Smaller batch sizes require applying MVP (minimal viable product) techniques. When IT gains the trust of the business to deliver more frequently, the business will be more open to focus on the minimal product set of highest priority items it believes can deliver business value. Accelerated delivery provides the ability to test business hypotheses and gain feedback from customers more quickly to determine what worked and what needs changed. The enterprises that can turn this crank more quickly gain a huge advantage in the marketplace.

Modern web architecture techniques (e.g. microservices, APIs), along with design techniques such as feature toggling to enable dark- and canary-launching allow applications to be deployed independently and with direct control of customer impacts. This increases development velocity due to simplifying and reducing the lifespan for managing feature or release branches in the code base.

Increased automation combined with a move to readiness based releases allows application releases to be deployed when ready rather than waiting for calendar-based change windows. They key here is not sacrificing quality, auditability or security but rather building those capabilities into your applications and delivery pipeline and requiring those certifications are earned before an application can be released.

About the Author / Carmen DeArdo

Carmen DeArdo is the director of Build Technologies at Nationwide Insurance. He is responsible for helping drive accelerated delivery utilizing Lean and DevOps techniques across the end-to-end delivery value stream. One current focus area is the creation of an integrated delivery pipeline capability.
December 20, 2016
Infrastructure’s Impact on Shifting Left

If you lead big honkin’ software development projects at a big honkin’ enterprise, you’re probably doing everything you can to shift QA left. That’s because the earlier you discover defects and shortfalls, the faster and less expensively you can bring great software to market.

Unfortunately, above a certain scale, it takes more than just better processes and a more aggressive culture to shift test and QA left. It also requires a different approach to the infrastructure you use to distribute your builds.

Infrastructure and the Problem of Scale

Most software developers understand the scale issue primarily as one of complexity. The bigger the project, the greater the chance that something won’t work with something else exactly as planned.

But large software projects also canget badly impeded by the sheer physics of build distribution.

Take, for example, a 10GB build that you have to distribute to five different domestic and offshore locations from your primary facility over your company’s 10Mbps MPLS connection. If your two of your locations are domestic and have 5Mbps connections, it will take five hours to get them each build. If the other three are offshore and have 2Mbps connections, it will take them each 12.5 hours to get their copies of the build.

That’s a “time tax” you’ll have to pay again and again as you keep having to ship massive files over the network to your globally distributed test/QA team. Worse yet, the more Agile and iterative you get, the more you keep paying this “time tax” over and over.

So hours expand into days, and days into weeks—constantly frustrating even your best efforts to more aggressively shift left.

Avoiding the Scale-Driven Time Tax

Here’s how you can successfully avoid the time tax that chronically prevents you from shifting your biggest projects further left.

Instead of continuing to use IT infrastructure that was never designed to support time-compressed software development at scale, you can instead implement a hub-and-spoke build distribution system that virtually eliminates network-related delays.

This hub-and-spoke model lets you maintain a “gold copy” of your current build(s) in the cloud—while providing all your remote locations with their own local copies that get continuously and automatically updated with any changes as they occur.

This model eliminates network-related bottlenecks while allowing your geographically dispersed teams to collaborate without tripping over each other’s work. As a result, you can shift left much more aggressively.

Of course, infrastructure decisions typically aren’t made by the same folks who lead software development. So the two groups have to collaborate closely to make sure the network doesn’t impede critical development processes.

But both parties have lots to gain—because hub-and-spoke saves infrastructure owners lots of money in storage, bandwidth costs and network acceleration hardware at the same time as it empowers development to shift further left.

That’s why DevOps leaders and enterprise infrastructure owners need to work together ASAP. The digital success of the business depends on it!

— Barry Phillips

October 28, 2016
Keep Applications Running by Testing Early, Continuously

Almost every product we buy or use today goes through some form of extensive testing before it’s released on the market; typically, these tests are designed to replicate the worst-case scenarios that a product may encounter under real-world conditions.

Automobiles, for example, go through extensive hot and cold weather testing in remote locations, and are driven over obstacle-laden tracks designed to push the limits of their durability. Even seemingly mundane objects such as children’s toys are exposed to abusive simulations to ensure that they don’t harbor any potentially hidden and harmful effects.

On the current landscape, applications are the face of your business, often serving as the battleground where consumer loyalty is won or lost. Given this reality, it seems crazy to think that any applications are released having only been exposed to a minimum of tests. Yet, in the name of getting to market quickly, that’s far too often the case.

As disciplines including agile and continuous delivery continue to increase the speed at which development teams can release software, it’s easy to put the crunch on testing in the rush to get an update released. But that’s a costly mistake in the long run, as research has shown that it is 50 percent less expensive to find an error in testing than to address the same error once it has reached production, and more than 90 percent cheaper when addressed in the development/coding phase.

Given the dramatic cost savings of addressing errors in development, clearly the goal of any organization focused on continuous delivery should be to shift testing left so a greater number of defects are found early in the development process. This is what we refer to as continuous testing, and it’s a huge part of the reason why I am so excited about CA’s acquisition of BlazeMeter.

Performance load testing should be a core element of any continuous testing strategy because it is a critical factor in determining system reliability. Similar to the worst-case scenario testing that consumer products undergo, performance testing can quickly uncover applications defects that might otherwise go unnoticed. After all, the manner in which a system operates with a few hundred concurrent users may be very different than how it runs with thousands of users.

BlazeMeter enables today’s development, operations and DevOps teams to conduct this critical performance testing earlier in the development process, saving countless hours and resources. The company is also competing—and winning—in a highly competitive marketplace where it has been pitted against much larger, more entrenched competitors.

Part of the reason BlazeMeter is winning is its focus on offering a SaaS-based solution that is highly cost-effective and easy to engage. Using this model, customers can try out the testing software with a few clicks, and then quickly move forward to broadly adopt the system. Another catalyst to its success: BlazeMeter was built using open source tools, such as Apache JMeter, used widely in the development community for performance and load testing. As a result, BlazeMeter can easily run and analyze JMeter and other open source-based load tests—such as Selenium, Gatling and Locust—within a commercial-grade performance testing system.

This combination of a SaaS delivery model and incorporation of open-source tools, augmenting strong performance testing functionality, make BlazeMeter very complementary to CA’s existing continuous delivery solutions, including its CA Service Virtualization, CA Test Data Manager, CA Agile Requirements Designer and CA Release Automation products.

At CA, we’re committed to providing the most comprehensive, integrated and open continuous delivery ecosystem, allowing our customers to release swiftly, test agilely and constantly increase reliably. The addition of BlazeMeter will further enable customers test continuously, thereby increasing agility and the overall quality of their software, to ensure that applications don’t break down on the side of the virtual road.

— Aruna Ravichandran

October 12, 2016
How the Lure of an ‘Easy Button’ Installer Traps Projects

As preamble for this post, I wrote about the Unicorn Installer challenge. The challenge is that user needs are too deeply heterogeneous for good reasons to create a vertically integrated way (aka an “easy button”) that works for all users. The problem is not just current diversity, it’s also about how we cope with needed changes in the future that would force users to straddle configurations.

Don’t despair! There is a practical solution to shared ops on project installers: composability.

If projects focused on maintaining install modules for their specific components, then the community could collaborate on well-bounded problems that should be infrastructure-agnostic. Even building in multiple scripting DSLs (Ansible, Chef, Puppet, Salt or Bash) is a much smaller-scope problem. Further, all of these DSLs have modularity concepts so it’s feasible to pull in community roles/modules/gains/plays into larger automation script. This approach allows us to decouple snowflaked infrastructure and prerequisite configuration from the project configuration.

At that point, the proverbial “easy button” becomes an opinionated set of project, prerequisite setup and infrastructure-specific steps. These then become vendor-specific buttons: “Press here for the fast AWS install with Dejour SDN and L8istSh9y O/S” as was showcased to great effect at Dockercon.

With that said, let’s look at how trying to create an “easy button” installer inside the project backfires.

Cloud Foundry chose to have an in-project installer, BOSH, that has an opinionated installation approach based on generated images. To work, the installer has to maintain an increasing number of BOSH-specific integrations and image generators. These efforts are redundant with other tools non-core to installing Cloud Foundry. Further complicating the story is Pivotal, a primary Cloud Foundry maintainer, which considered BOSH a product differentiator. This creates ecosystem conflict with other Cloud Foundry vendors. While BOSH helps create robust installations, it also drags along a lot of baggage that keeps those improvements from being more generally accessible. For all of the expected ease of installation for BOSH, Cloud Foundry is still considered difficult to install.

OpenStack installers are a very confusing story, with multiple competing in-project installers. Starting from the single-node DevStack, which provides all of the continuous integration/continuous delivery (CI/CD) testing to the confusing over-cloud/under-cloud OpenStack-On-OpenStack (aka Triple-O) concept. Neither of these initial in-project attempts created practical, universal installers. In fact, they’ve spawned waves of other installers and configuration scripts. Recently, OpenStack decided to allow many of these attempts into the project (aka “big tent”) without coordinated governance making it impossible for operators to determine which efforts they should be tracking. Having some efforts stamped with “community approved” only makes it more difficult to solve cross-platform issues. The effect is further balkanization, as vendors and advanced operators attempt to build working automation silos against competing interests.

As a positive example, Linux pre-seed and kickstart options are distribution-specific and support a healthy ecosystem. It would be easy to think of consolidating these installs into a single system; however, the likely result would be to lock together rivals and slow down innovation.

What we want is a thriving ecosystem around a project without endorsement as one being THE install process.

Why is lack of endorsement critical? It allows the project to advance without being slowed down by installation entanglements created when a single way is anointed. Worse, it discourages ecosystem activity around the project.

The first successful installers were targeted at a single vertical stack, such as Kubernetes on CoreOS or a version of Salt Kube-up on Google. These monolithic install approaches are helpful jump-starts but limit community diversity. Yet, if they are presented as “the right way,” then other participants will be discouraged from innovating. It only takes one variation in a vertically integrated process for the whole process to become exception hell.

Our early experience with Crowbar was that our initial installation ideas were not going to remain useful as the project and targets expanded. It was important that we could maintain Crowbar AND replace it with something better. The fundamental lesson is that operations around projects is an evolving practice.

Hindering the evolution by selecting winners early actually slows the development process.

This may be counter intuitive; most projects want to embrace “easy button” installers to reduce start-up costs. These critical builders of initial traction may actually undermine projects as they enter a operational maturity phase by creating a fidelity gap between initial and sustaining use cases.

— Rob Hirschfeld

October 10, 2016
Mainframes and the Myth of Bimodal in DevOps
Rewind 20 years ago when we lived in a business world where big companies with unlimited resources beat out smaller, more strapped companies. The digital era—also known as the “age of the customer”—is changing all that. Where big once beat small, fast now beats slow. The DevOps approaches of startups often make them more adept at continuously delivering the innovations and conveniences that earn customer raves. Bigger, established companies often feel disadvantaged‚ shackled by an outdated culture and inflexible processes that haven’t kept up with the pace of change.

Gartner coined the term “bimodal IT” as a way for older IT establishments to keep pace with agile startups. In the bimodal model, IT teams are separated into two distinct styles of work—one focused on supporting predictable, well-understood workloads with legacy systems and another that is more fast-moving, experimental and adaptable in meeting users’ constantly evolving demands and flourishing digital expectations.

But the bimodal concept is not well-aligned to the spirit of innovation underlying modern application development and delivery. This is especially true for bigger organizations using mainframes. Here’s why.

Successful DevOps Depends on Mainframe Agility

The goal of DevOps is to roll out high-performing software and updates, ahead of the competition and with maximum speed, ease and cost-efficiency. In a DevOps model, software delivery can only happen as quickly as the slowest team. Modern web and mobile applications tend to span multiple platforms in their end-to-end transaction—from front-end web servers (systems of engagement) all the way back to mainframes (systems of record). IT places a lot of emphasis on systems of engagement, often taking back-end systems of record for granted. The reality is, the value in systems of engagement cannot be realized without the systems of record—making it essentially all one system. Developers supporting modern applications must be able to maneuver across and between all platforms, especially within mainframe code and data.

Let’s look at a tactical example of this. According to industry research, 80 percent of the world’s corporate data—including customer data—continues to reside on mainframes. When a mobile web application is used to make a bank deposit, or purchase a product, for example, the transactional component of those applications typically is executed on back-end mainframes.

This means mainframe applications are crucial to development efforts. What happens when mainframe developers and testers are locked into waterfall processes or don’t have the modern tools necessary to understand application logic or access mainframe data for testing? Frustration mounts and the overall innovative effort is unnecessarily stalled.

Stability and Agility are not Mutually Exclusive

The longstanding myth that stability and agile cannot co-exist is simply untrue. Unfortunately, it’s at the core of negative perceptions surrounding the mainframe today—people think the mainframe environment is too slow and cumbersome to support agile development. Development teams may think if they have mainframes, the only approach it can support is waterfall.

The reality is, though, no customer will accept poor availability and performance in exchange for new features delivered quickly. People with hands-on experience painfully understand that slow, waterfall delivery methods produce far greater quality issues and missed expectations, resulting in competitive risk. Additionally, large enterprises taking the “slow and stable” route give startups a head start in beating them to market, with offerings that customers want now.

Alternatively, Agile process and DevOps tool chains enable scrum teams to test quality in terms of usability, function and stability early and often in the development cycle—when the time to remediate is the fastest and the cost the lowest. Agile process and DevOps tool chains build higher quality applications and enable continuous improvement in quality over time.

Re-Platforming is not Necessarily the Answer

Some DevOps proponents believe moving off the mainframe—to a supposedly more modern, commodity server-based distributed architecture—is the best way to circumvent perceived challenges to nimbleness and agility. But this is almost never the best answer. The mainframe retains its distinction as being the most powerful computing platform on the planet. It remains deeply embedded in large organizations, setting the standard for reliability, security and availability for mission-critical data and transactions. The new IBM z13 mainframe, for example, can handle the load of 100 Cyber Mondays every day, 365 days a year. Currently, mainframes process 30 billion business transactions per day. The post-modern mainframe is an engineering marvel unequaled as a system-of-record, business compute platform.

The mainframe also offers cost advantages. Studies have shown that the mainframe is more cost-effective than commodity servers in handling the massive computing volume increases instigated by mobile applications. This is because decreases in commodity server pricing have not kept pace with the increase in computing volumes. In addition, leading vendors have worked together to deliver intelligent cost management on the mainframe, enabling workloads to be distributed in a way that keeps mainframe licensing costs (MLCs) down.

In our view, “biomodal” advocates for the exact problem that must be solved. We think the answer is not replacing the mainframe, although if mainframe development is left unaddressed, this can present obstacles to DevOps. Rather, the answer is to evolve the mainframe to keep up with the pace of DevOps. There are mounting examples from large, high-performing organizations that it is possible to apply modern practices to the fullest extent, including mainframe applications. These organizations do this by giving developers the tools they need to work on mainframe code and data just as they would with any other programming language on any other platform, while meeting the escalating quality expectations of the digital age. For example:
- Mainstreaming the Mainframe Environment – Mainframe users need to move away from the antiquated “green screen” mainframe environment, adopting state-of-the-art interfaces and tools. This is the first step enabling developers to work more fluidly on the mainframe, in an environment that’s much more familiar and comfortable to them. Organizations that fail in this adoption will have a very difficult time enabling the next generation of development talent to leverage invaluable and irreplaceable mainframe code and data. This will place a huge hindrance on DevOps success.
- Leveraging Tools from the Java Development World – One example is visualization capabilities, which enable developers to quickly and easily identify critical interrelationships between apps and data. This helps them to sidestep unintended, undesirable impacts as they modify applications, increasing confidence and mitigating unnecessary delays.
- Make it Easier to Identify Mainframe Code-Level Issues – Developers need tools that enable them to identify problems with mainframe code; for example, when it is the source of a widespread user performance issue, or when it is consuming too many resources and driving up costs unnecessarily.
Conclusion

In our view, “bimodal” is not the solution, but a broken state of disadvantaged affairs for too many organizations. From a competitive vantage point, bimodal will only hurt DevOps teams and hold back the innovative efforts of the organizations they support.

The bimodal IT model is a road to inevitable failure for those enterprises seeking success in the digital economy. This is especially true for organizations that rely on mainframes. A smartly leveraged mainframe is an extremely valuable competitive asset that needs to be removed from its silo, evolved and smoothly integrated into modern, heterogeneous development environments.

It is possible for big, established companies to adopt DevOps, become agile, and move as fast as startups. But the ability to think and act like a startup requires some major fundamental changes, particularly at the level of software development. The good news is, the legacy technologies that are often perceived as impediments can actually be a tremendous competitive advantage, if they are harnessed and evolved in the right manner.

About the Author / Christopher O’Malley

Christopher O’Malley is CEO of Compuware. He has nearly 30 years of IT experience with past positions including CEO of VelociData, CEO of Nimsoft, EVP of CA’s Cloud Products & Solutions and EVP/GM of CA’s Mainframe business unit, where he led the successful transformation of that division. Connect with him on LinkedIn and Twitter.
September 23, 2016