devopsy.kinsta.cloud

Tag: culture

Challenges in DevOps Standardization
Many organizations still face challenges in their Devops adoption journey enterprise-wide, especially when there is no standardization in tools, non-repeatable processes and adhoc practices and culture barriers. Here are some of the challenges in standardization and possible solutions to achieve standardization in DevOps as part of DevOps transformation journey.

Business Case

A survey conducted within multiple customer teams revealed they face challenges in standardization of tools and streamlining the process gaps due to lack of end-to-end orchestration, manual intervention, legacy practices, inconsistent environments, longer provisioning time and several other factors. Taken together, they necessitate standardization in DevOps across the enterprise to make it repeatable, predictable and consistent.

Challenges in Standardization of DevOps

Tools Challenges
1. Multiple tools to evaluate with one or more overlapping features in the tools, learning curve involved to evaluate/do proof of concept and choosing the tools with short-term goals instead of long-term perspective.
2. Limited knowledge of tools, used from the perspective of automation in their specific area, resulting in siloed automation and using manual, ad hoc process.
3. Lack of out-of-box integration with most tools used in the CD pipeline and teams investing time to build custom integrations.
Process Challenges
1. There is no central DevOps team who can evaluate and recommend the right set of tools to be used, with individual project teams/groups try to set up their desired tools and serve their teams (Dev/QA/Ops), with no clarity on availability of infrastructure that can be shared across groups.
2. There is no defined process framework detailing the different tenets in DevOps, including what activities and approach are needed to move to desired state, leaving teams to achieve their desired state of automation without a structured approach.
3. There is a lack of knowledge on defining right KPI for each DevOps capability and how to measure the metrics to determine efficiency of the DevOps automation process and identify scope of optimization.
4. There is a lack of SME knowledge on the application, application stack and environment management flowing between Dev and Ops teams to arrive at a standardized process.
5. Multiple vendors are managing the different stages of an application and each following their own defined process for development life cycle.
Cultural Challenges
1. Standardization on metrics, process, practices, governance and organization-level control over DevOps implementation is challenging without a central authority (a central DevOps team) driving these practices across the enterprise.
2. Teams operate in silos and follow their own DevOps practices and think in silos without having a holistic view, resulting in broken processes that are inconsistent.
3. Teams still lack training in DevOps on the tools, practices to be followed in CI, continuous testing, continuous deployment and operations. Lack of DevOps skilled resources/expertise brings challenges in implementing end-to-end automation.
4. DevOps is considered a concept, with every DevOps implementation unique for every organization and no standard approach for adoption of DevOps, causing confusion where to start and how to start. In addition, there is a conflict of interest and power of influence from multiple stakeholders while standardizing on tools.
Proposed Solution(s)

DevOps Adoption Framework (DAF)

DevOps adoption framework (DAF) is a comprehensive framework for due-diligence, tools support and solution accelerators to create systematic and well planned-out transition road map and implement a continuous delivery toolchain and overcome the common challenges in DevOps implementation.

Choosing the Right Set of Tools and Stack

Tools selection should be based on the requirements and what best suits the need, decided based on various factors discussed in the attached whitepaper, and using one of the methodologies such as points given for out of box features and different factors/constraints.

DevOps Automation Platform with Template-Based Infrastructure and Toolchain Orchestration

DAAS (DevOps as a service) is an automation platform, which serves as a central platform to drive standardization across tool use, provisioning infrastructure, toolchain, managing the life cycle of VMs, bringing in central authority and governance, defining the KPI, monitoring and measuring the metrics, monitoring end-to-end pipeline giving real time analytics, providing self-service capabilities across multiple teams, bringing in audit and compliance, continuous improvement and drive consolidation and standardization.

Future Direction
- Increased focus on standardization of DevOps metrics with analytics and monitoring to identify the bottlenecks early and big data used for predictive analytics through the delivery pipeline.
- Rapid DevOps adoption across the enterprise, with standardization and consolidation critical factors for scale, compliance and utilization with self-service DevOps automation platform enabling end-to-end pipelines.
Conclusion

The critical success factors for enabling DevOps adoption across enterprise needs focus on central DevOps team driving DevOps adoption with governance enabled with DevOps adoption framework and consolidation of the processes, standardization on metrics and tools, enabled with a DevOps automation platform.

For a deeper look at the survey, download the white paper here.

About the Author / Lavanya Subbarayalu

Lavanya Subbarayalu is Senior Architect working with Technology Office in HCL Technologies. She has expertise in Azure, DevOps consulting & Microsoft technologies. She is associated with DevOps COE, working on design and Development of DevOps solutions and consulting tools. Connect with her on LinkedIn.
April 5, 2017
‘Not My Problem’: A Big Problem for DevOps Teams

Not my problem (NMP) – (n)
1.    a statement, or position, of apathy expressed by those who perceive they are external and unaffected by a negative predicament. While sometimes warranted, it is typically uttered by those who perceive themselves as powerless; can’t be bothered; are too lazy, or are selfish non-contributing leeches. See also “complete cop out”
2.    an attitude that will stymie attempts to implement DevOps in your organization and will thwart success
3.    (archaic) Actually not your problem

While perhaps it’s become more frequent in our culture of relative indifference, some of our oldest stories bear witness to how timeless the phenomenon “not my problem” is. For example, in the the biblical story of Cain murdering Abel, God asks Cain afterward, “Hey Cain, I can’t find Able. Do you know where he is?” Cain responds famously, “Am I my brother’s keeper? (NMP)” It crosses country borders and language barriers, too: A transliteration of the same sentiment in Polish is, “Not my circus, not my monkeys.”

And we’ve all been guilty of thinking it at work: “That’s not in my job description,” “I’ve got to focus on my own projects rather than bail that person out,” “That’s the Sales team’s role.” These thoughts pass uncharitably through most of our minds, despite most of our unwillingness to say something so callous to a person we witness being affected by our indifference.

It would be naïve to think that this cultural indifference was held at bay by DevOps teams, all united under the same goal of turning out quality product in the most efficient ways. Unfortunately, the NMP problem can pervade even the most motivated and scrappy teams, and it has a unique power to dismantle attempts at positive culture-building on those teams, which are necessarily built on the core values of sharing and learning.

To have DevOps teams that embody these fundamental characteristics, we have to shift the individual and team perceptions of the ownership of success and failure. It can no longer be an Ops or Dev or QA or InfoSec issue. It has to be OUR issue.

Here are some key items that I have found fundamental to combating NMP in my teams:

1.    Change Starts with You

If you don’t want to see NMP present in your team, you need to ensure you are not a contributing source. Before ending your day, dedicate at least a few minutes of reflection to the day’s interactions. Did you actively seek out and identify areas where you can be of assistance? Upon reflection, did you miss an obvious chance to help someone on your team? Keep a daily journal and try to list these things. This will help instill the behavior into your normal regimen.

2.    Culture is What You Tolerate, not What You Mandate

We have all been a part of “hoorah” speeches where we hear the “we are a team” speech; perhaps we have given them ourselves. We have also walked away from said speeches with a cynical demeanor, knowing that they are just words and everyone is out for themselves. While words are important, words without action are just noise. As a leader, you need to communicate your “No NMP” policy, and you need to act upon it. If you are the manager of a team, you must be willing to take corrective actions against people who continue to exhibit a NMP attitude. If you are a member of a team, you must be willing to hold your teammates accountable. Apathy kills. It takes time and patience to repair NMP behavior; sometimes, however, you might need to part ways with the source of NMP. As a manager this may mean letting someone go, or as a team member it can mean letting your team go.

3.    Everyone’s a Leader

As readers of my blog know, I do not equate leaders to managers. I am sure that most of my readers have lived that experience firsthand. But having a non-NMP team requires us all to be leaders, even if your manager is not. We must actively try to identify areas of weakness and and areas of improvement and seek out ways to address them and turn them into areas of strength. Sometimes that even means mentoring your manager. Yes, I know this can be tricky, but good managers will be receptive and open to constructive feedback. If you manager is not and is instead a source of NMP, see No. 2, above.

Many companies and teams have been eaten up with the cancer of NMP. There aren’t enough team-building events and “trust falls” to counteract a culture where some people break for the door while others work to resolve an issue. Because it is this behavior that causes the whole culture to collapse. It can’t be “All for One, and One for All (except Bob, because he’s a selfish jerk).”

It’s incumbent upon all of us, as team members and as managers, to fight the NMP battle. The culture of our teams most certainly is our problem.

About the Author / Adam Bowen

Adam is a technology leader with over 15 years of experience in technical, sales, and senior leadership roles. As the senior member of the Office of the CTO at Delphix, Adam works with other thought leaders executives of the Global F100 to eliminate key business constraints through innovative new processes, offerings and technologies. Adam serves as an evangelist on behalf of Delphix to communicate the company’s vision, values, and ideas with customers.

April 5, 2017
All Day DevOps: DevOps at Massive Scale
When you have a billion users, people notice. That’s where our story about DevOps and Yahoo! starts. For Kishore Jalleda and Gopal Mor, both engineers at Yahoo!, when something goes wrong on a Yahoo! page, people will notice. Correction: a lot of people will notice.

Of course, Yahoo!, like all services on the internet, constantly improves its products. In fact, it has 100+ iterations and experiments happening at any given time. Some changes bring new innovation to the forefront and others alter the user experience.

When iterations and experiments are served in front of loyal users who have become comfortable with a specific user experience, they sometimes react with a natural resistance to the change. When change causes, or appears to cause, breaks in service, the backlash can be crippling. Frequent spurts of backlash resulting from these changes became known to some Yahoo! Insiders as “the bad years.”

At the recent All Day DevOps conference, Kishore and Gopal shared how Yahoo! turned to DevOps practices to recover from “the bad years.” In their presentation, “Launching Products at Massive Scale: The DevOps Way,” Kishore said, “DevOps is about eliminating technical, process and cultural barriers between idea and execution, using software.” Specific to each of these, Kishore recommends the following:

Culture of ownership and excellence: Own life cycles within Development, fix root causes and have pride in the product.

Processes: Design or engineer processes to be fast and agile. Work should be iterative, support learning and provide fast feedback cycles. Let the machines to the heavy lifting.

Tools: Solve operations problems with software. Use open-source tools that are self-service, reusable, friendly and easy to use.

At Yahoo!, the DevOps practice is built on three functional pillars: deliver products to market quickly; prevent defects from reaching customers; and, repair production issues quickly.

Speaking of repairing production issues quickly, Gopal discussed resiliency. With its user base, downtime (or lack thereof) is critical to Yahoo! and the challenges are many:
- Distributed multilayer architecture
- 100s of subsystems
- Complex request flow
- Change is the only constant
While it may be counterintuitive, Gopal demonstrated how the combined system is weaker than the weakest subsystem.

To ensure optimum uptime, Gopal said we need to:
1. Analyze the entire range of failure types
2. Understand their rate and impact level
3. Plan to cover all failure types
4. Conduct fire drills – test, test and test.
Specifically, how does Yahoo! ensure high availability? It maintains four layers of resiliency in the serving stack:

Speculative retry – Deliver the page again after a predefined latency is exceeded. This addresses longtail latency and intermittent failures.

Per-module fallback – Cache non-personalized modules to front-end servers and serve cached content for failed modules. You need to ensure the cache is refreshed often, that you implement strong validation of the cache’s data and you check for backward compatibility if the TTL is high.

Fullpage failsafe – Cache the entire page without personalized data and ads and with minimal interaction. This is used when the entire page cannot be served. Yahoo! uses auto-scale AWS servers to serve these pages so its servers are not negatively affected.

Failwhale page – The “we will be right back” page that lets users know, yes, your internet is working, but our service isn’t – temporarily. Please try again. Obviously, a last resort, but a necessary one.

There is a lot here and you don’t have to have a service at the scale of Yahoo! to benefit for the experience. You can dive into Kishore and Gopal’s full All Day DevOps conference session (just 30 minutes) to learn more about their learnings from the DevOps front lines. The other 56 presentations from the All Day DevOps Conference are also available online, free-of-charge here.

This blog series is reviewing sessions from the All Day DevOps conference from November, which hosted over 13,500 registered attendees. Last week I discussed, “Docker, the New Ordinary.” Next week, look for “System Hardening Using Ansible,” from Akash Mahajan.

— Derek E. Weeks
March 13, 2017
CD 101 Podcast: How to Build the Modern Software Factory with Agile Development and Continuous Delivery
And now, for Episode 3 … you decide if it beats “Star Wars – Revenge of the Sith.”

I think we accidentally just set the bar pretty low, but …

… here we go. Let’s begin with a prerequisite pithy quote from an objective industry observer:

“If agile software development was the opening act to a great performance, continuous delivery is the headliner,” according to Kurt Bittner of Forrester Research.

Kurt explains this concept further by saying, “The pace at which consumers expect change is causing a crisis in application development and delivery … agile was a good start, but it wasn’t sufficient to drive better business results; continuous delivery forges the broken link in the value chain, connecting business strategy with business results.”

So there, you have it. At least according to Forrester, continuous delivery (CD) is one of the most important evolutions of agile methodology within the modern software factory. But, there’s more.

Andrew Binstock from Dr. Dobbs fairly well echoes this sentiment, saying: “It’s becoming clear that agile alone is no longer sufficient to keep up with the demands of the present day … software complexity and the need for constant updates are making it difficult for leading edge organizations to keep up without moving beyond agile and embracing continuous delivery.”

Episode 3 of our podcast series discusses the importance of Agile Transformation along WITH Continuous Delivery to automated some of the development, test, and release efforts to remove some of the bottlenecks downstream in the software development life cycle in the build, test and release phases.

Lean Agile + CD can create these 1+1=3 types of results.

Episode 3 of the Continuous Delivery 101 Podcast can be found by:
- Listening to it on your computer on YouTube
- Downloading it from iTunes store
- Downloading the podcast from your Podcast App by searching for “Continuous Delivery 101 podcast”
Thanks for listening!

— Scott Edwards
March 13, 2017
DevOps is Dead! Long Live DevOps!

Some are seekers on the quest for the one, true DevOps. They were misled. I’m here to say: Give it up. Whatever you find at the end of that journey isn’t it. That one true DevOps is dead. Dead and buried. The search is pointless or, kind of worse: The search misses the point altogether.

DevOps began as a sort of a living philosophy: about inclusion rather than exclusion, raises up rather than chastises, increases resilience rather than assigns blame, makes smarter and more awesome rather than defines process and builds bunkers. At any rate, it was also deliberately never strictly defined.

In 2010, it seemed largely about Application Release Automation (ARA), monitoring, configuration management and a lot of beginning discussion about culture and teams. By 2015, a lot of CI/CD, containers and APIs had been added. The dates are rough but my point is: That’s all still there but now DevOps discussions today also include service design and microservices. Oh, and the new shiny going by the term “serverless.” It is about all of IT as it is naturally adapting.

And this emergent, inclusive content is possible because it’s always just people talking practices and sharing how to optimally do things. Infra as code, “cattle, not pets,” webscale, version and automate all the things, horses not just unicorns, etc. Lots of conversations, insights sharing, presenting.

Good times.

There are more DevOps Days naturally occurring globally this year in 2017 than ever before. Note I wrote “naturally”—DevOps was always an organic movement, a grouped (and, whoa, global) response to increased app release pressures and the desire to select for teamwork over turf wars.

So we started with DevOpsDays and those blossomed all over. Then we got the Phoenix Project, the DevOps Enterprise Summit (now in US and EU formats!), and more great books including “Infrastructure as Code,” “System Reliability Engineering” and, finally, “The DevOps Handbook.” It’s all solid, informative stuff and I believe as a community we’re smarter, happier, and far more effective because of it.

Even those naysayer analysts—you know who you are—have come around to accepting the DevOps goodness.

DevOps lives when we continuously adapt to managing emergent properties spawning system-level feedback loops in our IT and across our businesses. To do better, faster, higher quality things to delight the customers.

Any discussion of DevOps is truly about continual IT innovation adaptation because the rate of IT change is also changing. Fast and faster. All that means starting with the roots like CLAMS and moving forward with containers and embracing the journey to quickly delivering better customer experiences.

So, the concept of DevOps is evolving, just as our understanding of what’s needed is evolving.

— Christopher Little

March 7, 2017
Release Automation: Delivering the Big Quality Payoff

Have you ever gone down the path of testing the latest release of an application only to find out that you had the wrong version?

Maybe you accidentally grabbed the wrong build or you didn’t have the correct configuration file for the environment you were testing against—or your database wasn’t updated correctly, or the script you were using was still pointing to a prior version of the software.

Whatever the case, we can agree that keeping track of all the various artifacts, database schema changes, configuration parameters and other components needed to accurately test and deploy an application is both time-consuming and prone to errors.

Today, many companies are looking to DevOps, continuous delivery and agile development methodologies to shorten their application delivery cycles and reduce the number of errors and the costs associated with application deployment processes. Nearly all enterprise companies are software companies, and their mere existence depends on delivering differentiating products and services via software to their customers.

It’s imperative to be able to react to customer demands and competitive pressures quickly, if not immediately, to stay ahead of the competition. Gone are the days of yearly release cycles.

Today we see companies deploying new capabilities into production environments on a monthly, weekly or even daily basis. At some companies, hundreds, if not thousands, of deployments are occurring across dev, test and production environments daily. The old methodologies we once used simply can’t support this type of volume and cadence.

Employing Release Automation

So, how are these companies transitioning to this new way of thinking? In a word—automation; specifically, by automating early and often.

Everything about your release process is a candidate for automation. This includes environment provisioning and configuration automation, build automation, test automation and release and deployment automation.

A customer of ours recently implemented CA Release Automation, and within eight weeks had automated 90 percent of its deployments. Deployments that once took more than five hours and a team of people now take a matter of minutes.

“This not only saves us a huge amount of time, but it also mitigates the risk of human error being introduced by manual processes,” said Tim Dallmann, platform engineer at Jewelers Mutual.

Another customer, a major airline, implemented CA Release Automation and immediately reduced their production outages from an average of 237 over a three-month period, down to 15, seven, six and five over the following four months.

And Molina Medicaid Solutions reduced their cost of human error by 90 percent, all the while responding quicker to emergency deployment requests and improving deployment quality.

Automation is critical, but I believe these results prove that implementing a framework to manage and deploy release processes in addition to automation works even better.

Increased Flexibility and Visibility

Continuous delivery helps break down barriers often seen between development and operations teams. It provides, even forces, a more collaborative approach and automation is a key enabler of this.

“One of the best things about automation is that it gives us the flexibility to skill-up end users—deployments in our test environments are no longer solely the responsibility of the operations or IT teams,” explained Dallmann.

With an application release automation (ARA) solution, artifact, build, configuration and release processes are no longer tracked with spreadsheets or embedded in complicated scripts.

Increased visibility into these processes ultimately enables all stakeholders to take a more active role in the release process, and access to real-time information can significantly reduce the amount of time and resources needed to resolve and recover from problems when they do occur.

Automation leads to:

Integrating with the Ecosystem

An enterprise ARA solution can help you create a single control point to streamline all your application delivery processes for better predictability and reduced risk. A critical component of this is the ability to integrate and orchestrate tools and technologies required along the continuous delivery pipeline, including source control, automated testing tools, provisioning and configuration management solutions and change management and performance management solutions.

Integration with an application lifecycle management solution can provide even further alignment, governance and traceability between development projects and what is ultimately being tested and promoted to production.

“By integrating the two solutions, we will have a complete audit trail of deployments to help us understand how, where and by whom stories have changed,” said Phil Nickolai, director of application services at Jewelers Mutual.

“What we really needed was a solution to streamline our whole way of working,” he said. “We had numerous people working on projects, competing for resources and causing complexity. Now we’re back in control and getting smarter every day.”

The Time Is Now

Let’s get rid of the problem that exists between the keyboard and information kept in spreadsheets, scripts or emails. Not only will you stop wasting time testing the wrong version of an application, but you can also ensure you have the right test data, the appropriate test cases and the correct configuration parameters for your environment.

Stress-inducing release weekends and nights become a thing of the past. Developers can spend more time on developing innovative capabilities that generate revenue and keep customers coming back for more.

— Tim Mueting

March 7, 2017
Be Here Now – Agile and DevOps in the Current State of Synergy

If there’s one thing that our culture does not lack, it’s a litany of pithy catch-quotes meant to remind us to focus on the task at hand; yet, this often runs counter to our tendency to fixate on matters of the past and future.

While we are so often encouraged to “live in the moment” and “seize the day”, this is hardly an easy task, especially as related to any sort of emerging phenomenon that compels us to improve on historical missteps, and predict how things should change for the better.

Register today for the 3^rd Annual DevOps Virtual Summit.

With the methodologies of agile and DevOps the tendency is, of course, to do the same, harping on the inefficiencies of previous software delivery models and painting an elaborate portrait of how related workflows are being transformed overnight.

The simple truth is that it’s easy to get excited about concepts that hold such tremendous potential in recasting an increasingly critical vital set of processes, and lose track of where things actually stand.

To note, while a recent research project published by Coleman Parkes found that over 80 percent of organizations have adopted some manner of agile and DevOps practices, and see them as critical in transforming their internal software factories, only a third have adopted the methods on a broad scale.

Point being, we’re still pretty early.

That said, we in the business of helping organizations adopt related culture and tooling have gone to great lengths to encourage everyone that this change should be pursued with dogged intensity. We have sliced and diced the data to illustrate how those organizations at the forefront of so-called digital transformation, enabled by agile and DevOps, are realizing the results in the form of increased velocity.

The same research finds that “advanced” adopters of the two concepts are, in fact, seeing significant improvement in time-to-market, customer satisfaction and new business growth.

But research is just that, and often not completely aligned with real-world scenarios. Someone who takes the time to answer a survey about such matters, one could reasonably expect, is the type of person far more likely to be involved in agile and DevOps activities.

My personal conversations with everyday practitioners [aka other parents on the sidelines at my kids’ sports and social events, many of whom work in IT] tell me that we’re much more oriented to the “pockets of innovation” phase at present, than any sense of pervasive adoption.

So this begs the two-part question – where do we really stand, and what do I really need to care about?

The only reliable source of record in my mind is the perspective of people already doing the work, beyond what I can discern amid basketball games and middle school theater. Real-world practitioners are the voices of reason that allow us to “be here now” and address the specifics of practical agile and DevOps implementation.

Over the past several years, I’ve found one of the best sources of such input to be the customer panels available at industry conferences and on webcasts. When it’s time for me to sit down and crank out some related content, I often go back to these recordings to remind myself where the rubber actually hits the road.

On Tuesday Mar. 7, we’ll host just such an event, the 3^rd Annual DevOps Virtual Summit, where agile and DevOps leaders from organizations including CNN, GM Financial and United Airlines will share their thoughts on how this ongoing transformation is playing out, and where the remaining challenges lie.

Along with those experts, we’ll also hear from Dr. Nicole Forsgren, one of the leading researchers in the space who is in seemingly constant contact with people leading the charge and working in the trenches putting agile and DevOps into practice.

As a marketer, and previously as a journalist, everything that I know comes from direct interaction with those colleagues willing to share their hands-on experience garnered via doing this work every day.

Register for the event and join us to find out where agile and DevOps actually stand today.

— Matt Hines

February 22, 2017
CD 101 Podcast Episode 2: The Agile Enterprise
And we’re back with another episode of the “Continuous Delivery 101 Podcast.”

In Episode 1, we covered the three waves of IT disruption—and how these disruptions, combined with the move toward digital transformation, are drastically changing every industry.

This, of course, underlined how accelerating speed-to-market is clearly more important than ever before.

In this second episode, we talk about Agile methodologies with expert Dan Rice, who addresses how these practices can enable business strategies to be implemented at greater speed.

Listen closely and you’ll learn more about what Agile actually means across various teams, roles, product owners and scrum masters, along with related points on DevOps and CD tooling.

All of these topics are great food for thought as you think about how you can move to build software faster and with increased quality. Yet, if enterprises only focus on these issues, they aren’t tackling or talking about the real problem in their organization that Agile can solve.

It’s time to take the conversation beyond the team level and talk about how work flows to every involved team, enterprise-wide—specifically, about how initiatives can expand across multiple teams, how to manage shared resources and how to pull in other dependent teams.

For Agile to work, it has to expand universally and be ingrained further at program level, to understand precisely how work is coordinated and flows across every domain.

You can listen to Episode 2 of the “Continuous Delivery 101 Podcast” via:
- YouTube
- iTunes store
- Or by searching for it in the iTunes Podcast App under “Continuous Delivery 101 Podcast”
Thanks for tuning in to the “Continuous Delivery 101 Podcast.” We hope you’ll continue the conversation by posting your comments below or in the comments section on YouTube or iTunes.

Meanwhile, keep tuning in: We’ll be back soon with another episode!

— Scott Edwards
February 21, 2017
4 Top Tips for Developing a New Feature

As a backend developer at CA BlazeMeter, I recently had the chance to work on a big feature for our web app. The feature required a massive change on our backend code, a big change on our frontend and multiple migrations. You know what I’m talking about—it was one of those projects that developers fear.

But, I gained a lot of valuable experience from the project. Not only did we deliver a kick-ass feature, I also learned the importance of teamwork and patience when developing something new. Armed with those two values, and the right tools and technologies, you can deliver a new feature quickly and efficiently.

Here are a few more things that I learned that should prove helpful whether you’re developing a major feature or one that only requires a minor code change.

Tip 1: Plan to understand the development requirements

The process of development starts way before you start coding, with planning. After the product manager decides what needs to be developed, R&D needs to understand the cost: how many days (or weeks) of development the feature will take, how many people will be needed and which APIs are required.

To do that, the product manager needs to conduct at least three “grooming” sessions with backend, frontend and QA leaders. The grooming sessions start with the product manager presenting the product flow and requirements.

Since no one wants to read a lot of text (except perhaps the awesome readers of this article), the product manager should prepare mockups. They don’t have to be designed or colorful, but they do need to show the team what the feature looks like in his or her vision. During our grooming sessions, we used Balsamiq, which lets you create screen sketches for how the product is supposed to look, feel and work.

Editing screen sketches using Balsamiq

After the initial presentation, the sessions should include a lot of open discussion—asking questions, checking for answers and examining compliance, etc. —until everyone is satisfied and understands what the feature requires from them and from others.

This is the time and place for frontend, backend and QA to give their opinion about the schedule and to alert and flag any potential problems in development or extreme use cases. It’s the role of the product team to answer questions and change the flow and requirements until there are no R&D rejections.

Afterward, the frontend and backend leaders need to create a feature development plan, sitting down to determine what data they need to transform the product vision into a real thing. Or in other words, which APIs are required for each flow or button, how long each step is supposed to take, including buffers, and the order of development. APIs are the agreed “contract” between the development teams. After determining the contract terms, each team can proceed individually.

I recommend you write your APIs in Swagger, which turns YAML files into detailed HTML pages with all the APIs.

Swagger Editor

This plan needs to be transparent to everyone working on the project. In fact, it’s even better if each project participant can add and describe scenarios. Tools such as TestLink and Atlassian do just that. Then, when the code is ready, developers can go over the list and see everything is covered.

We will need to stick to this schedule later on, especially when new demands unrelated to the feature come in, such as fixing bugs. Sticking to this plan is the best way to distinguish the wheat from the chaff of the development process.

Start writing automation ASAP

After planning, we want everyone to start working, and not only the backend. To do that, the backend can prepare mocks, either through outsourcing services such as JSON servers, which show inputs and outputs, or by writing dummy code.

I mentioned earlier that we used Swagger to write APIs. We also used it to generate a server in NodeJS/Python, like this:

Generating a JSON Server through Swagger

From this moment on, frontend can begin writing and building screens and QA can start writing API automated testing, by using tools suh as JMeter. At this point, frontend and QA have the response codes and response bodies; they know which features to put where; and which APIs they need and how to use them.

QA also know which data they’re missing, such as field names and field data. Sure, the backend code isn’t ready yet and all of the automated exams will return “red” instead of “green,” but the main issue is making sure everyone is as prepared as possible.

This includes load and performance testing. You can use open-source tools such as JMeter, which is enhanced by CA BlazeMeter. By uploading our test script into, configuring the number of virtual users and ramp-up time, determining geo-locations to test from and analyzing KPIs to determine trends and the health of our system, we can ensure our product is delivered to our customers without crashing.

This is also true about sending code to QA: Developers should test their code themselves and ensure that when they commit their code, it works 99.9 percent of the time.

By starting automation as soon as possible, you can minimize future bottlenecks in the development process.

Ensure the team communicates and works together

Like I said before, teamwork is the most basic requirement when creating a new feature. This is because new questions arise during development and because requirements change. Therefore, everyone in the team needs to be in sync all the time, from the product manager and the designer to the developers and QA.

Open communication lines are crucial because everyone is working on the same product at the same time, but each developer or QA should be working on a different part of the product and on different functionalities. Communication ensures each person’s work is compatible with the rest of the team’s work. This saves time, prevents frustration and redundancy and ensures maximum professionalism.

To make sure everyone knew what was happening, we used Slack and opened a dedicated channel for the creation of this feature. We also had daily (short!) standups where we discussed what we have and what is missing.

Opening a Slack channel

Communication also includes team and assignment management. JIRA, our primary issue tracking tool, offers Kanban-style, Scrum and fully customizable boards, which show the progress of each development task, including the developer assigned to it, and his or her exact status or phase.

This helps you understand what other people are working on, and just as importantly, where people are stuck. If you can, get everyone to work in an open space and put a large screen in the middle of it. This lets you stay in sync at all times.

We used JIRA to open and manage bugs as well. This was helpful when multiple QAs found the same bug. It’s important to carefully choose which tickets are opened and put them under one epic, so everyone stays focused.

JIRA Dashboard

Another important aspect of communication is ensuring everyone uses the same terminology, as sometimes people might use the same term to describe different things or different terms to describe the same things. To prevent mixups that could waste your time, make sure everyone is on the same page. For example, do you say “system admin” or “<name (ex. BlazeMeter)> admin”?

Work on the same development environments

It’s important that everyone works on identical development environment s and even on the same version of the tools and libraries. Otherwise, it could cause an (in)compatibility mess, including a lot of unnecessary back and forth and even might cause bugs to pop up. This environment needs to be as similar as possible to production, but only on a smaller scale.

Conclusion

Now look at the feature that came out of all that work here. I’m so grateful that I had the opportunity to work on it with an awesome team.

I hope these best practices help you the next time you develop a new feature. Don’t forget to keep going agile after development and delivery by running tests in a continuous environment such as Jenkins for every commit.

CA BlazeMeter works with Jenkins, ensuring you can load test in the cloud and analyze your data while shifting left. You can also use different APM tools, such as New Relic, AppDynamics and CA APM to alert you about errors before QA, which saves time.

— Dor Atias

February 7, 2017
Culture Comes First in Bringing DevOps to the Mainframe
One of the most difficult challenges in engineering, oddly enough, isn’t technical: the challenge is culture transition management. In my 20-plus years of managing engineering teams worldwide, I’ve led teams through countless organizational adjustments and process improvements toward high-performance and quality, beginning with QA segregation to CMMI to ITIL, AGILE and SAFe.

Today, mainframe organizations have emerged as the focal point for enabling DevOps capabilities and practices. Why the mainframe? Well for starters, business critical applications rely on mainframe data and business components. These applications are mission-essential and that sustained innovation requires solution sets that deliver continuous testing, continuous delivery and continuous feedback on the mainframe platform.

But culture comes first.

Understanding Your Key Players

In conversations with our customers we’ve observed an interesting dynamic emerging within mainframe application development teams. Within these teams are experienced developers who were instrumental in creating the applications that helped make their companies so successful. Over time, many have become set in their ways, adopting a motto of, “If it’s not broken, why fix it?” Such developers are not interested in trying out new things, including programming languages and tools used by the modern-day programmer.

New to the scene are the millennials who have also taken up the mantle of enterprise software engineer. They are a bit more open-minded and bring a different programming language profile, including Java, .NET and C. These developers might be learning how to code APIs with the support of management, and have likely dabbled the basics of how to code iOS apps. Millennial developers are typically much more active online and, as a result, prefer to leverage Twitter and other social channels to stay informed with the development community.

United We Stand

Who, then, should be involved in the journey to realizing DevOps? The answer, of course, is both. Because we’ve heard it reinforced among our users we talk with in the CA Communities every day, at user groups and at prominent mainframe community events such as CA World and SHARE.

You know it, because you probably see it in your organization as well. There are experienced developers: people who have deep knowledge of the applications in place, business intelligence, database tables, cross-application dependencies, batch execution windows and the general experience of how things work on the z/OS platform.

Most importantly, they also know how to avoid breaking what’s working.

And there are the millennial developers: people who are curious to learn about new programming languages, platforms, people who, with the advent of Agile development, and open source have played with things, tested them and are using it to enhance their skill sets.

These people prefer to use modern user interfaces and tools to increase their productivity. Sometimes they might be tentative about making mainframe changes, not willing to risk their careers by breaking poorly documented code.

Culture – Have it Both Ways

While the interests of millennial and experienced developers may at times seem conflicting, both have pivotal roles to play in application development and will benefit greatly from the common sense practices of DevOps and agility that are coming to the mainframe platform. To survive and thrive, teams must unify to facilitate the change process. Most importantly:
- Code must be accessed and modernized at the speed of business. Enterprise developers, testers and architects cannot be afraid to go talk to the “black box” that is the mainframe team.
- Project managers can’t worry that a six-week turnaround for small changes will kill their hopes of meeting their delivery windows.
- The delivery team—QA, release management and operations—can’t afford to sit around waiting for simple deployment steps to happen: when we can automate and integrate to increase deployment frequency.
The promise of DevOps and agility on the mainframe is to fix this, and that’s why our engineering teams are actively building out solution sets that extend DevOps practices to the mainframe, including continuous testing, continuous delivery and continuous feedback.

Curious about what it will take to bring DevOps to the mainframe? Catch some tips and best practices from Forrester Research principal analyst Rob Stroud, or hear from the leaders driving change in their organization in our CA World 2016 Mainframe DevOps panel.

— Aline Gerew
January 31, 2017