Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Working from home is here to stay, and Agile is still a failure. (more…)
Tag: waterfall
-
WTH? WFH is 6× Pre-Covid ¦ Plus: Agile Sucks (Redux)
-
Best of 2022: Agile/Scrum is a Failure – Here’s Why
As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.
Welcome to
The Long View—where we peruse the news and strip it to the essentials. Let’s work out what really matters.
This week, something a bit different: I’m devoting my entire column to Agile and Scrum. They’re increasingly getting a bad reputation, being associated with the worst aspects of toxic workplace culture: Sexism, racism, burnout, micromanagement. (more…)
-
Agile Sucks (Redux) | Plus: DevOps on Mars
Welcome to
The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Agile is bad, but “Wagile” is worse.
Plus: This prod is even worse than yours. (more…) -
All Day DevOps, Viewed from the Outside
This year was my first All Day DevOps event. I loved it, as much as I was able to attend. I was especially interested in how DevOps worked with government, as I work in state government. We don’t use DevOps. My guess is that if you work in state or local government, your group also may not use DevOps. This may also be true of some federal agencies.
I’ve not been fortunate enough to work in an organization that used DevOps. So, when I learned of the All Day DevOps event, I had to join.
I’m a DevOps outsider looking in. Where I work the waterfall project management paradigm is followed, mainly because waterfall includes the word, “requirements.” I’ve been told that scrum and, more generally, agile project management includes the concept of requirements, but they’re just not called that.
Since we use waterfall, our progress in delivering software solutions is lengthy. I appreciate the need to get a very good understanding of what the users want, but it can be difficult to meet so many needs in a timely manner. Consequently, many users will strike out on their own to come up with a software solution. In our state department, that means using Microsoft Access. Sometimes a division will get someone to write an Access app to quickly satisfy their needs, which they start to use. This works for a while until the demands on the app become too great and they come to IT begging for help.
I really should mention that not all of this is caused by IT being slower to respond than we would if we were using DevOps. A large part of this is a consequence of not having enough money to purchase a commercial, off-the-shelf software solution, if such exists. Fiscal limitations coupled with a long queue of projects to work on and a project management process that tends to be slower makes for aggravation on both sides.
Another thing I’ve noticed about my department is that both IT and development don’t work together as well as they could. I wouldn’t characterize it as adversarial—more like just not entirely trusting of each other. This IT/Dev department has more than 200 people throughout the state, so I find it interesting to see how it plays out. Before this job, I worked in a very small organization with just two IT people—and I was one of them. It was easy to coordinate and there was no divide between IT and development. In my current position, however, everyone has their assigned tasks: IT, development, business analyst or security. There is very little overlap. I very much would like to see more.
When I learned about the All Day DevOps event, I was thrilled with the prospect of being able to learn more about DevOps. Where I work, no one ever travels to any conference, so having All Day DevOps online was a huge bonus. Also, it was free to attend, which was essential. I caught a few of the sessions throughout the day. This may sound trite, but I was awestruck with the idea that All Day DevOps was going on worldwide! This demonstrates just how widespread the DevOps movement is. When I start promoting All Day DevOps to my colleagues, I will stress this fact.
I understand that All Day DevOps will be done again in 2018. I look forward to that because this year I learned of the event only the day before, which was hardly enough time to really prepare. If you’re like me, working in a government agency—or any organization, really—and you desire to do a better job of providing your users with software solutions they’ll enjoy and in a more timely manner, as well as work better with your colleagues, I recommend that you join in at the next All Day DevOps event. I’ll be there, so I hope to see you there as well.
About the Author / Rod Falanga
I’m a longtime software engineer working primarily in the Microsoft stack. Writing applications for both the desktop and web, with some experience in writing mobile apps. I’m also a longtime fan of the BBC’s Sci-fi show, “Doctor Who.” Follow me on Twitter @Doctor_Who. -
The 7C Holistic Approach to DevOps
The DevOps methodology was first adopted in 2007 with a bigger purpose of improving organizational efficiency by bridging the gap between software release and IT operations teams. This started as the core practice of continuous integration (CI) and continuous delivery (CD) with a prime focus on better quality of releases and more frequent releases. Now, DevOps is becoming the norm of the industry and is being embraced by multiple organizations as a digital transformation with a shared goal, involving all stakeholders, to serve the customers better with higher quality product release and more frequent releases.
DevOps has evolved beyond release and deploy and now involves product owners, project managers and all aspects of software development life cycle. This is the key reason it has evolved as a holistic approach starting from the service team interacting with the customers to product release and production monitoring. In this blog, we will explain how the 7C approach of DevOps is a natural evolution of DevOps and how it helps different organizations to perform better.
DevOps now spans across the entire organization involving customer service, marketing, product owners, project managers, program managers, and different levels of developers, QA, release or build teams and infrastructure team. As the main focus of DevOps is to serve the customer better and faster, it is important that all the stakeholders have better end-to-end visibility of operations, planning, integration, testing, monitoring, delivery and feedback. This is the 7C approach of DevOps, as illustrated in Fig 1. This requires an efficient integration of processes and tools to automate the seamless information exchange and execution. This approach also empowers all the stakeholders to better contribute in product’s success and efficient execution.
Fig. 1: 7Cs of DevOpsThe 7C holistic approach to DevOps also requires efficient collaboration between all stakeholders, empowering them to contribute more. The 7C approach removes all silos of information and execution.
Startups were fast to adopt DevOps as many of them were setting up things from scratch. But the latest industry trends show large enterprises embracing DevOps primarily with the focus on efficiency and better, more frequent releases.
The DevOps domain includes lots of tools and many of them are open source. This has provided a toolset for engineers to tinker with or experiment. This tends to have problem of its own, as too many toolsets in the flow create silos of information and execution, which creates confusion and inefficiency. This is becoming a problem and that’s why solutions are required which provide better integration and execution across multiple tools. The right solution for this will be a platform with deep integration capabilities.
A deep integration platform for DevOps will enable seamless integration and execution across different tools, even home-grown proprietary tools, to provide a holistic solution starting from customer initiation to production deployment and feedback. This approach is significantly important for large enterprises, as this enables them to have the flexibility of preserving their existing investments on tools and processes while also allowing them to selectively introduce new tools. Rather than trying everything new, large enterprises can manage changes while primarily focusing on efficiency, automation, collaboration, better quality of releases and more frequent releases.
This 7C approach gives same visibility to all stakeholders and everyone knows what feature is being released when. This approach also helps to move information/execution in both left and right direction with powerful traceability features, analytics and dashboards. Deep integration of information and execution also reduces dependency on other teams and promotes self-service.
Fig. 2 illustrates a sample DevOps flow where service team uses ServiceNow; software, product and project managers are using Jira for Agile management, developers are using Eclipse as their integrated development environment, GitHub is used for source code repository, QA team uses a test management tool for test and defect management, and finally the release team uses Chef and Jenkins to orchestrate build, release and deployment. In real-world examples, teams add more tools for code analysis, security and production monitoring. All such complex flows are supported within DevOps 7C holistic approach by utilizing the deep integration platform. The 7C approach is more productive as it allows faster detection, analysis and resolution of problems.
Fig 2. Sample DevOps Flow with Deep IntegrationThe 7C approach works well with both traditional waterfall software management as well as Agile software management—mainly because of better end-to-end visibility, efficiency and collaboration.
To conclude, 7C approach of DevOps is the natural evolution of DevOps, empowering the team to achieve better efficiency and execute better. This empowers all the stakeholders in the organization by providing better end-to-end visibility and powerful means of collaboration. Involving all stakeholders also creates a better environment of trust and collaboration. With this 7C approach, DevOps is not limited to build and release but it has spanned across the entire spectrum of software development lifecycle. This is especially highly beneficial for large enterprises as it enables them to preserve their prior investments and enhance their existing flows. This gives them leverage to selectively try new tools and technology while preserving what is working well. This is a great approach to expedite the enterprise level digital transformation.
About the Author / Devendra Joshi
Devendra Joshi is Sr. Director, Customer Solutions and Products at Kovair. He is a software professional with over 20 years of experience in enterprise software, social media, healthcare, SaaS companies, and EDA. He has worked with startups and large enterprises in engineering management, and product and solutions management roles. -
Break the Rules and Win at Digital Transformation
The rules of the game have changed.
In today’s digital world, it’s not about the big eating the small. It’s the fast that beat the slow and the nimble that outmaneuver companies constrained by old systems and business models.
What’s more, the finish line is always moving. The rapid pace of innovation and disruption means businesses must continuously transform themselves to stay ahead of competitors.
The takeaway for CIOs: Speed and agility are the name of the game.
IT must respond faster than ever to shifting business needs. To succeed at digital transformation, companies must rethink how they design, build, deploy and source enterprise applications. They must adopt new approaches that accelerate delivery and anticipate, even bake in, ongoing changes.
It’s time to update the IT playbook. Here are five best practices from digital leaders that are breaking with the past to deliver a successful future.
Accelerate and Iterate Software Delivery
Digital leaders outpace their rivals by adopting methodologies and mindsets that shorten software delivery cycles. They’re also really, really good at rapid, iterative change.
Shift to Speedier Methodologies
Digital leaders rely on Agile development methods—short sprints, scrum teams and user feedback loops—to accelerate the design and development of new applications. With its emphasis on rapid iteration and continuous improvement, Agile methodologies are a proven way for companies to churn out new innovations quickly. Studies show Agile approaches average 50 percent faster time-to-market than waterfall ones.
DevOps is also a catalyst for speed. Automation and tighter collaboration between development and operations make it easier to put new code into production. Led by such companies as Facebook, Google and Amazon, the world is moving away from “that 10-year-old solution everyone’s afraid to touch” to flexible platforms that are updated multiple times a day.
Adopt an Agile Mindset
Of course, being agile requires more than just new methodologies. It also demands a willingness to take risks, learn on the fly and adjust continuously. Digital leaders don’t just respond to change; they embrace and design for change.
Digital leaders test and then invest. They fail fast and then swiftly course-correct. The most successful experiment rapidly and incrementally so if they do stumble, they fall from the first floor, not the 10th. Digital leaders also think big. They don’t just digitize current processes. They completely re-imagine business models and technology systems—as Pitney Bowes did with its transformation into a digital commerce powerhouse or as GE did with its launch of the Predix industrial IoT platform.
Adopt New Application Architectures and Infrastructure
Digital leaders are building in flexibility with new application architectures and deployment models. They’re leaving legacy systems behind as they take to the cloud with microservices-based solutions.
Move to Microservices
Monolithic applications are hard to change, costly to maintain and ill-suited to today’s dynamic business environment. That’s why digital leaders are moving to an agile, microservices-driven architecture. Converts include Netflix, eBay, Amazon, Twitter, Gilt, Liberty Mutual and the UK Government Digital Service, to name just a few.
Assembling applications from small modular services—each a self-contained building block—allows digital leaders to transform quickly and at scale. They can mix and match services to fast-track new digital offerings and processes. They can tweak and scale services to adapt to changing business models and customer expectations. And they can push innovation across the enterprise by reusing services—all without adding armies of developers or getting locked into any one technology.
Step Up Cloud Migration
The cloud is increasingly the preferred foundation for digital transformation. Digital leaders gain speed and agility by plugging into on-demand services and computing resources. They short-circuit long design and procurement cycles; focus on innovation, not infrastructure; and deliver differentiated capabilities and customer experiences at a fraction of the time and cost.
Digital leaders are betting big on the cloud. According to IDC, at least half of IT spending will be cloud-based by 2018, reaching 60 percent of IT infrastructure spending and 60 percent to 70 percent of software, services and technology spending by 2020. Organizations across industries—such as Nike, Xerox, General Mills, Liberty Mutual, and the U.S. federal government—have migrated mission-critical workloads to the cloud to support new, more digital ways of doing business.
Use Novel, Unexpected Sources of Innovation
Digital leaders rely on open-source innovators and third-party collaborators to shorten development cycles. They’ve also abandoned long RFPs in favor of shorter, more effective proofs of concept (POCs).
Embrace Open Tech with Open Arms
Open technologies—including open source, open standards and open APIs—dominate the digital landscape. According to a study by North Bridge and Black Duck, open source software makes up more than 50 percent of the typical enterprise and application stack. Digital leaders including Ticketmaster, Box, eBay and Goldman Sachs rely on open-source software to speed solution delivery and draft off the innovations of millions of developers worldwide.
Digital leaders also innovate openly with others. They hold hackathons. They crowdsource. They co-create with customers and partners. They also contribute back to the community with projects such as Walmart Labs, Ericsson’s open-source hub and GE’s Industrial Dojo for the Cloud Foundry ecosystem.
Say RIP to RFPs
Digital leaders have pulled the plug on massive RFPs for complex, multiyear development projects. Instead, they favor faster, more flexible procurement processes that reward innovation and accelerate vendor selection. They conduct POC tests to evaluate alternative technologies. They use iterative prototyping to refine requirements and determine the best solution for the job. They keep their options open with shorter timelines and smaller deliverables.
Digital success calls for a more collaborative relationship with technology providers. It’s the whiteboard sessions, technical conversations and joint problem-solving that yield breakthrough results. That’s why progressive companies now approach the RFP like a “request for partnership.”
Integrate Data and Systems in Transformative Ways
Digital leaders use open platforms and APIs to integrate information, people and processes across their extended value chain. These connections lay the foundation for faster business flows and new digital products and services.
Connect Quickly with APIs
APIs (application programming interfaces) enable diverse applications to connect and share data. Digital leaders use API-led integration to combine services and expose information in ways that deliver unprecedented customer and business value. Within days and without lots of coding, they can roll out service innovations, revamp internal processes, enliven customer interactions, extend their digital reach and enable entirely new business models.
This year, 2017, has been dubbed the year of the API economy. ProgrammableWeb lists more than 17,700 APIs and hundreds of digital leaders are opening up their APIs so others can easily integrate and innovate with their technologies. API marketplaces from Mastercard, ADP, Ford and FedEx are just a few examples.
Achieve Digital Flow
Silos slow the flow of business. Disconnected processes, departmental barriers and gaps in information all impede speed and agility.
Digital leaders excel at silo-busting. They integrate disparate systems. They connect processes from end to end, across all customer touchpoints and organizational boundaries. They use open platforms to ensure that information—structured data and unstructured content—flows to the right person, at the right time, in the right context.
The seamless integration of information, people and processes allows digital leaders to be much more nimble and responsive. They can also leverage these connections to make huge improvements in customer experience—such as the bank that slashed loan origination steps by 70 percent or the insurer that processes claims 25 percent faster.
Bring Business and IT Closer Together
Digital leaders treat digital transformation as a partnership between business and IT, with a shared focus on meaningful outcomes and better experiences for customers and employees.
Engage and Empower Users
Digital leaders put people—customers, citizens, employees and partners—at center of their transformation efforts. They prioritize design thinking, a user-centric approach that shifts the focus from adding features to adding value. They take advantage of platform tooling to involve users in solution design and invest in dedicated UX teams to deliver better experiences faster. Capital One, for instance, regularly solicits customer input at its three innovation labs. Other proponents of design thinking include Citrix, Airbnb, BMW, DHL and Procter & Gamble.
Another winning strategy is to empower users with self-service tools that add business agility. Examples include self-service analytics for rapid decision-making and process services that make it easy for users to modify workflows as business needs change.
Take a More Strategic Role
Now that every company is a software company, IT is increasingly strategic to the enterprise. Successful CIOs have seized this opportunity and are making the most of it.
Forward-thinking IT leaders have recast IT in the role of business partner and change agent for digital transformation. They look for ways to drive growth, deepen customer engagement and shorten time to market. They help colleagues understand and realize the value of digital initiatives. They build bridges and buy-in by adopting a service mindset—what one CIO calls “ego-free IT.” The new measures of success aren’t IT-specific metrics; they’re meaningful business outcomes such as dollars saved, customers added and market share gained.
Conclusion
Every digital transformation is different. But the underlying challenge is the same. How can IT accelerate the pace of delivery and change simultaneously? By following the lead of digital trailblazers, any organization can gain the speed and agility needed to compete in a digital-first world—and win.
About the Author / Chris Wiborg
Chris Wiborg is Vice President of Product Marketing at Alfresco Software. In his role, Chris is responsible for setting and executing the go-to-market strategy for Alfresco’s Digital Business Platform. Connect with him on Twitter and Linkedin. -
Continuous Delivery vs. Continuous Deployment
I am often asked, “if we are using a continuous integration (CI) server and adopting a continuous delivery pipeline, where does continuous deployment fit in?” Good question. In a nutshell, continuous delivery engines such as CloudBees or Blue Ocean are designed to trigger the execution of CI workflows across the dev, test and prod environments. Continuous deployment is called as part of the CI workflow to do the actual release of code out to endpoints. Continuous deployment can be managed by application release automation tools or supported by one-off scripts. Regardless of how sophisticated the deployment step of your CI workflow is, it is still continuous deployment.
To understand both continuous delivery and continuous deployment, one must go back to their roots, continuous integration. CI is a key component of Agile practice. Agile developers understand the importance of adopting a methodology that can manage smaller incremental software changes to release software, and therefore business value, to end users quicker and cheaper. The activities of deploying to testing and production historically have been in the hands of testing and data center teams, separate from Development. But in the Agile practice, these activities cannot be separated from development. The concept of separate development, testing and production teams is a waterfall concept, not an Agile concept. In Agile, everyone is on the same team. This means that testing and production teams must become part of the Agile process.Mastering Agile’s last Mile
Ultimately, the goal of continuous delivery is to accelerate release cycles and get new code out to production fast. However, for most companies this is not happening. Developers and testers may have embraced continuous delivery, but when it comes to production, the code is moved to a staging area where it languishes until production is ready to perform the update. The last mile of Agile is not achieved. And we are back to where we started: waterfall.
There are many reasons for the production roadblock. Lets’ review the top four.
- Agents: Both open-source and commercial CI servers are agent-based. An agent based system works fine for developers and can be tolerated by testing. However, production teams are not that excited about installing yet another agent on to the production end points. For this reason, production team use their own method of deploying code. Therefore, the deployment scripts used for releasing code to dev and test are not used by production. The way they see it: Production does not want a deployment process that requires them to re-configure their data center environments with agent from a development tool.
- Scripts: Production teams want input and visibility. One-off scripts obfuscate the deployment process. For this reason, production teams will write their own scripts. Scripts simply do not have the intelligence needed to meet the audit, rollback and transparency requirements needed by production teams. Production teams perform many steps manually. This takes time and cannot be done in an agile cadence. The result: Great code languishes in staging areas. Production stays several releases behind what continuous test has approved.
- Cost, cost, cost: Most of what developers create in a continuous delivery workflow comes from open-source or consumption-based licensing models (think Jenkins and Jira). Tools that can achieve continuous deployment such as application release automation solutions are far to expensive for one development team to acquire and incorporate into the CI workflow. These types of tools are sold at the CIO level and cost upwards of $250,000. Production teams are often hesitant to purchases these tools, as the high cost equals high risk. And ‘one ring to rule them all’ does not work. This type of operational tool acquisition is waterfall-based and does not fit into the leaner Agile DevOps tooling. The result: One-off scripts continue to be used and dev and test. Production does not adopt agile.
- Sustainability: During waterfall, a production release was performed as infrequent as two times per year, or at most once per quarter (considered aggressive). In other words, production is staffed to support a production release on an infrequent basis. With agile, the demand for production releases increases 10-fold. Instead of doing at most four releases a year, they may be required to do four releases a month. Production cannot keep up the agile pace. They are staffed for waterfall.
Automating Continuous Deployment: Key to Maturing Continuous Delivery
Continuous delivery cannot be successful without continuous deployment. If you want to roll code out to end users faster, the deployment process itself must be modernized. Using the same one-off scripts that were used in the waterfall approach will not achieve your CD goals.
Continuous deployment automation is performed by a new set of agile DevOps tools called application release automation, or ARA. ARA solutions are designed to support a model-driven software deployment that allows development, test and production teams to use the same process without the need for one-off deploy scripts. ARA adds all of the features and functions needed to support continuous delivery and data center deployment requirements. ARA is designed to fully perform the heavy lifting of software releases including application packaging, release versioning, database updates, server configuration management, calendaring, roll-forward, rollback, security access and auditing.ARA solutions are called by a templated CI workflow that is defined by the development team. The workflow is moved up the pipeline and adapts to the changes at the various pipeline stages (System Test, Integration Test, Pre-Production, Production, etc.) Because the process is driven by the ARA tool and not a script, the changes are automatically addressed across the pipeline reducing the workload for all teams and supporting frequent changes that can be easily traced and audited down to the server inventory level, including the production environment.
Developers, Get Your Hands on Deployment Automation Tool
To achieve a repeatable CD pipeline that supports production, developers must embrace the automation of continuous deployment at development. The dysfunction however is the outrageously high cost of ARA tools. Developers don’t buy expensive operational solutions. More ARA solutions must be offered on a consumption-based model or offered as open source with the ‘out of the box’ integrations into CI/CD. When considering the purchase of an ARA solution, look for agentless models that can be implemented one project team at a time. Embracing this level of corporate process change must be done project by project. The operational big bang approach does not work well in an agile practice. Not everyone can use the same solution. However, if done correctly, there are big benefits.
ARA is a process that strengthens the foundation of the continuous delivery pipeline and is core to the agile DevOps initiative. It facilitates agile practices by allowing the testing teams and data center teams the tools needed to become more agile and accept the frequency of changes being pushed across the pipeline by Development. Ultimately, the goal is repeatability across the pipeline, pulling together Dev and Ops.
About the Author / Tracy Ragan
Tracy Ragan is CEO and Co-Founder of DeployHub. Tracy has extensive experience in the development and implementation of business applications. It was during her consulting experiences that she recognized the lack of build and release management procedures for the distributed platform that had long been considered standard on the mainframe and UNIX. Tracy has worked with hundreds of development teams in implementing a team-centric standardized build to release process. Connect with her on LinkedIn. -
DevOps Chat: Waterfall Will Not Disappear, with Flint Brenton, CEO Collabnet
One of the perks of being editor in chief of devopsy.kinsta.cloud is I get a chance to meet with so many leaders of the DevOps and IT industry. You can really learn a lot by listening to what these folks say. This DevOps Chat is a perfect example. I sat down with Flint Brenton, CEO of Collabnet. Collabnet is not some johnny-come-lately. The company has been helping developers and IT professionals for a long time. Flint talks about the fact that while DevOps is a great path for many organizations, it may not be the right way for every situation. Waterfall is not going away entirely.
Have a listen to my discussion with Flint and hear for yourself what he has to say on the subject. As usual, the streaming audio player is immediately below with the transcript below that. Enjoy!
Streaming Audio
Transcript of Audio
Alan Shimel: Hey, everyone. This is Alan Shimel, devopsy.kinsta.cloud, here for another DevOps Chat. Happy to be joined in this DevOps Chat by Flint Brenton, CEO of CollabNet. Flint, welcome.
Flint Brenton: Hey, Alan. It’s great to see you again.
Shimel: Great to see and hear you again. Flint, the last time I think you and I talked was back at the DevOps Enterprise Summit in San Francisco. I guess that had to be October or November.
Brenton: November. Actually, it was the same week that our new president was elected, if you remember.
Shimel: Yes. Yeah, a red letter day.
Brenton: Right.
Shimel: So, Flint, without getting any further into that, let’s turn the conversation to CollabNet. What—well, first of all, maybe some of our audience is not familiar with CollabNet in general, so why don’t we start with that? Can you give us a little CollabNet background?
Brenton: Yes, absolutely. So CollabNet is a SAS-based, cloud-enabled development platform focused exclusively on enterprise that delivers application lifecycle management capability for both Agile and waterfall development processes.
Shimel: Mm-hmm.
Brenton: The version technology to structure those builds and to maintain high quality and high quality control throughout the process. And then we also provide the DevOps infrastructure to build and deploy automatically whatever gets created in Azure or waterfall, with high quality, on time, and, to the extent we can help, under budget. So that’s really our focus.
Shimel: Sure. And you know what, Flint? I remember this about CollabNet. You guys have been in business now for a number of years, certainly before DevOps was what it is today, let’s call it, whatever it is today. How long have you been—
Brenton: Yeah, we’ve been in business—this is our 18th year.
Shimel: Absolutely. So, you know, DevOps is another technology innovation, if you will, that’s come down the road, but I think what—not unique, but what is one of the special qualities of CollabNet is is that you’re not out here saying that the whole world needs to go DevOps today, right, that everything—we should be all DevOps all the time. But you have customers who are still on waterfall and may very well remain on waterfall for the foreseeable future.
Brenton: Right. And we really invest very heavily in both waterfall and Agile development platforms and—although the growth rate for Agile’s higher than waterfall in enterprise, we still see roughly 70 [percent] to 75 percent of enterprises using a waterfall methodology. But the fastest growing segment in that area is clearly Agile.
Shimel: Sure. And I—obviously. But I think what’s important for our audience to recognize is even the biggest DevOps and Agile advocates or biggest—whatever you wanna call ’em, should recognize that there are still projects and places where waterfall might be the best way to skin the cat, to complete a project correctly. And it doesn’t have to be an all-or-nothing type of environment, if you will.
Brenton: That’s exactly right. You know, the industries that tend to be highly regulated, with a high need for security, compliance, and scalability, tend to lean more towards waterfall development. And, of course, once you started a process where you’ve got an established team that’s been using waterfall for a long period of time, they tend to prefer to continue with that. But it’s new projects, new projects where maybe time to market outweighs the need for compliance and security and scale—those tend to go Agile in an enterprise. But, either way, regardless of development methodology, the things you need beyond the tool chain or the process of building software is fairly similar, right? The way you use version control is fairly similar. The way you use DevOps technologies is also very similar.
And what we found is customers who are well grounded in Agile and waterfall—typically, in enterprise, they have both—getting your act together in DevOps and making sure you have a strategy, you created a culture to embrace and execute that strategy, then having the processes and tools in place to bring it to reality is critical because you don’t really get the return on investment for anything you develop, regardless of the methodology, until you put the build into production. And that’s what DevOps, at the end of the day, is all about.
Shimel: Sure is. Excellent. Excellent point, Flint. Flint, let me turn things a little bit ’cause, as I mentioned to you off camera, off mic, 15 minutes here goes pretty quick. Want to talk about some recent news around CollabNet in the industry. I understand you guys were just included in a Gartner Magic Quadrant, correct?
Brenton: Yes. Yes, thank you for bringing that up. Yeah, we’re very excited about it. It’s always a reflection of the investment our team has made in Agile planning and execution. So we’ve always been considered a leader when it comes to waterfall development, but, as we just mentioned, for enterprise, our enterprise customers are really demanding that we provide both, both equal value in waterfall and Agile. So we invested heavily in the last few years in Agile, and, as a result, we were just included in their Magic Quadrant as a visionary for Agile planning and execution, and we’re very excited about it.
Shimel: That’s nice. That is great. And I know everyone always wants to be in the upper right, but, really, having my background, being entrepreneurial and having started businesses, I’d almost rather be in that visionary quadrant than the upper-right leader quadrant and certainly in the visionary quadrant over, let’s say, the challenger quadrant, which is the upper left, because I think, with visionary, I mean, the recognition is that you’re setting a—you’re pushing the envelope, right? You’re pushing the vision—hence, “visionary.”
Brenton: Right.
Shimel: So congratulations to you guys. I know, in speaking with your team, you have rights for download of the quadrant, so what we’ll try to do is put into the show notes here a link to that where people can download the quadrant report and see if—check it out for themselves.
Brenton: Yeah, that’d be perfect and I’m sure that your listeners would very much appreciate that.
Shimel: Sure will. So, Flint, beyond the quadrant inclusion, though, what else do you see that’s happening with CollabNet or in the market in general?
Brenton: We’re starting now to get a number of customer wins that are embracing our motto of providing a cloud-based service where you can consume our integrated ALM, version control, and DevOps solution either in Amazon or on our own private cloud and, in many cases, a combination of on-prem and off-prem.
Shimel: Right.
Brenton: So we now have a number of customers who have completely moved their development process into one of our cloud-enabled environments, and it’s very exciting.
Shimel: That is exciting.
Brenton: It’s very exciting. And the great (thing) about that is, if it’s in the cloud, it’s actually easier for us to include, Alan, both open-source and compare it to components that we don’t own or control because then the need to test to get everything in your data center is lessened and our ability to innovate to your advantage is much greater, so we’re very excited about that.
Shimel: Absolutely. So, Flint, interestingly enough, I was reading an article this morning—I forgot where I saw it but it was a recent survey where 65 percent of enterprises still keeping the majority of their infrastructure on their own data center, not on a public cloud, not a hybrid situation. Is that about what you guys are seeing but is the arrow pointing up to the cloud? What’s your take on that?
Brenton: Now that’s a great question, Alan, and it’s absolutely what we’re seeing. So a thing to keep in mind, though, is “What is the variable that’s changing and what is the velocity of that change?” Right? So I would say, two years ago, that 65 percent number was probably 90-plus, right? So, as enterprises get more and more comfortable with the security and compliance options in the cloud, whether it’s in a public cloud or private cloud, they can see the tremendous advantages of driving down the cost of development and actually increasing the speed of innovation by going to the cloud.
I’ll give you an example. U.S. government, we do a tremendous amount in federal, and so both the Obama administration and now it’s transitioned to the Trump administration has really been pushing the department of defense to do a couple things. One, innovate faster, and so, if you just think about the U.S. government is barraged by cyberattacks from all over the world. Right?
Shimel: Mm-hmm.
Brenton: And so your ability as a professional, in NSA or CIA or the FBI, to deal with that is you want a complete toolkit of things you can use, and you want to choose the tools you need to deal with those cyberattacks, but you also need to do it quickly. So if you can use open source, it helps. If you can deploy it in a cloud, it dramatically increase—or decreases the time to deployment. And, three, it also tremendously helps you on the economics side. So even something as highly regulated and structured as the US federal government, there are advantages sometimes in doing things in a public cloud to help, and so we’re starting to see variations of that theme also in industry, primarily around enterprise, where there’s flexibility, increased ability to innovate, and improved economics by moving workflow to the cloud.
Shimel: Yep. You know, Flint, living in the tech world and you lived the world—you live that life; I live that life—I think sometimes we get jaded in that we’re in this bubble of, “What’s the hot new thing?” and it’s DevOps and it’s whatever’s next after DevOps, and then sometimes you get a reminder—65 percent still not in the cloud. And you say, “But I thought everyone’s in the cloud.” And I think it does good for us to remember that there’s a big world out there that moves at their own pace and their own—
Brenton: Right.
Shimel: Same thing with waterfall. Right? I think people would be—people in our audience from devopsy.kinsta.cloud would probably be surprised to find out how many organizations still either haven’t adopted DevOps or are primarily waterfall shops.
Brenton: You know, the key, when you’re a solution provider like CollabNet, is you need to meet people where they are. Right? So, if they wanna be on-prem, you need to be able to deliver them superior value on-prem. If they want to move part of their workload to the cloud, you need to be able to help lead them and sponsor that transition at that pace by which they’re comfortable. Right? So I learned a long time ago the key to success in this business is to give people what they want, right? He who can do that best with a quality solution at a great price is gonna win. Right? So that’s just the way it works, right?
Shimel: Absolutely—and that hasn’t changed, by the way, through all of these innovations, and I think it was true 100 years ago and it’s true today. Or maybe even more than 100.
Brenton: You bet.
Shimel: Anyway, Flint, listen, I think we’re about out of time here. I wanna be respectful of your time, so what we will do is let’s have you on again soon when we have some more time and we can delve into this. Are you gonna be at DevOps Enterprise Summit maybe in London?
Brenton: You bet. Yeah.
Shimel: Or San Francisco?
Brenton: Well, I always enjoy speaking with you, Alan. You’re a great guy and you’re—
Shimel: Thanks.
Brenton: —you’re a fun guy off camera too, I will add.
Shimel: [Laughs]
Brenton: So I look forward to our next adventure together. Thank you for the—
Shimel: Absolutely. Hey, Flint—
Brenton: All right.
Shimel: Flint Brenton, CEO of CollabNet, this week’s guest on DevOps Chat, thanks for joining us. This is Alan—
Brenton: My pleasure, Alan. Thank you.
Shimel: Thank you.
Brenton: All right. Buh-bye.
Shimel: This is Alan Shimel for DevOps Chat.
-
Mainframes and the Myth of Bimodal in DevOps
Rewind 20 years ago when we lived in a business world where big companies with unlimited resources beat out smaller, more strapped companies. The digital era—also known as the “age of the customer”—is changing all that. Where big once beat small, fast now beats slow. The DevOps approaches of startups often make them more adept at continuously delivering the innovations and conveniences that earn customer raves. Bigger, established companies often feel disadvantaged‚ shackled by an outdated culture and inflexible processes that haven’t kept up with the pace of change.
Gartner coined the term “bimodal IT” as a way for older IT establishments to keep pace with agile startups. In the bimodal model, IT teams are separated into two distinct styles of work—one focused on supporting predictable, well-understood workloads with legacy systems and another that is more fast-moving, experimental and adaptable in meeting users’ constantly evolving demands and flourishing digital expectations.
But the bimodal concept is not well-aligned to the spirit of innovation underlying modern application development and delivery. This is especially true for bigger organizations using mainframes. Here’s why.
Successful DevOps Depends on Mainframe Agility
The goal of DevOps is to roll out high-performing software and updates, ahead of the competition and with maximum speed, ease and cost-efficiency. In a DevOps model, software delivery can only happen as quickly as the slowest team. Modern web and mobile applications tend to span multiple platforms in their end-to-end transaction—from front-end web servers (systems of engagement) all the way back to mainframes (systems of record). IT places a lot of emphasis on systems of engagement, often taking back-end systems of record for granted. The reality is, the value in systems of engagement cannot be realized without the systems of record—making it essentially all one system. Developers supporting modern applications must be able to maneuver across and between all platforms, especially within mainframe code and data.
Let’s look at a tactical example of this. According to industry research, 80 percent of the world’s corporate data—including customer data—continues to reside on mainframes. When a mobile web application is used to make a bank deposit, or purchase a product, for example, the transactional component of those applications typically is executed on back-end mainframes.
This means mainframe applications are crucial to development efforts. What happens when mainframe developers and testers are locked into waterfall processes or don’t have the modern tools necessary to understand application logic or access mainframe data for testing? Frustration mounts and the overall innovative effort is unnecessarily stalled.
Stability and Agility are not Mutually Exclusive
The longstanding myth that stability and agile cannot co-exist is simply untrue. Unfortunately, it’s at the core of negative perceptions surrounding the mainframe today—people think the mainframe environment is too slow and cumbersome to support agile development. Development teams may think if they have mainframes, the only approach it can support is waterfall.
The reality is, though, no customer will accept poor availability and performance in exchange for new features delivered quickly. People with hands-on experience painfully understand that slow, waterfall delivery methods produce far greater quality issues and missed expectations, resulting in competitive risk. Additionally, large enterprises taking the “slow and stable” route give startups a head start in beating them to market, with offerings that customers want now.
Alternatively, Agile process and DevOps tool chains enable scrum teams to test quality in terms of usability, function and stability early and often in the development cycle—when the time to remediate is the fastest and the cost the lowest. Agile process and DevOps tool chains build higher quality applications and enable continuous improvement in quality over time.
Re-Platforming is not Necessarily the Answer
Some DevOps proponents believe moving off the mainframe—to a supposedly more modern, commodity server-based distributed architecture—is the best way to circumvent perceived challenges to nimbleness and agility. But this is almost never the best answer. The mainframe retains its distinction as being the most powerful computing platform on the planet. It remains deeply embedded in large organizations, setting the standard for reliability, security and availability for mission-critical data and transactions. The new IBM z13 mainframe, for example, can handle the load of 100 Cyber Mondays every day, 365 days a year. Currently, mainframes process 30 billion business transactions per day. The post-modern mainframe is an engineering marvel unequaled as a system-of-record, business compute platform.
The mainframe also offers cost advantages. Studies have shown that the mainframe is more cost-effective than commodity servers in handling the massive computing volume increases instigated by mobile applications. This is because decreases in commodity server pricing have not kept pace with the increase in computing volumes. In addition, leading vendors have worked together to deliver intelligent cost management on the mainframe, enabling workloads to be distributed in a way that keeps mainframe licensing costs (MLCs) down.
In our view, “biomodal” advocates for the exact problem that must be solved. We think the answer is not replacing the mainframe, although if mainframe development is left unaddressed, this can present obstacles to DevOps. Rather, the answer is to evolve the mainframe to keep up with the pace of DevOps. There are mounting examples from large, high-performing organizations that it is possible to apply modern practices to the fullest extent, including mainframe applications. These organizations do this by giving developers the tools they need to work on mainframe code and data just as they would with any other programming language on any other platform, while meeting the escalating quality expectations of the digital age. For example:
- Mainstreaming the Mainframe Environment – Mainframe users need to move away from the antiquated “green screen” mainframe environment, adopting state-of-the-art interfaces and tools. This is the first step enabling developers to work more fluidly on the mainframe, in an environment that’s much more familiar and comfortable to them. Organizations that fail in this adoption will have a very difficult time enabling the next generation of development talent to leverage invaluable and irreplaceable mainframe code and data. This will place a huge hindrance on DevOps success.
- Leveraging Tools from the Java Development World – One example is visualization capabilities, which enable developers to quickly and easily identify critical interrelationships between apps and data. This helps them to sidestep unintended, undesirable impacts as they modify applications, increasing confidence and mitigating unnecessary delays.
- Make it Easier to Identify Mainframe Code-Level Issues – Developers need tools that enable them to identify problems with mainframe code; for example, when it is the source of a widespread user performance issue, or when it is consuming too many resources and driving up costs unnecessarily.
Conclusion
In our view, “bimodal” is not the solution, but a broken state of disadvantaged affairs for too many organizations. From a competitive vantage point, bimodal will only hurt DevOps teams and hold back the innovative efforts of the organizations they support.
The bimodal IT model is a road to inevitable failure for those enterprises seeking success in the digital economy. This is especially true for organizations that rely on mainframes. A smartly leveraged mainframe is an extremely valuable competitive asset that needs to be removed from its silo, evolved and smoothly integrated into modern, heterogeneous development environments.
It is possible for big, established companies to adopt DevOps, become agile, and move as fast as startups. But the ability to think and act like a startup requires some major fundamental changes, particularly at the level of software development. The good news is, the legacy technologies that are often perceived as impediments can actually be a tremendous competitive advantage, if they are harnessed and evolved in the right manner.
About the Author / Christopher O’Malley
Christopher O’Malley is CEO of Compuware. He has nearly 30 years of IT experience with past positions including CEO of VelociData, CEO of Nimsoft, EVP of CA’s Cloud Products & Solutions and EVP/GM of CA’s Mainframe business unit, where he led the successful transformation of that division. Connect with him on LinkedIn and Twitter. -
Test-First Development: Processes and Tools for Success
A lot of software development teams are talking about “test-first” methodologies. The practice involves moving testing up into the very earliest stages of development so automated tests can be written before code. Making this seemingly minor shift can result in much higher quality software and greater efficiencies. Integrating development and testing avoids needless late-stage bugs and complicated redevelopment efforts. It can help software development organizations focus on usability and features and achieve faster time to market. And because of test-first’s reliance on automated tests, organizations gain reusable tests, saving time and increasing standardization for testing and quality assurance overall.
It’s no surprise, then, that 51 percent of organizations have begun using test-first methodologies, including 37 percent in the past year, according to a recent global survey of 200 software testers conducted by our company. But, despite the enthusiasm for this practice, there are concerns: 44 percent of survey respondents said their biggest fear about moving to this approach was forcing developers to contribute tests prior to completing code, while 19 percent said they worried about removing traditional testing checkpoints on the way to production. Other barriers include the lack of standard practices or tools. That said, test-driven development promises to bring ample benefits to software development teams, the business and, importantly, customers.
Test-first development is recognized across three broad terms today: behavior-driven development (BDD), test-driven development (TDD) and acceptance test-driven development (ATDD). TDD is an umbrella term for the practice, although it has traditionally focused more on technical, unit testing. BDD and ATDD incorporate user requirements and business benefits along with technical testing. BDD differentiates through its adoption of specific automation frameworks and language for writing and running tests.
Key Changes for Successful Test-First Development
Getting started requires change across a few critical areas for most teams. First, evaluate your team for hiring and skills training needs. On the one hand, developers will need to learn basic testing skills, so they can write good tests and run tests before they develop code. Meanwhile, testers must become more proficient in automated testing tools, which usually entails acquiring a fundamental knowledge of programming. Many companies will need to hire key contributors from outside; in which case, leaders should look for well-rounded individuals with past experience using test-first methodologies successfully. An individual who is an expert at one skill, or who is too independent and can’t collaborate well, won’t be an ideal member of a team experimenting with test-first approaches. Leaders also may need to adjust by balancing the ranks more closely between developers and testers so that one group doesn’t rule over the other. Hiring additional experienced testers frequently will be necessary.
Training of existing staff is paramount, best achieved through a combination of methods including internal training, mentoring, online courses and paired development, where developers and testers can sit side by side and collaborate. More than half of the survey respondents said developers and testers would be jointly responsible for automation test creation within their organizations.
Beyond cross-training, the cultural challenge of moving to test-first methodologies can be overwhelming. It requires the development of incentives for creating quality products, not just delivering new features at speed. It requires an appreciation and respect for the practice of testing. Unfortunately, testing too often is viewed as a necessary cost of doing business, rather than bringing a clear benefit to the organization and its customers. That mind shift is not only inaccurate, but it doesn’t fly with progressive, Agile thinking. It will take strong leaders to help traditional teams understand the need to change and modernize practices. A leader may forego the buy-in process and instead decide to lay down the law to staff: Staff must move to TDD or BDD, or else. But the results won’t be ideal. Smart developers will find loopholes, such as writing tests that aren’t comprehensive or fully aligned with project goals, so the code will pass more quickly.
On the process front, organizations that have moved or are moving from Waterfall to Agile and DevOps will have an easier time adopting test-first methodologies. In fact, it’s pretty difficult to implement test-first processes and tools in a Waterfall environment because of the siloed nature of functions and the long release cycles. The survey results support this premise: 55 percent of respondents said their testing process was Scrum/Kanban-driven before transitioning to test-first, while 21 percent said their approach was iterative-requirements driven and 18 percent was Waterfall-requirements driven.
Along with that shift, test-first teams will need to adopt new metrics that align with this new way of doing business. Instead of looking solely at velocity, development leads should measure how long it takes the team to move a product from feature approval to code release. With a focus on quality and testing, companies also should measure and improve upon bugs, such as number of severe defects post-production. Those types of business and customer-oriented metrics align with the new set of Agile metrics that that many teams are working for today.
Test-first teams will need to adopt DevOps toolsets, which are designed to push code out to the pipeline faster. Specific technologies that go hand in hand with test-first development include:
- Continuous integration (CI) platforms such as Jenkins and Team City, to assist in running the automated tests immediately for quick build feedback.
- Container technology for rapid deployment of test and pre-production environments, to capitalize on having features production-ready soon after coding.
- Test management platforms including qTest Scenario or Cucumber Pro.
- Test automation frameworks including Cucumber, SpecFlow , and other open source tools which enable easier support for customization and collaboration.
There’s a lot to think about when moving toward test-first development. While it may seem overwhelming at first, all of this is achievable in a reasonable time frame if you have the right dedicated resources to see it through. One-third of survey respondents said it took them less than three months to complete the transition to a test-first approach, while 30 percent said it took less than one year. The payoff for software teams, in terms of improving quality, customer experience and reducing costs of innovation, can deliver higher customer loyalty and even faster growth in the marketplace.
About the Author / Kevin Dunne
Kevin Dunne is vice president of Business Development & Strategy at QASymphony.