devopsy.kinsta.cloud

Tag: developers

Are We Leaving Developers Out of DevOps Spinoffs?

SecOps. DataOps. NetOps. Reading these terms, you get the sense that the key to IT efficiency is to make IT Ops work with everyone else. But that’s a mistake, because it leaves developers out of the picture.

It’s no secret that the DevOps movement has generated a number of other *Ops initiatives, including those listed above. The driving idea behind all of them is that IT Ops teams should work more closely with security teams (in the case of SecOps), data analytics teams (in the case of DataOps) or networking teams (in the case of NetOps). These are only a sampling of *Ops movements; the list could go on.

To be sure, there are advantages to gain by having IT Ops collaborate more closely with other parts of the organization. But there are an equal number of advantages in doing the same thing with developers.

To optimize security operations, you need developers, IT Ops and the security team to work together. If Dev is left out of the picture, it’s much harder for the other two parts of this equation to make sure that the code the organization deploys and manages is secure.

For similar reasons, DataOps, NetOps and almost all the other *Ops you can come up with work well only when Dev also participates. The people who actually design and write the applications are important.

Rethinking Ops

Why does this matter? My point is not merely that terms like SecOps and DataOps are misleading because they don’t mention developers.

On the contrary, giving equal weight to developers is important because if you don’t, you risk isolating developers. Some are already complaining that DevOps is “killing” them. The trend of defining new *Ops practices that don’t mention developers is not helping.

So, maybe we should start mentioning and thinking about developers more prominently when we are talking about ways to extend the DevOps philosophy to other parts of the IT organization. This is already being done somewhat in the case of SecOps, which you sometimes hear called SecDevOps. But Google lists nearly 13.5 times as many results for “SecOps” as for “SecDevOps.” The numbers are similar if you search for “NetOps” as compared to “NetDevOps.” And I’ve never heard anyone say “DataDevOps.”

The takeaway is this: Developers are as important as IT Ops to DevOps. Don’t leave them out.

— Chris Tozzi

November 15, 2016
Foster Creative Workers, not Assembly Lines

Why do we keep building assembly lines of creative people?

Why do tech companies still structure their developer workforces around ideas invented during the industrial revolution over a century ago? It’s ignorant, lazy and, worst of all, it pisses everyone off. Most of these archaic ideas must die if we are to take fully enlightened advantage of the impending internet revolution.

Internet Revolution? What?

Before I explain my thoughts about what old ideas are holding our workers back. it is important to understand the one big similarity between the workers of the industrial age and developers of today. Both are the backbone of an exploding economy that has changed nearly everything about the way we live. Without these two types of workers, both revolutions (industrial and internet) would collapse. But this is where the similarity ends.

For me, the most important difference between the two is that developers are highly creative and highly skilled, and are required to be highly intelligent workers. This creative worker requires a very different environment to ensure “productivity.”

Let’s dive deeper into this idea, and see if we can uncover the problems with sticking to century-old concepts.

‘Work’ in the Industrial Revolution

In 1913, Henry Ford installed the first assembly line for the mass production of automobiles. From there the industrial revolution exploded. Quality of life boomed. Life-changing products became affordable. Life expectancy increased. Everything was going up and up. (Except, of course, animals and the environment, but I’m going to stay on the subject of work.)

The industrial revolution redefined the way we measure work. The world moved to a strict schedule of productivity, and was no longer tied to the weather or the sun. Hans Rosling shows it the best here.

But more than 100 years have passed. We are very clearly going through another work revolution, yet we still cling to the archaic concepts of the Industrial Revolution. The nature of the work we need is drastically different. Ideas about 9-to-5 work hours, productivity, vacation and work/life balance just don’t apply to the work we are trying to do today.

This was work back then, done by people who could do the task without creativity.

If Henry Ford lived today, what would he do? Would he make an assembly line of computers on which programmers would write line of code and pass it down the line to the next programmer, who would do the same, until you had a complete application? If you think that’s stupid, why are we working like that? Here is a bug, go fix it! Shouldn’t we be reinventing these 100-year-old concepts instead of applying them as they are?

Industrial Revolution ‘Work’ Causes an Unhappy Workforce

Before the Industrial Revolution, most manufactured products were made individually by hand. A single craftsman or team of craftsmen would create each part of a product.

Now that most assembly lines are automated, today’s work requires individuals to be craftspeople once again. We no longer hire developers just to tell them what to do and how to do it. Developers, like all creatives, don’t like to be told what to do. Creative workers like to be given a problem and they love creating a solution.

We no longer need to make sure that our employees are working. We need to make sure that they are inspired.

Take, for instance, the job of an overseer/manager. In the industrial age, this type of person made sure everyone was on task and meeting quotas. They were task masters, and they were often feared. Today, small startups rarely hire task masters (or as we now call them, project managers) to ensure worker productivity right out of the gate. Even if a company does hire PMs, they are no longer micromanagers; they are process optimizers. They make sure that the team is fully informed about what problems need to be solved so they can reach their highest creative potential. Today, if a tech company goes down the route of micromanagement, it loses its most talented people. This change tells us something: Today’s tech workers want to be engaged. They want to solve problems. They want to work for the sake of the creativity.

No More 9 to 5 and Hourly Rate

An hourly rate only makes sense when you need someone to complete the same task x times a day. If you need to assemble a thousand cars a month, then you need to make sure you can guarantee numbers. This makes hiring and firing simple: If workers can’t fill their quota, then they are replaced with those who can.

If you work in the tech industry today, it’s obvious why the above concept is idiotic. Ask any creative worker—software developer or otherwise—when they are most productive and if it helps to be given tasks and blindly assume deadlines. They will quickly tell you that what matters most is the breakthrough, and that could happen at any time of the day. I can’t tell you how many breakthroughs I’ve had on weekends, or in the last thoughts before sleep. And once the breakthrough happens, we developers furiously code toward the finish line to test the validity of our breakthrough. No one needs to tell us to get the work done. We WANT to get the work done and we understand the business requirements; don’t worry, we won’t slip—assuming, of course, that the work is engaging and interesting.

Let’s separate the concept from the workforce for a second. No painter sells his or her art based on the hours spent creating it. The price is based upon the value of the work in of itself. We give an intrinsic value to the final product.

A developer could make a million-dollar contribution in a single day. Another developer could make a series of contributions over two years that made it possible for that one developer’s million-dollar solution. Both are equally valuable and necessary.

The creative workforce should be valued similarly. It is impossible to measure productivity-ensuring work eight hours a day, five days a week. A million-dollar contribution achieved in a single day or over two years—both are equally valuable and necessary. Neither can be reached by giving tasks. They can only be reached by giving problems, and as much time as it takes to solve that problem.

Creative Problem-Solving

In a task-based scenario one software engineer might fix a bug quickly, but then it crashes the system next time it runs. In a problem-solving scenario, the creative thinker would figure out that this particular bug is a symptom of a large underlying problem that needs to be fixed to prevent an even larger failure. This type of creativity we seem to attribute to the arts alone. However, engineering is not different: Here is what the founder of YC, Paul Graham, who said in his amazing book, “Hackers and Painters”:

“Like painting, most software is intended for a human audience. And so hackers, like painters, must have empathy to do really great work.”

We must remember that just like making a new product is creativity, writing code at its core is creative work. After all, it is not the application of the paint that is creativity. It is how you decide to apply the paint that takes a creative mind. Similarly, it is not the act of coding that is important. It is how you code that is the creative part. In software development, the creativity scale may vary per problem, but if the creative aspect is not nurtured then we encourage mere task completion.

How Do We Fix These Problems?

First, we must work on our definitions. Now that we understand where the crux of the problem is, we can start re-engineering how we communicate our concepts of success and failure. I discuss my ideas about this in my previous article, “4 Inspiration-Killing Concepts That Need Redefining.”

See you there.

About the Author / Devrim Yasar

Devrim is the CEO of Koding, a San Francisco-based software development automation platform vendor. Connect with him on LinkedIn and Twitter.

November 14, 2016
How to Bridge the Developer Gap

It’s time to bring cloud application development back into the organization and bridge the developer gap by training your own people.

For years, companies have outsourced many IT functions, treating computing as a commodity, perhaps thanks in large part to Nick Carr’s publication, “IT Doesn’t Matter.” Carr predicted that computing would become a utility. And while spinning up new servers in the cloud might be almost as simple as switching on a light, the functions specific to your business—your competitive DNA—aren’t flipping on automatically. Not even close.

Computing is not a commodity. It’s the cognitive engine of your business. The companies that are killing it right now view their computing environments the way we look at the human brain and nervous system. Their businesses generate, collect and process signals, and act on the intelligence influenced by those signals. Case in point: GE is building its own operating system—a central nervous system for its industrial applications. This is large-scale cognition.

You would not outsource your brain or spinal cord. You couldn’t survive without those crucial components of your overall nervous system. Yet for decades, companies outsourced key functions—technology and processes for understanding their businesses—because they didn’t think of technology as a core competency.

Now that they’re bringing it back in-house, companies are faced with another problem: a looming developer gap. The majority (64 percent) of IT professionals and executives taking part in the Cloud Foundry’s 2016 Identifying the Developer Gap report agreed that the developer shortage is a real threat. Most respondents (57 percent) state that this shortage has already impacted their ability to hire skilled people.

Every new technology comes with a labor gap. The way to close the cloud application skills gap is to bring the knowledge back into the organization—to train people who are part of the central nervous system of your business.

Business computing in the digital world is not just a faster way to process paper. When you think of your organization as an organism and integrate intelligence from all its parts, you behave more intelligently. Building apps to enable that integration is the difference between treating computing as a commodity vs. cognition.

Our study showed that by a nearly 2:1 margin, companies are choosing training over hiring or outsourcing as the preferred method for addressing a shortage of skills in their own companies. I have had multiple conversations with CIOs in the last two months in which they have said, “We have this working with a team of 40 developers. Now we need to make it work for 1,000. We aim to certify over 200,000 developers on Cloud Foundry in the next three years.”

The best way to bridge this developer gap is to train your own people. Think of them as essential parts of the whole. Enable them to use technology to adapt and streamline your business so you can compete in the digital world.

About the Author / Sam Ramji

Sam Ramji is CEO of the Cloud Foundry Foundation. A 20-year veteran of the Silicon Valley and Seattle technology scenes, Sam brings a wealth of business, product and open source experience to the foundation. He was head of strategy for API powerhouse Apigee, designed and led Microsoft’s open source strategy, and drove product strategy for BEA WebLogic Integration. Previously he held a range of software engineering positions at firms including Broderbund, Fair Isaac and Ofoto. He is a member of multiple industry advisory boards and volunteers with the Sierra Club. Sam is happiest in the mountains, where he loves to hike, run and play with his dog. In his spare time, he likes to play guitar, read fantasy and sci-fi novels, and spend time with his wife. Connect with him on Twitter.

November 7, 2016
Making Performance More Than a Best Practice

Yes, you read that correctly: SRE means that strong application performance—speed, reliability and availability—no longer is simply a best practice. It is a must. Site reliability engineering, or SRE, is a term Google coined several years ago, and while there is no universal definition, SRE can be thought of as a discipline that blends software engineering with IT operations management. At the heart of the SRE model is a site reliability engineer who knows how to code as well as handle ops. The SRE concept is similar to DevOps, which is intended to bridge the divide between development and operations teams, but until now has not caught on the same way DevOps has.

The Need for SRE

The rise of the digital economy is now propelling end-user application performance to the top of the IT priority list. According to a recent EMA Survey, 55 percent of IT professionals rank end-user experience monitoring as their top application management priority. At the same time, growing IT complexity, including the cloud and multi-platform applications, are making it more difficult than ever to ensure strong user performance and identify the root source of problems before they escalate. We believe this performance imperative is driving the SRE resurgence, giving DevOps teams the necessary “teeth” to govern and resolve performance-related disconnects, swiftly and decisively.

Performance issues traditionally have been a lightning rod of conflict for even the most well-oiled DevOps teams. Developers, relentlessly focused on rolling out new products and features, keep throwing their “completed” work over the fence to operations. Operations teams, conversely, are focused on minimizing change and maintaining stability as much as possible. These opposing desires naturally create friction.

Performance problems only exacerbate this. Developers, who have already moved on to the next must-have feature, say it’s operations’ fault. Operations points their fingers at the developers, saying that poorly written code is to blame. No one is assuming responsibility and the developers and operations teams are stuck in a stalemate. Meanwhile, users are suffering, the call center is being inundated, revenues likely are circling the proverbial drain and the situation is quickly reaching a boiling point.

SRE is valuable because it finally brings an end to this suffering through a set of clear, irrefutable, mathematically based rules to govern all DevOps efforts and decisions. If a new piece of software achieves a targeted performance level, further rollouts can proceed. If not, future releases will halt until the first piece of software is fixed. The site reliability engineer, who is well-versed in both development and operations, serves as an independent arbiter who provides a clear assessment of where the root cause lies—on the development or operations side of the house. With guidelines firmly established and a central, unbiased person in charge of determining the right course of action, the developer/operations friction can be avoided and problem resolution can commence.

In an SRE model, developers and IT operations also share a staffing pool. If there’s budget for one hire, and a product is not delivering an agreed-upon performance level, the hire goes to the IT operations. If the product is delivering the required performance level, the hire can go to development, which will help the development team roll out new features and products.

The real beauty of SRE is that both sides have a vested interest in achieving strong application performance, ideally right out of the gate. If whichever side owns the problem does not fix it, they can forfeit any hopes of adding manpower to their team, at least in the near-term. There’s another more serious implication: for developers, there will be no further development; and for operations teams, less software to support may translate to lower headcount requirements.

An SRE approach has many advantages, including fostering a vested interest in performance across the entire DevOps team, and holding feet to the fire. Performance cannot be an afterthought—performance must be considered and measured from the very beginning phases of ideation and development all the way through production and post-production modifications. Users don’t care how fast or efficient your DevOps team was if the ultimate work product does not perform—in fact, this means the whole effort likely was in vain. Like a strict adjudicator, SRE may just be what DevOps teams need to bring to an end to performance-related squabbles once and for all.

About the Author / Mehdi Daoudi

Mehdi Daoudi is CEO and co-founder of Catchpoint Systems, provider of web performance testing and monitoring solutions. Before Catchpoint Systems, he spent 10+ years at DoubleClick and Google, where he was responsible for Quality of Services, buying, building, deploying, and using monitoring solutions to keep an eye on an infrastructure that delivered billions of transactions daily. Connect with him on LinkedIn and Twitter.

November 4, 2016
Why PR, Legal and HR Matter for Continuous Delivery

By most definitions, DevOps involves making developers and IT Ops teams work better together. But integrating other parts of an organization, such as legal and public relations, into DevOps is also important. Here’s why.

DevOps has become popular because it helps to speed the delivery of software. The goal is to reach continuous delivery, which means code is changed, tested, staged and delivered to users on a (nearly) continuous basis.

If you think only from a technical perspective, then getting to continuous delivery of software involves only technical people. Developers, IT Ops admins and software testers are all you need.

DevOps in the Big Picture

Yet, to deliver software continuously on a reliable basis, you need to think beyond just technical processes and personnel. Effective DevOps workflows have to involve all of the other parts of the organization that play a role in creating software, delivering it to users and maintaining it once it is in production.

For example, PR teams need to be plugged into the software delivery chain so they know when new features are coming down the pipeline and can prepare to promote them. Legal professionals should play a role as well, since they’ll know when code changes require tweaks to licensing, compliance procedures and so on.

The list can go on. In most organizations, human resources has a role to play in DevOps, since personnel changes can affect the continuous delivery chain. Supply chain managers should be involved, too, since they have a hand in provisioning the infrastructure on which DevOps teams rely.

To be sure, these various non-technical teams don’t need to be as plugged into the DevOps workflow as deeply developers and admins do. Code can be written and tested without their perpetual involvement. Still, at a minimum, all parts of the organization should at least be aware of the state of the delivery chain. Otherwise, visibility and communication break down.

Taking this big-picture approach to DevOps is important because ultimately, despite the relatively narrow purview implied by the term DevOps itself, the value of DevOps lies in adding efficiency and promoting better communication across the entire organization in ways that optimize software delivery.

So, if you want to make the most of DevOps, start thinking beyond just Dev and Ops.

— Chris Tozzi

November 2, 2016
Four Inspiration-Killing Concepts That Need Redefining

Software engineering is a creative process, and creativity cannot be measured by the hour. It can be measured only by the value of the contribution. Yet, a majority of companies still measure productivity by hourly contributions. Companies treat the development process as if it still existed in the days of the assembly line, and all we needed to do was to affix 100 widgets a day. This mindset pisses off developers, kills inspiration and causes developers to quit.

So, how do we measure progress in our companies? How do we ensure that we can increase our probability of success? These are necessary questions for businesses, and it’s easy to see why companies cling to the rules of the past. It’s all we know. And it is a big gamble to change. After all, shareholders and such.

It will take true visionaries to rewrite the rules. And although many new companies are doing wonders for changing the way we define work, we still have a long way to go. So the question now stands: What does the ideal dev process look like in the future? It might be helpful to attribute new words to the dev process to help us gain better understanding.

New definitions for the Internet Era

Task → Contribution

Approval → Collaboration

Workplace → Remote

Workers → Culture

Task → Contribution

Remove the worker from the assembly line, and give them a seat at the decision makers’ table.

This trend has already started in the open-source community. What used to be a to-do or a task is now an “issue” on GitHub or GitLab. If you think about it, task implies, “You got to do this whether you like it or not!” It’s involuntary but necessary involvement; you being creatively engaged with the task is optional. We embraced this unnecessary definition and are justifying it for no reason.

We have established that creative workers don’t like to be told what to do and how. We should have contribution meetings where creative workers decide what they would like to work on. Unfortunately, many decisions are made in meetings where business owners pre-decide needs and delegate to managers to distribute tasks among engineers. If you want to lose creativity in your team and make everyone feel bad, this top-down approach works exceptionally well. We see the Band-Aid solutions to this top-down approach all over the place now. The manager needed to find ways to motivate workers. How? Ping pong tables! Free laundry! Ice cream! You get the idea. Most creative people see right through that garbage. It’s only novel to inexperienced recent grads who have yet to understand how the real world works.

You will not have an engaged workforce unless you empower them and let them decide how to give value to the company. The easiest way to achieve this is to communicate transparently to the company what the business requires and give a detailed description as to the reasoning. From there sit back and watch the engineers chime in and say, “Hey, can I do that?” or, “What about this other problem we have?” I believe we could then change the perception from, “This is what I’m expected to do,” to “This is what I’m contributing to the company and I’m passionate about it.”

Approval → Review

This change is also apparent in the open-source world, when you submit your changes to a source code repository, your peers can take a look at it and see if they can contribute to its quality. In the past, this process was called “approval” and now is called “review,” where in fact your peer is approving the piece of code you’ve written. Approval implies. “I have authority over you,” whereas review says, “I’m going to help you get this done.”

Workplace → Remote

Dave Gray makes some really interesting points about how we make decisions in his book, “Liminal Thinking: Create the Change You Want by Changing the Way You Think.” He suggests that we only have the ability to make theories, judgments and beliefs based on our personal experiences. And even in our experiences we are only able to process a minuscule sliver of the information available.

David Gray’s diagram of how we come to our understanding of what the obvious choices is in a situation.

If we apply this idea to the way we feel about our workplace, then we can quickly see the we are reducing the number of creative solutions available to our workforce. If everyone one of your creative workers gets to work from 9 to 5 at the same desk doing relatively the same thing daily, they will start narrowing their understanding of what is possible. Just imagine what different solutions might arise if your employee is sitting outside at a lake or in a crowded bar. Their experience would change, thus their well of experience deepens, and their concepts of what is possible, multiplies. Make sure your developers experience new things, both as individuals and as teams. This doesn’t just mean offsites for team-building. It means, let them follow their gut, and work from wherever they need from time to time. This doesn’t mean let them go solo and unplug for days at a time. They need to stay plugged into the team and be a contributing part to the culture, but they need to balance that with building fresh perspectives so they can contribute to the collective experience of the whole.

Workforce → Culture

Yes, I know. “Culture” is one of those buzzwords that now cause people to roll their eyes so far back in their head that they fall out. But it’s important to know why the concept is important in relation to the old idea of “the workforce.” When management thinks of its developers as “workers,” the developers are immediately dehumanized and turned into pawns. They are expendable and indistinguishable as creative minds. They act out of fear from punishment and rarely take chances.

We are no longer cogs in the corporate machine. Our ability to take risks can make the difference between being one of the 99 startups that fail, or the one tech company that succeeds. Managing culture as opposed to a workforce means that we can focus on enabling workers to be as creative as possible. We can focus on nurturing collaboration and honest feedback. A healthy culture is one that is transparent and communicative. A workforce could give a rat’s ass about making positive change. A culture is a living breathing thing that everyone contributes to and improves. A culture is owned by the collective. A workforce is owned by a man in a suit at the top of the tower. Screw the man. Power to the people.

Conclusion

Now that we have an understanding of why we must redefine our terminology, it’s time to start defining practical ways to make the switch for each term. In my upcoming writings I will start to lay out practical ways to make the switch in your company.

About the Author / Devrim Yasar

Devrim is the CEO of Koding, a San Francisco-based software development automation platform vendor. Connect with him on LinkedIn and Twitter.

October 24, 2016
Continuous Integration vs. Continuous Delivery: There’s an Important Difference

If you read DevOps blogs today, it can be easy to assume that Continuous Integration (CI) and Continuous Delivery (CD) are synonymous terms. People often use them that way. But that’s a mistake.

As an example of conflation of the terms CI and CD, take this page, which purports to be a “history of CI/CD tools.” The author does provide a truly excellent history of CI platforms.

But CI tools are the only subject that the page discusses. It’s not a history of CI and CD. It’s just a history of CI platforms. To work the history of CD into the picture would require discussion of how software testing and delivery practices have changed over time, not just an overview of the history of CI software over the past several years.

I don’t mean to knock the page. It’s a great read, and the mistake the author makes in treating CI and CD as synonymous is hardly rare. He or she is just repeating a common confusion. You see the same lack of distinction between CI and CD on Docker’s discussion of CI/CD, for instance.

The Difference between CI and CD

It’s unclear how this conflation of the two terms originated. But it’s easy enough to clear up.

Simply put, CI is the process of integrating code into a mainline code base. In almost all scenarios today, CI is done using platforms designed specifically for the purpose. Implementing CI is, therefore, as simple as using the right tools.

CD is more complicated. CD is about the processes that have to happen after code is integrated for app changes to be delivered to users. Those processes involving testing, staging and deploying code. There is no single type of tool that does all these things. These processes take different forms, depending on the culture of the team and the type of app it is creating.

In actuality, then, CI and CD share little in common. Besides starting with the same word (“continuous”) and being different stages in modern software delivery pipelines, they don’t have much to do with one another.

Why the Difference Matters

Why is it important to steer clear of this confusion? The main reason is because, by conflating CI with CD, organizations can end up doing only the former but thinking they have implemented the latter.

Setting up a CI server such as Jenkins or Bamboo to integrate code changes continuously does not mean you are doing CD. It just means you are using a CI server.

To get to CD, you also have to organize your software testing, staging and deployment processes in a way that automates them as much as possible and makes them continuous. Doing so requires more than simply turning on a certain type of tool.

How you actually do CD is fodder for a separate, much longer article (or you could just read the book about it). But recognizing that CI is not CD, and avoiding confusing formulations including CI/CD, is a start.

— Chris Tozzi

October 20, 2016
How Agile Are Your User Stories?
Around the world, enterprises are adopting the concept of Agile development. This methodology is based on a set of principles for software development by which requirements and solutions evolve through the collaborative effort of teams. User stories form the foundation of the Agile requirements process, and teams gather to create them by exploring how users will interact with their solutions. To facilitate planning and discussion, teams typically jot down stories on pieces of paper and then physically arrange and rearrange them. It is a curiously old-school process for such a forward-thinking method.

User stories are so vital because they give developers just enough information to estimate the level of effort needed to implement the functionality described by the story. When written well, they can be powerful, because they help developers and testers view requirements from a customer’s perspective. They provide context and an understanding of what motivates the people who will use the solutions they deliver.

Struggles with Stories

Agile represents a significantly different workflow than developers have previously used. As enterprises work to expand and mature their Agile practices across their projects and teams, they struggle to figure out how user stories fit, because:
- Developers and testers may prefer user stories over a lengthy, text-heavy business requirements document, but user stories are new to most business stakeholders and business analysts. As organizations try to transition to Agile, these individuals struggle to write high-quality user stories that accurately and sufficiently describe all customer needs. They get bogged down writing and managing them and lose focus on the bigger picture.
- The number of user stories can get out of hand quickly due to complexity. Most enterprise projects involve the development and enhancement of multiple, integrated systems to deliver functionality for a variety of roles. Users can take many paths that often interconnect and overlap with other processes. This can mean hundreds of user stories for a project of any size. Manually recording and managing the wealth of information that the development team needs is not feasible, and user stories can be missed or misinterpreted in the process.
- Critical nonfunctional requirements—such as those for compliance, security and performance—complicate the adoption of user stories because they weren’t designed to represent those types of requirements. User stories scribbled on sticky notes or created in an Excel spreadsheet don’t support the rigor enterprises need for audits, change management, history and traceability.
Enterprises want to reap the benefits of Agile for their complicated, expensive, high-risk projects, but they want to manage complexity and risk as well. They need a robust Agile requirements tool to help them, especially for the creation of reliable, consistent and high-quality user stories.

Making the Shift to an Agile Requirements Tool

Just as approaches to development have changed, so has the workplace, necessitating a new approach to teamwork. PricewaterhouseCoopers noted in its paper, “Adopting an Agile Methodology, Requirements Gathering and Delivery“: “As companies shift from small projects and teams engaged in Agile to more complex projects and potentially distributed teams, there is a need to shift from paper and spreadsheets to tools that provide workflow, persistence and traceability.”

The power of best-in-class requirements management tools is that they enable teams to use user stories for development while making it easier for business stakeholders and business analysts to create them. These tools eliminate the need to manually create user stories within the development teams’ Agile tools. They bridge the gap between traditional and Agile requirements, enabling enterprises to scale Agile while managing enterprise concerns.

The powerful capability sets of tools such as this help teams to:
- Generate high-quality user stories and tests automatically from process flows. Product owners and business analysts use customer journeys to automatically generate user stories and acceptance criteria with the click of a button. They push these artifacts into the development teams’ Agile management tool of choice, where developers and testers also have access to related requirements information, such as regulatory information, visual models and constraints, supporting a comprehensive understanding. User stories are reliable and consistent, and there is no longer a need to spend time and money to teach business stakeholders or business analysts to write them.
- Work together to visually define customer journeys. This allows stakeholders to “tell their stories” as they work with product owners and business analysts to collaboratively define customer journeys. Using the familiar construct of user models—with steps, decision points, actors and condition statements—the entire team collaborates to record and analyze processes in a shared workspace. Teams maintain a focus on strategic objectives when making prioritization decisions and spend less time managing a huge list of user stories manually.
- Reap the benefits of Agile while using enterprise-level capabilities for visualization, traceability and reuse. Support for visual models and the ability to relate them to one another and other requirements artifacts helps teams establish the precise traceability they need to ensure full requirements and test case coverage. It also supports improved change management and decision-making, ultimately leading to higher-quality software. Customer journey models can be reused across projects and teams, as can user stories and other requirements artifacts, saving time and improving consistency.
Great strides have been made in Agile requirements management tools recently, but many enterprises are still using sticky notes to create and organize their user stories because they don’t know about what these tools can do. A next-generation tool enables organizations to scale Agile across the business and create a common ground for IT and business to meet. With capabilities including visualization, collaboration, traceability, management analytics and reuse, teams are more effective and applications are delivered faster.

About the Author / Ruth Zive

Ruth Zive is Vice President of Marketing at Blueprint. She is a metrics-driven marketing strategist who has worked for two decades serving B2B clients in the technology, healthcare and financial services industries. At Blueprint, Ruth is responsible for product marketing, analyst relations, branding, demand generation and inside sales initiatives. Connect with her on LinkedIn and Twitter.
October 18, 2016
So Long to Configuration Management Systems?

When I was an operations architect at Ning way back in 2009, we were struggling to manage our deployments and configurations of more than 50 different application servers, or what the cool kids today call microservices. Applications were deployed across three distinct environments: development, staging and production. None of these environments were exactly the same as another, and differences in these dependencies, together with human errors, frequently broke our deployments.

Ning was nowhere near the first company to encounter such problems. In the mid-1990s, as the public internet emerged, much of what was understood at the time about IT operations was challenged by the need to scale internet-facing systems. For many years preceding the internet boom, the maximum number of users was strictly limited to the number of employees in a company. That all changed with the World Wide Web, where the need to serve thousands and millions of simultaneous users easily exceeded the computing capacity of even the largest and most expensive computers.

As three-tier application architectures were broken down into services and distributed across dozens or hundreds of individual servers, this proliferation of servers proved to be far too much for operations teams to handle. The worst problems lived within application deployment processes, where environments and requirements were ever-changing in successive releases of software. The lifetime of a server was a long one. Once deployed, servers were subjected to repeated changes by operations and developers in what would come to be called “configuration drift.” The combined effects of changes, human errors and unknown states would create disastrous outages that damaged brands and destroyed online revenue. It would take sites days and even weeks to recover from these outages. Some would never recover.

It was in the midst of this chaos that modern configuration management systems were born. CFEngine was the first to emerge in academia, where the need to manage multiple UNIX systems was felt early on. Later, Puppet, Chef, SaltStack and Ansible emerged directly from web-scale systems. Configuration management systems fulfilled several important needs. First, they abstracted the differences between operating systems, providing a high-level language for the system administrator to describe the desired state. Second, they automatically synchronized the desired state across large numbers of servers, reducing costly human errors. Finally, they provided idempotency to ensure that destructive operations were not applied unnecessarily.

There is never a free lunch. The benefits of better abstraction were offset to a degree by the need to build and maintain evermore complicated recipes. Recipes frequently broke from time to time as package management features were updated, renamed or otherwise changed from the time the recipe was first written. Even when recipes worked, they were frequently broken by differences in environments or runtime errors. The configuration management systems themselves began to suffer from scale challenges, and evolved message bus architectures to improve their speed and responsiveness. By overcoming many of these obstacles, configuration management systems have now become a first-line tool for managing web-scale systems.

But back at Ning, modern tools such as Puppet and Chef were still in their infancy and simply could not cope with the scale of our 60,000+ node virtualized environment. Ning had a lot of brilliant software engineers floating around, and they set about creating their own deployment automation system called Galaxy, and later a meta-orchestrator called Cosmos. Galaxy and Cosmos were great tools, but they required constant care and feeding and suffered from severe scale problems. There was just a lot of complexity in dealing with these automation systems while also trying to scale a top-tier social network for a hot startup company.

As we began to analyze and post-mortem our deployment failures, it became clear that a majority of the problems stemmed from host and environment configuration errors. We needed a simpler way to deploy our software. Virtual machine images were a promising way to ensure consistency between environments with the tried and true “gold master image” approach. The problem with this approach was that building full system virtual machines was difficult in automation terms. Once deployed, virtual machines are also subject to configuration drift. Furthermore, while we were eager to adopt virtualization to get better resource isolation, we were wary of the overhead that hypervisors would create. Our benchmarks showed that if we adopted virtual machines, we would need to increase our physical server capacity to accommodate the overhead.

At the time, Linux containers were already available in the kernel, but the distributions had almost no support for managing them. At the insistence of our VP of Operations, I cooked up an alternative solution based on jails, writing a small hack of a program called Warden to build and manage them. (Note: Totally unrelated to the Warden in Cloud Foundry). Jails gave us bare metal performance, resource isolation and a way to consistently deploy the entire environment for each application server. With Warden, we could build jail images on the fly automatically and deploy them consistently with our existing tools, galaxy and cosmos. Hey, it was a Band-Aid for a difficult situation, but my experience with building Warden flipped a switch in my brain. I had seen the way lightweight virtualization could eliminate complexity, and I knew right then this was the way of the future.

Today, with the success of Docker we are finally experiencing sweeping changes in the processes and tools we use to build, deploy, and manage distributed applications. At the same time, we are about to enter a brief period of containers disillusionment as early stage solutions get washed out. When that is over, what comes afterward is something of a golden era for distributed applications. I call it the container application platform, and its somewhere between what we think of as platorm as a service (PaaS) and Serverless today. In that not-too-distant future, containers will be the default output of the software build process, and where that container gets deployed will hardly be an afterthought. This unicorn magic will come from the ability to introspect containers to determine their environment dependencies, and a fully integrated tool chain that deals with deployment, scaling and dependency resolution automatically. I will give you just one guess which company is in the best position to own that future. “One ring to rule them all.”

What happens to our beautiful configuration management systems? There has been much debate about this among smart people over the last few years. Most of these arguments boiled down to some kind of coexistence model. Indeed, configuration management systems are useful for dealing with a wide range of operational problems outside of the container platform. While it’s no surprise that Chef, Puppet, Salt and Ansible all moved quickly to include Docker support, I think we will see these tools begin to focus more on what Rob Hirschfeld calls achieving “ready state”. That is to say, everything that still needs to be done before you can lay down your container platform also still needs to be automated. Right now, that is still a great deal of work, and it’s a pretty safe place to draw the line on using the right tool for the right job.

About the Author / James Thomason

James Thomason is chief technology officer at HyperGrid from Dell, where as CTO he led the vision, technology strategy and product road map for Dell Cloud Marketplace. He joined Dell through the acquisition of Gale Technologies in 2012, where as CTO he headed the product road map, architecture and engineering of the company’s flagship cloud and converged infrastructure automation software, GaleForce. Prior to joining Gale Technologies, Thomason was the CTO and Founder of Virtiv, a San Francisco cloud automation startup focused on Linux virtualization, acquired by Gale Technologies in 2011. For over 17 years, as a specialist in distributed systems and large-scale infrastructures, Thomason has been an entrepreneur and innovator at a number of notable Silicon Valley startups, including Exodus Communications, Digital Island, Netli, NetVMG, Netscaler, 3Leaf Systems and Ning. Connect with him on LinkedIn / Twitter.

October 14, 2016
What to Expect from DevOps Enterprise Summit San Francisco 2016
Many have asked me what I’m looking forward to at DevOps Enterprise Summit 2016 in San Francisco—it is the third year of our conference, and I think it will be our best conference yet! Here are three of the things I’m looking forward to most:

Where Are They Now?

First off, we will have seven speakers back to present for the third time at the conference. Having this pattern of repeat speakers is a bit unusual for a conference, but I love that we’re following these leaders on their continuing journey. For me, it’s almost as if we have front row seats of an unfolding documentary of the ongoing transformations that they are helping drive in their large, complex organization.

Why do I think this is important? My belief is that in the DevOps Enterprise community, many of us are on a common journey, increasingly being asked to help elevate the state of the practice for the entire technology organization, often involving thousands of engineers. To see the journey unfolding for other pioneering leaders is often a glimpse into our own future.

The third-time speakers include:

Heather Mickman from Target,

Scott Prugh and Erica Morrison from CSG,

Tapabrata Pal from Capital One,

Carmen DeArdo and his colleagues Jim Grafmeyer and Cindy Payne from Nationwide Insurance,

Terri Potts from Raytheon,

Courtney Kissler in her new role at Starbucks, and

Jason Cox, who is giving the closing keynote about the DevOps journey at Disney. (So stick around!)

Each of these leaders will recap on their DevOps journey so far, whether and how their roles have changed, the new challenges that have emerged since last year, and what they’ve done to overcome those challenges.

We’ll also be hearing from four people who have had a tremendous influence on the entire DevOps community: John Allspaw, CTO at Etsy; Mark Imbriaco, who led the Ops organizations at 37signals, Heroku, GitHub, LivingSocial and DigitalOcean; Adrian Cockcroft, who led the migration of Netflix into the public clouds; and Michael Nygard, author of the seminal book “Release It!”

Transformations from Different Industry Verticals

We always want to hear about DevOps transformations at all stages of the journey—I’m particularly looking forward to three of them that that are outstanding on their own merits, but are also interesting because of the industry they operate in: airline, hospitality and insurance. These are three industry verticals that we don’t typically associate with DevOps.

Susanna Brown and Ben Chan will talk about how DevOps is changing how American Airlines works (and doing it in the middle of its merger and integration with US Airways), Matt O’Keefe will talk about speed becoming the prime directive at Hyatt Hotels and Resorts, and Opal Perry will talk about her efforts to transform culture at Allstate Insurance (which, like Nationwide Insurance, operates in an industry famous for its centuries-old, risk-averse traditions and decision-making styles).

I’m also excited about another conference first for us, which is the joint presentation from Suzette Johnson and Robin Yemen, from Northrop Grumman and Lockheed Martin, respectively. I met Johnson at an Agile conference for the Federal Government community, and I was so inspired by the work that they’ve done within their firms and the broader community at large to improve outcomes for federal programs—both civilian and military.

New Session Format: DevOps Workshops

As adult learners—and this is especially true for leaders—we tend to learn more effectively from observation, experimentation and group exercises and group learning, as opposed to didactic lectures and descriptions of theory.

I’ve observed that the DevOps Enterprise community is truly a group of people who are helping each other. To help further enable the spontaneous and serendipitous interactions that make great communities great, we are adding a set of sessions called DevOps Workshops.

These are 60-minute sessions that run alongside the conference talks, and each workshop will focus on one of the community’s top problem areas:
- How can we better lead change in our organizations?
- What are the modern architectural and technical practices that every leader in our organization needs to know?
- How do we design our organization to best enable DevOps outcomes?
- What are ways that we can more effectively bridge the information security and compliance gap?
Each of these workshops will have a 10-minute introduction, which will be followed by 45 minutes of small-group discussions. And to help ensure that each small group achieves its learning objective, as well as to ensure open and healthy dialogue, we have decided to have trained facilitators use the Lean Coffee format.

Lean Coffee helps achieve this through a democratic selection of topics and rigorously enforced time-boxing of each topic. I’ve always been amazed at how effective this technique is at creating good outcomes.

(Why Lean Coffee? In my personal experience, all it takes is a bad facilitator or one person who loves hearing themselves talk to ruin the small-group session for everyone. I can’t think of a better mechanism than Lean Coffee to neutralize these risks.)

In addition to the amazing lineup of repeat speakers, expert talks, and new stories, my hope is that these DevOps Workshops will be of tremendous value and create fantastic learning and networking opportunities for all attendees. The collaboration and relationships that result from each of the DevOps Enterprise Summits never ceases to amaze me, so we’d like to give attendees a variety of opportunities to engage with our speakers and with each other throughout all three days.

I’m looking forward to the best DevOps Enterprise Summit ever! I hope to see you there!

— Gene Kim
October 13, 2016