devopsy.kinsta.cloud

Tag: devops

What the Auto Industry Can Teach IT Development

The latest and greatest advances in technology often draw the spotlight, but the truly great changes in business aren’t driven by technology alone. They’re enabled by a change in culture that’s supported and accelerated by technology.

Consider the assembly line in the auto industry. Automation, specialized tooling and robotics have provided a boost to quality and efficiency, but these advances only mattered because of a change in U.S. culture that emphasized speed, acceleration and economies of scale. Henry Ford capitalized on these cultural constructs to implement an innovative set of tools that revolutionized auto production. Toyota led the next evolution in auto manufacturing with its implementation of the Andon cord and Kaizen continuous improvement (CI) but similarly, these tools depend on a culture of trust and respect for the workers to be useful.

In the field of software development, a similar shift is taking place through the DevOps movement that’s leading to faster, more efficient and higher quality application development. Technology has made DevOps easier to implement, but these new tools only add value when they are accompanied or even implemented via a change in culture across the entire organization and all technology areas. This includes items such as visualizing the delivery value stream to identify areas for improvement, increased trust and collaboration across silos and a CI mindset.

A key aspect of DevOps is automation, and automation is only possible when patterns can be created, which requires eliminating any variances in practices and capabilities in all aspects of the organizations that don’t add value. For example, if every application is a snowflake in terms of its architecture and configuration, infrastructure automation won’t be possible. If release planning, management and deployment don’t have standard work associated with them, then again, an integrated delivery pipeline capability cannot be implemented.

Nationwide’s Journey to DevOps

At Nationwide, we began a change in how we developed applications almost a decade ago. At the time, the term was “agile.” We shifted application development to focus on smaller iterations of work with automated build and testing practices. While this proved valuable, we realized this was only part of the journey of accelerating our lead time from customer concept through delivery of business value.

We started the journey to implement a Lean-based DevOps model that encompasses the entire development value stream, from the incoming business request to application deployment and monitoring. Our goal was to work more closely with the business units to continuously plan, design, develop, test and deploy incremental improvements in our application-based products.

Nationwide has seen demonstrable benefits on this journey. Critical defects have declined by 80 percent, system availability is up 70 percent and on-time delivery of applications has risen to 90 percent from 60 percent. More importantly, we’re starting to see results in our business as we refine our ability to test business hypotheses and measure the results in terms of increased revenues.

Infrastructure to Support the Cultural Shift

Just as Ford’s auto assembly line utilized advanced technology to be successful (e.g. electricity enabled optimal placement of machines versus steam power), accelerated delivery is utilizing the development of new technologies. To advance DevOps, we use IBM Rational and UrbanCode software to manage and deploy applications. The new technology, though, has only worked because we have committed to both a technology and cultural change across our entire organization. We have more than 5,000 IT professionals on 200 different teams. Every team—whether it’s responsible for the mainframe, distributed, web or mobile— needs to utilize the same standard work approach to application development.

Let me give you one example of how that works: Biweekly show-and-tells are a core part of our approach. For new mobile applications, the changes after two weeks might be dramatic, but even for the larger applications running on our IBM z Systems mainframes, the teams have something to share. They break the mainframe application work into smaller units and demonstrate updates they’ve made that fit into more long-term initiatives.

The Autobahn for Application Development

While applications and products are not all developed at the same speed (e.g. a change to a mobile app versus a regulated change to back-end banking system), there is a need for these systems to utilize similar approaches and a common delivery pipeline with visibility of dependent changes, to be implemented as quickly as possible.

An example of this is a highway like the Autobahn accommodating different types of drivers and vehicles, such as a Porsche built for speed and a semi-truck built to carry cargo. The rules of the highway allow vehicles to drive at different speeds safely without having stoplights by providing exit and entrance ramps, speed limit minimums and visibility of other vehicles along with “standard rules of the road.”

Likewise, we want our IT delivery capability to accommodate variable speeds of IT based on the business needs and the development teams’ abilities to go faster by implementing practices such as smaller batch sizes, APIs and microservices, feature flags and performing automated testing and readiness validation. While distributed and mobile applications might be the Porsches on the road, they can only go as fast as the mainframe and legacy applications, or semi-trucks and other traffic will allow.

Consequences of Resisting Change

No company will get through the next decade without being disrupted, either externally or internally via a continuous improvement and innovation mindset. Those that embrace Lean and DevOps practices across the entire IT organization will have a steady stream of new services and improvements that outpaces the competition and transforms their business. Those who don’t will join former Fortune 500 companies that waited too long to realize that their competitor’s Porsches and semi-trucks were too far along to catch up with.

About the Authors / Carmen DeArdo and Jason Patterson

Carmen DeArdo is the director of Build Technologies at Nationwide Insurance. He is responsible for helping drive accelerated delivery utilizing Lean and DevOps techniques across the end-to-end delivery value stream. One current focus area is the creation of an integrated delivery pipeline capability.

Jason Patterson is a Lean and Agile coach at Nationwide Insurance. He works with development and support teams across the company to help recognize process gaps and opportunities in their delivery stream. His background in development fuels that energy, while he resists the urge to automate every broken or unnecessary process. After nearly a decade at Nationwide and seven years experimenting with extreme programming, scrum and agile methods in a software development department at Indiana University, Jason is still learning.

August 17, 2016
Simplify and Streamline Hybrid Cloud with DevOps

The cloud is not new. The virtues of the cloud have been proclaimed incessantly for years, and most businesses have embraced cloud servers and services on some level. The realities of data security and the need to leverage existing investments in IT infrastructure, however, generally result in some sort of hybrid cloud setup, with servers and applications that span both local data centers and cloud providers. Hybrid cloud deployments can be complex and volatile, which is where DevOps comes in.

I wrote a report recently for devopsy.kinsta.cloud that takes a closer look at the pros and cons of hybrid cloud, and the symbiotic relationship with DevOps. “DevOps Enabling the Hybrid Cloud” examines how DevOps tools and practices streamline management of such an environment. It also looks at some of the security implications and how DevOps can address those concerns.

The report explains, “Speed and flexibility are imperative for organizations that want to remain competitive. Organizations that strive to lead understand that IT must be viewed as a strategic advantage—a powerful tool for driving innovation and creativity. The hybrid cloud provides a bridge from legacy hardware and assets to cutting-edge concepts such as containers and serverless computing for organizations that do it right.”

One truism addressed in the paper is that customers—whether they’re external customers of your company or the users within the company that rely on IT services and applications—don’t know or care where your applications or data are delivered from. They know that they want stuff to just work when they need it to work. It’s up to IT to determine the most efficient and effective way to achieve that goal, and to provide a seamless, frustration-free experience. Sometimes that means relying on cloud servers and services, and sometimes it means using local infrastructure.

I spoke with Justin Foster from Trend Micro about the overlap of hybrid cloud and DevOps, and specifically about unique security issues with hybrid cloud environments. He stressed that legacy tools and traditional approaches to security are not sufficient for the more dynamic hybrid cloud environment. As explained in the report, “Security has to be baked in from the beginning and throughout the development life cycle.”

A pure cloud infrastructure is unlikely for any established company. It may be achieved eventually through attrition, but there’s no getting around the fact that there already are servers and applications in place. The hybrid cloud approach is the right solution in most cases, and DevOps tools and practices are essential to help manage and secure the hybrid cloud effectively.

Click here to download the report: “DevOps Enabling the Hybrid Cloud.”

August 17, 2016
What DevOps Skills are Organizations Looking for?
DevOps is a cultural, philosophical and organic approach toward automating workflow and getting products to market more efficiently and effectively.

DevOps engineers are in top demand right now. According to studies by Gartner, 25 percent of top global 2000 organizations will have adopted DevOps as a mainstream strategy by the end of 2016.

DevOps is both a cultural movement and a philosophy … which can make it difficult to define the specific skill set you need a DevOps team to possess in order to function successfully.

Of course, an excellent way to show fluency in DevOps is by earning a DevOps Institute certificate—for example, DevOps Foundation^®.

That being said, there are some “mission-critical” DevOps skills that organizations are searching for pretty universally right now.

1. Experience with infrastructure automation tools and fluency in web languages

Experience with tools such as Ansible, SaltStack, Puppet, Docker, Chef and Windows PowerShell DSC as well as fluency in languages including Ruby, Python, Java and PHP are must-have skills for DevOps pros.

Organizations are searching for DevOps candidates who are highly proficient in many of these tools and languages and open to increasing their competencies further when given the opportunity to do so.

2. Expertise in “soft skills”

DevOps evolved as a way to enhance collaboration across project life cycles and connect front-end and back-end professionals on either side of the software development and operations spectrum.

When software developers and operations staff (system engineers, systems admins, DBAs, etc.) collaborate effectively, communicate honestly and work as a cohesive team, the software development life cycle (SDLC) is streamlined, risk factors are mitigated, and projects are completed and deployed with a minimum of bugs, quality defects and glitches.

In addition, it can be useful for organizations to have DevOps specialists on their team who are comfortable in the role of client consultant, able to interact effectively and build positive business relationships with customers while addressing their needs and wants.

Organizations look for DevOps candidates who are skilled in interpersonal communication, with an arsenal of listening, negotiation, problem-solving and team-building skills at the ready.

3. Familiarity with CI (continuous integration) tools

Continuous integration is a DevOps methodology that continually merges source code updates from all developers on a specific software build, notifying the team of any failures in the process.

This offers a real-time window into the developing software system, allowing immediate and constant engagement of all team members throughout the project lifecycle.

CI tools DevOps candidates should be familiar with include CruiseControl, CruiseControl.NET, Jenkins, Bamboo, Hudson, ThoughtWorks’ Go, Urbancode’s Anthill Pro, Microsoft’s Team Foundation Server and Jetbrains’ Team City.

4. Bonus Skills

Other skills hiring managers are on the lookout for in the DevOps space include:
- Extensive knowledge of software build cycles
- Experience deploying code
- Experience in software architecture
- Familiarity with application programming
- Database management
- System design
Useful experience and certifications include:
- Project Management
- Risk Management
- Scrum Master
The good news is that the industry’s need for DevOps team members with diverse and evolving skill sets is in sync with what DevOps candidates want from their workplace. According to the 2014 State of DevOps Report from Puppet Labs, an atmosphere conducive to learning is a primary component of job satisfaction among DevOps engineers.

About the Author / Robin Borough

Robin Borough is President of Omnikron Systems, Inc., which offers IT and management training and IT staffing services. As an advisor to technology professionals and executives for over 25 years, Robin has built a world-class reputation as a “go to” career strategist in the IT industry, one who’s capable of successfully guiding executives and specialists, at any level, through transition.
August 16, 2016
Does Modeling Have a Place in DevOps?
As organizations adopt DevOps, they not only release smaller changes sooner, but they also change the way they develop software. Additionally, applications no longer are seen as giant monoliths, but rather as dispersed across dozens of services that may be built on different platforms (from Node.JS to .NET), depending on the team and technology du jour. To complicate matters even more, as technology is changing, so are business requirements.

This is where an application model comes in. It saves time, ensures quality and increases organizational knowledge of the application and its components that are constantly being changed and rebuilt. This article will explain what an application model is, who is involved in creating one and how to start using it.

What is an Application Model

The application model is not a specific set of unified modeling language (UML) diagrams or models used in a direct 1-to-1 relationship with the code. Rather, it is a high-level document that furthers understanding of the application and helps provide insights about that application.

The model needs to be approachable to personnel from many different business roles, including developers, project managers, IT administrators and even users. The array of people who might use the model should give a rough idea about technical level.

What makes the application model so different, so useful and so important is that it defines not just the field of the application but also the specific domain, which is also why it can be called a domain model.

Field vs. Domain

For example, a core function of banks is to underwrite loans, but they do many different types of loans. A field of underwriting could be as specific as commercial underwriting for loans over $5 million. This field will have many similarities no matter the bank because underwriting is a regulated practice; therefore, the information used to make a decision is mostly identical.

However, the field isn’t specific enough to create an application model because it doesn’t include the unique factors of a specific bank—things such as the amount of weight a system gives to any number of loan factors (credit score, cash on hand, etc.), processes that are unique to an individual bank or even specific nomenclature that bank uses. All of these factors go into creating the application model.

In the end, the team gathered will build a complete model of what the application is and how it currently works. It’s important that the model being created reflects the actual application and not an idealized version of it. Because many business applications are quite large, it often makes sense to have a broad model for the entire application and several more intricate models for specific sections or features.

Bite-sized chunks

For example, a bank will have different criteria for commercial underwriting depending on size, income and even type of business. While the overall application covers all the different types of businesses being underwritten, different sections could easily be divided by requirement and modeled individually as part of the whole.

Lastly, it’s important that the model is reflected in the code base for easy understanding by the development team. If the model is meant to serve as a central hub for conversations about changing and improving the application, the decisions and insights found during those conversations can be relayed and documented for the development team more easily and much better if there is a tight relationship between the model and the code base.

How to Create the Model

As mentioned earlier, the model must be approachable and understandable to individuals across many different business roles and with different backgrounds because it is the thing around which conversations about change and improvements happen. So, the first logical step in creating the model is assembling the team.

The core team should share several attributes for success. They should all (no matter their business title) be very familiar with the application and the current processes; this is not the place to learn about how things work.

At a minimum, the team should include developers, operations and project managers, but knowledgeable users can also be invaluable. Everyone should also be above average communicators because they will be working to create understanding with the core team and relaying that understanding to non-team members.

Communication Matters

Communication is an extremely important skill for any member of the model team to have. For example, there could be an incredibly gifted developer who has vast knowledge of the application being modeled, but who speaks the local language as a second language and without advanced and nuanced proficiency. This is fine when changes that they are working on are documented through a ticket system and can be understood by reading the ticket and looking at the code base; however, it’s not appropriate for the modeling team.

At best, that developer will slow down the entire modeling process because they will have to ask many questions to fully understand what is being said by non-developers. At worst, that developer will not ask questions to get the understanding needed, and therefore will not be able to share their insights with the modeling team nor adequately explain the model to fellow developers.

Once a team has been created, the best start is to talk about the application itself as it is. The goal is to create a shared understanding of how the application works and how it’s being used. Some good starting topics are:
- What does it do well, what doesn’t it?
- What’s missing from the application?
- What’s in the app that’s not being used?
- What other applications does this interact with and which processes?
- Is the application being misused (e.g.: a field labeled one thing used for another)?
- What is the process and data flow?
- How much overlap is there between data and objects?
- How granular are security requirements?
- What are the application deployment specifics?
- Which users will need access?
- Which APIs will be used?
- What are the Performance and Health Monitoring requirements?
Conversing around a whiteboard is often helpful for quick sketches and lists that everyone can see. You’ll note that there are both non-technical and semi-technical topics above, so a good project manager who is familiar with both ends of the application can be invaluable in facilitating discussion.

Miscommunication

For example, if an underwriter who works on business loans was describing the process to a group modeling an underwriting application, they might inform the group that a credit rating is one of the factors that is used in determining loan worthiness and needs to be recorded in the application. Credit ratings are something that everyone who has ever had a credit card, home loan or even auto insurance is familiar with. However, it also presents an opportunity for miscommunication.

Individual loans (for things such as a mortgage) use a credit rating score that can go up to 850. A developer might know this from personal experience and create a field on in the application to record a credit score. Restraints on the field could be numeric only, with a range of 300 to 850 (this would help ensure that only valid scores were entered). However, businesses are given credit ratings with letters and symbols—not numbers—such as AAA, for example.

In this example, if there were no follow-up questions and no examples used even though multiple people on the team were using the same term and thought they were using it correctly, there is still miscommunication that would result in an error in the final software that would then need to be fixed.

As discussions progress (and it will take time), what is being developed by the modeling group is a shared understanding of the application, as well as shared knowledge of how to talk about it. This allows words to be used precisely with the assurance that everyone on the team is using them in the same way.

Sometimes this understanding is referred to as a universal language because through questions, miscommunications, explanations, demonstrations and discussions, what was created by the core group is a way to easily and fluently talk about the application without any misunderstanding.

Using this shared understanding documenting the application should be achievable. Because the application model isn’t a set series of diagrams or maps, there isn’t a definitive template for creating one, but many teams have found that a combination of data flow, activity, use case, sequence and component diagrams are at a high enough level that everyone can understand and use them, while still being technical enough to be useful for the development team. It can also be helpful for text documents of notes, snippets of sample code and even photos of whiteboard drawings to be part of the model.

Of course, a series of diagrams and notes can be messy to use and organize, so oftentimes a mind-map can be helpful as a method of keeping track of the disparate parts and diagrams and maintaining ease of use.

Modeling for Development

Having a model is all well and good, but how does the model help with development? By defining the application not just from the developer point of view, insights will have been gained about what needs to be tweaked, what needs a major overhaul and what needs to be demolished completely.

By not just knowing what needs to be changed but why it’s broken and what lead to the break makes changing the software not just an issue on a ticket, but a complete picture. Also, noting application traits such as data overlap, object reuse and layers of abstraction allow the core development team to be able to develop and iterate quickly without worrying about changing functionality in an unplanned way, thereby saving on testing time and costs.

Modeling to Save Time

For example, if a new federal regulation were introduced, and several new pieces of information had to be logged for loans of a specific size (or greater), the domain model might show:
- That some information was already being collected and only needed minor updates.
- None of the information was being collected at that size loan; but it was for a different size, and that function could easily be added to that section of the application.
- That data wasn’t being collected on loans of any size, and a conversation might ensue to determine if it was worth including the same data requirements on loans other than newly regulated.
- That information could be calculated based on existing procedures and would only need to be accurately logged and kept.
Surely, a model is only useful so long as it’s current. Any changes to the application need to be entered into the model. An outdated model can be worse than having no model at all since conversations that occur around it are started with false data and assumptions. If a model isn’t maintained, then all of the organizational time and effort that went into it and shared knowledge it created are lost.

Maintaining the model is key to ensuring its usefulness long after it is first “complete.” Once a model is created, it also can be helpful for the members of the modeling team to present and discuss it with their department coworkers. Sharing knowledge beyond just a few people ensures that no matter what personnel changes happen or what development trends and methodologies are implemented in the future, the model can carry on and maintain usefulness as a teaching and documentation tool.

Modeling for Operations

As applications and infrastructure become more and more intertwined, it becomes critical that both Developers and Operations are involved with creating an application model. Any changes that will be considered based on the model will have to take into account factors such as security, application monitoring and the host of other criteria that Ops deals with.

Designing Changes with Devs and Ops

When evaluating an organization for a loan, the credit rating of that organization will need to be acquired. There are vendors that perform the task of rating other organizations, and individual banks will have preferred vendors for that metric.

If the task of gathering these ratings were to be automated, a developer might want to implement a simple process wherein the application queries the vendor through its API and waits for a reply. This is simple and straightforward, and only requires two pieces of data: the query and the report. However, Operations would certainly step in because this scenario would be a security risk. It would either involve two-way communication or require leaving the communication channel open for an extended period of time.

A better solution that could be created with input from operations might involve an additional piece of data: a query ID. The interaction would then consist of multiple steps such as a query being made, a query ID getting returned and, then, at regular intervals, the vendor being sent the query ID until the report is ready.

Operations will be involved in the application development, either from the start where it can help define practices and procedures that the development team must follow, or before the application is released and it raises red flags and security issues the development team must comply with before the changes are released. It’s much more efficient for Operations to be involved from the start.

Get Modeling

Organizations have proprietary software because it is a competitive advantage for them to create their own applications based on their business needs. Because business needs are constantly changing and evolving, the software that encompasses them must change and adapt as well.

Finding methods that allow for quicker changes while ensuring better change quality should be a goal in any organization. Creating an application model, and the shared understanding that goes with it, can be an invaluable part of any application that faces maintenance or overhaul.

About the Author / Scott Reece

Scott Reece is a huge fan of the greater visibility and communication enabled by DevOps. As a Friend of Astah Scott has been successfully closing project communication gaps by using models and team building. With more than 10 years of project management experience, Scott currently serves as a Partner Director at Inedo.
August 16, 2016
Unconventionality within Constraints

What do dependency resolution, situational awareness and superheroes have in common? Meet Chris Corriere, a DevOps/software engineer at Autotrader speaking on creative ways to maximize usage of all of the above. Mark Miller, community advocate and senior storyteller at Sonatype, caught up with Chris to learn more about what his team is up to.

Corriere: I’m Chris Corriere, and I’m a DevOps engineer at AutoTrader.

Miller: Can you give us an overview on how you’re using Nexus?

Corriere: We use Nexus for dependency resolution. Part of that is to insulate our enterprise infrastructure from general third-party dependencies. We’re able to filter that down with our internal repo and make sure that our developers are using things that are pre-approved, that we’re allowed to track and update. We also use it to move around internal projects that have cross-dependency between applications in-house.

Miller: How are you enforcing pre-approved dependencies?

Corriere: They are generally around who has access into Nexus and who is allowed to improve and pull things into Maven Central. But there are ways around that. There are ways to sneak things into builds. If a developer really wants to get something in then, sometimes when you’re troubleshooting, you find some of those clever solutions. There’s definitely an element of striking balance between that, of making sure they’re able to experiment and get the tools in they need to get their job done, but not exposing any liability or risk in the process by pulling something in that’s not trusted.

[soundcloud url=”https://api.soundcloud.com/tracks/270488651″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=true” width=”100%” height=”450″ iframe=”true” /]

Miller: If somebody isn’t using a binary repository, what could you tell them that would get them to use one?

Corriere: Consistency within the build environment is one of the main draws to me, that and reusability of code. I think that doubles back into situational awareness—depending on how they’re measuring their organization, what they’re looking for in terms of improvement, and where there’s room the optimize. There’s probably a lot of interruption in build, a lot of different permutations. I’ll say in different solutions, it could be more standardized and more consistent across teams. You can get by without it. But you’re probably doing a lot more work for the same end result. Without a repo, there’s a lot more drama, a lot more heartache involved.

—

“Without a repo, there’s a lot more drama, a lot more heartache involved.”

—

Miller: I would think that the more developers you had, the more you would actually have a real need for it.

Corriere: I would say the most creative solution I’ve seen is teams get pretty consistent results with using source control and builds at particular revisions. But in that instance you have to have the build to succeed. You have to get through the build, and that’s going to take time even if it doesn’t break. Your repository is putting everything after that step where it’s ready to go. You know it’s built. You know it’s safe. It didn’t get into the reboot. It didn’t make it through the filter.

Miller: Have you guys done anything cool with Nexus?

Corriere: I wouldn’t say anything too out of the ordinary. There’s some overlap into how we’re leveraging it that looks more like configuration management, which is unconventional. Depending on the project, you’re able to check some things from Nexus and how they will stand up [against] a lot of your environmental dependencies outside of just a job application. It can dip into the version of Java you’re running, different inch grips, and other things that nest into each other. That gets you to sort of square one to launch a lot of our standard internal apps. Again, back to the situational awareness, given some of the constraints of the teams we were working with, it’s something I ran across and it felt a little left handed at first. But when you stop and look at it, they were able to get the ball moved down the field within the constraints that they were given. At the end of the day, it works.

Miller: Are you using any continuous integration, continuous delivery?

Corriere: Yeah. I don’t think that’s a boolean of a value. We’ve got a lot of it end over life cycles, and we’re running continuous builds. We’ve got some in-house tooling around that, and we also run Jenkins. There’s always room to accelerate, improve and optimize it. We’re moving to more continuous delivery. We really want to start getting things out closer to prod. But you always start with the left side and move out from there, and try to debate quality from step one where there’s lower risk. I’ve seen AutoTrader make a lot of good strides in that over the past couple years.

—

“There’s always room to accelerate, improve, and optimize it. ”

—

Miller: Final question, if you were going to be a superhero, would it be in Dev, Ops or Sec?

Corriere: I’m going to go with security because I think there’s some development and operations involved in that if you’re doing it right. I don’t think you should be blue team or red team. I think team purple is where it’s at. You really got to come at it from both ways to get the job done today.

—

“I don’t think you should be blue team or red team … purple is where it’s at. You really got to come at it from both ways to get the job done today.”

—

Miller: Excellent. Thank you, Chris.

Corriere: My pleasure.

If you loved this interview and are looking for more great stuff on DevOps, I invite you to download the “Continuous Delivery and DevOps Reference Architectures,” highlighting real-world DevOps practices from organizations including PayPal, IBM, Adobe, USPTO and many more.

August 15, 2016
The ‘Continuous’ Illusion

Continuous delivery is an illusion. That doesn’t make it insubstantial—illusions can have a tremendous impact. Like the first motion picture: the locomotive felt so real to first audiences that they fell from their chairs, screaming. But of course, the locomotive was an illusion, merely rushing past on screen.

A good magician, like a good inventor, will tell you that the secret to a powerful illusion is hard work on top of a good idea. Indeed, good magicians and inspired technologists have a lot in common. Each is always on the hunt for the next big insight, the transformative angle, building on existing industry techniques to create something even greater. But each knows the secret to truly wowing their audience, whether for a one-night-only show, or an ongoing relationship. What really gets the job done is the vast, complicated, hard-won machinery ticking away behind the scenes.

Today’s No. 1 magic trick for making great applications, a directive now pervasive across every industry, is continuous delivery. Right now, continuous delivery provides that wow. It takes app updates from a customer stumbling block to a seamless benefit. It takes feature delivery from “maybe in a year” to “is tomorrow soon enough?” It allows Dev teams to wave their magic wands and transform the impossible into the effortless.

And, of course, continuous delivery is an illusion. Remember that movie locomotive, leaping out of the screen? Every instant of its motion was its own little frame of celluloid. And the true magic was the process that bound each and every frame together, delivering them onto the screen with perfect timing.

Continuous delivery is the very same sort of illusion. The steady stream of new features, new capabilities and bug fixes must be created one by one by a vast army of talented developers—not unlike the thousands of talented stencilers who first brought color to motion pictures. And the true heart of the process is the automated systems that support this army, and which ensure the coordinated delivery of its products to the end user.

The heart of continuous delivery is the machinery of automated testing and deployment. The approach behind continuous delivery encourages teams to produce software in short cycles, ensuring that the software can be reliably released at any time. For these releases to be reliable, continuous testing is integrated into the software delivery pipeline, for immediate feedback on any bugs an update would generate. Then, when a proposed change has passed the automated tests, it is deployed to production automatically. This software development life cycle (SDLC) allows companies to achieve the business goal of truly responsive development. The team sees an opportunity, confirms the solution and then delivers that solution to customers extremely rapidly.

Each stage of continuous delivery—develop, test, deploy—has its own technical challenges, but the testing step in particular poses a unique bottleneck.

In the development stage, developers primarily require code. Code is relatively lightweight, and can be managed through industry-standard tools such as Github. Developers need their laptops, but in general code management poses a limited operations support challenge.

The deployment stage can be more challenging for ops. The newest version of the code must be deployed to a wide population of final platforms, which may vary in hardware, software, location, permissions and so on. However, the target systems are at least those in ordinary use. The server the new page runs on or the smartphone that installs the application update are pre-existing systems, and they are managed by their users. So Ops teams are not required to support specialized hardware to deploy. Furthermore, in the past few years a number of innovative technologies, such as Ansible and Docker, have emerged to simplify this process further.

Testing, on the other hand, combines the challenges of both of these stages. Continuous testing requires dedicated resources for developers and QA. But these resources are full-scale application environments, including not just code but data, operating systems and even appropriate hardware. And because best practices encourage developers to write new tests for all new code, the amount of regression testing required for every code change can only grow. That means that testing must grow ever more parallel, or suffer increasing delays—and any delay in testing can undermine overall continuous delivery.

To preserve the Continuous Illusion, a development team needs on-demand access to scalable testing environments, including their data. Data in particular is often overlooked in test environment creation, but it must be delivered to every test instance. That means test data management (TDM) is an inescapable part of any successful continuous strategy.

TDM is relatively self-explanatory: it refers to the process of creating, managing and delivering production (or production-like) data to non-production data environments. Every organization does some TDM, though many do not consider it a separate practice. But to achieve the sort of seamless efficiency that can support the Continuous Illusion, teams must adopt a modern TDM practice.

Legacy TDM relies on multi-stage, manual delivery chains. Developers and testers file tickets, and then they wait. Storage admins, DBAs, backup admins, security personnel and other staff each have to spend a huge portion of their time servicing the ever-growing stream of tickets, handing off environment creation from one function to another. Meanwhile, developers and testers are left in the lurch, and continuous delivery grinds to a halt.

Once, this process might have been unfortunate but necessary, imposed by technological restrictions. But modern solutions can tackle these challenges and overcome them. Too often, teams with poor TDM have simply neglected to implement the solutions that they need.

The most effective TDM practices will use a data management solution on par with other continuous delivery pipeline tools. The TDM solution will be able to stand up data environments in minutes and at massively parallel scale. It will rely on production data and automatic security, rather than synthetic data generation or data subsets (both of which requires substantial upfront planning and cannot hit the full suite of edge cases). It will integrate readily with existing continuous delivery workflow tools, allowing data to be automatically provided whenever a test job runs.

There are some solutions in the market today that meet these criteria. Adopting one can represent your opportunity to tighten test cycles and deliver code faster—regardless of how fast your software timelines are today.

To deliver the Continuous Illusion, every component of the continuous delivery pipeline must be operating at peak efficiency. A powerful TDM solution is a key requirement for allowing your customers to sit back, relax, and enjoy the magic of your product.

About the Author / Louis Evans

Louis Evans is a DevOps evangelist with Delphix. He’s excited about the potential for superior TDM to enhance overall software quality. He is a huge Alan Turing fan, and partial to a well-placed semicolon.

August 15, 2016
Case Study: Stytch Before, After DevOps Culture, Tools

Cloud analytics provider Stytch leverages its cloud-based analytics for customers who want to centralize, fuse and model their data. “Our mission is to make it easier and faster for business users to get insights and make decisions,” said Taylor Frost, cloud operations engineer at Stytch.

Before the company implemented a DevOps approach, its primary software platform and some services were large monoliths maintained by disconnected teams. Stytch tested new features in isolated Jenkins jobs and performed quality testing and product releases manually in siloed realms that were out of sight and out of mind, Frost said.

Stytch’s Dev, Ops and QA were walled from each other, isolating the knowledge base of activities in any one group from members of the other two groups. Stytch teams did not share this information, and sister teams could not benefit from the insights this knowledge held.

What’s more, developers did not have transparency into the products and tools that manipulated the cloud-based infrastructure or the build pipelines. “This fostered an attitude of ‘it’s an (Ops/Dev/QA) problem’ and discouraged a sense of ownership of or responsibility for success, which led to an increasingly lengthy feedback cycle and slower, less frequent builds,” Frost noted.

Stytch lacked the orchestration and the down-in-the-weeds level of control it needed to enable real-time changes to labyrinthine back-end technologies without repercussions that could cause the customer base to suffer, and to speed and simplify deployment, she said. Standards such as Terraform or CloudFormation did not provide the fine-grained control of deployment that Stytch needed.

The company’s Ops was in charge of Python software libraries, but other groups including Dev and QA needed access. Stytch also has a performance team that the company was eager to include in Python library development.

Stytch After DevOps

Stytch is using DevOps to unite teams and divide the various software monoliths into microservices for continuous integration (CI) and continuous delivery. The company has already fed its core software platform through the new build pipeline a number of times, and continues to create even more automated and efficient deployment processes.

As it works to create homegrown Python orchestration libraries, Stytch is establishing a stronger grip on deployment. All teams connected to the pipeline including Dev, QA and Performance are actively expanding these libraries.

Stytch increasingly intermarries its Dev, QA, Ops and Performance teams and enables and rewards collaboration across the stretch of the DevOps pipeline. “Our goal is transparent, scripted change integration, from check-in to testing and deployment. We empower our developers to take responsibility for their work, from check-in to production deployment,” Frost said.

Stytch uses frequent meetings of a continuous delivery committee with members from all four groups to update the status of the pipeline. The company is swapping out Jenkins for Travis CI for builds and tests, making Dev more hands-on and able to gauge its work from start to finish.

“Developers are iterating on their pipelines in the same way that they iterate on their projects,” Frost said. Stytch is able to find and fix problematic code earlier in the pipeline while accelerating development cycles. Customers are pleased with the faster rate of innovation on the software platform they use.

New Infrastructure, Training = More Results

Stytch has made cross-pollination of expertise common for pipeline teams. “We have an annual training allowance and our engineer-led ‘Ministry of Propaganda” holds regular afternoon sessions to introduce new technology and practices,” Frost said.

As CI and continuous delivery processes grow, absorb and automate manual steps, related activities across the pipeline become more translucent for everyone. Both the Travis CI platform and the use only of scripted product releases and scripted tests for performance are increasing that transparency.

The metamorphosis that has spread throughout the teams with direct access to the DevOps pipeline now extends to the rest of the company, as needed, such as with marketing. Frost provided an example:

“Our marketing team was struggling to make good progress with regular releases of our WordPress-based website. Our engineers were fielding regular requests for basic tasks—starting and stopping servers, updating configuration files and retrieving logs,” she said. “We brought our engineers and marketing together on this. The result was a simple command-line tool that enabled our nontechnical marketing staff to perform complex actions on their own, including promoting their changes to the production environment.”

August 15, 2016
8 Reasons to Go ‘All In’ and Attend Jenkins World

Jenkins World 2016, the world’s largest gathering of Jenkins users, is coming to the Santa Clara (California) Convention Center, Sept. 13-15. The lead organizing sponsor is CloudBees, along with a number of premier Jenkins ecosystem vendors including BlazeMeter, Undo and Black Duck. Jenkins World will offer attendees opportunities to learn, explore and network. This year’s theme is “All IN” as Jenkins users, experts and thought leaders prepare to go all in on DevOps.

Need more convincing? Below are eight reasons to go all in at Jenkins World 2016:

Hear keynotes from industry leaders — Kohsuke Kawaguchi, founder of the Jenkins project, kicks off the conference with the opening keynote this year. Other keynotes include Sacha Labourey, CEO of CloudBees, and Gary Gruver, former DevOps exec at Macys.com and HP and industry author. Rumor has it that Gene Kim may make a guest appearance, too!

Attend training/workshop add-on options — Come to Jenkins World as an attendee, leave as a Jenkins master. Join us a day early to attend Jenkins certification training and/or learn the fundamentals of Docker and Jenkins. Additional workshops, covering topics such as plugin development, Jenkins certification and automating pipelines with the Pipeline plugin, are available.

FREE certification! Your Jenkins World registration also provides you with the option to take a certification exam completely free! Did we mention it was FREE? All you have to do is sign up and study up!

Brush shoulders with the Jenkins stars — Get access to some of the best Jenkins experts in the world; attend their sessions and network with them. The sessions cover a range of topics including infrastructure as code, security, containers, pipeline automation, best practices, scaling Jenkins and community development projects.

Visit a variety of vendors — At the expanded Sponsor Expo you can check out a range of technologies and services that help you optimize software delivery with Jenkins.

Pick up your next read — CloudBees Senior Consultant Viktor Farcic has recently published his book, “The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices.” Be one of 200 lucky attendees to get a free copy and be sure to catch his session Sept. 13. We will even be hosting a book-signing with Viktor.

Do you follow CommitStrip? They will be onsite and you can help them paint a custom Jenkins-themed mural!

Meet the Butler and snap a selfie – You may have seen the Butler on @CloudBees engaged in some Xtreme social media adventures to get himself to Jenkins World, but at Jenkins World you’ll have the chance to meet him in person. Don’t forget to snap a pic with him at the social media station and share with your friends.

All of this and so much more awaits you at Jenkins World. Go “All In” and register now! Use this code JWDEVOPSCUST and get 20 percent off your conference registration.

See you in Santa Clara!

August 12, 2016
Test-First Development: Processes and Tools for Success
A lot of software development teams are talking about “test-first” methodologies. The practice involves moving testing up into the very earliest stages of development so automated tests can be written before code. Making this seemingly minor shift can result in much higher quality software and greater efficiencies. Integrating development and testing avoids needless late-stage bugs and complicated redevelopment efforts. It can help software development organizations focus on usability and features and achieve faster time to market. And because of test-first’s reliance on automated tests, organizations gain reusable tests, saving time and increasing standardization for testing and quality assurance overall.

It’s no surprise, then, that 51 percent of organizations have begun using test-first methodologies, including 37 percent in the past year, according to a recent global survey of 200 software testers conducted by our company. But, despite the enthusiasm for this practice, there are concerns: 44 percent of survey respondents said their biggest fear about moving to this approach was forcing developers to contribute tests prior to completing code, while 19 percent said they worried about removing traditional testing checkpoints on the way to production. Other barriers include the lack of standard practices or tools. That said, test-driven development promises to bring ample benefits to software development teams, the business and, importantly, customers.

Test-first development is recognized across three broad terms today: behavior-driven development (BDD), test-driven development (TDD) and acceptance test-driven development (ATDD). TDD is an umbrella term for the practice, although it has traditionally focused more on technical, unit testing. BDD and ATDD incorporate user requirements and business benefits along with technical testing. BDD differentiates through its adoption of specific automation frameworks and language for writing and running tests.

Key Changes for Successful Test-First Development

Getting started requires change across a few critical areas for most teams. First, evaluate your team for hiring and skills training needs. On the one hand, developers will need to learn basic testing skills, so they can write good tests and run tests before they develop code. Meanwhile, testers must become more proficient in automated testing tools, which usually entails acquiring a fundamental knowledge of programming. Many companies will need to hire key contributors from outside; in which case, leaders should look for well-rounded individuals with past experience using test-first methodologies successfully. An individual who is an expert at one skill, or who is too independent and can’t collaborate well, won’t be an ideal member of a team experimenting with test-first approaches. Leaders also may need to adjust by balancing the ranks more closely between developers and testers so that one group doesn’t rule over the other. Hiring additional experienced testers frequently will be necessary.

Training of existing staff is paramount, best achieved through a combination of methods including internal training, mentoring, online courses and paired development, where developers and testers can sit side by side and collaborate. More than half of the survey respondents said developers and testers would be jointly responsible for automation test creation within their organizations.

Beyond cross-training, the cultural challenge of moving to test-first methodologies can be overwhelming. It requires the development of incentives for creating quality products, not just delivering new features at speed. It requires an appreciation and respect for the practice of testing. Unfortunately, testing too often is viewed as a necessary cost of doing business, rather than bringing a clear benefit to the organization and its customers. That mind shift is not only inaccurate, but it doesn’t fly with progressive, Agile thinking. It will take strong leaders to help traditional teams understand the need to change and modernize practices. A leader may forego the buy-in process and instead decide to lay down the law to staff: Staff must move to TDD or BDD, or else. But the results won’t be ideal. Smart developers will find loopholes, such as writing tests that aren’t comprehensive or fully aligned with project goals, so the code will pass more quickly.

On the process front, organizations that have moved or are moving from Waterfall to Agile and DevOps will have an easier time adopting test-first methodologies. In fact, it’s pretty difficult to implement test-first processes and tools in a Waterfall environment because of the siloed nature of functions and the long release cycles. The survey results support this premise: 55 percent of respondents said their testing process was Scrum/Kanban-driven before transitioning to test-first, while 21 percent said their approach was iterative-requirements driven and 18 percent was Waterfall-requirements driven.

Along with that shift, test-first teams will need to adopt new metrics that align with this new way of doing business. Instead of looking solely at velocity, development leads should measure how long it takes the team to move a product from feature approval to code release. With a focus on quality and testing, companies also should measure and improve upon bugs, such as number of severe defects post-production. Those types of business and customer-oriented metrics align with the new set of Agile metrics that that many teams are working for today.

Test-first teams will need to adopt DevOps toolsets, which are designed to push code out to the pipeline faster. Specific technologies that go hand in hand with test-first development include:
- Continuous integration (CI) platforms such as Jenkins and Team City, to assist in running the automated tests immediately for quick build feedback.
- Container technology for rapid deployment of test and pre-production environments, to capitalize on having features production-ready soon after coding.
- Test management platforms including qTest Scenario or Cucumber Pro.
- Test automation frameworks including Cucumber, SpecFlow , and other open source tools which enable easier support for customization and collaboration.
There’s a lot to think about when moving toward test-first development. While it may seem overwhelming at first, all of this is achievable in a reasonable time frame if you have the right dedicated resources to see it through. One-third of survey respondents said it took them less than three months to complete the transition to a test-first approach, while 30 percent said it took less than one year. The payoff for software teams, in terms of improving quality, customer experience and reducing costs of innovation, can deliver higher customer loyalty and even faster growth in the marketplace.

About the Author / Kevin Dunne

Kevin Dunne is vice president of Business Development & Strategy at QASymphony.
August 12, 2016
Why Your Organization Needs DevOps

The recently released fifth annual “State of DevOps Report” by Puppet has once again brought with it fresh insights into what drives high-performance teams. The report, which comes at a time when DevOps has ceased to be a buzzword, provides proof of why more and more businesses across the world are adopting DevOps to improve their organization’s performance, revenues and profitability. DevOps has truly come of age and this year’s “State of DevOps Report” clears any doubts fence-sitters may have about the effectiveness of the DevOps approach.

The 2016 report reasserts that unparalleled IT and organizational performance is a result of a team effort spanning development and operations. The report also highlights the importance of improving the entire product life cycle—from initial product planning, to quality and security assurance, to customer feedback—to speed up delivery while improving quality, security and business outcomes. You can read the full report here.

In today’s competitive business environment, DevOps has emerged as a vital solution that helps organizations meet ever-growing industry demands. The “2016 State of DevOps Report” helps us understand how the practices associated with DevOps affect IT performance and organizational performance. Let us examine the findings of the report to ascertain how DevOps can benefit your organization.

Why Does Your Organization Need DevOps?

The “2016 State of DevOps Report” makes it plain and simple that companies that incorporate DevOps practices get more done. According to the report, high-performing IT organizations deploy 200 times more frequently than low performers, with 2,555 times faster lead times. They have 24 times faster recovery times and three times lower change failure rates. High-performing IT teams spend 50 percent less time remediating security issues and they spend 22 percent less time on unplanned work and rework, the report states.

Performance

If we boil down the report findings into actual benefits of adopting DevOps, we could say that DevOps enables continuous software delivery with less complex problems to fix and faster resolution of problems. This essentially means that you can deploy faster and more effectively in more stable operating environments and add value rather than just fixing or maintaining.

DevOps can revolutionize the performance levels of organizations and allow them to deploy at a rate never seen before. It has certainly enabled organizations such as Etsy, Netflix, Facebook, Amazon, Twitter and Google deploy hundreds and thousands of code per day while delivering world-class stability, reliability and security. You can find out how different organizations have implemented DevOps to their benefit in this YouTube video.

Security

Higher performance is not the only payoff in the DevOps culture. Organizations that adopt DevOps practices are known to have better information security. The “2016 State of DevOps Report” clearly states that organizations that integrate information security objectives into daily work spend 50 percent less time remediating security issues than low performers. This has a positive impact on IT performance and security outcomes.

Any organization, irrespective of size, can employ DevOps to achieve high performance levels and meet business goals while cutting down on costs related to fixes and remediation security issues.

How DevOps Transforms Teams

Consider this scenario: In an organization that does not apply the DevOps methodology, the development team writes code and develops a product, which is then sent to the testing team that verifies if the code meets the requirements. The operations team steps in to run the product only after this process is complete. The DevOps philosophy was created to break the silos that teams operate in and effect better collaboration and performance.

The culture shift DevOps brings about in organizations is substantial, and since culture is a key factor responsible for a company’s performance, application of the DevOps philosophy provides measurable and non-measurable results. A few of them are:

Employee Engagement

DevOps is neither a method nor a tool, it’s a culture! In organizations that observe the DevOps culture, the development and operations teams work outside silos, releasing products faster, thereby increasing operational efficiency. This is made possible by the operations team giving the developers constant feedback about the code and troubleshooting problems with the user experience in mind. This continuous learning and delivery practice allows cross-functional teams collaborate as one team with one goal, thus facilitating higher performance levels. The improved organizational culture and enhanced employee engagement can help organizations not only in retaining great talent but also in recruitment.

Organizational Success

The “2016 State of DevOps Report” effectively explores the relationship between employee engagement and organizational success and goes on to show that there is a strong correlation between these two factors. The report states that employees in high-performing organizations are more than twice as likely to recommend their organization to a friend as a great place to work.

Organizations can greatly benefit from adopting the DevOps culture. The “2016 State of DevOps Report” shows that high-performing organizations are widening their lead against lower-performing organizations. This can be attributed to the increased productivity and creativity of highly engaged employees that helps them provide better experiences to customers.

Organizations can achieve great business results by reinvesting the human time, creativity and energy saved by reducing rework and downtime through DevOps. Herein lies the ROI of DevOps—the competitive advantage that organizations get by investing in a philosophy that positively affects bottom-line results!

About the Author/ Pallavi Poojary

Pallavi Poojaryis a writer with Edureka, an online learning company with more than 250,000 learners spread across the globe. She writes on hot technologies and skills that are in demand today, including DevOps, Big Data and Project Management, to name a few. She is a believer in transformation of careers through learning

August 11, 2016