devopsy.kinsta.cloud

Tag: agile

5 Ways Continuous Integration Reduces Risk in Development

***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

There’s a lot of talk about DevOps and Continuous Integration in the tech space right now, but sometimes the implementation of those methods can be tricky.

An effective Continuous Integration strategy requires DevOps teams to manage risks in software development with a range of practices.

Here are five processes you can use to make Continuous Integration work for you.

Automated and Self-Testable Builds

DevOps professionals can use separate machines for software integration to eliminate the tight coupling between the build process and Integrated Development Environment (IDE) that leads to last-minute integration concerns.

Everything needed to build the software should be contained in a separate version control repository.

Dedicated Continuous Integration servers and open source build automation tools can run these builds through continuous testing processes and deploy the product in the required environments.

Automated deployment further eliminates the bottleneck of waiting for other team members to deploy testable versions of the latest software.

Synchronize Version Control Repository

Continuous integration is only possible as long as Dev, Ops, QA and the wider IT team is tightly integrated, collaborative and supportive of each other’s efforts.

From a tooling perspective, effective collaboration requires a separate-but-unified master database updating changes in real-time.

All database artifacts — including scripts, procedures and other assets should be placed in this version control repository.

DevOps teams can enable Continuous Database Integration by using automated database integration, local database sandbox and version control repository to share database assets, as well as empowering developers with capabilities to modify the database and making DBAs a part of the Dev team.

Build a Stronger Foundation and Expand Test Coverage

Limiting test cases to individual assets executed at every check-in to the version control repository ensures the continuously integrated builds are defect-free in combined form.

Tests should be repeatable and performed continuously in virtualized systems replicating unavailable testing environments.

Continuous Integration methodologies allow you to determine how much of the code was actually tested without much manual intervention.

Although some code coverage tools can assess the percentage of test coverage by package or class, Continuous Integration systems ensure this coverage is updated in real-time.

For instance, DevOps teams can run these tools as part of the Continuous Integration server build script every time the version control repository is updated with a code change.

With this information, untested code would never get through to the next stage and the latest test will be conducted across all areas as required.

Enhance Project Visibility

With automated build testing and deployment using a centralized code repository, build feedback mechanisms update Devs and Ops with the latest test results and the necessary progress information.

In cases when the same level of automation for complex builds seems impractical, DevOps teams using manual procedures must establish a feedback loop to bring everyone on the same page with real-time communication and notification of accurate performance and results.

These capabilities ensure the same high standard of reporting is maintained for manual testing practices as available for automated build testing systems.

Follow Consistent Dev Standards

Quality code forms from the ground up, with adherence to common code and architectural standards established for an organization’s specific requirements.

Automated inspection tools enable Devs working on short and quick builds to maintain their focus on quality from the very foundation of the software itself.

DevOps teams should assess every integration build for potential defects that surface after integrating several iterative builds but should be taken care of far earlier during the development stages.

In Conclusion

The right toolset is key to Continuous Integration, as is the cultural change that facilitates this DevOps practice.

Continuous Testing presents several areas of improvement for DevOps teams to develop iterative builds in individual, isolated branches and then merge the latest builds in a unified, single and well-synchronized Master branch.

The idea is to ensure all risks are mitigated beforehand so DevOps professionals are not bogged down with endless needs to rework the product — and that gives them time to focus on true Continuous Integration.

June 9, 2015
Continuous Testing: What exactly is it?
***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

The aptly named DevOps practice of Continuous Testing synchronizes Testing/QA with Dev and Ops processes optimized to achieve business and development goals.

A few years ago when Agile was all the rage, lightning-fast Dev sprints left the Ops and QA personnel playing catch-up due to inadequate and sluggish testing practices with minimal coverage across frequent builds.

The widespread culture of Agile development has accelerated development while software testing still lags behind, forcing organizations to cut corners in QA or slow down Dev processes entirely.

Instead of speeding up the conveyor belt beyond control or slowing down development to kill efficiencies, Continuous Testing in DevOps encourages a systematic approach toward process improvement.

Continuous Testing goes beyond automation and encompasses all practices — including tooling and cultural change — that help mitigate risks before progressing to subsequent Software Development Lifecycle stages.

Continuous testing and Automation

The approach to Continuous Testing can vary and follow diverse pathways to ensure the best user experience is delivered, free of defects.

It’s barely viable to repeat all tests every time a new feature is added, so the Continuous Testing strategy fosters a company-wide cultural change to achieve four capabilities: Test early, test faster, test often and automate.

End-to-end test automation practices are intended to integrate QA into existing fast-paced Dev and Ops processes as a means to create continuity while maintaining faster development cycles.

It’s important to understand the role of automation in Continuous Testing. Automation alone does not enable continuity in testing, but it helps provide a qualitative assessment of risk and practice actionable tasks to mitigate these risks throughout the SDLC.

In essence, automated testing constitutes the detection process for software issues and defect prevention, whereas Continuous Testing addresses the wider challenge of improving the effectiveness of these detection sensors.

What does Continuous Testing Look Like

This example demonstrates how collaboration, automation and tooling enable end-to-end testing that’s performed faster, more often and better aligned with Agile Dev and Ops practices.

Continuous Testing starts within development processes as Devs use open source tools such as Selenium to test the functionality of their code.

Tools such as GitHub or other internal repositories can then be used to store tests and version together with the software code. DevOps teams can use the same tests to integrate frequent builds of the software with the Load Impact API.

Once the code reaches preproduction, dedicated QA teams can use these tests by modifying dynamic test parameters as appropriate. In the production environment, Ops can reuse these tests for acceptance testing and ongoing post production monitoring.

Once the application goes live, integration with existing APM tools help keep tabs on server-side app metrics.

Continuous Testing in this example is broken up into at least four environments: Development environment, Continuous Integration environment, QA Pre-production and Performance testing environment, and the Staging environment. Additional post-live monitoring further enable end-to-end continuous testing.

Key Elements of Continuous Testing

Parasoft, a global software solutions company, is one of the great thought leaders in DevOps and Continuous Testing. The company has laid out some key elements of the process in a fantastic e-book, and they’re definitely worth sharing here:

Risk Assessment: Covers risk mitigation tasks, technical debt, quality assessment and test coverage optimization to ensure the build is ready to progress toward next SDLC stage.

Policy Analysis: Ensure all processes align with the organization’s evolving business and compliance demands. Primary objectives include:
- Identifying trends associated with injection of dangerous patterns within the code
- Augmenting defect prevention practices for high risk areas
- Isolating risk in target areas
Requirements Traceability: Ensure true requirements are met and rework is not required. An object assessment to identify which requirements are at risk, working as expected or require further validation.

Advanced Analysis: Using automation in areas such as static code analysis, change impact analysis and scope assessment/prioritization to prevent defects in the first place and accomplishing more within each iteration.

Test Optimization: Ensure tests yield accurate outcomes and provide actionable findings. Aspects include Test Data Management, Test Optimization Management and Test Maintenance.

Service Virtualization: Early, frequent and everywhere testing requires access to real-world testing environments. Service visualization enables access to the virtual form of the required testing stages, cutting the waste time to test environment setup and availability.

An effective Continuous Testing strategy incorporates all of these elements and forces an organization-wide cultural change to synchronize Dev, Ops and QA/Testing as part of the true DevOps philosophy.
May 27, 2015
Surprise! Broad Agreement on the Definition of DevOps
For the past few years, the conventional wisdom has been that DevOps has no agreed upon definition. Instead, everyone has their own interpretation. However, the conventional wisdom is wrong.

While everyone supplies slightly different words, the same working understanding of DevOps is nearly universal. Our understanding of DevOps goes back to a handful of common touch-points. There is the “CAMS” (Culture, Automation, Measurement and Sharing) acronym popularized by John Willis and Damon Edwards. CAMS has been extended repeatedly – Eveline Oehrlich of Forrester is now discussing CALMSS. Gene Kim offers his “3 Ways” and Patrick Debois has codified “4 Areas”. Allspaw and Hammond had 10 deploys a day, and Etsy made technologists fascinated by a virtual crafts bazaar.

Analysts, authors, industry, and community leaders all agree. The ideas are big and mutually supportive. In short:
- DevOps exists to help the business win
- The scope is broad, but centered on IT
- The foundations are found in Agile and Lean.
- Culture is very important
- Feedback is fuel for innovation
- Automation helps
Why: To help the business win

The most popular DevOps book today is The Phoenix Project, with a subtitle “A Novel about IT, DevOps and Helping Your Business Win”. DevOps is not a navel gazing IT methodology for making IT better for IT. It is focused on how stuff involving the technologists can deliver better results for the business.

As Patrick Debois writes in his excellent post,”DevOps Areas – Codifying devops practices“:

As rightly pointed out by Damon Edwards, devops is not about a technology, devops is about a business problem. The theory of Constraints tells us to optimize the whole and not the individual ‘silos’. For me that whole is the business to customer problem, or in lean speaks, the whole value chain.

At IBM, our definition of DevOps calls out enabling the business to “seize market opportunities”.

Today, technology is critically important to how a wide variety of companies compete. Retailers who neglected e-commerce have gone out of business. Logistics companies like UPS find a competitive edge by creating software to shave miles off delivery routes. In the automotive space, Ford is at tech conferences recruiting companies to write apps for its cars and Tesla’s first fix for their flammable battery problem was an over the air software update.

DevOps is a response to what the IT is experiencing – from a back-office concern that needed to “keep the lights on” to key part in how the business competes every day.

Who: A Broad Scope, beyond development and operations.

Yes, DevOps is a portmanteau that brings development and operations together. Yes, the classic DevOps concern is the Wall of Confusion between development and operations. However, with the focus on business value, the scope must include everyone who is involved in taking an idea through IT to business value.

In recent years, there’s been something of a cottage industry of community leaders and analysts writing, “No, no it really should be Dev_____Ops” where the blank is filled in by their own specialty. Examples: DevQAOps, DevOpsSec, DevSecOps, BizDevOps and of course Bussdevtestqanetsecnetops.

When a Finance department must approve the provisioning of each new test environment, slowing the delivery of new applications by weeks or months, there is a DevOps problem that leaves narrow confines of development and operations. It leaves IT. Patrick’s quote above talks about optimizing the whole value chain. In that post he describes DevOps as being about “collaboration, optimization across the whole organization. Even beyond IT (HR, Finance…) and company borders (Suppliers)”.

Based On: Agile and Lean

In answering why he didn’t support a manifesto for DevOps, Patrick Debois points people back to the Agile manifesto and asks them to reread it more broadly. Agile’s shift towards a faster delivery of software, and focus on the end goals make it a natural spiritual ancestor for DevOps. However, from the perspective of Ops teams, Agile represented the barbarians at the gates. Agile generally took root in development side of the house, enabling it to create something the business wanted faster. But with operations still moving at a pace suited to waterfall, pressures grew and the need for something like DevOps became clear.

Where much of DevOps energy seems to come from its Agile roots, the intellectual side of the movement is more rooted in Lean. It seems no accident that many of the Phoenix Project’s lessons are taught away from the cube farm and instead from the factory floor. While we can intuit that faster feedback is better, and the “continuous” in CI and CD is convenient because it lets us map feedback to fewer changes, it is in the Lean literature where we encounter Little’s Law. It provides the queue theory mathematics supporting the idea that smaller batch sizes supports shorter cycle time. And so Reinertsen’s “The Principals of Product Development Flow: 2nd Generation Lean Production Development” has become popular in DevOps circles and is well cited by Jez Humble in his new Lean Enterprise book.

Culture: Collaboration and Experimentation

The Agile and Lean roots show through in the DevOps culture as well. Lean and Agile are both focused on people first, systems second and heavily cross-train or work in cross-functional teams. They also emphasize continual improvement (kaizen) through approaches like retrospectives.

In DevOps circles, collaboration is preached as an antidote to thinking in siloes which has resulted in antagonism between development and operations. For the systems thinking required to deliver for the business, teamwork is required. Developers make better decisions when they take advantage of operations knowledge (Patrick’s Area 3) and operations does better when application knowledge is embedded with them (Area 4).

The other cultural emphasis is a push towards experimentation, constant learning and improvement. This is Gene Kim’s third way of DevOps. The Lean Startup ideas of Eric Reis permeate the DevOps world. Intuit has made great strides in showing how these approaches work in established companies by running experiments in the midst of tax season resulting in improved business outcomes.

These cultural trends come together in things like blameless postmortems discussed by Etsy and Jake Loomis in Web Operations. When things go wrong, organizations need to understand why and take corrective action. In troubled corporate cultures, the messenger reporting the problem might be shot, and the hunt to assign blame for the problem will begin. Healthy organizations look to learn from failure and try to improve the systems that allowed (or guided) people to make a series of mistakes that lead to the failure.

Feedback: Yes. Lots please!

Experimentation, adaptation and learning tend to do better when there is ample feedback. Otherwise, there is too little to learn from. Developers tend to be more productive as well with rapid feedback. Their thirst to know whether the latest change diminished quality led to the practice of continuous integration and the tools that support CI and continuous delivery. That CD ends up facilitating faster delivery of value to the customer is something of a happy accident.

Amplifying feedback loops is Gene’s “second way“. The “CAMS” concepts of measurement and sharing point to feedback. You gather lots of information, and make sure people see it so they have an opportunity to learn from it. And when Elisabeth Hendrickson presents at DevOps Enterprise 2014, as Cloud Foundry’s Director of Quality Engineering at Pivotal, she describes her job as “to see to the care and feeding of the feedback cycles“.

As expected with a collaborative culture with ample sharing, feedback must cross the traditional silo lines. That means the awesome production monitoring that had lived in Ops land, is radiated back to development. This is “Area 2” in Patrick Debois codification.

Automation: Of course.

The “A” in CAMS is automation. Patrick’s “Area 1” points to continuous delivery techniques, well described by Dave Farley and Jez Humble in their book. A Lean initiative looking at IT will find waste in wait times, errors, and more and move towards automation as a natural result. Feedback craves automated deployment of changes, and automated tests or production monitoring that validates them. Automation is what makes a lot of DevOps possible. It doesn’t make cooperation between development and operations happen, but tensions are reduced when Ops gets out of the business of arguing about the SLA they have with Dev around server provisioning and instead provide Dev a button to press.

DevOps isn’t about automation. Automation is just the natural result of DevOps.

Additional Resources not cited above

DevOps for Dummies – by Sanjeev Sharma

8 Best Practices for DevOps – Ashok Reddy’s overview of the Forrester Report

About the Author/ Eric Minick

Eric Minick is a Technical Evangelist. He joined IBM through its acquisition of UrbanCode. He has spent the last ten years helping organizations – large and small – adopt continuous integration, delivery and now –DevOps. He is a frequent speaker on the topic, and co-authored Application Release and Deployment For Dummies.

Prior to consulting, Eric was a developer, test automation engineer and support engineer, and has contributed to multiple generations of UrbanCode products. Follow him on Twitter at @ericminick.
May 13, 2015
Documenting DevOps: Agile, Automation and Continuous Documentation
Chris Riley former GigaOm analyst, O’Reilly author, devopsy.kinsta.cloud featured writer and founder of Fixate I/O has written an outstanding report on the key role documentation must play in the modern software development lifecyle. Agile or DevOps, automated or not, we live in a continuous IT era. So why shouldn’t documentation be continuous as well? Riley’s analysis includes key findings including:
- Even the definition of documentation is subjective. Despite several decades of implementation, the methods, drivers, and even language of documentation are nearly limitless. Standards such as ITSM help stabilize the look, feel, and process of documentation, but it does not help an organization understand how to use that documentation and integrate it into development.
- The modern software delivery pipeline introduces new complexities in administration. New tooling in the DevOps space has improved application efficiency and quality, but in some cases, this has also reduced the manageability of the environment. This creates new challenges to knowing what you have, how it functions, and how to extend it into the future.
- As the delivery pipeline moves to a continuous stream of releases, classic documentation processes become impossible. Documentation must happen automatically within the process, or not at all.
- In order for documentation to be useful, it needs to be actionable.
Download the complete report on continuous documentation for an in depth analysis, conclusions and guidance:
Documenting DevOps
Name
This field is for validation purposes and should be left unchanged.
Name*
First Last
Company
Email*
May 13, 2015
DevOps Teams: Containers Will Help You Become Future Ready

Container frameworks are emerging as a superior choice to create, manage, and run distributed applications in a scalable and secure fashion. Docker’s growth is testimony to the fact. There are 100 million downloads of the Docker Engine, 75,000 Dockerized applications, 770 contributors, and 50,000 third-party projects on GitHub using Docker in less than two years.

The State of Containers 2015 Docker Adoption Survey found that more than 70 percent of respondents are either using Docker or evaluating for future use. Given Docker’s overwhelming popularity, here are three trends that DevOps teams should prepare for:

–THIS STORY HAS BEEN MOVED TO CONTAINER JOURNAL–

About the Author / Varma Kunaparaju

Varma Kunaparaju is CTO at Vistara and a seasoned technologist with over 20 years of experience leading engineering teams to deliver enterprise software. Reach him on Twitter @varma1

May 8, 2015
The Business Value of Continuous Testing

***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

Enterprise software development models have presented many innovative solutions for IT to align development efforts with business goals.

Adopting new software development lifecycle methodologies is as much a business decision as an IT decision. From an IT perspective, this means the ability to produce profitable products at lower cost without compromising quality.

Maintaining quality through the code changes to add features and push new releases has been a struggle until the inception of the DevOps-driven Continuous Testing model, which transforms costly, time-consuming IT practices into profit centers.

The DevOps Patch to Agile Pitfall

Having already adopted lean Dev and Ops strategies, Agile enterprise IT seems to have unlocked unprecedented efficiency and productivity by delivering better business value at a rapid pace.

But in all the mayhem and chaos in jumping on the bandwagon of the Agile development, IT teams ended up stockpiling a mass of false assumptions.

Simply implementing Agile processes and iteratively adding functionalities doesn’t necessarily streamline development.

With this approach, teams ultimately ignored the true Agile philosophy of adapting, collaborating and planning, and ended up following an accelerated and reinvented version of Waterfall in hopes to develop fast, deliver early and manage issues effectively.

A change was needed somewhere between design and development, and release and production phases where IT teams were failing to identify software defects and perform fixes fast enough.

The lack of early, adequate, parallel and continuous testing was transforming IT into a cost center by drawing unnecessary expenses and time investments in reworking software to meet desired quality standards.

The DevOps approach aims to address this challenge by facilitating continuous testing as an ongoing effort to identify and fix software defects early and deliver the right quality at the right time at the right cost.

Continuous testing presents the following value propositions to raise business bottom line and open doors to unprecedented revenue opportunities:

Informed Decision Making

Business expectations about the reliability and speed of application delivery vary throughout the SDLC.

Continuous Testing creates a centralized system identifying failures and defects that present business risk to the organization.

Continuous Testing as a consistent effort provides adequate visibility and useful, real-time data. This can be a huge advantage to businesses, especially when it comes to application performance monitoring and testing.

The assessment of stability and validity of software builds enables Ops and business decision-makers to trade off decisions between risk and time of release, cost investments, functionality, scope and quality.

Business Agility and Competitive Differentiation

Enterprises across all industries have development shops delivering software applications as core components of their service offerings.

These departments, often driven by DevOps, must focus on churning out new, stable and high-quality software and race to deliver a stream of features that complement primary service offerings.

For instance, mobile check deposit was a leading driver for bank selection a few years ago and financial institutions offering this online service (while doing it well) early gained valuable competitive differentiation.

In the DevOps world, Continuous Testing creates a safety net for DevOps teams to develop, test and release new features to the market faster without compromising quality. This safety net works as a trusted testing environment, suite and tool to assess the impact of code changes accurately and quickly.

With these capabilities, organizations gain the agility to bring new products to market and compete on innovation and not on the scale of business operations and investments.

All in all, great DevOps levels the playing field between the eight-person startup and the massive enterprise competing in the same space. Who releases faster with a better product is more dependant on the process, not the funding.

Stronger Business Performance

IT performance is business performance. It’s measured in terms of the quality, productivity and functionality of IT solutions delivered for internal business operations or as marketable products.

The primary objective of Continuous Testing is to ensure high quality software products are delivered on time, cost and scope. To achieve these goals, IT teams need capabilities such as automation-driven, self-testing builds in a clone of production environments.

QA is expected to leverage frameworks and automation tools that allow testing early during the lifecycle. That shrinks time spent on test setup and maintenance, using service virtualization to replicate unavailable testing environments and establishing a comprehensive quality validation process with continuous integration systems.

These practices work as core elements of Continuous Testing to test faster, sooner, more often and continuously, thereby boosting IT and business performance.

These business values not only generate unprecedented new revenue streams but also unlock cash trapped within the business IT.

The resulting increase in revenue encourages organizations to change their game entirely and pursue new DevOps-driven business models in improving service delivery, scaling operations immediately and competing on fronts of business innovation.

May 5, 2015
Overcoming Siebel Complexity to Achieve Agile and Continuous Deployments
Siebel releases are complex. Executing one well is difficult. Extracting databases, migrating SRF repositories, deploying non-repository items and managing all the planned and unplanned outages is no small feat. Oh yeah, and there’s still that little thing called production. These deployments can easily take days and are updated infrequently, because the business can’t afford any downtime.

Now there is a solution to meet the business needs – Automating your deployments with Application Release Automation.

Join us on April 14 to learn from Anne Kootstra, Siebel CRM Consultant at Accenture, how they used Application Release Automation to shorten downtime of Siebel and increase quality and stability at a huge telecom provider in the Netherlands.

Attend this complimentary webinar to learn how to:
- Enable agile releases and deploy Siebel updates in just minutes
- Reduce planned and unplanned downtime
- Deploy the right artifacts and configuration settings for even the most complex environments
- Prevent unauthorized access to production environments
- Free up Siebel administrators and developers from release and deployment tasks
Recording

Your Host:

Alan Shimel, Editor-in-Chief of devopsy.kinsta.cloud. An often-cited personality in the security and technology community and a sought-after speaker at industry and government events. Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.

Panelists:

Ralf Paschen is currently running the solutions marketing department at Automic Software. Over the past years he has held a various number of positions in the software industry. Coming from a technical datacenter job role he moved into presales, sales to product management and product marketing in several software companies. He was responsible for the complete SAP portfolio for Automic and is now building the solutions marketing department to give customers and partners a clear understanding about the innovation value of the unique One Automation Platform. You can connect with Ralf on LinkedIn: Ralf Paschen or Twitter: @RalfPaschen.

Anne Kootstra is with Accenture since 9.5 years and as Senior Siebel consultant responsible for huge customer implementations of Siebel CRM. He is part of the Accenture Netherlands delivery center and specialized in Siebel development and delivery.
March 31, 2015
DevOps at REA Group

“DevOps is 100 years old, and probably invented by a British engineer in the 1920s,” according to Nigel Dalton, CIO at REA Group, the company behind realestate.com.au and other leading real estate sites in Australia, Hong Kong and Europe.

He was referring to the work of Frank Woollard at Morris Motors in the UK. In 1925, Woollard produced a paper titled Some Notes on British Methods of Continuous Production in which he wrote “The ideal of continuous flow must be present from the design and raw material stages up to and even beyond the sales stage.” (This paper can be found in Bob Emiliani’s annotated reprint of Woollard’s Principles of Mass and Flow Production.)

“There’s nothing new under the sun with DevOps — we’re doing the same thing [as Woollard],” said Dalton. “We’re reproducing practices that are 100 years old.”

They might be even older, he said, pointing to research suggesting China’s Terracotta Warriors were made using flow manufacturing around 200BC.

While the principles are old, “the cutting edge today is DevOps.”

REA’s DevOps journey began around three years ago when the company’s applications and website covering commercial and residential sales, rentals and shared accommodation were being moved to a new Java-based platform. It was “an epic migration” involving a large team from REA and consultancy ThoughtWorks, and one that triggered consideration of how to do things even better.

People like REA’s Trent Hornibrook (then infrastructure lead, but now delivery lead) realised greater changes were needed, Dalton said. “I doubt that bosses could tell people to do DevOps — they’d find a million reasons to avoid it. It’s a practitioner, grass-roots thing.”

Hornibrook likened the situation that results from separate development and operations groups to working on a car production line and finding that a door doesn’t fit the body. Production grinds to a halt, but that could have been avoided with better collaboration further back in the process, so it’s largely about making the feedback cycle quicker.

The idea of a monthly development cycle where operations signs off at the end of the month “couldn’t go on in a world where things were moving faster,” said Dalton.

The company was still organised in functional groups: IT development, IT operations, marketing, sales, and so on. But as it moved to customer-segmented operations, these divisions were no longer appropriate.

So REA reorganised into business units covering its commercial, residential and property developer customers. “We got rid of IT operations, and put operations people into the teams,” he said.

The only centralised IT development workers are the global infrastructure and architecture team of around 12 people who build tools and measurement technology, manage the company’s use of Amazon Web Services, and provide consulting services. “They’re not high and mighty enterprise architects, they’re hands-on.”

The approximately 180 developers are part of the delivery teams that focus on the entire lifecycle of their applications, Hornibrook explained. “We’ve driven the autonomy and accountability down to individual teams,” and this has resulted in high levels of staff engagement.

Scaling IT is difficult, he observed, so it is necessary to provide that autonomy. While it does mean some work is duplicated, that’s a worthwhile trade-off for the increased throughput.

Dalton agreed, pointing out that the real estate business is changing rapidly, and REA’s ability to match that speed makes the organisation more resilient.

“The business benefits of adopting DevOps are significant,” said Dalton. The lean approach focuses on value, flow and waste, and DevOps improves flow and reduces waste. It is not logical to block flow by addressing operational issues at the end of a cycle — it’s much better to embed that function along the development process.

Another factor that allows IT at REA to keep pace with the business is that “we don’t have vendor dependency.” Rather than being locked in to the development cycles, the company uses a lot of open source software. “The tooling around DevOps is incredible… I reckon we use 50 tools that I know about across the teams,” Dalton said.

Similarly, REA employees are actively involved in Meetups, DevOps Days and other community events. In addition to gaining and sharing knowledge, this helps make REA an employer of choice, said Hornibrook.

The introduction of DevOps does require attention to various organisational issues, including governance. For example, the idea of separation of duties as a defence against certain kinds of malpractice goes out the window. “It’s a good concept, but an old concept,” Dalton said, and one that can be largely replaced by peer checking. He suggested it was much easier to make ‘unauthorised’ changes to code 20 years ago when it was commonplace for developers to work independently on a routine or module, but greater peer involvement and teamwork has made a big difference

“It’s all about outcomes,” he said. “We have controls.”

The adoption of DevOps does mean “the guardians of compliance have to learn more skills,” Dalton said. Tools such as GitHub record the changes that are made, but auditors need to learn how to use them. “All the evidence is there, they just have to find it.”

“Think about security at the beginning of the process and it’ll probably be better,” he added.

And when it comes to the functionality and quality of the software developed, accountability is an important motivator: if a particular component breaks, then the people who caused the problem will be the ones that have to fix it.

“DevOps is human behaviour and collaboration, yet it’s deeply technical,” said Dalton. REA’s biggest problem is finding people who will work collaboratively and take responsibility. Also — and how many times have we heard this about IT staff? — “Communication skills are really vital.”

Another important attribute that Hornibrook looks for is malleability, because “change is so confronting.”

“A focus on individuals within a work culture and mindset is a key element that prevents the adoption of DevOps,” he said. An important first step for someone in a role like his is to shepherd employees into a shared vision and purpose.

That, suggested Dalton, is helped by the way REA does not have a highfalutin mission statement, just a simple raison d’être: ‘Our purpose is to empower people by making property simple, efficient and stress-free.’

There’s also a benefit from one of the most basic Agile tools — cards on walls. This turns a binary ‘you versus me’ situation such as one person asking another whether a certain change has been made correctly into a ‘you and me versus the cards’ situation, and that’s a very different dynamic, according to Dalton.

The deeper collaboration is spreading from within IT into other functions such as marketing and sales. Dalton calls this ‘product DevOps,’ and added “we can lead that if we get our own house in order.” It helps that REA has already adopted Agile principles across several functional areas, including the call centre and the legal team.

“That’s how this organisation succeeds at pace and with flow,” he said. “Learn, learn, learn, with a relentless focus on the customer.”

That customer focus addresses the question of whether REA is building the right products, Hornibrook pointed out. Dalton agreed, observing “with DevOps and Agile, we can make the wrong thing really fast, but that’s not helpful.”

March 31, 2015
Service virtualization – The hows and the whys.

Close your eyes and take a deep breath. You are on an island, smelling the iodine coming from the sea, the sun is gently caressing your cheeks and you are sipping on a cocktail in the middle of the afternoon. Your phone is off, in a place you forgot and you do not have direct access to the internet. You are a sysdamin and for the first time since the beginning of your career, you take a break, an actual break to reload your batteries and you are not even thinking about work because you KNOW. You know it is all going to be alright, as it has been since you implemented service virtualization in a really DevOps oriented way.

What is service virtualization?

First of all, let me say that service virtualization is not something new but many are still unaware of what it actually is and only a few good products are available.

You can find a really good explanation of what it is in this post from Chris Riley: https://devopsy.kinsta.cloud/blogs/service-virtualization-window-advanced-devops/ but I will try to sum it up.

In order to avoid confusion, it is also necessary to briefly explain mocks and stubs. Mocks are usually programmers written tests that check the behaviour of a specific piece of code like which methods are called, in which order and if sub processes were created. Stubs on the other hand check that for any given input, the expected result is outed. I often refer to them as logic tests and I/O tests.

Service virtualization has nothing to do with mocks or stubs. It mimics the entire behaviour of an application as if it was an actual one deployed on a server and connected to a database, backend etc. In order to enable this, data will be recorded from QA (and anonymized) and the whole behaviour of the application will be recorded. Going through security layers, calls to a database, updating some values, even timings can be recorded for performance testing. Absolutely every single operation is stored and if the test data is well chosen at the time of recording, all cases can be covered.

For those of you who are into gaming, think of it as an emulator. You do not need a SNES, Genesis or Neo Geo to play your favourite oldies, instead, all you need is an emulator software and the ROMS. The emulator will behave like a gaming console on your computer, reproducing inputs and outputs with high fidelity and translating CPU instructions into different ones your computer can understand.

If you are not into gaming, and that is a shame if you ask me, think about virtual machines. When you install a Fedora in a confined environment on top of your favourite edition of Windows (or Mac OS), what you do is pretty similar to service virtualization as well. Your Guest OS will send instructions to the different components of your computer which the virtual machine will catch, translate into commands your Host OS can understand and send to the concerned components. It sounds complicated but it works like a charm.

What it can do for you.

As it operates on local servers or even better, on containers (check out Docker if you have not done so yet), it is virtually always on. Downtime is a thing from the past and updates can be scheduled and performed when people are not at work. How much of your project you want to cover is up to you and your needs but once you have started the process, why not go all the automated way? If you do, that means that your programmers will have access 100 percent of the time to environments that are up, running and that reflect actual services.

What is the point of having test environments then you might ask. And here is the answer you can give stakeholders: none. You can achieve good environment resilience, actual data tests and higher-than-ever response time through service virtualization. If well applied, and I really want to emphasize on that, service virtualization makes good ol’ test-environments obsolete.

Once again here, the benefits mentionned by Chris Riley will speak for themselves: https://devopsy.kinsta.cloud/blogs/service-virtualization-window-advanced-devops/

Taking DevOps one step further.

Like all great things, service virtualization comes at a cost. In order to fine tune it to your requirements, it is going to take great efforts. You have to decide which services, ideally all of them, you will provide your programmers, which data sets are representatives of a functional application and of course, install and configure it all. The biggest benefit that you can identify here already is that developers, testers and operations have to be involved and work in a close collaboration. This is your opportunity to make different teams understand what others do. Get them all together, make them work together in a form they are not necessarily familiar with: cross-teams brainstorming.

Brainstorming, if well done, can be a very powerful tool to come up with creative solutions to problems that were not identified as such beforehand. It has to be directed though. To brainstorm doesn’t mean talking nonsense about non topic related things. Having a neutral person in the room who focuses the discussion around a few pre-established points and who keeps track of time is the way to go. Scrum masters are generally good at this since they have an overview of the project and its problems and can quickly identify the right persons for a task.

Getting developers to update their own service virtualization server with the latest data from QA means better communication with testers. Operations have a critical role here too since they setup the environments and give the how-to to the different teams.

Another glass sir?

Once your teams work together on an almost 24/7 available server and understand how it works, all you have to do as a sysadmin, is to relax a bit for once and focus your brain and time on how to improve processes even more, enhance the quality of deliverables and tighten up the now-existing bonds with your colleagues. Oh yeah, and if you’re on holidays, stop worrying about that potential catastrophic phone call, just worry about yours and yourself.

March 20, 2015
The right answer was “process”
The question is “why is there no one-size-fits-all automation and orchestration solution”?

Automation, a component critical for continuous deployment particularly as it extends beyond app operations and into the broader environment to encompass network and security operations, too, continues to be a custom build kind of thing. There’s no single automation tool, framework or pattern that can be generally applied to a significant number of environments. Mi casa es no su casa, after all. That’s because environments don’t actually look like power point slides; your environment is not the simplistically diagrammed firewall –> load balancer –> web server typically found in decks extolling the virtue of one solution over another. They are far more complex, comprising not just multiple tiers of interdependent and interconnected apps and services, but multiple chains of infrastructure services spanning the gamut of security, availability, mobility, performance and access control. The specific flow through and between these services is unique. Indeed, the intimate relationship between network latency and app performance is well known, and thus the navigation through the critical yet complex site of services required to deliver an application could easily be considered a competitive advantage if it’s able to keep performance under the 5 second toleration limit on response time apparently now the standard for consumer satisfaction.

So it’s no surprise that automation and orchestration, responsible for automatically setting up the triggers and routes required for application traffic to correctly traverse the right set of services for optimal performance, would not be a simple, turn-key solution. One size does fit all. It must, like its business process focused counterpart – BPM – be customizable and easily changeable.

BPM? Are you seriously?

I’m dead seriously. BPM – business process management – has been around far longer than automation and orchestration of IT systems and its focus is, as its name implies, on process. Automation is included, but it is the orchestration of business processes that often change in the face of variable business conditions that makes it an invaluable tool for optimizing business and a source of competitive advantage. “The ability to change quickly is essential. Businesses change their key processes 4-7 times per year. The driver for change can be internal or external. New opportunities can arise. New partners or customers need you to support a different way of doing business. Federal or international regulations can require you to change your processes.” (Making the Case for BPM, Jim Rudden)

Consider for a moment this data from a 2012 Bearing Point survey on the goals and impact on culture:

Top BPM goals are increased efficiency (93%), increased transparency (90%) and the realization of standardization potentials (85%). A number of BPM goals have an impact on corporate culture, e.g. increased transparency (rank 2), increased customer orientation (rank 5), disclosure of weaknesses/ bottlenecks (rank 6) and the reduction of mistakes/ error rates (rank 9).

– Bearing Point Business Process Management Survey 2012

If many of these goals and impacts sound familiar to you, they should. Because what DevOps ultimately needs to focus on is exactly the same as the focus of BPM – process. In an application world, these processes require applications and services. IT must be able to support these process changes in the application and service infrastructure.

BPM (Business Process Management) has known this for years, and it is the orchestration of the process – the workflow – from task to task, from approver to doer that generates the efficiency gains achieved with that technology.

Indeed, adopters of DevOps who focus on the “big” picture would do well to consider the similarities to BPM which touts as benefits such things as:

Process quality improvements

Continuous process improvement

Reduces costs

Improves business agility

It would be difficult if presented with that list of benefits with no attribution to determine whether it was referring to DevOps or BPM. That’s because the operationalization of business through process orchestration is not all that different from the operationalization of IT through process orchestration.

And processes are not, nor should they be, turn key and one-size fits all. They should be tailored to address the needs of a specific operational or business goal, i.e. deploy an application or secure an order from a customer, based on the systems in use to achieve those goals. BPM shines where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. DevOps will shine where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. It isn’t about racing through a given task with automation. Speed isn’t efficiency and, if you do the math, a focus on automation without similar focus on process can lead to even greater inefficiency.

And that’s why there is no one size fits all automation and orchestration tooling; because the goal isn’t automating tasks, it’s orchestrating process. Coordinating workflows. Ferreting out inefficiencies.

Orchestration is the glue that binds automated tasks together. For example, recently a missed route update caused some unexpected down time during what was considered a routine* decommissioning of gear. A missed route update. In other words, someone forgot. That task might have been automated, it might have just needed to be kicked off. But it wasn’t.

That’s the kind of thing that would likely not happen if the process was governed by some form of orchestration; if there was a clear list of what needed to be updated or changed based on the removal of that particular component. If orchestration was involved to provide visibility and oversight of the operational processes needed to perform such a routine task. This does not mean the tasks should have been automated; in fact automation is not strictly required to achieve the visibility and oversight necessary because processes are about workflows and hand offs, not specific tasks completed during the steps that comprise a process.

Which leads us right back to Business Process Management. Take a deeper dive into the aforementioned Bearing Point survey and you’ll find more similarities with DevOps:

Crucial elements for sustainable BPM are clearly defined roles (95%), top management support (92%) and a clear mandate (92%)

Typical implementation barriers such as the lack of resources (53%), sponsorship (32%) and experience (25%) need to be identified early and addressed in a structured way

BPM as a continuous (=sustainable) approach is already adopted by 59% of companies, coordinated and implemented either centrally or locally

Involvement of stakeholders (90%), process oriented thinking (90%), organizational institutionalization (82%) and a well-communicated concept (75%) are elements of communication and change management which must accompany the establishment of sustainable BPM

That’s because both are ultimately about process, with automation being only one of the tools in the tool chest that support implementation of a continuous approach to enabling agility in the process layer. Indeed, about 38% of the companies in the survey use software to support their BPM initiatives while another 37% do not. A spreadsheet with the right Six Sigma calculations can be as powerful as any automation framework, toolset or API in identifying the dead space between tasks that introduce inefficiencies and delay application deployments.

BPM has, like DevOps, brought tangible, measurable results to the table. Bearing Point found that almost three quarters of the surveyed companies (72%) were able to achieve measurable benefits through BPM such as process lead time (72%), process cost (68%) and error rate improvements (67%).

So rather than focusing too much on automation, adopters and proponents of DevOps would do well to up-level the conversation and start looking at process orchestration and improvement as the goal. The successes BPM has had on the business side of the equation tell us that doing so will provide a better path to achieving success with DevOps in the operational side of the equation.

* I’d argue that in today’s environment of hyper-sensitivity to availability that nothing that touches a service or gear in the critical path of an application is “routine” but, hey, it is what it is.
March 12, 2015