devopsy.kinsta.cloud

Author: Stephen Withers

DevSecOps: Security Needn’t be Sacrificed for Speed

With the right practices, security needn’t be an impediment to continuous development and rapid improvement.

The same cycle keeps repeating, according to IT security and risk management consultancy Sense of Security co-founder and CTO Murray Goldschmidt. A new technology is hyped, leading to rapid adoption, but security is left as an afterthought. It happened with WiFi and voice over IP (VoIP), he says, and is currently happening with the internet of things (IoT). But it doesn’t have to be that way.

DevSecOps: DevOps + Security

DevOps is good for making things better, faster. But there tends to be a culture clash between those talking about speed, velocity or agility and those concerned with issues such as control points.

So Goldschmidt and his colleagues are showing their clients how security can be integrated into DevOps (DevSecOps) in an automated manner without affecting velocity.

Most people with development, operations or cloud backgrounds aren’t well-versed in security, he suggests, so Sense of Security shows clients how DevSecOps means security and DevOps can run in parallel.

“It’s not that complicated,” he says, but it’s something most people don’t think about.

Organizations often have a vulnerability management life cycle to identify and treat issues. After all, that’s part of various industry standards.

The typical approach is an annual penetration test to find issues, which can then be addressed.

A better way is to move to automated testing, and there are various commercial and open-source products to do that job. Goldschmidt says he sees little resistance to this idea.

The next step is to go from scheduling these vulnerability scans on a quarterly basis to weekly or daily—or even to run them continuously.

Citing the Struts vulnerability that caused a big problem for Equifax, he notes the challenge is to identify vulnerabilities in your systems before an attacker does. Continuous automated scanning means the window of vulnerability is very narrow.

This idea is simple enough, he says, but for some reason it doesn’t occur to most people until someone brings it to their attention.

When a scan reveals a vulnerability, its significance (severe, medium, etc.) is used to prioritize remediation.

“DevOps guys understand those triggers,” he observes: If you give them the data, they can take the necessary actions. For example, that might mean shutting down the servers in a small web farm one at a time, applying the patch, restarting and moving on to the next. They could all be patched and running by the end of the day.

Patching Isn’t Everything

However, “it is unfair to assume organizations can patch things quickly” every time, he says. For example, the Struts vulnerability required more than just a patch—the code running on it had to be recompiled and tested, and that process could take months in a large organization.

So organizations should not rely on the idea that patching is the fix for a vulnerability. While patches will be applied eventually, if you can make it difficult to actually take advantage of any exploits, the risk of data breaches is very low.

One very easy-to-implement example of these measures is using a stateful firewall to ensure inbound-only access to web servers. By disallowing outbound access—”the channel through which data is lost”—there is no path for data to leave.

Goldschmidt also recommends using tools to examine in-house code and the open-source and commercial code that is used with it. Such tools can detect most common problems, including openings for SQL injection, cross-site scripting and token misuse.

He describes these tools as being the equivalent of “spellcheck for developers,” and says it reduces the likelihood of putting insecure code into production.

The market includes products from Veracode (now part of CA), Sonatype and Qualys.

Qualys, he notes, provides differential reporting. This makes it obvious that something has changed, and “that is the key to automation.” Whether the system itself has changed or new information has been discovered about vulnerabilities or configuration issues, action is required.

Furthermore, the ease of making configuration changes that could have security implications makes continuous scanning even more important.

These checks should not be limited to individual modules. Rather, they should be repeated—preferably automatically—as modules are assembled to help developers avoid poor code and to check the overall product.

RASP Away Some Attacks

Runtime application self-protection (RASP) is another technology Goldschmidt advocates. The idea is to detect abnormal and possibly malicious activity. One example is that normal operation would see an application communicating with its application server, and the application server communicating with the database. If the application suddenly makes a direct request to the database—as happens in an SQL injection attack—action is clearly required.

Depending on the situation, RASP might raise an alert or immediately terminate the application, among other possibilities.

The use of these and other technologies and techniques should allow organizations to run at the pace implied by DevOps without sacrificing security.

— Stephen Withers

November 20, 2017
Case Study: DevOps Key to AWS Cloud Migration

DevOps practices halved the time it took this SaaS company to migrate to AWS

Australia-based Whispir provides communications applications as a service. One of its goals is to get products into users’ hands as soon as possible, and that requires development, operations and product staff working together, said Jonathan Swift, head of product and operations.

Whispir formerly operated from a co-location data center, a model that continued as it expanded internationally. But the time and effort needed to run the systems was not cost-effective for an organization of its size, so the company switched largely to AWS. Governance issues faced by some clients force the company to still run its own systems in certain geographical locations.

In the process, “[We] learned a lot about our app and how to make it cloud-ready,” said Swift. “Cloud isn’t free, cloud isn’t cheap” and is potentially more expensive than placing servers in co-lo data centers if the software isn’t built correctly.

For example, it makes sense to make the most use of a physical server as it is largely a fixed cost. But in the cloud, the fewer resources you use, the less it costs.

Crises Lead to Spikes

On the other hand, a lot of Whispir’s business is crisis-focused (eg, Caltex uses Whispir at its Kurnell oil refinery in Sydney to send emergency messages), and it takes a certain amount of time to spin up additional instances to deal with spikes in message volumes. “That was a real challenge,” said Swift.

Those crisis-communications clients expect Whispir to have sufficient resources to meet sudden demand. That’s an expensive proposition with co-located hardware, whereas Ccloud resources can be scaled up quickly, though not instantaneously.

Seasonality has to be taken into account. One relatively predictable driver of crisis messaging is Australia’s bushfire season. Unfortunately, this overlaps Christmas, which is a peak time for marketing messages. So Whispir plans to run at around 200 percent of its normal baseline capacity during the fire season. This is easily done with cloud infrastructure, which Swift describes as “a beautiful thing” in this context.

Some organizations are scared by multi-tenant environments. The ability to stand up dedicated instances on AWS is a great feature that Whispir can offer, especially to its clients in the financial sector.

The company’s DevOps practices had been focused on the co-location environment, so even though kanban still “works very well for us” after moving to AWS, some previous choices had to be revisited. In particular, new monitoring tools were needed to cope with the elasticity inherent in the cloud and because the focus expanded from the hardware to the whole stack.

Selected tools included the Zabbix open-source monitoring platform and Sumo Logic for trend analysis.

Earlier this year, Whispir moved all of its Australian customers into a “pure AWS” environment. There are some great tools for migration, Swift said, “but it’s not as straightforward or seamless as you might think.” He recommends doubling any initial estimate of the time required for such projects.

There were some nasty surprises: The first month’s bill from AWS was “huge” and it took a couple of months to determine the cause, even with the aid of AWS’s “really beneficial” premium support service.

Security requirements in various jurisdictions had to be taken into consideration, with updates to various procedures to ensure the new cloud environment was compliant. For instance, the procedures required when a physical system was decommissioned were well-understood, but how could they be translated into a cloud environment? One specific example is that an in-house storage device can be destroyed when it reaches the end of its life, but in the cloud it’s largely a matter of relying on the provider’s controls.

Avoid Lock-In

But Swift warned that “the challenge is to not lock yourself in” to a particular cloud provider. For Whispir, which operates a mixed cloud/on-premises environment (no cloud provider offers the right onshore facilities in New Zealand and the hardware that Whispir has installed in Singapore is too new to be replaced by IaaS), that is a requirement rather than a strategic choice.

Whispir would have avoided AWS-specific features that would make the platform easier to use and more difficult to get out of, despite the fact that using provider-specific toolsets can yield significant cost savings.

Whispir is keeping an eye on Azure—”great tools, great partnerships, great ecosystems,” observed Swift—and may migrate to Microsoft’s cloud in around 18 months.

A particular advantage is that Azure Stack can be run on private hardware, which could be useful if Whispir decided to enter a market where Microsoft Azure wasn’t present.

Either way, the intention is to go 100 percent cloud to the extent that is possible.

‘Infrastructure as Code’ Means More Power for Ops

The move to the cloud has made Whispir’s operations people more powerful, he observed. The infrastructure-as-code concept allows for rapid deployment, enables operations to contribute more to product development, reduces risk and enables the company to be more responsive to the market.

Furthermore, dealing with on-premises hardware still very much a waterfall situation, thanks to the delays and dependencies involved. But moving to IaaS means infrastructure issues sit comfortably within the kanban model.

IaaS also means that various network-related tasks simply disappear, but other issues arise in their place. For example, some external services used by Whispir (eg, SMS gateways) require the use of VPN connections from a known IP address, but with cloud infrastructure the best you may be able to do is specify a range of addresses. Whispir’s short-term fix was to route such connections through a specific server in the co-lo center, and in the longer term it found providers with less-restrictive arrangements.

Moving from co-lo to cloud would have been “incredibly hard” without DevOps, Swift said. It would have taken at least twice as long, he noted, and the company wouldn’t be able to run the mixed environment as well as it does.

“There’s no way we would have been so successful” without DevOps, he said.

— Stephen Withers

November 3, 2017
DevOps Proves No Coaster Ride for Roller
Keeping amusement parks running and the fun flowing is Roller‘s main job. The company provides software for the leisure and entertainment industries, notably theme parks, amusement parks and trampoline parks. Its core functions are ticketing, POS and CRM. Clients include Coney Island Luna Park, Aussie World and Twinlakes Park.

When Roller decided to expand and redevelop its operations in 2015, DevOps was the main attraction. CTO Andrew Brodie led the company’s push, having worked on Roller’s original implementation in his previous role as co-founder and technical director of digital product design studio Bravo!

Automation of Roller’s processes was a major goal. Even small organizations should adopt good practices, Brodie says, and that includes automating everything that can be automated (within reason) to the point of being able to click one button to deploy a system.

Before DevOps

In its early days, Roller used Microsoft tools including Web Deploy to deliver a monolithic application that met the requirements of a minimum viable product. Roller initially ran on Ninefold, a now-defunct Australian IaaS platform.

When the company attracted additional funding in 2015, it was able to expand its development team with people who had various skills including DevOps. Today, it comprises around 50 people—12 developers and various user experience, quality assurance and project management specialists.

The newfound expertise enabled the redevelopment of the system as a series of applications and microservices and eliminating most frequently performed manual operations.

Tools

Roller’s deployment process had become a bottleneck, so the company invested time in adopting the most appropriate tools. Several came from HashiCorp, including Packer (for creating machine images), Terraform (cross-provider infrastructure management), Consul (service discovery, configuration and orchestration) and Vault (centralized storage and control of distributed secrets such as keys).

After Ninefold’s demise, Roller became heavily involved with AWS, which “had everything we needed” in the way of APIs and toolsets “to code out our infrastructure,” Brodie says. But even though “we’re really happy with Amazon,” the company has avoided using exclusively AWS tools to retain the ability to move to other platforms such as Azure.

Implementing load balancing, auto scaling groups and so on as code took about a month’s effort. “It was a great move for us,” he says.

Other tools used by Roller include the Chocolatey package manager, and Microsoft’s PowerShell and its Desired State Configuration feature. After a machine image is tested, that’s exactly what moves into production, providing confidence that it will work as expected.

Roller uses blue-green deployment with autoscaling groups to make it easy to roll back if necessary. Deployment or rollback takes less than 10 minutes. When individual problems arise, Roller’s practice is to kill the unhealthy server and bring up a new one.

The company maintains separate deployment pipelines for each application and service, using Atlassian’s Bitbucket distributed version control system and its Pipelines feature to automate the deployment of checked-in code.

Roller can deploy code within an hour if necessary, but currently works to a weekly release schedule. The goal is to move to multiple releases in a day.

Resources

Major Australian jobsite Seek served as an inspiration for Roller’s DevOps path. “We’ve always looked at the bigger players,” Brodie notes. In addition, Meetups (“there’s something for everyone”) and conferences (he specifically mentioned DDD Melbourne and NDC Sydney) have been invaluable in being able to talk with other companies about their experiences. “We’re keen to stay ahead of the latest practices,” he says, so the company encourages its people to attend such events.

Other resources he finds useful include Stack Overflow and Plural. Plus, he recommends “The DevOps Handbook” by Gene Kim, Jez Humble, Patrick Debois and John Willis, which he describes as “a fantastic resource,” providing practical advice particularly around issues at the organizational level.

Tips

When asked for advice for those starting a new business or major project, Brodie recommends pragmatism:
- Consider DevOps from the start “but don’t overcook it.” In other words, begin with the basics and evolve from there.
- Aim for continuous improvement—people will keep generating new ideas, but picking the right time to implement them is important.
- Look for a mentor, and talk to people about the tools they’re using.
- Take advantage of outside organizations’ capabilities. For example, Rackspace can help architect an application for AWS.
— Stephen Withers
August 22, 2017
Yahoo7 Uses DevOps to Speed Website Redevelopment

Yahoo7 is a joint venture between Yahoo and Australian entertainment and TV company Seven West Media. Seven is one of Australia’s three major TV networks.

The company has completely replaced its technology stack and is in the process of revamping its seven consumer sites to take advantage of the new platform. The first cab off the rank is the Be entertainment and lifestyle site, but “the rest will come pretty quickly,” according to CTO Paul Russell.

“We had an aging website based on a proprietary framework,” he said, which has been replaced with a platform based on node.js and other open-source frameworks, including the Yahoo-backed Screwdriver continuous delivery platform.

There is also a focus on adopting reusable components across all the sites. It is more efficient to select, for example, an image gallery that gives the best user experience and deploy it across all the sites. That doesn’t mean all the new sites will have the same look; it is still possible to have distinctive layouts for each of them, Russell said.

A move to cross-functional teams and Scrum “was a pretty big change for us” that took approximately two years, he said. There are three teams, each with four or five developers, a product owner, a tech lead, a Scrum master and a tester. A user experience specialist is added when required.

The reorganization required “a mindset change,” but having people doing different types of work side by side did help. But the adoption of common KPIs “really has made a big difference,” as it keeps people focused on the things that really matter.

“We’ve got a fairly mature DevOps model” and the current project is the company’s latest step, said Russell.

Augmenting Screwdriver with automated performance testing on a range of metrics including latency and the number of requests per second that can be handled captures any problems at the early stages. This was “a really big step for us,” he said, especially as Yahoo7’s news content leads to significant spikes in demand, and also because the entire stack was being replaced.

The scope of the project meant it was important to have experienced staff (top engineers were needed to pull everything together under Screwdriver, he noted), although some PHP developers were cross-trained for the technical skills required for the new platform. Russell said he is prepared to train people who come with the right attitude, but even though Yahoo7 is regarded as an attractive employer, “it’s definitely a challenge” to get the right people.

Functional testing is being automated for some of the more complicated parts of the site, such as the video player, to improve confidence in each release.

Russell believes organizations need to be clear about what they expect to accomplish by adopting DevOps practices, and warns that it requires a change of mindset which is unlikely to be achieved unless the people involved buy into the concept and are all on the same page.

In particular, the idea of allowing the development team to run releases can be challenging to those used to traditional operations practices, but “it’s absolutely worth it,” he said. For example, the increased release pace means the company can respond more quickly when opportunities arise; Yahoo7 now can deploy two releases per day when necessary.

There are also benefits from the way the teams take greater day-to-day ownership of the systems they build and run. This—combined, we suspect, with the shared KPIs—brings about a situation where rules imposed from above are largely replaced by good working practices created within the teams.

A final tip from Russell to anyone considering the adoption of DevOps: You need the right tools and practices in place if it is to be successful.

— Stephen Withers

August 3, 2017
From Artisanal to Factory: The IT Pendulum is Swinging Back

Splunk Vice President of IT Markets and Chief Technology Advocate Andi Mann sees DevOps moving from an artisanal approach to an IT factory.

This represents a reversal: Historically, developers operated in a factory-like environment, passing chunks of work down the line, but that caused problems because “different managers owned different sections of the production line.”

Agile and DevOps were a response to that issue, he said, resulting in “the artisan developer … the full-stack developer.”‘

The trend is now flowing in the opposite direction, with Splunk customers wanting to see “the whole flow.” Furthermore, enterprise DevOps is too big for an artisanal approach.

“I worry about that [reversal],” he said, but technology can help artisanal quality within an IT factory. Collaboration tools such as permanent video links between different locations enable specialists to work together and understand each others’ work.

“That’s a different kind of factory,” said Mann, in which development, security, QA, operations and other staff do not need physical proximity to work closely together. This is particularly important for large enterprises, he observed, as you cannot pack 1,500 developers into a small space.

One example he cites to show this approach can work is a telco with its developers spread across multiple locations in Sydney and Singapore using DevOps practices to achieve continuous delivery. Another is one of the largest UK retailers getting weekly releases from developers in London and Mumbai, with QA work done at another location in India.

“Executives I talk to are very heavily committed to improving cycle time,” said Mann. But they are also concerned with quality (e.g., problems should be detected and fixed before they reach the user, and the percentage of changes sent for rework should be minimized) and business impact. Executives want to link DevOps to their own KPIs: “Are they moving the needle on my business requirements” such as shareholder value and reputation?

“You get the changes you drive by your measures,” he said, so it is up to managers to make sure their measures make sense in the broader context. It may be time for a balanced scorecard to be applied to DevOps, with metrics in all three areas, he suggested.

These metrics could include code coverage, error rates, customer complaints or trouble tickets raised, plus business impact measures such as click-through rates, cart abandonment, subscriptions cancelled, open rates (for mobile apps) and customer sentiment.

Mann noted executive demand for the adoption of DevOps, even where “they don’t even know what they’re talking about.” He feels it is really about a desire for visibility into the development pipeline, and “that’s really challenging.”

Part of the problem seems to come from the tension between traditional IT (especially “ITIL-owned shops”) and new-generation executives with titles such as head of digital.

Slavish adherence to ITIL practices won’t allow the required speed, may not achieve the expected level of compliance and business goals probably won’t be met, he suggested.

“ITIL came out of a fear of failure,” but failure is no longer catastrophic. “If you’re not failing, you’re not trying, you’re not innovating.” What’s important is to fail fast, fail small, and fail cheap—and DevOps allows IT to fail in low-impact ways. Even when an organization is making significant changes to software, the effects of failure can be contained through measures such as canary releases (where the new versions are initially made available to a small subset of users).

Furthermore, customers are getting used to the idea that technology isn’t and doesn’t need to be perfect, largely because of their experiences with speech recognition services such as Siri, he said.

— Stephen Withers

April 7, 2017
Australian DevOps Company to Open US Office in 2017

Australia-based DevOps as a managed service provider base2Services is preparing to establish a U.S. office in mid-2017.

Despite being based in Melbourne, Australia, base2Services already has five customers in North America—including media company All Day and online video specialist Sardius Media—that contribute nearly 20 percent of the company’s revenue. Further expansion in the region will require staff in a similar time zone to potential customers, said managing director Arthur Marinis.

“Our service is unique in the fact that everything we do and say is aimed at helping development teams at existing companies, big or small, leverage the cloud and automate. They are the ones with the knowledge of how vital it is to speed up development times and to have the ability to change quickly. Those are the ones searching for DevOps around the world.”

He added, “They have found we are able to provide the right consulting and managed services to help them migrate to the cloud, optimize their cloud infrastructure and implement DevOps processes. They aren’t nervous about our location as they use the cloud, and most are used to working remotely, it helps that we also provide 24/7 support.”

base2Services’ policy of charging a fixed monthly price to development teams to help them build and manage scalable and agile cloud-based solutions presumably helps.

So far, only one potential customer was put off by the idea of working with an Australian company, Marinis said. But, “Our cultures aren’t all that different,” and culture is particularly important for DevOps.

The company often is asked to prove its expertise, in which case its preference is to organize a meeting between its technical staff and their counterparts at the prospective customer. “That solves the issue in most cases,” he said.

Location, Location, Location

Although most of its customers are in California, base2Services currently plans to locate the new office in either Boulder, Colorado; Kansas City, Kansas; or Austin, Texas. Marinis described Kansas City as a high-growth area for DevOps, and said inland areas of the United States offer a good supply of talent at more affordable prices than California. Another benefit of these cities is that their time zones provides a better crossover with eastern Australia.

Indeed, the time difference between the United States and Australia has proved beneficial to North American customers, Marinis said. Australian-based staff can talk to their client’s developers and then work overnight (according to an American clock) and have the results ready when the developers arrive at work the next morning. “It actually helps them move faster.”

It also makes it easy for base2Services to provide out-of-hours support.

All the U.S. staff will be hired locally, with the first appointment expected before Christmas. An experienced member of the Australian team will work in the U.S. office for six months to teach the recruits “the base2 way,” Marinis explained.

He said the company was still focused on Australia and wanted to grow its customer base by 200 percent over the next 12 to 18 months. It focuses on startups and SaaS providers, and there are many more of those in the United States, he observed. In time, base2Service’s U.S. business could be five times the size of its home operation, Marinis predicted, though he added, “We’re very confident that we will remain Australia-based. This is where our hearts are.”

— Stephen Withers

December 14, 2016
DevOps: Achieving Process Without Too Much ‘Process’

While there’s no fundamental conflict between IT service management (ITSM) and DevOps, DevOps does bring new and different techniques for getting work done and interacting with the rest of the organization.

ITSM people who are used to frameworks such as ITIL and COBIT tend to regard the IT department as a “service provider” and the rest of the organization as “the customer.” But DevOps has come from web-scale organizations “where IT is the business,” and that means the prevailing view is more like the Toyota Production System mentality, wherein the aim is to move things along a production line with maximum speed and maximum quality, says Matt Hooper, product evangelist at LANDESK.

DevOps thus has already broken down some of the barriers between “IT” and “business”—practitioners realize they have to determine what the business needs, and then provide it. But Agile or Agile-like practices also are part of this approach, so there’s lots of experimentation, learning and redeployment in “lots of tiny little steps,” and, consequently, there’s not much risk of going very far wrong with any of them.

An issue is that this conflicts with ITSM’s view of life cycles, partly because the whole idea is to merge development and operations—for instance, by automatically spinning up software-defined infrastructure as needed.

What’s needed, Hooper suggests, is “service management with a digital mindset” rather than ITIL’s “human mindset.” It’s all about feedback, he says, but for ITSM and DevOps to sit comfortably together there must be a realization that feedback can come from humans or systems.

And even though the command and control mindset tried to avoid failures, systems still failed. So a better approach is to accept that “stuff breaks” and make small changes at a time, respond to issues as they arise and keep learning. In the longer term, this will lead to more durable systems, he says.

It’s a myth that you’ll be able to stop bad things happening, because people and systems aren’t perfect. DevOps, in contrast, accepts that things will go wrong but aims to respond quickly—for example, by keeping developers close to operations so they can see problems occurring, realize what mistakes they made, and put them right.

“That’s challenging to a traditional ITSM person,” Hooper says. It is important to accept that failure is an option, even though massive failure isn’t.

(Before you ask, he accepts that certain systems need to be extremely reliable. But others provide room for a somewhat experimental approach, in which case DevOps helps avoid overengineering.)

While some ITIL practitioners see value in DevOps’ continuous deployment approach, their premise is still that IT is a “service provider” rather than a “partner,” but the business side of the organization wants the latter. You don’t get home from a business trip and tell your spouse you were 99.98 percent faithful while you were away, he observes, and that sort of SLA-based thinking works against becoming a learning organization, achieving good demand management and so on.

ITIL v2 served a good purpose, but ITIL v3 arrived too late, Hooper suggests. The growing influence of the Lean movement means systems are going through rapid change cycles, and too much “process” leads to waste and lower performance. DevOps and software-defined technologies provide ways to better handle rapid change—infrastructure can be spun up and back down in seconds, providing opportunities for self-healing systems with better scalability and reliability.

Where ITIL focuses on meetings and human processes, DevOps prefers to use analytics to determine the right time to release and then whether then release was successful. “The feedback loop is an essential part of DevOps strategy,” just as it is with ITIL, but ITIL has a more human-centric idea of feedback.

“ITSMers tend to hide in the back closet” waiting for the business to tell them what to do, but DevOps takes a more active approach, he suggests.

August 29, 2016
DevOps is Becoming Mainstream in Australia

DevOps has taken root in Australia, where its practitioners are showing a high level of sophistication.

According to Paul Muller, vice president of Strategic Marketing, Software, at Hewlett Packard Enterprise Asia Pacific and Japan, interest in DevOps at Australian organizations rose, fell back and then rose again—a pattern that reminds us of Gartner’s Hype Cycle.

DevOps has progressed from the stage where its devotees were seen as a ‘radical fringe’ fighting against the organizations where they work, and is quickly becoming part of mainstream IT, he said, observing he has even heard the CEO of a bank talking about DevOps. (The banking sector in Australia is highly concentrated, with the ‘Big Four’ collectively accounting for around 80 percent of the market, with the customer-owned, regional and foreign-owned banks taking the rest.)

More generally, executives increasingly realize “their business is only as good as their software,” he said.

He sees innovation occurring in what are sometimes regarded as very traditional sectors (automotive, banking and finance, to name two) as companies realize that they face—or soon will face—competition from “software-savvy startups.”

The success of DevOps shows “Adam Smith was wrong”—that is, overspecialization is bad, he observed.

“The sophistication here is higher than I would have expected,” and higher than in most places outside Silicon Valley, said Muller.

The Australian DevOps community benefits from the huge amount of volunteer effort that goes into organizing events such as meetups. HPE is helping with its ‘DevOps Days’ that expose the company’s customers to HPE’s DevOps culture and practice, allowing them to “live a day in the life of this modern process” and take a better understanding back to their own organization.

One DevOps mistake he has seen made that generally results in failure is what he calls “processes of fetish,” where people simply try to copy another organization’s practices instead of developing a true understanding of DevOps.

He also drew attention to the shortcomings of measuring productivity in terms of lines of code or function points delivered. That overlooks, for example, the contribution of a person who does a lot of coaching.

Muller drew particular attention to the importance of scaled agile to larger organizations with multiple internal and/or external teams that need to work together to successfully complete projects. In such environments, problems arise if it is not used.

To be successful, DevOps requires the right collection of people, the right measurement and reward system, and the right tools (such as HPE ALM Octane, which can among many other things handle the synchronization of data across multiple teams), he said.

But it’s not just about implementation: organizations and teams still need to generate ideas that can be put into production. There are tools to help that process, Muller said, and HPE is teaching its innovation methodology to other organizations.

Many businesses spruik their innovation credentials and want their developers to focus on innovation. Agile and DevOps help free up developers’ minds—along with those of their colleagues in business units—to do just that, Muller said.

July 11, 2016
How to Kick-Start Your DevOps Adoption

A recent survey by CA Technologies found Australian organizations are below par in terms of implementing measures to adopt and achieve successful DevOps strategies.

According to Ashok Vasan, vice president of DevOps Solutions Strategy at CA Technologies Asia Pacific and Japan, organizations are generally aware of the DevOps methodology and its value, but they either don’t know how to start or they’ve begun but then failed to move to rapid or broad adoption.

A lot of companies ask CA about the prerequisites for a successful DevOps adoption, he said. It needs to be considered as more than just an IT change, and success requires an understanding and acknowledgement that the business side must lead the transition.

Starting the DevOps Adoption Journey

The business perspective is about launching new products or services and gaining market share. To do that, companies will ask IT to not only deliver certain facilities, but deliver them quickly, so the business should see the adoption of DevOps as a first step in achieving those goals. If senior executives are looking at a particular strategic initiative, there is a greater possibility of getting the required resources across the organization, including IT.

Ideally, that first project will be relatively small, yet visible and significant to the rest of the organization.

It is important to establish a common set of goals and shared objectives, covering areas including development, testing, APIs, product management and product marketing. “Everybody needs to be measured in the same goal,” said Vasan.

Then it is time to look at the IT capabilities and activities needed to make it happen. Where are the bottlenecks? What can be automated? How should testing be performed?

The next step is to identify the weakest link in the existing product or service chain, and work to improve that. If the release process is the bottleneck, then automating that is a good early step to take. Vasan has seen speed improvements of 70 percent to 80 percent from the introduction of release automation.

One of the main reasons for the improvement is that release management can be a complex task, and automation reduces the incidence of errors, sometimes to the point of practically eliminating them. No errors means no time is wasted on remediation, so significant time savings are possible.

Reduced downtime is obviously a benefit to the business, but it can even pay dividends in terms of employees’ work/life balance. Making the release process quicker and smoother means there is no need to do it overnight, on weekends or during holidays.

Indeed, there is a very high return on automation efforts in general, he said. It is important to realize that it doesn’t matter how strong the rest of the links are: For example, even if the marketing people are doing everything expected of them, if a new app receives low user ratings, the organization has failed.

Successful DevOps Needs More than Tools

Back on the IT side, a technology platform should be designed and implemented so the people involved have the right tools, the right scalable technology and so on. A commitment to tooling is necessary to realize the benefits of DevOps, Vasan said. But it’s not just about tech, he warned, but rather a combination of the right processes, the right technology and the right people skills for successful DevOps adoption.

Where an organization is struggling to gain traction with DevOps, Vasan recommends making sure that the process really is business-led; checking that the right employees have been assigned and the right vendors and consultants engaged; ensuring good project management practices; using the lessons learned from initial projects to inform the rest of the organization (possibly by establishing a DevOps center of excellence to spread the word to other divisions and projects); and using an enterprise scorecard to help keep everyone on the same journey.

The rate of change demanded by today’s environment can be an issue for some organizations, said Vasan, but Australian banks have been very progressive and have done a lot to deliver world-class IT products.

Even outside of the banking industry, “there are pockets of [DevOps] excellence … that can serve as examples for others in Australia.” There are “exciting” things happening in the government sector, he added.

CA is working to pass on to a wider audience the experience and best practices developed by pioneers in the DevOps field. “We have seen how companies have started to work in all these [technical] areas, and we take these experiences to our customers.”

March 25, 2016
Microservices: Easier Than You Think

“Microservices are easier than you think and more valuable than you’ve heard.” Those are the words of Ben Gracewood, chief delivery officer at cloud POS and omnichannel retail provider Vend. His company has used microservices to trim a monolithic PHP application, improving performance and changing the way the company approaches updates.

Approaching Microservices

Vend is currently used by 15,000 retailers in more than 180 countries. It is aimed at inventory-based retailers—think housewares, fashion, grocery and so on. In addition to the usual POS and payments facilities, Vend handles loyalty schemes, gift cards and online storefronts. The company is based in New Zealand, with offices in London, Melbourne, San Francisco and Toronto. Engineering and DevOps is done in Auckland.

Its global customer base means there is no maintenance window, and so live updates are essential.

Vend’s HTML5 and iOS clients include caching and a local database to allow offline operation (the data is subsequently resynchronized), primarily to allow users to keep working in situations where there is no Internet connection. The main system was originally a monolithic PHP application.

Vend executives had discussed the idea of migrating to microservices for about a year. But when the company’s vice president of Architecture experimented by writing one read-only microservice that improved performance by orders of magnitude, “This was the ‘a-ha’ moment for us,” Gracewood says.

Later, when Vend migrated from Rackspace to Amazon Web Services, it used a routing microservice that directed traffic to whichever cloud held that particular customer’s data at the time. That allowed a progressive migration with no disruption for customers.

Those initial successes showed Vend it should move more seriously in that direction. Nine months later, the company had 12 microservices in production.

Vend is now adding them at the rate of about two per month, and the monolithic PHP system has been slimmed and surrounded by a cloud of microservices. An Nginix-based routing layer allows existing URLs to be redirected to microservices where appropriate.

The company has followed DevOps practices from the outset, but its first few microservices were deployed and managed as special cases. Now, both the PHP monolith and all the microservices are within the DevOps tent.

Adoption Dos and Don’ts

When it comes to adopting microservices, “It’s never too soon to start,” says Gracewood. “It doesn’t have to be all or nothing,” despite what you might hear from other microservices veterans—everyone has been partway through their journey at some stage.

Gracewood’s advice is to build your first microservice and do what you have to in the development of your DevOps practice. Then keep iterating.

Don’t fall into the trap of thinking a microservice has to be 100 percent complete before it can be deployed, he says. Start by developing the minimum functionality required at the outset, and then increasing its sophistication over time. Since a change to a microservice affects a relatively small part of the entire system, the continuous update model is easier to apply to microservices than monolithic applications.

That point also has an organizational impact: A microservices architecture and DevOps support Gracewood’s philosophy of trusting engineering and development teams to implement the right things at the right time. With monolithic software, a mistake or a poor decision can have wide-ranging repercussions, so considerable oversight is required. But the nature of microservices keeps issues localized, so if an issue does somehow slip through the testing process, it is easier to revert to the previous version.

Adopting DevOps and microservices does require a certain amount of education, Gracewood notes. Engineering teams must understand the command line, Docker and so on. Making the move calls for “lots of tooling or lots of training,” he says. To that end, Vend has built some custom tools.

Gracewood’s final tip for organizations thinking of adopting microservices: “Log everything [and] apply metrics to everything,” right from the start. You need to know how people are using your system. In particular, identifying the most heavily used microservices will reveal the areas where efforts to improve performance are likely to yield the best returns.

March 2, 2016