devopsy.kinsta.cloud

Author: Stephen Withers

API Security by Design

“APIs are nothing new,” said Secure Code Warrior co-founder and CTO Matias Madou, but they have recently become more widely used.

And where they were once a local mechanism, they are increasingly used in a distributed manner, partly because of changes to application architectures. Another reason is that users are increasingly likely to access systems from outside the organization, whether they are customers, suppliers or employees working at home.

Potentially anyone and anything can access an API, so developers and architects should therefore assume that APIs are exposed, and treat requests accordingly.

This means addressing issues such as authentication and authorization, and generally ensuring that the only data exposed by the API is the data that’s supposed to be exposed.

So assume every API call is malicious, suggested Madou.

Start at the Beginning

Achieving API security starts with the design and architecture, so jumping in and writing code immediately is a mistake. Unfortunately, starting with a completely blank slate is unusual, and some code will likely already exist. In that case, it is especially important that the design makes provisions for possible weaknesses.

If you want secure APIs, then ensure that everyone working on the project – architects, developers, testers and so on – are “very security savvy,” said Madou.

The need to improve skills is often triggered when the company or one of its competitors experiences a problem.

So teams should work to identify any existing problems, realize they need to proactively avoid the underlying issues, and undergo appropriate training, he says.

The trouble is, “security and developers are not often the best of friends,” Madou said. So, developers tend to think that security is the responsibility of their security colleagues. But there should be a common goal: making reliable and secure software.

“Security needs to help developers,” he said. One way of achieving this is to appoint a ‘security champion’ to act as a bridge between the security and development teams. An embedded security person can help developers realize that security is an enabler, not a blocker.

Help When Help’s Needed

Embedding communication from the security team into developers’ workflows is another part of the process. Slack is one possible conduit. Security people need to be wherever developers discuss issues. Then, they can take advantage of opportunities to say things like, “it looks like you’re having problems with SQL injection. Here’s some training material that might help.”

Developers live in their IDE, so that’s where integration should occur. But design such systems to help developers, not to control them. “Developers are people, not robots controlled by machines,” Madou warned.

Another consideration is that it’s important to avoid unnecessarily interrupting developers. Such systems should therefore predict what a developer is about to do and deliver appropriate guidance.

For example, if someone codes a database connection, the system could offer the statements that securely retrieve information. Then, the developer can copy and paste them instead of starting from scratch each time. Using new code every time carries the risk of introducing vulnerabilities.

“It’s tricky to make these systems,” Madou admited, but he predicted they will become increasingly common in the coming years.

May 25, 2021
Consider Telemetry When Rearchitecting Applications

Australia and New Zealand have seen a big move to the public cloud, and “… when I talk to customers, I see huge interest in rearchitecting applications rather than doing a lift-and-shift right into public cloud,” said Scott van Kalken, F5 ANZ solution architect for DevOps and modern applications.

That migration to the public cloud has led to massive interest in telemetry. When you go from a monolithic application to a distributed app, there may be 100 or even 1,000 APIs involved. Without telemetry, you don’t know what’s going on with those APIs. But the telemetry needs to be consistent.

F5’s 2021 State of Application Strategy report found 75% of respondents considered telemetry, especially about application security and delivery, important for meeting business outcomes, and 95% of respondents said they are missing insights from their existing monitoring and analytics solutions.

Van Kalken recounted a conversation with a practitioner as an example. This practitioner gets yelled at when problems arise with partner APIs used by the application he manages. So, even though these APIs are not technically his responsibility, he needs visibility into their status. Only by combining internal telemetry with telemetry from these partner APIs can he make sense of what’s happening within the application ecosystem.

And it’s not just a question of whether an API is available or not. Problems can arise when it is responding slowly. “‘Is that normal? Is that normal for this time of day?’ I’ve got to have some sort of telemetry data available to me to make decisions,” van Kalken said.

Enabling Security Automation with Telemetry

An aspect that’s particularly interesting to him is the ability to automate or otherwise take action based on security telemetry data.

A simple example is a security event that reveals some kind of systems abuse. Automation can add a firewall rule to block the associated traffic.

As organizations increasingly adopt event-driven architectures, events and responses occur at the application, network, infrastructure and other levels. “The minute that you automate those event-driven architectures with security, it opens up a world of possibilities.”

It’s not just about automating security. Imagine an API gateway that serves a variety of APIs to customers. Telemetry might reveal that two of the APIs are starting to slow down; in which case, “I might kick off a workflow that just deploys more of them so that we can handle the load. But if you don’t have that telemetry, you can’t do that,” he said.

In practice, this involves telemetry, analytics and automation.

“When we show [customers] how you can make use of the analytics in a really easy way, their eyes light up,” van Kalken said.

Multi-Cloud Migration

A further consideration is that rearchitecting applications isn’t just about splitting monoliths into smaller pieces and redeploying them as microservices. There’s also a desire to run them across multiple public clouds to reduce risk, and also to deploy applications at the edge where they are closer to their users.

The 2021 State of Application Strategy reported that just over three-quarters of respondents are already using or have plans to use the edge to capture benefits related to application deployment, performance and data availability.

“This is really important, because as an application developer, I want to deploy my application once and I want to say, ‘I want this distributed to all of these places around the globe’,” van Kalken said.

But if the application is deployed in, say, five locations, then from a telemetry perspective, it is five applications, not one.

“I’ve got five sets of telemetry, because I’ve effectively got five different applications deployed out there. It’s the same app, but it’s in five locations, and the consumers are different in every location; the way that they interact is different and the telemetry that I’m getting is going to be different,” he said. “So, I need to actually suck up all of that telemetry, and make it meaningful in some way,” he explained.

Telemetry Enables Clearer Vision

“A distributed cloud architecture is right for these modern applications. We recently bought a company called Volterra, and this is one of the areas that they play in — distributed applications, and specifically, abstracting multi-cloud and multi-cluster Kubernetes application deployments,” he said. This area of focus will be critical in the future, van Kalken said.

He predicts that as organizations move to 5G, applications will become increasingly distributed. If that does happen, it will be extremely important to receive telemetry from all components of the application so it can be analyzed and used to support intelligent decisions, as part of the distributed application deployment.

May 12, 2021
How To Address DevSecOps Skills Shortages

“If it’s scarce, it drives the price up,” said McAfee director of systems engineering Sahba Idelkhani. And DevSecOps talent is scarce, and expensive.

Restrictions on international travel are having an effect on immigration in some countries, notably Australia. But there’s practically nothing employers can do about that.

But some aspects of the supposed skills shortage are, in part, self-imposed. One example is recruitment ads that specify “5+ years’ experience required” (or something similar) when that level of experience is only a desirable characteristic.

Such practices don’t just restrict the field of candidates for that particular vacancy, they also make it hard to find entry-level positions. If candidates can’t get a foot in the door, how can they gain the experience that employers say they want?

DevSecOps Change is Needed

Idelkhani suggests that organizations need to change their hiring practices and put more effort into reskilling existing employees.

In particular, if you’re looking for DevSecOps people, he suggests it’s better to teach developers about security than it is to bring security specialists (who are also in short supply) up to speed about development. The result is less rework, reduced risk and shorter development timelines.

However, there is a risk that security specialists will tell developers what they did wrong. Proactively educating developers about security is a more productive approach.

Cloud Academy is one of many sources for online security training courses and includes hands-on labs, sometimes involving the same tools that criminals use to breach security. Other tips Idelkhani offered include adopting a “build secure” mindset to reduce the security effort needed after systems have been created.

A similar perspective can be applied to entry-level positions.

Idelkhani has hired new graduates, even though it requires effort to guide and train them to the point of being productive.

He has even adopted this approach with experienced people that have no background in security, including a former CTO and mobile developers and a pre-sales engineer.

Revamp job descriptions so more people can apply, urges Idelkhani, as aptitude and skill are more important than specific experience, which can be taught.

For example, instead of a large financial services organization recruiting someone with experience in incident response for cloud systems, it would probably be better to hire an incident responder and then sponsor their cloud training, he suggests.

Furthermore, a diversity program increases the supply of candidates, quite apart from the other benefits it may bring.

DevSecOps Training

Employers sometimes suggest training is a waste of time and money because people simply take their new skills elsewhere in search of higher pay.

But Idelkhani reports that his staff turnover has, in fact, fallen over the last three years, and no one has left after being trained.

He puts this down to establishing an “emotional connection” between employee and employer, at least in part.

However, organizations do have to accept the need to pay people what they are worth as they master new skills. But standard HR practices don’t suit skills development situations, he warns.

People usually won’t move from their current role or company for a 10% to 20% pay raise, Idelkhani said, but would be more willing to do so for the 200% to 300% increase that comes from being considered an accomplished worker rather than a mere trainee.

At McAfee, he is able to argue for ‘pay for value’ raises for his team; that task is generally harder at larger organizations, he said.

“It needs a rethink [of] HR practices,” he added.

Planning

Substantial raises tend to require approval from higher up in the hierarchy, and that requires a clear justification for the increase. He suggests this should be done in terms of the capabilities that are being built, not based on the numbers alone.

Managers should look at the skills they are going to need in order to deliver on the business’s strategic goals and plans, and decide how best they can be obtained. An expected DevSecOps vacancy might best be addressed by hiring a cloud developer and training them in the necessary security skills, or by retraining a person with potential who is already on the payroll.

“It’s a long-term process,” though, that requires a lot of management effort, Idelkhani said. Using this method means identifying people’s interests and skills, aligning them with company goals, identifying the skills gaps and then developing those needed skills. This requires a training budget and getting one approved means having clear plans.

But once you have shown the value of this approach, senior management is less likely to question it in the future.

April 2, 2021
SOFIHUB Relies on DevOps to Support Independent Seniors

How DevOps is helping one company ensure an active older population can remain independent

SOFIHUB home is designed to support older people who wish to live as independently as possible. However, its sensor-based system needed an upgrade to improve the user experience.

The SOFIHUB technology connects wirelessly to sensors placed around the home and learns the individual’s habits. When it detects deviation from the user’s routine or other issues, it asks if they are OK. Unless the answer is affirmative, it automatically contacts a nominated carer.

However, the existing products were created with a risk-aversion mindset and built to standards rather than user requirements, which sometimes impacted the user experience. “The market required a change,” said SOFIHUB CTO Ian Aitken.

For example, personal alarm pendants intended to detect falls generated four false alarms for every real fall they detected. A common cause of false alarms was the pendant accidentally hitting a piece of furniture.

In any case, only 4% to 12% of pendant owners actually used their devices, he said, mostly because they forgot to put them on after showering or dressing in the morning. Plus, many often regard pendants negatively, referring to them as “cowbells.”

So SOFIHUB changed its approach, placing movement sensors around the home to detect when the person moves from one room to another.

The SOFIHUB System

The sensors communicate via Z-Wave with the Raspberry Pi-based SOFIHUB home, which processes the data. The Raspberry Pi is capable, stable and cheap, said Aiken. But the finished product required additional custom electronics to provide power management and other functionality. The designers also added a microcontroller that monitors the Raspberry Pi and restarts it if necessary.

Caregivers receive alerts if the person is late to bed, late to rise or spends longer than usual in the bathroom, or if there is a lack of movement for longer than usual. Machine learning algorithms developed in conjunction with Monash University detect these patterns.

SOFIHUB is also able to deliver reminders such as those for appointments and birthdays or, working with a medicine cabinet sensor, when it’s time to take medications.

The optional SOFIHUB beacon is a panic alarm that can also generate alerts when it detects the user has left a geofenced area or has fallen. Since the hardware is basically a mobile phone without a dial pad, caregivers can call it to converse with the user.

Where the use of conventional pendants was about 10%, the use of the SOFIHUB beacon is around 60%, partly because the system reminds people to keep the beacon charged and take it with them. For example, it suggests they pick up the beacon from the charger once it has reached 80%.

The system transfers data collected by the SOFIHUB home to cloud-based infrastructure for longitudinal analysis and other purposes.

SOFIHUB Under the Hood

The company developed and must maintain several layers of software: code for the main board and the microcontroller within SOFIHUB home, code for managing Wi-Fi and cellular communications, and the code that runs in the cloud. SOFIHUB uses containers and infrastructure as a service (IaaS) rather than platform as a service (PaaS), so it can easily move from AWS to another host if necessary.

“We’re reasonably sophisticated,” said Aitken. For example, the company created a separate domain for a new corporate customer in around a day and a half, thanks to the level of automation within its DevOps processes.

The SOFIHUB service is also available through alarm center companies to ensure someone is always available to react in emergencies. Onboarding a new alarm monitoring company is usually a quick process, even if custom features are required. Sometimes implementing old protocols still used by some alarm centers takes longer.

“We are a highly skilled team of developers” and database experts, said Aitken. This “small, very agile team” works without dedicated operations staff, although two people specialize in building and maintaining implementation scripts.

The AppDev Toolkit

The team has adopted widely used tools (eg, Jira, Confluence, Slack, GitHub and CircleCI), and a well-developed release mechanism sees code start in a developer’s instance before moving into the AWS environment for additional testing.

From there, it progresses to the user acceptance testing environment to deal with any UI and browser-specific issues. After that, the code goes into production.

SOFIHUB does not do automated testing. Aitken said that’s because it actually slows development to an unacceptable degree, roughly doubling development time due to the effort needed to write the tests.

New releases occur every Wednesday with two fortnightly development cycles running side by side. Aitken said he expects the pace will slow as the product becomes mature and foresees monthly releases starting sometime in 2021.

September 21, 2020
Government Has a DevOps Lesson for Private Sector
The missions of government organizations and DevOps are well-aligned, according to F5 system engineer Scott Van Kalken, who works with many organizations at various levels of government around Australia.

That’s because government organizations provide services to the community, and DevOps is about providing services to a community of users.

He sees more use of DevOps in the government sector than in corporations.

Increasingly, DevOps is embedding security from the start of a project. “That’s really exciting for me,” he said, as it shows DevOps is working and allows faster iteration with security issues being picked up earlier.

For example, the COVID-19 pandemic has resulted in more people working at home. That makes security considerations even more important than ever.

All levels of government have seen massive increases in remote working capacity and functionality. Applying DevOps principles to security practices has allowed them to adapt rapidly to those increased levels of working from home.

One large department took just three or four days to go from 1,000 to 20,000 people working from home. Van Kalken said this was possible partly because they “automate absolutely everything they are doing,” allowing rapid implementation of security policies. They quickly stood up additional cloud infrastructure with the right security settings by using DevOps practices. Similarly, DevOps teams can use APIs to easily and quickly apply security policies to SaaS applications.

Cross-disciplinary Cooperation and Coordination

One government department that Van Kalken is involved with was new to DevOps but very good at getting representatives of the relevant groups into the same room. They all focused on reaching the desired outcome with “no turf wars” and agreed that establishing appropriate security policies was part of that process.

In his experience, the private sector tends to be more siloed, which can make it more difficult to make the right security-related choices from the outset.

For government organizations with more mature DevOps practices, “it’s just an incremental ramp-up,” he said. The collaboration between the various functions makes a big difference to successful outcomes.

So the role of the security organization changes from being akin to a nightclub bouncer (“No, you can’t”) to that of a partner in the delivery of services.

The private sector can learn from this, Van Kalken suggested. Government organizations have a “laser-like alignment” on service delivery. In contrast, the differing and competing objectives of various departments or functions within corporations may be a handicap.

Most organizations have adopted one of the published security frameworks, but project teams have the responsibility to decide how to comply with them. Government entities are different, he said, in that they must at a minimum meet the requirements of the Australian Government Information Security Manual (ISM). That leads to more alignment between departments and functions within the organization.

DevOps and Security Hygiene

From a more general security perspective, “governments are targeted in various ways” as they increasingly make services available online, he said.

“Government is no different to every retailer, every bank, and so on,” in the sense that all the usual threats apply, said Van Kalken. But some players specifically attack government organizations because of the types of data held about individuals. Fortunately, DevOps practices assist in responding rapidly to changing situations.

As a simple example, organizations can quickly deploy new policies to distributed firewalls by using DevOps principles including automation. That gives skilled individuals more time to deal with any exceptions that occur.

Organizations can apply similar practices to improve security hygiene by more quickly rolling out new versions of commercial software.

“That element of speed is something very new to the security space,” he said.

Recommendations

Van Kalken’s recommendations are:
- Get all the right people—including security—together in the project team from the outset.
- Ensure they all understand the objective.
- “All swim in the same direction.”
- Make sure everyone continues to work collaboratively.
The rest “will just happen if you do these things,” he said.
August 11, 2020
DevOps at Longtail UX

Longtail UX is an Australian SaaS startup that boosts traffic to e-commerce sites by automating the selection of relevant long-tail search terms and the creation of custom landing pages for each of those terms.

The company has fully embraced the Infrastructure-as-Code approach, according to Troy Jendra, senior systems architect at Longtail UX.

By using Terraform and Ansible for automation, “we can release whenever we want,” said Jendra. In practice, that can be at intervals between two minutes and as long as necessary at the other extreme.

Jendra said he had not used Ansible before joining Longtail UX. He used Puppet in a previous role, but found it “painful.”

“Ansible was a breath of fresh air,” confessed Jendra. It makes it easier to write configurations, and uses fewer CPU cycles because it does not need to run continuously. He explained that Terraform’s role is to spin up sufficient infrastructure so Ansible can run.

A recent project added support for CJK (Chinese, Japanese, Korean) characters to Longtail UX as part of a push into the Asia market, and that made it necessary to clone part of the infrastructure. Thanks to Ansible and Terraform, that task was easy to perform and completed in just half an hour. Thus, Longtail UX brought the new capability to market more quickly than would otherwise have been possible.

Ansible and Terraform together form “a cornerstone of our disaster recovery plans,” said Jendra. Fully automating recovery to improve on the current worst-case time of a few hours would take more time and effort than justified.

Apart from that, “it’s important to automate as much as we can,” said Andreas Dzumla, Longtail UX co-founder and CEO.

Avoiding Vendor Dependence

Even though Longtail UX runs on AWS, a deliberate decision was taken to avoid the vendor-dependency that would come from using AWS-specific services, such as AWS CloudFormation, in place of Terraform.

For the same reason, the company keeps things as simple as possible and sticks to AWS basics, such as EC2. It has recently started using Spot Instances to reduce the cost of running certain batch processes.

Jendra keeps a watching brief on AWS Lambda, and while the company does not currently have a use case he does see a time when it will jump from EC2 to Lambda.

“We’re pretty locked into AWS,” he admitted. However, the company uses AWS-specific services only where the benefits clearly outweigh the lack of portability. He likened the situation to 1990s attitudes toward using Microsoft-specific technologies.

Growth

The adoption of Kubernetes is not on the cards due to the relatively small size of Longtail UX’s DevOps team.

“We’ve doubled the size of our development team in the last six months,” said Jendra. It now comprises seven developers, a project manager and himself.

Dzumla expects it will double again in the next 12 months.

Jendra predicts that improvements to Longtail UX’s development stack will make it easier to on-board new hires.

“We currently utilize a Vagrant/VirtualBox dev stack (hooked into the aforementioned Ansible configuration), but find that very rough around the edges, and I only see it get worse as more attention is focused on Docker,” said Jendra.

DevOps is a philosophy, not a role, he emphasized. “Either everyone is doing DevOps or it doesn’t work.”

Fortunately, that understanding goes to the top of the company.

“Everyone needs to pitch in,” said Dzumla. “We do a lot of things that haven’t been done before, so we need senior people who don’t merely expect to do what they’re told.”

— Stephen Withers

December 20, 2019
DevOps at a Distance

How do you successfully follow DevOps practices when the two parts of your team are 5,000 miles (8,000km) and two time zones apart?

Airwallex is a global fintech founded in Melbourne, Australia just four years ago. It is Australia’s fastest-growth tech unicorn with a valuation of over $1 billion thanks in part to investments from companies such as Tencent Holdings, Sequoia Capital, Square Peg, Horizon Ventures, DST Global and MasterCard.

The business came into being when co-founders Jack Zhang and Max Li–who both worked in the banking sector–started a cafe in Melbourne’s Docklands area and realized that expensive foreign exchange rates hurt their profit margins when they imported products such as coffee cups.

Zhang was a software developer working in investment banking and foreign trading, and he knew technology could be applied to the problem.

Four years later, the company operates a network that allows SMEs to make easy and affordable payments to more than 130 countries and in 50 currencies.

Airwallex has 400 employees spread across offices in Hong Kong, London, Melbourne, Shanghai, Shenzhen, San Francisco, Singapore and Tokyo. That headcount includes 50 engineers in Melbourne and 84 in Shanghai.

“Communications between the two teams is incredibly important,” said Peter O’Donovan, Airwallex engineering lead.

So the approach has been to treat all of his people as if they were remote workers.

Tools

Extensive use of Slack–even when both people are in the same office–keeps everyone in the loop, he explained.

Similarly, Zoom videoconferencing is used for standups, and physical agile walls are replaced with digital equivalents.

These tools also allow for the growth of skills across regions, and help ensure everyone is “working to a common pattern.”

Such shared channels for collaboration and communication help accommodate overlapping working hours (Melbourne and Shanghai are three hours apart when daylight saving is in effect in Australia), as well as the need for engineers to be on call to deal with emergencies.

This approach puts everyone on a similar footing, whether they work in one of the offices or remotely.

DevOps leaders need to “focus on building relationships first,” he advised. Frequent video conferences are one way to build bonds with people in different locations. Understanding each others’ mindsets, establishing common ways of working and setting clear expectations are essential, and “doing it from day one is important.”

Scaling up

Airwallex plans to double the size of its engineering staff in both cities during 2020. O’Donovan points out that these established standards and practices help with the onboarding process and allow new hires to become productive more quickly.

That is especially true as the company has selected tools that most people will find familiar: in addition to Slack and Zoom, Airwallex also uses GitHub and G Suite. “These tools are easy to adopt and easy to consume.”

“Melbourne is a challenging market,” said O’Donovan, because DevOps skills are in demand and Airwallex wants to hire and retain quality talent. But there’s a virtuous circle–helping staff build a strong skill set and surrounding them with other talented people makes it a better place to work, so attraction and retention becomes less of a problem.

O’Donovan sees DevOps as a key to scaling the company. Having all the teams using the same tools and patterns yields economies of scale by avoiding unnecessary repetition of work, and it allows developers to focus on functionality rather than plumbing.

Furthermore, regulation and compliance is particularly important in the finance sector, and those considerations need to be built into the processes. “If you can do that seamlessly, you’re doing a good job,” said O’Donovan.

For example, making vulnerability scanning and the segregation of duties the defaults in the CI/CD pipeline addresses such issues in an automated and auditable manner.

You need “the right mindset for finance,” he warned. Building in compliance from the start works; trying to add it as an afterthought does not.

Don’t segregate people, advised O’Donovan, because everyone needs to be working collaboratively toward the same goal. With the right tools and automation, a cross-functional team can deliver better code, faster.

— Stephen Withers

November 7, 2019
DevOps Needs Collaboration and a Safe Place for Success

When it comes to DevOps adoption, “[many organizations] tell a similar story,” said Scott Van Kalken, systems engineer at F5 Networks. It takes more time and effort to develop the right culture than you probably expect.

Van Kalken has a long history in tech, including development and operations roles, and has spent several years practicing DevOps. He is part of the open source and meetup communities, and organizes some of the largest DevOps events in Australia.

“DevOps typically starts in a corner of the organization, where people want to deliver high-quality software really quickly,” said Van Kalken.

When that team gets good results, the organization decides to scale up its DevOps efforts, but tradition and silos get in the way.

“At least two things are needed to overcome those obstacles and achieve DevOps success,” suggested Van Kalken.

Collaboration and Safe Places

The first thing you need to achieve DevOps success is a collaborative team environment. One of the central ideas of DevOps is you don’t throw software over the wall between functions, such as development and testing, so there needs to be an understanding and acceptance that everyone in the team is on the same side.

Second is the provision of a safe place to fail and the realization that failure is just an iterative step on the path to implementing the best ideas.

A financial services company that Van Kalken works with was adopting GitOps but needed to convince the service management team this was the right move.

“Initially, it was quite challenging because service management thought it [GitOps] was uncontrolled chaos, even though that isn’t the case,” he said.

Its support was gained after running a series of workshops that showed the underlying processes were actually the same, even though the tools used to implement them were changing.

Service management is still the final gatekeeper, but it now approves releases in the repository rather than on paper.

Maturity

“There is a maturity in accepting change does involve risk,” said Van Kalken.

Things such as GitOps provide a quick and easy way of reverting to a previous release if something goes wrong. So the “safe place to fail” idea can be implemented in a way that protects the organization in a practical sense as well as staff members in a psychological (and career protecting) sense.

Another way DevOps can protect the organization is that frequent releases generally involve relatively minor changes, which in turn have a smaller blast radius in the event something goes wrong. Again, this is about having the maturity to recognize risk and dealing with it, rather than trying to avoid it in the first place.

A sometimes overlooked aspect of increasing the release cadence is the effect on users, who are being asked to cope with frequent changes.

Van Kalken pointed out that increasing the frequency does not necessarily mean multiple releases a day–it could be from two releases a year to three or four, if that’s what suits the organization.

Not all changes have a direct impact on users. Some are under the hood, improving performance or addressing rarely-encountered bugs. But when users’ interaction with the system is changed, he suggested canary deployments as a way of checking the acceptability of the new approach among a larger pool of users than those brought into the development process, before it is released to the entire user population.

“This approach also has a place in DevSecOps as another way of limiting the blast radius,” he said.

Accepting Failure

Perhaps the biggest challenge to an organization’s culture is the adoption of chaos engineering because if you’re going to kill a container or flood a network with data in order to check that the wider system can cope, you absolutely need to be in a safe place to fail.

You also need to realize everyone involved–including developers, security people and those involved in the business side–wants to achieve high availability, and that means designing systems that can handle (partial) failures.

It isn’t enough to do this in pre-deployment testing. Ongoing testing provides confidence that production systems will continue to cope with such failures.

“It’s pretty cool when you get it right,” said Van Kalken. However, “your culture has to be OK with this iterative approach.”

“DevOps is really just a methodology to achieve a better outcome,” stated Van Kalken. That involves collaboration, iteration and embedding security and other considerations up front. But the wider organization needs to understand and tolerate this, and the risk that goes with it.

“Education, awareness and experience can all contribute to dispelling the traditional view that risk and failure are inherently bad,” he suggested.

Are the Metrics Appropriate?

A related point is that whether one outcome is better than another, depends on the metrics adopted. “[Metrics are] probably the biggest things organizations need to change,” said Van Kalken.

For example, a customer who ordered a pizza doesn’t care whether a particular system involved in taking the order, cooking the pizza and then delivering it, achieved a certain uptime target. They just want the pizza to arrive promptly.

When that doesn’t happen, he believes a collaborative approach is needed to determine why the delivery was late, exactly what caused the problem and how to quickly roll out a fix to stop it recurring.

When it comes to practical advice, Van Kalken emphasized the need to get people together early in a project, otherwise there is a risk that sub-groups will demonize each other. In addition, you get more buy-in and everyone accepts they have to help with the pedaling, not just sit back and be passengers.

“If people aren’t talking, you have a bad culture,” he said. But he acknowledged it can be difficult to get past existing adversarial relationships. Breaking the ice between teams that haven’t previously engaged with each other can also be tricky.

Returning to an earlier theme, he stressed the importance of adopting an analytical, rather than punitive, approach when failure occurs. Everyone involved must know the organization accepts risk and understands the consequences.

— Stephen Withers

October 1, 2019
Security: Are You Doing DevOps Right?

Phil Kernick, co-founder and CTO of cybersecurity specialist CQR Consulting, has no fundamental problem with DevOps, but asks, from a security perspective, “How many people do it right?”

If DevOps is going to work and produce secure systems, then developers must take responsibility for security. It’s not something that can be treated as an additional process.

But abstract responsibility isn’t sufficient. It’s rare for developers to have security tools in their kit and an understanding of their results, said Kernick.

Learn From History

As former systems administrator and past president of the Systems Administrators Guild of Australia (SAGE-AU, now the Information Technology Professionals Association), Kernick suggested that DevOps pretends that the discipline of systems administration does not exist and merely treats infrastructure as if it were a software library.

Spinning up an instance is easy, but do you know that it is properly designed and maintained? This is a real concern, given the number of people working as developers without professional training, Kernick explained.

Repeatable loops and short development cycles do make sense. But are you confident all the individual pieces being assembled were built properly?

That raises questions about the software supply chain. Where did that Docker container actually come from? Is it well-maintained?

The process needs to be managed, yet, people at some organizations are “just grabbing things from anywhere,” Kernick said.

“It almost integrates the ‘not my problem’ attitude with development,” he noted.

Developers tend to care about whether a component works from a functional perspective, but an untrustworthy piece of code will compile and run.

Developers working in a DevOps environment need to understand the tools that help ensure code is secure, and they need to understand infrastructure, said Kernick.

‘Code Reuse is Bug Reuse’

DevOps makes the old joke that “code reuse is bug reuse” even worse, said Kernick, citing the example of a piece of vulnerable code shared on the Stack Overflow site and reused more than 100,000 times in GitHub-resident projects.

“I don’t want DevOps running in my bank,” he said. What he does want is well-engineered, secure, well-run, certified software.

Sacrificing security might save some time and get a new product on the market sooner, which might be acceptable if there are no real risks to the user.

It is easy to tell if a system is really important, as it involves tradable value, safety or privacy. Obvious areas include banking and health. Similarly, it is clear that a website that allows access without requiring any credentials is unimportant.

But there’s a wide middle ground where it’s “really hard” to determine a system’s importance because it is so difficult to imagine how criminals might exploit it.

For example, Kernick once bought a DVD cataloging app to help avoid duplicating items already in his very extensive collection. He thought this fell into the “unimportant” category. But—fortunately, as it turned out—he took the usual precaution of using a unique password for the associated service. Sometime later, he received a sextortion email that proffered that password as supposed proof that the criminal had gained control of his computer.

Worryingly, when he told the developer what had happened, it did not accept that there was a problem. The company said the password quoted in the malicious email wasn’t his password for the service—proving in the process that it was storing passwords in plain text!—even though he explained he changed his password as soon as he received the message. Predictably, he uninstalled the app.

Clearly, Kernick didn’t know what development methodologies that company used, but that’s not the point. The anecdote shows that a seemingly unimportant system can quickly become important to the user in situations outside the developer’s control. Consequently, security is always important.

DevOps and Security

He suggests that as long as an organization has the right security and governance processes in place, it won’t matter what development methodologies are used. So it can safely adopt DevOps or whatever else is most efficient in the circumstances.

It is possible to use DevOps practices to achieve speed without sacrificing security, Kernick said. But to do that, organizations need to take advantage of advances in software development without throwing away hard-won security experience.

— Stephen Withers

May 20, 2019
DevOps at Scale

According to Accenture‘s APAC practice lead for Agile, DevOps and Test Automation Mirco Hering, there is general agreement that DevOps at scale is a good thing, but the problem is getting from where organizations are to where they want to be.

Obstacles include having plenty of technology skills but little or no technology management (i.e., people and process) skills, or being driven heavily by technology vendors and ending up with a collection of tools that don’t all work well together.

Hering has found value stream mapping to be a very successful technique for implementing DevOps at scale. It allows people from all parts of the business to participate in the development of a map of the software delivery process (and, thus, gets different parts of the organization talking to each other), it provides important context for any external consultants that may be engaged, and it provides a visual representation of the real situation (not a formalization that doesn’t reflect reality).

It also provides an opportunity to explore what the participants think DevOps is really about. (Hering’s favorite definition is “building better solutions faster.”)

Get the First Bite Right

Once the map has been created and agreed, the next step is to determine what should be done first. “Find something that is meaningful but not impossible,” he recommends.

In other words, avoid biting off more than you can chew, but also avoid projects that are so specific that they won’t seem relevant to other parts of the organization. And avoid those involving technology that’s likely to be retired soon, because there might not be time for the results of the project to be appreciated.

It’s important to understand how you can show progress is being made, and organizations that understand the idea of rigorous continuous improvement are generally better at this.

There is a particular problem that arises when senior leadership changes: it is important to be able to explain what’s going on and demonstrate the benefits, such as the actual savings made.

Hering advocates the use of same rigor for applications supporting IT delivery as for business applications. For example, if software development systems or quality assurance systems go down, development and deployment come to a halt while the problem is rectified. The changes that are held up clearly have value to the organization (otherwise there’s no reason to make them), so any delay comes at a cost that can be minimized by applying the same types of backup and failover capabilities that are used to protect business systems. Other aspects of IT that can be applied to ‘doing IT’ include big data.

Use IT to Support IT

He points to the IT4IT reference architecture for managing the business of IT, saying “I think it is rarely implemented at this point, but at least someone’s thinking about it.”

Doing DevOps and Agile at scale requires de-scaling the technology, said Hering. That is, where hundreds of people might once have worked on a single piece of software that was released in a ‘big bang,’ the adoption of microservices and other approaches means small teams work on different parts of the overall project and independently deploy the results.

“That’s a huge jump,” he said. “A different capability is needed to do that.” Part of that capability involves resolving dependencies by using technology so they don’t need to be handled by people or processes.

But practitioners sometimes miss the point when they hear about organizations deploying hundreds of releases a day, and then they make the assumption that the practices and technologies required are not relevant to them because they don’t want to deploy that frequently.

Hering argues that it is really about having a deployment process that can support as many releases as you care to make, with “so little risk that it’s as natural as breathing.”

Can You Really do Continuous Delivery?

Many practitioners say they do continuous delivery, but Hering tests them by asking whether they could deploy a system at 4 a.m. the next day. If they are happy with doing that, they probably are on top of continuous delivery; if not, there’s probably something wrong, such as one or more manual steps that still haven’t been automated. Similar litmus tests can be created to cover other aspects of DevOps activity, he suggests.

Processes that are robust enough to detect any problems before the altered code reaches the customer or user empower DevOps teams to make small changes as required (whether the triggers were user requests, a realization that performance could be improved, or to implement a new business or compliance requirement, for example) without putting the business at risk.

Accenture has developed its own DevOps platform—a collection of open source based tooling—that provides a common reference point, and Hering recommends other organizations do something similar, warning that “it requires a difficult conversation about what is federated and what is centralized” because there needs to be a balance between the two extremes.

Changes to DevOps capabilities should be treated as scientific experiments, Hering suggests. The hypothesis might be that increasing unit testing coverage will decrease the number of defects, so you increase the coverage in a particular project (experiment), measure the defects (results), and draw a conclusion. But “very few organizations are that mature,” he conceded.

Training is Important

Another danger is that a lot of the information that practitioners hear (eg, at conferences) and read (e.g., in blogs) has been curated by organizations’ marketing teams, and that makes it too easy to focus on specific products or technologies rather than focussing on the issues that are really important to the organization.

So training is important to make sure everyone involved understands the concepts behind DevOps, and how to do unit testing and continuous integration. An increasing number of organizations say they want to adopt DevOps and Agile practices, but they don’t always understand the concepts and sometimes don’t really want to move out of their comfort zones.

Other pieces of general advice from Hering include asking why the organization is looking into DevOps (i.e., what are the goals and expected outcomes), making sure someone is going to drive the project, running cross-team workshops with a neutral facilitator to break down existing silos, and taking advantage of your peers’ experience in other organizations rather than trying to re-invent the wheel.

Very importantly, don’t try to move to DevOps at scale in one fell swoop because it is too much to swallow whole. Rather, take a piecemeal approach by focussing on a few aspects at a time and how they fit in with the next steps of your current project.

Hering’s book, “DevOps for the Modern Enterprise: Winning Practices to Transform Legacy IT Organizations,” was published this year by IT Revolution and is available from Amazon. He blogs about software delivery, Agile and DevOps at Not A Factory Anymore.

— Stephen Withers

September 17, 2018